diff options
| author | GitLab Bot <gitlab-bot@gitlab.com> | 2020-05-20 14:34:42 +0000 |
|---|---|---|
| committer | GitLab Bot <gitlab-bot@gitlab.com> | 2020-05-20 14:34:42 +0000 |
| commit | 9f46488805e86b1bc341ea1620b866016c2ce5ed (patch) | |
| tree | f9748c7e287041e37d6da49e0a29c9511dc34768 /doc/ci | |
| parent | dfc92d081ea0332d69c8aca2f0e745cb48ae5e6d (diff) | |
| download | gitlab-ce-9f46488805e86b1bc341ea1620b866016c2ce5ed.tar.gz | |
Add latest changes from gitlab-org/gitlab@13-0-stable-ee
Diffstat (limited to 'doc/ci')
51 files changed, 3074 insertions, 2790 deletions
diff --git a/doc/ci/README.md b/doc/ci/README.md index 30971d422cc..fce0ad15b70 100644 --- a/doc/ci/README.md +++ b/doc/ci/README.md @@ -37,14 +37,17 @@ your app. For a complete overview of these methodologies and GitLab CI/CD, read the [Introduction to CI/CD with GitLab](introduction/index.md). -<i class="fa fa-youtube-play youtube" aria-hidden="true"></i> -For a video demonstration of GitLab CI/CD, see [Demo: CI/CD with GitLab](https://www.youtube.com/watch?v=1iXFbchozdY). +<div class="video-fallback"> + Video demonstration of GitLab CI/CD: <a href="https://www.youtube.com/watch?v=1iXFbchozdY">Demo: CI/CD with GitLab</a>. +</div> +<figure class="video-container"> + <iframe src="https://www.youtube.com/embed/1iXFbchozdY" frameborder="0" allowfullscreen="true"> </iframe> +</figure> ## Getting started GitLab CI/CD is configured by a file called `.gitlab-ci.yml` placed -at the repository's root. The scripts set in this file are executed -by the [GitLab Runner](https://docs.gitlab.com/runner/). +at the repository's root. This file creates a [pipeline](pipelines/index.md), which runs for changes to the code in the repository. Pipelines consist of one or more stages that run in order and can each contain one or more jobs that run in parallel. These jobs (or scripts) get executed by the [GitLab Runner](https://docs.gitlab.com/runner/) agent. To get started with GitLab CI/CD, we recommend you read through the following documents: @@ -74,25 +77,32 @@ for all the attributes you can set and use. NOTE: **Note:** GitLab CI/CD and [shared runners](runners/README.md#shared-specific-and-group-runners) are enabled in GitLab.com and available for all users, limited only to the [user's pipelines quota](../user/gitlab_com/index.md#shared-runners). -## Configuration +## Concepts -GitLab CI/CD supports numerous configuration options: +GitLab CI/CD uses a number of concepts to describe and run your build and deploy. -| Configuration | Description | +| Concept | Description | |:--------------|:-------------| | [Pipelines](pipelines/index.md) | Structure your CI/CD process through pipelines. | | [Environment variables](variables/README.md) | Reuse values based on a variable/value key pair. | -| [Environments](environments.md) | Deploy your application to different environments (e.g., staging, production). | +| [Environments](environments/index.md) | Deploy your application to different environments (e.g., staging, production). | | [Job artifacts](pipelines/job_artifacts.md) | Output, use, and reuse job artifacts. | | [Cache dependencies](caching/index.md) | Cache your dependencies for a faster execution. | +| [GitLab Runner](https://docs.gitlab.com/runner/) | Configure your own GitLab Runners to execute your scripts. | + +## Configuration + +GitLab CI/CD supports numerous configuration options: + +| Configuration | Description | +|:--------------|:-------------| | [Schedule pipelines](pipelines/schedules.md) | Schedule pipelines to run as often as you need. | | [Custom path for `.gitlab-ci.yml`](pipelines/settings.md#custom-ci-configuration-path) | Define a custom path for the CI/CD configuration file. | | [Git submodules for CI/CD](git_submodules.md) | Configure jobs for using Git submodules.| | [SSH keys for CI/CD](ssh_keys/README.md) | Using SSH keys in your CI pipelines. | -| [Pipelines triggers](triggers/README.md) | Trigger pipelines through the API. | +| [Pipeline triggers](triggers/README.md) | Trigger pipelines through the API. | | [Pipelines for Merge Requests](merge_request_pipelines/index.md) | Design a pipeline structure for running a pipeline in merge requests. | | [Integrate with Kubernetes clusters](../user/project/clusters/index.md) | Connect your project to Google Kubernetes Engine (GKE) or an existing Kubernetes cluster. | -| [GitLab Runner](https://docs.gitlab.com/runner/) | Configure your own GitLab Runners to execute your scripts. | | [Optimize GitLab and Runner for large repositories](large_repositories/index.md) | Recommended strategies for handling large repos. | | [`.gitlab-ci.yml` full reference](yaml/README.md) | All the attributes you can use with GitLab CI/CD. | diff --git a/doc/ci/caching/index.md b/doc/ci/caching/index.md index 12267b4ab9f..16cabae353e 100644 --- a/doc/ci/caching/index.md +++ b/doc/ci/caching/index.md @@ -39,8 +39,9 @@ runtime dependencies needed to compile the project: - `artifacts`: **Use for stage results that will be passed between stages.** Artifacts are files generated by a job which are stored and uploaded, and can then - be fetched and used by jobs in later stages of the **same pipeline**. This data - will not be available in different pipelines, but is available to be downloaded + be fetched and used by jobs in later stages of the **same pipeline**. In other words, + [you can't create an artifact in job-A in stage-1, and then use this artifact in job-B in stage-1](https://gitlab.com/gitlab-org/gitlab/-/issues/25837). + This data will not be available in different pipelines, but is available to be downloaded from the UI. The name `artifacts` sounds like it's only useful outside of the job, like for downloading diff --git a/doc/ci/ci_cd_for_external_repos/bitbucket_integration.md b/doc/ci/ci_cd_for_external_repos/bitbucket_integration.md index 3437f867cb8..2836f9932c6 100644 --- a/doc/ci/ci_cd_for_external_repos/bitbucket_integration.md +++ b/doc/ci/ci_cd_for_external_repos/bitbucket_integration.md @@ -29,7 +29,7 @@ To use GitLab CI/CD with a Bitbucket Cloud repository: The web hook URL should be set to the GitLab API to trigger pull mirroring, using the Personal Access Token we just generated for authentication. - ```text + ```plaintext https://gitlab.com/api/v4/projects/<PROJECT_ID>/mirror/pull?private_token=<PERSONAL_ACCESS_TOKEN> ``` diff --git a/doc/ci/cloud_deployment/index.md b/doc/ci/cloud_deployment/index.md index f70998a5f49..8883ad5dfd6 100644 --- a/doc/ci/cloud_deployment/index.md +++ b/doc/ci/cloud_deployment/index.md @@ -1,4 +1,7 @@ --- +stage: Release +group: Progressive Delivery +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers type: howto --- @@ -36,7 +39,7 @@ Some credentials are required to be able to run `aws` commands: 1. You can now use `aws` commands in the `.gitlab-ci.yml` file of this project: - ```yml + ```yaml deploy: stage: deploy image: registry.gitlab.com/gitlab-org/cloud-deploy/aws-base:latest # see the note below diff --git a/doc/ci/directed_acyclic_graph/index.md b/doc/ci/directed_acyclic_graph/index.md index d4b87648f49..8722efd3b40 100644 --- a/doc/ci/directed_acyclic_graph/index.md +++ b/doc/ci/directed_acyclic_graph/index.md @@ -4,7 +4,8 @@ type: reference # Directed Acyclic Graph -> [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/issues/47063) in GitLab 12.2. +> - [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/issues/47063) in GitLab 12.2. +> - [Feature flag removed](https://gitlab.com/gitlab-org/gitlab/-/issues/206902) in GitLab 12.10. A [directed acyclic graph](https://www.techopedia.com/definition/5739/directed-acyclic-graph-dag) can be used in the context of a CI/CD pipeline to build relationships between jobs such that diff --git a/doc/ci/docker/using_docker_build.md b/doc/ci/docker/using_docker_build.md index 427f61deb29..f992af6c8a5 100644 --- a/doc/ci/docker/using_docker_build.md +++ b/doc/ci/docker/using_docker_build.md @@ -476,7 +476,7 @@ which can be avoided if a different driver is used, for example `overlay2`. On Ubuntu systems, this is done by editing `/etc/modules`. Just add the following line into it: - ```text + ```plaintext overlay ``` diff --git a/doc/ci/docker/using_docker_images.md b/doc/ci/docker/using_docker_images.md index 90e5c77063e..51139da2d16 100644 --- a/doc/ci/docker/using_docker_images.md +++ b/doc/ci/docker/using_docker_images.md @@ -26,7 +26,20 @@ test them on a dedicated CI server. To use GitLab Runner with Docker you need to [register a new Runner](https://docs.gitlab.com/runner/register/) to use the `docker` executor. -A one-line example can be seen below: +An example can be seen below. First we set up a temporary template to supply the services: + +```shell +cat > /tmp/test-config.template.toml << EOF +[[runners]] +[runners.docker] +[[runners.docker.services]] +name = "postgres:latest" +[[runners.docker.services]] +name = "mysql:latest" +EOF +``` + +Then we register the runner using the template that was just created: ```shell sudo gitlab-runner register \ @@ -34,9 +47,8 @@ sudo gitlab-runner register \ --registration-token "PROJECT_REGISTRATION_TOKEN" \ --description "docker-ruby:2.6" \ --executor "docker" \ - --docker-image ruby:2.6 \ - --docker-services postgres:latest \ - --docker-services mysql:latest + --template-config /tmp/test-config.template.toml \ + --docker-image ruby:2.6 ``` The registered runner will use the `ruby:2.6` Docker image and will run two @@ -197,7 +209,7 @@ default: image: ruby:2.6 services: - - postgres:9.3 + - postgres:11.7 before_script: - bundle install @@ -207,6 +219,12 @@ test: - bundle exec rake spec ``` +The image name must be in one of the following formats: + +- `image: <image-name>` (Same as using `<image-name>` with the `latest` tag) +- `image: <image-name>:<tag>` +- `image: <image-name>@<digest>` + It is also possible to define different images and services per job: ```yaml @@ -217,14 +235,14 @@ default: test:2.6: image: ruby:2.6 services: - - postgres:9.3 + - postgres:11.7 script: - bundle exec rake spec test:2.7: image: ruby:2.7 services: - - postgres:9.4 + - postgres:12.2 script: - bundle exec rake spec ``` @@ -239,7 +257,7 @@ default: entrypoint: ["/bin/bash"] services: - - name: my-postgres:9.4 + - name: my-postgres:11.7 alias: db-postgres entrypoint: ["/usr/local/bin/db-postgres"] command: ["start"] @@ -271,7 +289,7 @@ variables: POSTGRES_INITDB_ARGS: "--encoding=UTF8 --data-checksums" services: -- name: postgres:9.4 +- name: postgres:11.7 alias: db entrypoint: ["docker-entrypoint.sh"] command: ["postgres"] diff --git a/doc/ci/docker/using_kaniko.md b/doc/ci/docker/using_kaniko.md index 5a6f7490266..587f1f91f72 100644 --- a/doc/ci/docker/using_kaniko.md +++ b/doc/ci/docker/using_kaniko.md @@ -82,6 +82,7 @@ store: ```yaml before_script: + - mkdir -p /kaniko/.docker - echo "{\"auths\":{\"$CI_REGISTRY\":{\"username\":\"$CI_REGISTRY_USER\",\"password\":\"$CI_REGISTRY_PASSWORD\"}}}" > /kaniko/.docker/config.json - | echo "-----BEGIN CERTIFICATE----- @@ -89,6 +90,18 @@ store: -----END CERTIFICATE-----" >> /kaniko/ssl/certs/ca-certificates.crt ``` +## Video walkthrough of a working example + +The [Least Privilege Container Builds with Kaniko on GitLab](https://www.youtube.com/watch?v=d96ybcELpFs) +video is a walkthrough of the [Kaniko Docker Build](https://gitlab.com/guided-explorations/containers/kaniko-docker-build) +Guided Exploration project pipeline. It was tested on: + +- [GitLab.com Shared Runners](../../user/gitlab_com/index.md#shared-runners) +- [The Kubernetes Runner executor](https://docs.gitlab.com/runner/executors/kubernetes.html) + +The example can be copied to your own group or instance for testing. More details +on what other GitLab CI patterns are demonstrated are available at the project page. + <!-- ## Troubleshooting Include any troubleshooting steps that you can foresee. If you know beforehand what issues diff --git a/doc/ci/environments.md b/doc/ci/environments.md index fdd2791aa1d..b1dc9af6b5b 100644 --- a/doc/ci/environments.md +++ b/doc/ci/environments.md @@ -1,984 +1,5 @@ --- -type: reference +redirect_to: 'environments/index.md' --- -# Environments and deployments - -> Introduced in GitLab 8.9. - -Environments allow control of the continuous deployment of your software, -all within GitLab. - -## Introduction - -There are many stages required in the software development process before the software is ready -for public consumption. - -For example: - -1. Develop your code. -1. Test your code. -1. Deploy your code into a testing or staging environment before you release it to the public. - -This helps find bugs in your software, and also in the deployment process as well. - -GitLab CI/CD is capable of not only testing or building your projects, but also -deploying them in your infrastructure, with the added benefit of giving you a -way to track your deployments. In other words, you will always know what is -currently being deployed or has been deployed on your servers. - -It's important to know that: - -- Environments are like tags for your CI jobs, describing where code gets deployed. -- Deployments are created when [jobs](yaml/README.md#introduction) deploy versions of code to environments, - so every environment can have one or more deployments. - -GitLab: - -- Provides a full history of your deployments for each environment. -- Keeps track of your deployments, so you always know what is currently being deployed on your - servers. - -If you have a deployment service such as [Kubernetes](../user/project/clusters/index.md) -associated with your project, you can use it to assist with your deployments, and -can even access a [web terminal](#web-terminals) for your environment from within GitLab! - -## Configuring environments - -Configuring environments involves: - -1. Understanding how [pipelines](pipelines/index.md) work. -1. Defining environments in your project's [`.gitlab-ci.yml`](yaml/README.md) file. -1. Creating a job configured to deploy your application. For example, a deploy job configured with [`environment`](yaml/README.md#environment) to deploy your application to a [Kubernetes cluster](../user/project/clusters/index.md). - -The rest of this section illustrates how to configure environments and deployments using -an example scenario. It assumes you have already: - -- Created a [project](../gitlab-basics/create-project.md) in GitLab. -- Set up [a Runner](runners/README.md). - -In the scenario: - -- We are developing an application. -- We want to run tests and build our app on all branches. -- Our default branch is `master`. -- We deploy the app only when a pipeline on `master` branch is run. - -### Defining environments - -Let's consider the following `.gitlab-ci.yml` example: - -```yaml -stages: - - test - - build - - deploy - -test: - stage: test - script: echo "Running tests" - -build: - stage: build - script: echo "Building the app" - -deploy_staging: - stage: deploy - script: - - echo "Deploy to staging server" - environment: - name: staging - url: https://staging.example.com - only: - - master -``` - -We have defined three [stages](yaml/README.md#stages): - -- `test` -- `build` -- `deploy` - -The jobs assigned to these stages will run in this order. If any job fails, then -the pipeline fails and jobs that are assigned to the next stage won't run. - -In our case: - -- The `test` job will run first. -- Then the `build` job. -- Lastly the `deploy_staging` job. - -With this configuration, we: - -- Check that the tests pass. -- Ensure that our app is able to be built successfully. -- Lastly we deploy to the staging server. - -NOTE: **Note:** -The `environment` keyword defines where the app is deployed. -The environment `name` and `url` is exposed in various places -within GitLab. Each time a job that has an environment specified -succeeds, a deployment is recorded, along with the Git SHA, and environment name. - -CAUTION: **Caution**: -Some characters are not allowed in environment names. Use only letters, -numbers, spaces, and `-`, `_`, `/`, `{`, `}`, or `.`. Also, it must not start nor end with `/`. - -In summary, with the above `.gitlab-ci.yml` we have achieved the following: - -- All branches will run the `test` and `build` jobs. -- The `deploy_staging` job will run [only](yaml/README.md#onlyexcept-basic) on the `master` - branch, which means all merge requests that are created from branches don't - get deployed to the staging server. -- When a merge request is merged, all jobs will run and the `deploy_staging` - job will deploy our code to a staging server while the deployment - will be recorded in an environment named `staging`. - -#### Environment variables and Runner - -Starting with GitLab 8.15, the environment name is exposed to the Runner in -two forms: - -- `$CI_ENVIRONMENT_NAME`. The name given in `.gitlab-ci.yml` (with any variables - expanded). -- `$CI_ENVIRONMENT_SLUG`. A "cleaned-up" version of the name, suitable for use in URLs, - DNS, etc. - -If you change the name of an existing environment, the: - -- `$CI_ENVIRONMENT_NAME` variable will be updated with the new environment name. -- `$CI_ENVIRONMENT_SLUG` variable will remain unchanged to prevent unintended side - effects. - -Starting with GitLab 9.3, the environment URL is exposed to the Runner via -`$CI_ENVIRONMENT_URL`. The URL is expanded from either: - -- `.gitlab-ci.yml`. -- The external URL from the environment if not defined in `.gitlab-ci.yml`. - -#### Set dynamic environment URLs after a job finishes - -> [Introduced](https://gitlab.com/gitlab-org/gitlab/issues/17066) in GitLab 12.9. - -In a job script, you can specify a static [environment URL](#using-the-environment-url). -However, there may be times when you want a dynamic URL. For example, -if you deploy a Review App to an external hosting -service that generates a random URL per deployment, like `https://94dd65b.amazonaws.com/qa-lambda-1234567`, -you don't know the URL before the deployment script finishes. -If you want to use the environment URL in GitLab, you would have to update it manually. - -To address this problem, you can configure a deployment job to report back a set of -variables, including the URL that was dynamically-generated by the external service. -GitLab supports [dotenv](https://github.com/bkeepers/dotenv) file as the format, -and expands the `environment:url` value with variables defined in the dotenv file. - -To use this feature, specify the -[`artifacts:reports:dotenv`](yaml/README.md#artifactsreportsdotenv) keyword in `.gitlab-ci.yml`. - -##### Example of setting dynamic environment URLs - -The following example shows a Review App that creates a new environment -per merge request. The `review` job is triggered by every push, and -creates or updates an environment named `review/your-branch-name`. -The environment URL is set to `$DYNAMIC_ENVIRONMENT_URL`: - -```yaml -review: - script: - - DYNAMIC_ENVIRONMENT_URL=$(deploy-script) # In script, get the environment URL. - - echo "DYNAMIC_ENVIRONMENT_URL=$DYNAMIC_ENVIRONMENT_URL" >> deploy.env # Add the value to a dotenv file. - artifacts: - reports: - dotenv: deploy.env # Report back dotenv file to rails. - environment: - name: review/$CI_COMMIT_REF_SLUG - url: $DYNAMIC_ENVIRONMENT_URL # and set the variable produced in script to `environment:url` - on_stop: stop_review - -stop_review: - script: - - ./teardown-environment - when: manual - environment: - name: review/$CI_COMMIT_REF_SLUG - action: stop -``` - -As soon as the `review` job finishes, GitLab updates the `review/your-branch-name` -environment's URL. -It parses the report artifact `deploy.env`, registers a list of variables as runtime-created, -uses it for expanding `environment:url: $DYNAMIC_ENVIRONMENT_URL` and sets it to the environment URL. -You can also specify a static part of the URL at `environment:url:`, such as -`https://$DYNAMIC_ENVIRONMENT_URL`. If the value of `DYNAMIC_ENVIRONMENT_URL` is -`123.awesome.com`, the final result will be `https://123.awesome.com`. - -The assigned URL for the `review/your-branch-name` environment is visible in the UI. -[See where the environment URL is displayed](#using-the-environment-url). - -> **Notes:** -> -> - `stop_review` doesn't generate a dotenv report artifact, so it won't recognize the `DYNAMIC_ENVIRONMENT_URL` variable. Therefore you should not set `environment:url:` in the `stop_review` job. -> - If the environment URL is not valid (for example, the URL is malformed), the system doesn't update the environment URL. - -### Configuring manual deployments - -Adding `when: manual` to an automatically executed job's configuration converts it to -a job requiring manual action. - -To expand on the [previous example](#defining-environments), the following includes -another job that deploys our app to a production server and is -tracked by a `production` environment. - -The `.gitlab-ci.yml` file for this is as follows: - -```yaml -stages: - - test - - build - - deploy - -test: - stage: test - script: echo "Running tests" - -build: - stage: build - script: echo "Building the app" - -deploy_staging: - stage: deploy - script: - - echo "Deploy to staging server" - environment: - name: staging - url: https://staging.example.com - only: - - master - -deploy_prod: - stage: deploy - script: - - echo "Deploy to production server" - environment: - name: production - url: https://example.com - when: manual - only: - - master -``` - -The `when: manual` action: - -- Exposes a "play" button in GitLab's UI for that job. -- Means the `deploy_prod` job will only be triggered when the "play" button is clicked. - -You can find the "play" button in the pipelines, environments, deployments, and jobs views. - -| View | Screenshot | -|:----------------|:-------------------------------------------------------------------------------| -| Pipelines |  | -| Single pipeline |  | -| Environments |  | -| Deployments |  | -| Jobs |  | - -Clicking on the play button in any view will trigger the `deploy_prod` job, and the -deployment will be recorded as a new environment named `production`. - -NOTE: **Note:** -If your environment's name is `production` (all lowercase), -it will get recorded in [Value Stream Analytics](../user/project/cycle_analytics.md). - -### Configuring dynamic environments - -Regular environments are good when deploying to "stable" environments like staging or production. - -However, for environments for branches other than `master`, dynamic environments -can be used. Dynamic environments make it possible to create environments on the fly by -declaring their names dynamically in `.gitlab-ci.yml`. - -Dynamic environments are a fundamental part of [Review apps](review_apps/index.md). - -### Configuring incremental rollouts - -Learn how to release production changes to only a portion of your Kubernetes pods with -[incremental rollouts](environments/incremental_rollouts.md). - -#### Allowed variables - -The `name` and `url` parameters for dynamic environments can use most available CI/CD variables, -including: - -- [Predefined environment variables](variables/README.md#predefined-environment-variables) -- [Project and group variables](variables/README.md#gitlab-cicd-environment-variables) -- [`.gitlab-ci.yml` variables](yaml/README.md#variables) - -However, you cannot use variables defined: - -- Under `script`. -- On the Runner's side. - -There are also other variables that are unsupported in the context of `environment:name`. -For more information, see [Where variables can be used](variables/where_variables_can_be_used.md). - -#### Example configuration - -GitLab Runner exposes various [environment variables](variables/README.md) when a job runs, so -you can use them as environment names. - -In the following example, the job will deploy to all branches except `master`: - -```yaml -deploy_review: - stage: deploy - script: - - echo "Deploy a review app" - environment: - name: review/$CI_COMMIT_REF_NAME - url: https://$CI_ENVIRONMENT_SLUG.example.com - only: - - branches - except: - - master -``` - -In this example: - -- The job's name is `deploy_review` and it runs on the `deploy` stage. -- We set the `environment` with the `environment:name` as `review/$CI_COMMIT_REF_NAME`. - Since the [environment name](yaml/README.md#environmentname) can contain slashes (`/`), we can - use this pattern to distinguish between dynamic and regular environments. -- We tell the job to run [`only`](yaml/README.md#onlyexcept-basic) on branches, - [`except`](yaml/README.md#onlyexcept-basic) `master`. - -For the value of: - -- `environment:name`, the first part is `review`, followed by a `/` and then `$CI_COMMIT_REF_NAME`, - which receives the value of the branch name. -- `environment:url`, we want a specific and distinct URL for each branch. `$CI_COMMIT_REF_NAME` - may contain a `/` or other characters that would be invalid in a domain name or URL, - so we use `$CI_ENVIRONMENT_SLUG` to guarantee that we get a valid URL. - - For example, given a `$CI_COMMIT_REF_NAME` of `100-Do-The-Thing`, the URL will be something - like `https://100-do-the-4f99a2.example.com`. Again, the way you set up - the web server to serve these requests is based on your setup. - - We have used `$CI_ENVIRONMENT_SLUG` here because it is guaranteed to be unique. If - you're using a workflow like [GitLab Flow](../topics/gitlab_flow.md), collisions - are unlikely and you may prefer environment names to be more closely based on the - branch name. In that case, you could use `$CI_COMMIT_REF_NAME` in `environment:url` in - the example above: `https://$CI_COMMIT_REF_NAME.example.com`, which would give a URL - of `https://100-do-the-thing.example.com`. - -NOTE: **Note:** -You are not required to use the same prefix or only slashes (`/`) in the dynamic environments' -names. However, using this format will enable the [grouping similar environments](#grouping-similar-environments) -feature. - -### Configuring Kubernetes deployments - -> [Introduced](https://gitlab.com/gitlab-org/gitlab/issues/27630) in GitLab 12.6. - -If you are deploying to a [Kubernetes cluster](../user/project/clusters/index.md) -associated with your project, you can configure these deployments from your -`gitlab-ci.yml` file. - -The following configuration options are supported: - -- [`namespace`](https://kubernetes.io/docs/concepts/overview/working-with-objects/namespaces/) - -In the following example, the job will deploy your application to the -`production` Kubernetes namespace. - -```yaml -deploy: - stage: deploy - script: - - echo "Deploy to production server" - environment: - name: production - url: https://example.com - kubernetes: - namespace: production - only: - - master -``` - -When deploying to a Kubernetes cluster using GitLab's Kubernetes integration, -information about the cluster and namespace will be displayed above the job -trace on the deployment job page: - - - -NOTE: **Note:** -Kubernetes configuration is not supported for Kubernetes clusters -that are [managed by GitLab](../user/project/clusters/index.md#gitlab-managed-clusters). -To follow progress on support for GitLab-managed clusters, see the -[relevant issue](https://gitlab.com/gitlab-org/gitlab/issues/38054). - -### Complete example - -The configuration in this section provides a full development workflow where your app is: - -- Tested. -- Built. -- Deployed as a Review App. -- Deployed to a staging server once the merge request is merged. -- Finally, able to be manually deployed to the production server. - -The following combines the previous configuration examples, including: - -- Defining [simple environments](#defining-environments) for testing, building, and deployment to staging. -- Adding [manual actions](#configuring-manual-deployments) for deployment to production. -- Creating [dynamic environments](#configuring-dynamic-environments) for deployments for reviewing. - -```yaml -stages: - - test - - build - - deploy - -test: - stage: test - script: echo "Running tests" - -build: - stage: build - script: echo "Building the app" - -deploy_review: - stage: deploy - script: - - echo "Deploy a review app" - environment: - name: review/$CI_COMMIT_REF_NAME - url: https://$CI_ENVIRONMENT_SLUG.example.com - only: - - branches - except: - - master - -deploy_staging: - stage: deploy - script: - - echo "Deploy to staging server" - environment: - name: staging - url: https://staging.example.com - only: - - master - -deploy_prod: - stage: deploy - script: - - echo "Deploy to production server" - environment: - name: production - url: https://example.com - when: manual - only: - - master -``` - -A more realistic example would also include copying files to a location where a -webserver (for example, NGINX) could then access and serve them. - -The example below will copy the `public` directory to `/srv/nginx/$CI_COMMIT_REF_SLUG/public`: - -```yaml -review_app: - stage: deploy - script: - - rsync -av --delete public /srv/nginx/$CI_COMMIT_REF_SLUG - environment: - name: review/$CI_COMMIT_REF_NAME - url: https://$CI_COMMIT_REF_SLUG.example.com -``` - -This example requires that NGINX and GitLab Runner are set up on the server this job will run on. - -NOTE: **Note:** -See the [limitations](#limitations) section for some edge cases regarding the naming of -your branches and Review Apps. - -The complete example provides the following workflow to developers: - -- Create a branch locally. -- Make changes and commit them. -- Push the branch to GitLab. -- Create a merge request. - -Behind the scenes, GitLab Runner will: - -- Pick up the changes and start running the jobs. -- Run the jobs sequentially as defined in `stages`: - - First, run the tests. - - If the tests succeed, build the app. - - If the build succeeds, the app is deployed to an environment with a name specific to the - branch. - -So now, every branch: - -- Gets its own environment. -- Is deployed to its own unique location, with the added benefit of: - - Having a [history of deployments](#viewing-deployment-history). - - Being able to [rollback changes](#retrying-and-rolling-back) if needed. - -For more information, see [Using the environment URL](#using-the-environment-url). - -### Protected environments - -Environments can be "protected", restricting access to them. - -For more information, see [Protected environments](environments/protected_environments.md). - -## Working with environments - -Once environments are configured, GitLab provides many features for working with them, -as documented below. - -### Viewing environments and deployments - -A list of environments and deployment statuses is available on each project's **Operations > Environments** page. - -For example: - - - -This example shows: - -- The environment's name with a link to its deployments. -- The last deployment ID number and who performed it. -- The job ID of the last deployment with its respective job name. -- The commit information of the last deployment, such as who committed it, to what - branch, and the Git SHA of the commit. -- The exact time the last deployment was performed. -- A button that takes you to the URL that you defined under the `environment` keyword - in `.gitlab-ci.yml`. -- A button that re-deploys the latest deployment, meaning it runs the job - defined by the environment name for that specific commit. - -The information shown in the **Environments** page is limited to the latest -deployments, but an environment can have multiple deployments. - -> **Notes:** -> -> - While you can create environments manually in the web interface, we recommend -> that you define your environments in `.gitlab-ci.yml` first. They will -> be automatically created for you after the first deploy. -> - The environments page can only be viewed by users with [Reporter permission](../user/permissions.md#project-members-permissions) -> and above. For more information on permissions, see the [permissions documentation](../user/permissions.md). -> - Only deploys that happen after your `.gitlab-ci.yml` is properly configured -> will show up in the **Environment** and **Last deployment** lists. - -### Viewing deployment history - -GitLab keeps track of your deployments, so you: - -- Always know what is currently being deployed on your servers. -- Can have the full history of your deployments for every environment. - -Clicking on an environment shows the history of its deployments. Here's an example **Environments** page -with multiple deployments: - - - -This view is similar to the **Environments** page, but all deployments are shown. Also in this view -is a **Rollback** button. For more information, see [Retrying and rolling back](#retrying-and-rolling-back). - -### Retrying and rolling back - -If there is a problem with a deployment, you can retry it or roll it back. - -To retry or rollback a deployment: - -1. Navigate to **Operations > Environments**. -1. Click on the environment. -1. In the deployment history list for the environment, click the: - - **Retry** button next to the last deployment, to retry that deployment. - - **Rollback** button next to a previously successful deployment, to roll back to that deployment. - -#### What to expect with a rollback - -Pressing the **Rollback** button on a specific commit will trigger a _new_ deployment with its -own unique job ID. - -This means that you will see a new deployment that points to the commit you are rolling back to. - -NOTE: **Note:** -The defined deployment process in the job's `script` determines whether the rollback succeeds or not. - -### Using the environment URL - -The [environment URL](yaml/README.md#environmenturl) is exposed in a few -places within GitLab: - -- In a merge request widget as a link: -  -- In the Environments view as a button: -  -- In the Deployments view as a button: -  - -You can see this information in a merge request itself if: - -- The merge request is eventually merged to the default branch (usually `master`). -- That branch also deploys to an environment (for example, `staging` or `production`). - -For example: - - - -#### Going from source files to public pages - -With GitLab's [Route Maps](review_apps/index.md#route-maps) you can go directly -from source files to public pages in the environment set for Review Apps. - -### Stopping an environment - -Stopping an environment: - -- Moves it from the list of **Available** environments to the list of **Stopped** - environments on the [**Environments** page](#viewing-environments-and-deployments). -- Executes an [`on_stop` action](yaml/README.md#environmenton_stop), if defined. - -This is often used when multiple developers are working on a project at the same time, -each of them pushing to their own branches, causing many dynamic environments to be created. - -NOTE: **Note:** -Starting with GitLab 8.14, dynamic environments are stopped automatically -when their associated branch is deleted. - -#### Automatically stopping an environment - -Environments can be stopped automatically using special configuration. - -Consider the following example where the `deploy_review` job calls `stop_review` -to clean up and stop the environment: - -```yaml -deploy_review: - stage: deploy - script: - - echo "Deploy a review app" - environment: - name: review/$CI_COMMIT_REF_NAME - url: https://$CI_ENVIRONMENT_SLUG.example.com - on_stop: stop_review - only: - - branches - except: - - master - -stop_review: - stage: deploy - variables: - GIT_STRATEGY: none - script: - - echo "Remove review app" - when: manual - environment: - name: review/$CI_COMMIT_REF_NAME - action: stop -``` - -Setting the [`GIT_STRATEGY`](yaml/README.md#git-strategy) to `none` is necessary in the -`stop_review` job so that the [GitLab Runner](https://docs.gitlab.com/runner/) won't -try to check out the code after the branch is deleted. - -When you have an environment that has a stop action defined (typically when -the environment describes a Review App), GitLab will automatically trigger a -stop action when the associated branch is deleted. The `stop_review` job must -be in the same `stage` as the `deploy_review` job in order for the environment -to automatically stop. - -You can read more in the [`.gitlab-ci.yml` reference](yaml/README.md#environmenton_stop). - -#### Environments auto-stop - -> [Introduced](https://gitlab.com/gitlab-org/gitlab/issues/20956) in GitLab 12.8. - -You can set a expiry time to environments and stop them automatically after a certain period. - -For example, consider the use of this feature with Review Apps environments. -When you set up Review Apps, sometimes they keep running for a long time -because some merge requests are left as open. An example for this situation is when the author of the merge -request is not actively working on it, due to priority changes or a different approach was decided on, and the merge requests was simply forgotten. -Idle environments waste resources, therefore they -should be terminated as soon as possible. - -To address this problem, you can specify an optional expiration date for -Review Apps environments. When the expiry time is reached, GitLab will automatically trigger a job -to stop the environment, eliminating the need of manually doing so. In case an environment is updated, the expiration is renewed -ensuring that only active merge requests keep running Review Apps. - -To enable this feature, you need to specify the [`environment:auto_stop_in`](yaml/README.md#environmentauto_stop_in) keyword in `.gitlab-ci.yml`. -You can specify a human-friendly date as the value, such as `1 hour and 30 minutes` or `1 day`. -`auto_stop_in` uses the same format of [`artifacts:expire_in` docs](yaml/README.md#artifactsexpire_in). - -##### Auto-stop example - -In the following example, there is a basic review app setup that creates a new environment -per merge request. The `review_app` job is triggered by every push and -creates or updates an environment named `review/your-branch-name`. -The environment keeps running until `stop_review_app` is executed: - -```yaml -review_app: - script: deploy-review-app - environment: - name: review/$CI_COMMIT_REF_NAME - on_stop: stop_review_app - auto_stop_in: 1 week - -stop_review_app: - script: stop-review-app - environment: - name: review/$CI_COMMIT_REF_NAME - action: stop - when: manual -``` - -As long as a merge request is active and keeps getting new commits, -the review app will not stop, so developers don't need to worry about -re-initiating review app. - -On the other hand, since `stop_review_app` is set to `auto_stop_in: 1 week`, -if a merge request becomes inactive for more than a week, -GitLab automatically triggers the `stop_review_app` job to stop the environment. - -You can also check the expiration date of environments through the GitLab UI. To do so, -go to **Operations > Environments > Environment**. You can see the auto-stop period -at the left-top section and a pin-mark button at the right-top section. This pin-mark -button can be used to prevent auto-stopping the environment. By clicking this button, the `auto_stop_in` setting is over-written -and the environment will be active until it's stopped manually. - - - -NOTE: **NOTE** -Due to the resource limitation, a background worker for stopping environments only -runs once every hour. This means environments will not be stopped at the exact -timestamp as the specified period, but will be stopped when the hourly cron worker -detects expired environments. - -#### Delete a stopped environment - -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/22629) in GitLab 12.9. - -You can delete [stopped environments](#stopping-an-environment) in one of two -ways: through the GitLab UI or through the API. - -##### Delete environments through the UI - -To view the list of **Stopped** environments, navigate to **Operations > Environments** -and click the **Stopped** tab. - -From there, you can click the **Delete** button directly, or you can click the -environment name to see its details and **Delete** it from there. - -You can also delete environments by viewing the details for a -stopped environment: - - 1. Navigate to **Operations > Environments**. - 1. Click on the name of an environment within the **Stopped** environments list. - 1. Click on the **Delete** button that appears at the top for all stopped environments. - 1. Finally, confirm your chosen environment in the modal that appears to delete it. - -##### Delete environments through the API - -Environments can also be deleted by using the [Environments API](../api/environments.md#delete-an-environment). - -### Grouping similar environments - -> [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/7015) in GitLab 8.14. - -As documented in [Configuring dynamic environments](#configuring-dynamic-environments), you can -prepend environment name with a word, followed by a `/`, and finally the branch -name, which is automatically defined by the `CI_COMMIT_REF_NAME` variable. - -In short, environments that are named like `type/foo` are all presented under the same -group, named `type`. - -In our [minimal example](#example-configuration), we named the environments `review/$CI_COMMIT_REF_NAME` -where `$CI_COMMIT_REF_NAME` is the branch name. Here is a snippet of the example: - -```yaml -deploy_review: - stage: deploy - script: - - echo "Deploy a review app" - environment: - name: review/$CI_COMMIT_REF_NAME -``` - -In this case, if you visit the **Environments** page and the branches -exist, you should see something like: - - - -### Monitoring environments - -If you have enabled [Prometheus for monitoring system and response metrics](../user/project/integrations/prometheus.md), -you can monitor the behavior of your app running in each environment. For the monitoring -dashboard to appear, you need to Configure Prometheus to collect at least one -[supported metric](../user/project/integrations/prometheus_library/index.md). - -NOTE: **Note:** -Since GitLab 9.2, all deployments to an environment are shown directly on the monitoring dashboard. - -Once configured, GitLab will attempt to retrieve [supported performance metrics](../user/project/integrations/prometheus_library/index.md) -for any environment that has had a successful deployment. If monitoring data was -successfully retrieved, a **Monitoring** button will appear for each environment. - - - -Clicking on the **Monitoring** button will display a new page showing up to the last -8 hours of performance data. It may take a minute or two for data to appear -after initial deployment. - -All deployments to an environment are shown directly on the monitoring dashboard, -which allows easy correlation between any changes in performance and new -versions of the app, all without leaving GitLab. - - - -#### Linking to external dashboard - -Add a [button to the Monitoring dashboard](../user/project/operations/linking_to_an_external_dashboard.md) linking directly to your existing external dashboards. - -#### Embedding metrics in GitLab Flavored Markdown - -Metric charts can be embedded within GitLab Flavored Markdown. See [Embedding Metrics within GitLab Flavored Markdown](../user/project/integrations/prometheus.md#embedding-metric-charts-within-gitlab-flavored-markdown) for more details. - -### Web terminals - -> Web terminals were added in GitLab 8.15 and are only available to project Maintainers and Owners. - -If you deploy to your environments with the help of a deployment service (for example, -the [Kubernetes integration](../user/project/clusters/index.md)), GitLab can open -a terminal session to your environment. - -This is a powerful feature that allows you to debug issues without leaving the comfort -of your web browser. To enable it, just follow the instructions given in the service integration -documentation. - -Once enabled, your environments will gain a "terminal" button: - - - -You can also access the terminal button from the page for a specific environment: - - - -Wherever you find it, clicking the button will take you to a separate page to -establish the terminal session: - - - -This works just like any other terminal. You'll be in the container created -by your deployment so you can: - -- Run shell commands and get responses in real time. -- Check the logs. -- Try out configuration or code tweaks etc. - -You can open multiple terminals to the same environment, they each get their own shell -session and even a multiplexer like `screen` or `tmux`. - -NOTE: **Note:** -Container-based deployments often lack basic tools (like an editor), and may -be stopped or restarted at any time. If this happens, you will lose all your -changes. Treat this as a debugging tool, not a comprehensive online IDE. - -### Check out deployments locally - -Since GitLab 8.13, a reference in the Git repository is saved for each deployment, so -knowing the state of your current environments is only a `git fetch` away. - -In your Git configuration, append the `[remote "<your-remote>"]` block with an extra -fetch line: - -```text -fetch = +refs/environments/*:refs/remotes/origin/environments/* -``` - -### Scoping environments with specs - -> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/2112) in [GitLab Premium](https://about.gitlab.com/pricing/) 9.4. -> - [Scoping for environment variables was moved to Core](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/30779) to Core in GitLab 12.2. - -You can limit the environment scope of a variable by -defining which environments it can be available for. - -Wildcards can be used, and the default environment scope is `*`, which means -any jobs will have this variable, not matter if an environment is defined or -not. - -For example, if the environment scope is `production`, then only the jobs -having the environment `production` defined would have this specific variable. -Wildcards (`*`) can be used along with the environment name, therefore if the -environment scope is `review/*` then any jobs with environment names starting -with `review/` would have that particular variable. - -Some GitLab features can behave differently for each environment. -For example, you can -[create a secret variable to be injected only into a production environment](variables/README.md#limiting-environment-scopes-of-environment-variables). - -In most cases, these features use the _environment specs_ mechanism, which offers -an efficient way to implement scoping within each environment group. - -Let's say there are four environments: - -- `production` -- `staging` -- `review/feature-1` -- `review/feature-2` - -Each environment can be matched with the following environment spec: - -| Environment Spec | `production` | `staging` | `review/feature-1` | `review/feature-2` | -|:-----------------|:-------------|:----------|:-------------------|:-------------------| -| * | Matched | Matched | Matched | Matched | -| production | Matched | | | | -| staging | | Matched | | | -| review/* | | | Matched | Matched | -| review/feature-1 | | | Matched | | - -As you can see, you can use specific matching for selecting a particular environment, -and also use wildcard matching (`*`) for selecting a particular environment group, -such as [Review Apps](review_apps/index.md) (`review/*`). - -NOTE: **Note:** -The most _specific_ spec takes precedence over the other wildcard matching. -In this case, `review/feature-1` spec takes precedence over `review/*` and `*` specs. - -### Environments Dashboard **(PREMIUM)** - -See [Environments Dashboard](environments/environments_dashboard.md) for a summary of each -environment's operational health. - -## Limitations - -In the `environment: name`, you are limited to only the [predefined environment variables](variables/predefined_variables.md). -Re-using variables defined inside `script` as part of the environment name will not work. - -## Further reading - -Below are some links you may find interesting: - -- [The `.gitlab-ci.yml` definition of environments](yaml/README.md#environment) -- [A blog post on Deployments & Environments](https://about.gitlab.com/blog/2016/08/26/ci-deployment-and-environments/) -- [Review Apps - Use dynamic environments to deploy your code for every branch](review_apps/index.md) -- [Deploy Boards for your applications running on Kubernetes](../user/project/deploy_boards.md) **(PREMIUM)** - -<!-- ## Troubleshooting - -Include any troubleshooting steps that you can foresee. If you know beforehand what issues -one might have when setting this up, or when something is changed, or on upgrading, it's -important to describe those, too. Think of things that may go wrong and include them here. -This is important to minimize requests for support, and to avoid doc comments with -questions that you know someone might ask. - -Each scenario can be a third-level heading, e.g. `### Getting error message X`. -If you have none to add when creating a doc, leave this section in place -but commented out to help encourage others to add to it in the future. --> +This document was moved to [another location](environments/index.md). diff --git a/doc/ci/environments/environments_dashboard.md b/doc/ci/environments/environments_dashboard.md index f82728cd587..4a72c0d6d62 100644 --- a/doc/ci/environments/environments_dashboard.md +++ b/doc/ci/environments/environments_dashboard.md @@ -1,4 +1,7 @@ --- +stage: Release +group: Release Management +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers type: reference --- diff --git a/doc/ci/environments/incremental_rollouts.md b/doc/ci/environments/incremental_rollouts.md index 0fa0af6a9fb..016a6ac7cad 100644 --- a/doc/ci/environments/incremental_rollouts.md +++ b/doc/ci/environments/incremental_rollouts.md @@ -46,7 +46,7 @@ application will be deployed to a single pod while the remaining 9 will present First we [define the template as manual](https://gitlab.com/gl-release/incremental-rollout-example/blob/master/.gitlab-ci.yml#L100-103): -```yml +```yaml .manual_rollout_template: &manual_rollout_template <<: *rollout_template stage: production @@ -55,7 +55,7 @@ First we [define the template as manual](https://gitlab.com/gl-release/increment Then we [define the rollout amount for each step](https://gitlab.com/gl-release/incremental-rollout-example/blob/master/.gitlab-ci.yml#L152-155): -```yml +```yaml rollout 10%: <<: *manual_rollout_template variables: @@ -86,7 +86,7 @@ countdown and then deploy. First we [define the template as timed](https://gitlab.com/gl-release/timed-rollout-example/blob/master/.gitlab-ci.yml#L86-89): -```yml +```yaml .timed_rollout_template: &timed_rollout_template <<: *rollout_template when: delayed @@ -95,13 +95,13 @@ First we [define the template as timed](https://gitlab.com/gl-release/timed-roll We can define the delay period using the `start_in` key: -```yml +```yaml start_in: 1 minutes ``` Then we [define the rollout amount for each step](https://gitlab.com/gl-release/timed-rollout-example/blob/master/.gitlab-ci.yml#L97-101): -```yml +```yaml timed rollout 30%: <<: *timed_rollout_template stage: timed rollout 30% diff --git a/doc/ci/environments/index.md b/doc/ci/environments/index.md new file mode 100644 index 00000000000..b6ec30b5571 --- /dev/null +++ b/doc/ci/environments/index.md @@ -0,0 +1,991 @@ +--- +stage: Release +group: Release Management +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +type: reference +disqus_identifier: 'https://docs.gitlab.com/ee/ci/environments.html' +--- + +# Environments and deployments + +> Introduced in GitLab 8.9. + +Environments allow control of the continuous deployment of your software, +all within GitLab. + +## Introduction + +There are many stages required in the software development process before the software is ready +for public consumption. + +For example: + +1. Develop your code. +1. Test your code. +1. Deploy your code into a testing or staging environment before you release it to the public. + +This helps find bugs in your software, and also in the deployment process as well. + +GitLab CI/CD is capable of not only testing or building your projects, but also +deploying them in your infrastructure, with the added benefit of giving you a +way to track your deployments. In other words, you will always know what is +currently being deployed or has been deployed on your servers. + +It's important to know that: + +- Environments are like tags for your CI jobs, describing where code gets deployed. +- Deployments are created when [jobs](../yaml/README.md#introduction) deploy versions of code to environments, + so every environment can have one or more deployments. + +GitLab: + +- Provides a full history of your deployments for each environment. +- Keeps track of your deployments, so you always know what is currently being deployed on your + servers. + +If you have a deployment service such as [Kubernetes](../../user/project/clusters/index.md) +associated with your project, you can use it to assist with your deployments, and +can even access a [web terminal](#web-terminals) for your environment from within GitLab! + +## Configuring environments + +Configuring environments involves: + +1. Understanding how [pipelines](../pipelines/index.md) work. +1. Defining environments in your project's [`.gitlab-ci.yml`](../yaml/README.md) file. +1. Creating a job configured to deploy your application. For example, a deploy job configured with [`environment`](../yaml/README.md#environment) to deploy your application to a [Kubernetes cluster](../../user/project/clusters/index.md). + +The rest of this section illustrates how to configure environments and deployments using +an example scenario. It assumes you have already: + +- Created a [project](../../gitlab-basics/create-project.md) in GitLab. +- Set up [a Runner](../runners/README.md). + +In the scenario: + +- We are developing an application. +- We want to run tests and build our app on all branches. +- Our default branch is `master`. +- We deploy the app only when a pipeline on `master` branch is run. + +### Defining environments + +Let's consider the following `.gitlab-ci.yml` example: + +```yaml +stages: + - test + - build + - deploy + +test: + stage: test + script: echo "Running tests" + +build: + stage: build + script: echo "Building the app" + +deploy_staging: + stage: deploy + script: + - echo "Deploy to staging server" + environment: + name: staging + url: https://staging.example.com + only: + - master +``` + +We have defined three [stages](../yaml/README.md#stages): + +- `test` +- `build` +- `deploy` + +The jobs assigned to these stages will run in this order. If any job fails, then +the pipeline fails and jobs that are assigned to the next stage won't run. + +In our case: + +- The `test` job will run first. +- Then the `build` job. +- Lastly the `deploy_staging` job. + +With this configuration, we: + +- Check that the tests pass. +- Ensure that our app is able to be built successfully. +- Lastly we deploy to the staging server. + +NOTE: **Note:** +The `environment` keyword defines where the app is deployed. +The environment `name` and `url` is exposed in various places +within GitLab. Each time a job that has an environment specified +succeeds, a deployment is recorded, along with the Git SHA, and environment name. + +CAUTION: **Caution**: +Some characters are not allowed in environment names. Use only letters, +numbers, spaces, and `-`, `_`, `/`, `{`, `}`, or `.`. Also, it must not start nor end with `/`. + +In summary, with the above `.gitlab-ci.yml` we have achieved the following: + +- All branches will run the `test` and `build` jobs. +- The `deploy_staging` job will run [only](../yaml/README.md#onlyexcept-basic) on the `master` + branch, which means all merge requests that are created from branches don't + get deployed to the staging server. +- When a merge request is merged, all jobs will run and the `deploy_staging` + job will deploy our code to a staging server while the deployment + will be recorded in an environment named `staging`. + +#### Environment variables and Runner + +Starting with GitLab 8.15, the environment name is exposed to the Runner in +two forms: + +- `$CI_ENVIRONMENT_NAME`. The name given in `.gitlab-ci.yml` (with any variables + expanded). +- `$CI_ENVIRONMENT_SLUG`. A "cleaned-up" version of the name, suitable for use in URLs, + DNS, etc. + +If you change the name of an existing environment, the: + +- `$CI_ENVIRONMENT_NAME` variable will be updated with the new environment name. +- `$CI_ENVIRONMENT_SLUG` variable will remain unchanged to prevent unintended side + effects. + +Starting with GitLab 9.3, the environment URL is exposed to the Runner via +`$CI_ENVIRONMENT_URL`. The URL is expanded from either: + +- `.gitlab-ci.yml`. +- The external URL from the environment if not defined in `.gitlab-ci.yml`. + +#### Set dynamic environment URLs after a job finishes + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/issues/17066) in GitLab 12.9. + +In a job script, you can specify a static [environment URL](#using-the-environment-url). +However, there may be times when you want a dynamic URL. For example, +if you deploy a Review App to an external hosting +service that generates a random URL per deployment, like `https://94dd65b.amazonaws.com/qa-lambda-1234567`, +you don't know the URL before the deployment script finishes. +If you want to use the environment URL in GitLab, you would have to update it manually. + +To address this problem, you can configure a deployment job to report back a set of +variables, including the URL that was dynamically-generated by the external service. +GitLab supports [dotenv](https://github.com/bkeepers/dotenv) file as the format, +and expands the `environment:url` value with variables defined in the dotenv file. + +To use this feature, specify the +[`artifacts:reports:dotenv`](../pipelines/job_artifacts.md#artifactsreportsdotenv) keyword in `.gitlab-ci.yml`. + +<i class="fa fa-youtube-play youtube" aria-hidden="true"></i> +For an overview, see [Set dynamic URLs after a job finished](https://youtu.be/70jDXtOf4Ig). + +##### Example of setting dynamic environment URLs + +The following example shows a Review App that creates a new environment +per merge request. The `review` job is triggered by every push, and +creates or updates an environment named `review/your-branch-name`. +The environment URL is set to `$DYNAMIC_ENVIRONMENT_URL`: + +```yaml +review: + script: + - DYNAMIC_ENVIRONMENT_URL=$(deploy-script) # In script, get the environment URL. + - echo "DYNAMIC_ENVIRONMENT_URL=$DYNAMIC_ENVIRONMENT_URL" >> deploy.env # Add the value to a dotenv file. + artifacts: + reports: + dotenv: deploy.env # Report back dotenv file to rails. + environment: + name: review/$CI_COMMIT_REF_SLUG + url: $DYNAMIC_ENVIRONMENT_URL # and set the variable produced in script to `environment:url` + on_stop: stop_review + +stop_review: + script: + - ./teardown-environment + when: manual + environment: + name: review/$CI_COMMIT_REF_SLUG + action: stop +``` + +As soon as the `review` job finishes, GitLab updates the `review/your-branch-name` +environment's URL. +It parses the report artifact `deploy.env`, registers a list of variables as runtime-created, +uses it for expanding `environment:url: $DYNAMIC_ENVIRONMENT_URL` and sets it to the environment URL. +You can also specify a static part of the URL at `environment:url:`, such as +`https://$DYNAMIC_ENVIRONMENT_URL`. If the value of `DYNAMIC_ENVIRONMENT_URL` is +`123.awesome.com`, the final result will be `https://123.awesome.com`. + +The assigned URL for the `review/your-branch-name` environment is visible in the UI. +[See where the environment URL is displayed](#using-the-environment-url). + +> **Notes:** +> +> - `stop_review` doesn't generate a dotenv report artifact, so it won't recognize the `DYNAMIC_ENVIRONMENT_URL` variable. Therefore you should not set `environment:url:` in the `stop_review` job. +> - If the environment URL is not valid (for example, the URL is malformed), the system doesn't update the environment URL. + +### Configuring manual deployments + +Adding `when: manual` to an automatically executed job's configuration converts it to +a job requiring manual action. + +To expand on the [previous example](#defining-environments), the following includes +another job that deploys our app to a production server and is +tracked by a `production` environment. + +The `.gitlab-ci.yml` file for this is as follows: + +```yaml +stages: + - test + - build + - deploy + +test: + stage: test + script: echo "Running tests" + +build: + stage: build + script: echo "Building the app" + +deploy_staging: + stage: deploy + script: + - echo "Deploy to staging server" + environment: + name: staging + url: https://staging.example.com + only: + - master + +deploy_prod: + stage: deploy + script: + - echo "Deploy to production server" + environment: + name: production + url: https://example.com + when: manual + only: + - master +``` + +The `when: manual` action: + +- Exposes a "play" button in GitLab's UI for that job. +- Means the `deploy_prod` job will only be triggered when the "play" button is clicked. + +You can find the "play" button in the pipelines, environments, deployments, and jobs views. + +| View | Screenshot | +|:----------------|:-------------------------------------------------------------------------------| +| Pipelines |  | +| Single pipeline |  | +| Environments |  | +| Deployments |  | +| Jobs |  | + +Clicking on the play button in any view will trigger the `deploy_prod` job, and the +deployment will be recorded as a new environment named `production`. + +NOTE: **Note:** +If your environment's name is `production` (all lowercase), +it will get recorded in [Value Stream Analytics](../../user/project/cycle_analytics.md). + +### Configuring dynamic environments + +Regular environments are good when deploying to "stable" environments like staging or production. + +However, for environments for branches other than `master`, dynamic environments +can be used. Dynamic environments make it possible to create environments on the fly by +declaring their names dynamically in `.gitlab-ci.yml`. + +Dynamic environments are a fundamental part of [Review apps](../review_apps/index.md). + +### Configuring incremental rollouts + +Learn how to release production changes to only a portion of your Kubernetes pods with +[incremental rollouts](../environments/incremental_rollouts.md). + +#### Allowed variables + +The `name` and `url` parameters for dynamic environments can use most available CI/CD variables, +including: + +- [Predefined environment variables](../variables/README.md#predefined-environment-variables) +- [Project and group variables](../variables/README.md#gitlab-cicd-environment-variables) +- [`.gitlab-ci.yml` variables](../yaml/README.md#variables) + +However, you cannot use variables defined: + +- Under `script`. +- On the Runner's side. + +There are also other variables that are unsupported in the context of `environment:name`. +For more information, see [Where variables can be used](../variables/where_variables_can_be_used.md). + +#### Example configuration + +GitLab Runner exposes various [environment variables](../variables/README.md) when a job runs, so +you can use them as environment names. + +In the following example, the job will deploy to all branches except `master`: + +```yaml +deploy_review: + stage: deploy + script: + - echo "Deploy a review app" + environment: + name: review/$CI_COMMIT_REF_NAME + url: https://$CI_ENVIRONMENT_SLUG.example.com + only: + - branches + except: + - master +``` + +In this example: + +- The job's name is `deploy_review` and it runs on the `deploy` stage. +- We set the `environment` with the `environment:name` as `review/$CI_COMMIT_REF_NAME`. + Since the [environment name](../yaml/README.md#environmentname) can contain slashes (`/`), we can + use this pattern to distinguish between dynamic and regular environments. +- We tell the job to run [`only`](../yaml/README.md#onlyexcept-basic) on branches, + [`except`](../yaml/README.md#onlyexcept-basic) `master`. + +For the value of: + +- `environment:name`, the first part is `review`, followed by a `/` and then `$CI_COMMIT_REF_NAME`, + which receives the value of the branch name. +- `environment:url`, we want a specific and distinct URL for each branch. `$CI_COMMIT_REF_NAME` + may contain a `/` or other characters that would be invalid in a domain name or URL, + so we use `$CI_ENVIRONMENT_SLUG` to guarantee that we get a valid URL. + + For example, given a `$CI_COMMIT_REF_NAME` of `100-Do-The-Thing`, the URL will be something + like `https://100-do-the-4f99a2.example.com`. Again, the way you set up + the web server to serve these requests is based on your setup. + + We have used `$CI_ENVIRONMENT_SLUG` here because it is guaranteed to be unique. If + you're using a workflow like [GitLab Flow](../../topics/gitlab_flow.md), collisions + are unlikely and you may prefer environment names to be more closely based on the + branch name. In that case, you could use `$CI_COMMIT_REF_NAME` in `environment:url` in + the example above: `https://$CI_COMMIT_REF_NAME.example.com`, which would give a URL + of `https://100-do-the-thing.example.com`. + +NOTE: **Note:** +You are not required to use the same prefix or only slashes (`/`) in the dynamic environments' +names. However, using this format will enable the [grouping similar environments](#grouping-similar-environments) +feature. + +### Configuring Kubernetes deployments + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/issues/27630) in GitLab 12.6. + +If you are deploying to a [Kubernetes cluster](../../user/project/clusters/index.md) +associated with your project, you can configure these deployments from your +`gitlab-ci.yml` file. + +The following configuration options are supported: + +- [`namespace`](https://kubernetes.io/docs/concepts/overview/working-with-objects/namespaces/) + +In the following example, the job will deploy your application to the +`production` Kubernetes namespace. + +```yaml +deploy: + stage: deploy + script: + - echo "Deploy to production server" + environment: + name: production + url: https://example.com + kubernetes: + namespace: production + only: + - master +``` + +When deploying to a Kubernetes cluster using GitLab's Kubernetes integration, +information about the cluster and namespace will be displayed above the job +trace on the deployment job page: + + + +NOTE: **Note:** +Kubernetes configuration is not supported for Kubernetes clusters +that are [managed by GitLab](../../user/project/clusters/index.md#gitlab-managed-clusters). +To follow progress on support for GitLab-managed clusters, see the +[relevant issue](https://gitlab.com/gitlab-org/gitlab/issues/38054). + +### Complete example + +The configuration in this section provides a full development workflow where your app is: + +- Tested. +- Built. +- Deployed as a Review App. +- Deployed to a staging server once the merge request is merged. +- Finally, able to be manually deployed to the production server. + +The following combines the previous configuration examples, including: + +- Defining [simple environments](#defining-environments) for testing, building, and deployment to staging. +- Adding [manual actions](#configuring-manual-deployments) for deployment to production. +- Creating [dynamic environments](#configuring-dynamic-environments) for deployments for reviewing. + +```yaml +stages: + - test + - build + - deploy + +test: + stage: test + script: echo "Running tests" + +build: + stage: build + script: echo "Building the app" + +deploy_review: + stage: deploy + script: + - echo "Deploy a review app" + environment: + name: review/$CI_COMMIT_REF_NAME + url: https://$CI_ENVIRONMENT_SLUG.example.com + only: + - branches + except: + - master + +deploy_staging: + stage: deploy + script: + - echo "Deploy to staging server" + environment: + name: staging + url: https://staging.example.com + only: + - master + +deploy_prod: + stage: deploy + script: + - echo "Deploy to production server" + environment: + name: production + url: https://example.com + when: manual + only: + - master +``` + +A more realistic example would also include copying files to a location where a +webserver (for example, NGINX) could then access and serve them. + +The example below will copy the `public` directory to `/srv/nginx/$CI_COMMIT_REF_SLUG/public`: + +```yaml +review_app: + stage: deploy + script: + - rsync -av --delete public /srv/nginx/$CI_COMMIT_REF_SLUG + environment: + name: review/$CI_COMMIT_REF_NAME + url: https://$CI_COMMIT_REF_SLUG.example.com +``` + +This example requires that NGINX and GitLab Runner are set up on the server this job will run on. + +NOTE: **Note:** +See the [limitations](#limitations) section for some edge cases regarding the naming of +your branches and Review Apps. + +The complete example provides the following workflow to developers: + +- Create a branch locally. +- Make changes and commit them. +- Push the branch to GitLab. +- Create a merge request. + +Behind the scenes, GitLab Runner will: + +- Pick up the changes and start running the jobs. +- Run the jobs sequentially as defined in `stages`: + - First, run the tests. + - If the tests succeed, build the app. + - If the build succeeds, the app is deployed to an environment with a name specific to the + branch. + +So now, every branch: + +- Gets its own environment. +- Is deployed to its own unique location, with the added benefit of: + - Having a [history of deployments](#viewing-deployment-history). + - Being able to [rollback changes](#retrying-and-rolling-back) if needed. + +For more information, see [Using the environment URL](#using-the-environment-url). + +### Protected environments + +Environments can be "protected", restricting access to them. + +For more information, see [Protected environments](protected_environments.md). + +## Working with environments + +Once environments are configured, GitLab provides many features for working with them, +as documented below. + +### Viewing environments and deployments + +A list of environments and deployment statuses is available on each project's **Operations > Environments** page. + +For example: + + + +This example shows: + +- The environment's name with a link to its deployments. +- The last deployment ID number and who performed it. +- The job ID of the last deployment with its respective job name. +- The commit information of the last deployment, such as who committed it, to what + branch, and the Git SHA of the commit. +- The exact time the last deployment was performed. +- A button that takes you to the URL that you defined under the `environment` keyword + in `.gitlab-ci.yml`. +- A button that re-deploys the latest deployment, meaning it runs the job + defined by the environment name for that specific commit. + +The information shown in the **Environments** page is limited to the latest +deployments, but an environment can have multiple deployments. + +> **Notes:** +> +> - While you can create environments manually in the web interface, we recommend +> that you define your environments in `.gitlab-ci.yml` first. They will +> be automatically created for you after the first deploy. +> - The environments page can only be viewed by users with [Reporter permission](../../user/permissions.md#project-members-permissions) +> and above. For more information on permissions, see the [permissions documentation](../../user/permissions.md). +> - Only deploys that happen after your `.gitlab-ci.yml` is properly configured +> will show up in the **Environment** and **Last deployment** lists. + +### Viewing deployment history + +GitLab keeps track of your deployments, so you: + +- Always know what is currently being deployed on your servers. +- Can have the full history of your deployments for every environment. + +Clicking on an environment shows the history of its deployments. Here's an example **Environments** page +with multiple deployments: + + + +This view is similar to the **Environments** page, but all deployments are shown. Also in this view +is a **Rollback** button. For more information, see [Retrying and rolling back](#retrying-and-rolling-back). + +### Retrying and rolling back + +If there is a problem with a deployment, you can retry it or roll it back. + +To retry or rollback a deployment: + +1. Navigate to **Operations > Environments**. +1. Click on the environment. +1. In the deployment history list for the environment, click the: + - **Retry** button next to the last deployment, to retry that deployment. + - **Rollback** button next to a previously successful deployment, to roll back to that deployment. + +#### What to expect with a rollback + +Pressing the **Rollback** button on a specific commit will trigger a _new_ deployment with its +own unique job ID. + +This means that you will see a new deployment that points to the commit you are rolling back to. + +NOTE: **Note:** +The defined deployment process in the job's `script` determines whether the rollback succeeds or not. + +### Using the environment URL + +The [environment URL](../yaml/README.md#environmenturl) is exposed in a few +places within GitLab: + +- In a merge request widget as a link: +  +- In the Environments view as a button: +  +- In the Deployments view as a button: +  + +You can see this information in a merge request itself if: + +- The merge request is eventually merged to the default branch (usually `master`). +- That branch also deploys to an environment (for example, `staging` or `production`). + +For example: + + + +#### Going from source files to public pages + +With GitLab's [Route Maps](../review_apps/index.md#route-maps) you can go directly +from source files to public pages in the environment set for Review Apps. + +### Stopping an environment + +Stopping an environment: + +- Moves it from the list of **Available** environments to the list of **Stopped** + environments on the [**Environments** page](#viewing-environments-and-deployments). +- Executes an [`on_stop` action](../yaml/README.md#environmenton_stop), if defined. + +This is often used when multiple developers are working on a project at the same time, +each of them pushing to their own branches, causing many dynamic environments to be created. + +NOTE: **Note:** +Starting with GitLab 8.14, dynamic environments are stopped automatically +when their associated branch is deleted. + +#### Automatically stopping an environment + +Environments can be stopped automatically using special configuration. + +Consider the following example where the `deploy_review` job calls `stop_review` +to clean up and stop the environment: + +```yaml +deploy_review: + stage: deploy + script: + - echo "Deploy a review app" + environment: + name: review/$CI_COMMIT_REF_NAME + url: https://$CI_ENVIRONMENT_SLUG.example.com + on_stop: stop_review + only: + - branches + except: + - master + +stop_review: + stage: deploy + variables: + GIT_STRATEGY: none + script: + - echo "Remove review app" + when: manual + environment: + name: review/$CI_COMMIT_REF_NAME + action: stop +``` + +Setting the [`GIT_STRATEGY`](../yaml/README.md#git-strategy) to `none` is necessary in the +`stop_review` job so that the [GitLab Runner](https://docs.gitlab.com/runner/) won't +try to check out the code after the branch is deleted. + +When you have an environment that has a stop action defined (typically when +the environment describes a Review App), GitLab will automatically trigger a +stop action when the associated branch is deleted. The `stop_review` job must +be in the same `stage` as the `deploy_review` job in order for the environment +to automatically stop. + +You can read more in the [`.gitlab-ci.yml` reference](../yaml/README.md#environmenton_stop). + +#### Environments auto-stop + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/issues/20956) in GitLab 12.8. + +You can set a expiry time to environments and stop them automatically after a certain period. + +For example, consider the use of this feature with Review Apps environments. +When you set up Review Apps, sometimes they keep running for a long time +because some merge requests are left as open. An example for this situation is when the author of the merge +request is not actively working on it, due to priority changes or a different approach was decided on, and the merge requests was simply forgotten. +Idle environments waste resources, therefore they +should be terminated as soon as possible. + +To address this problem, you can specify an optional expiration date for +Review Apps environments. When the expiry time is reached, GitLab will automatically trigger a job +to stop the environment, eliminating the need of manually doing so. In case an environment is updated, the expiration is renewed +ensuring that only active merge requests keep running Review Apps. + +To enable this feature, you need to specify the [`environment:auto_stop_in`](../yaml/README.md#environmentauto_stop_in) keyword in `.gitlab-ci.yml`. +You can specify a human-friendly date as the value, such as `1 hour and 30 minutes` or `1 day`. +`auto_stop_in` uses the same format of [`artifacts:expire_in` docs](../yaml/README.md#artifactsexpire_in). + +##### Auto-stop example + +In the following example, there is a basic review app setup that creates a new environment +per merge request. The `review_app` job is triggered by every push and +creates or updates an environment named `review/your-branch-name`. +The environment keeps running until `stop_review_app` is executed: + +```yaml +review_app: + script: deploy-review-app + environment: + name: review/$CI_COMMIT_REF_NAME + on_stop: stop_review_app + auto_stop_in: 1 week + +stop_review_app: + script: stop-review-app + environment: + name: review/$CI_COMMIT_REF_NAME + action: stop + when: manual +``` + +As long as a merge request is active and keeps getting new commits, +the review app will not stop, so developers don't need to worry about +re-initiating review app. + +On the other hand, since `stop_review_app` is set to `auto_stop_in: 1 week`, +if a merge request becomes inactive for more than a week, +GitLab automatically triggers the `stop_review_app` job to stop the environment. + +You can also check the expiration date of environments through the GitLab UI. To do so, +go to **Operations > Environments > Environment**. You can see the auto-stop period +at the left-top section and a pin-mark button at the right-top section. This pin-mark +button can be used to prevent auto-stopping the environment. By clicking this button, the `auto_stop_in` setting is over-written +and the environment will be active until it's stopped manually. + + + +NOTE: **NOTE** +Due to the resource limitation, a background worker for stopping environments only +runs once every hour. This means environments will not be stopped at the exact +timestamp as the specified period, but will be stopped when the hourly cron worker +detects expired environments. + +#### Delete a stopped environment + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/22629) in GitLab 12.9. + +You can delete [stopped environments](#stopping-an-environment) in one of two +ways: through the GitLab UI or through the API. + +##### Delete environments through the UI + +To view the list of **Stopped** environments, navigate to **Operations > Environments** +and click the **Stopped** tab. + +From there, you can click the **Delete** button directly, or you can click the +environment name to see its details and **Delete** it from there. + +You can also delete environments by viewing the details for a +stopped environment: + + 1. Navigate to **Operations > Environments**. + 1. Click on the name of an environment within the **Stopped** environments list. + 1. Click on the **Delete** button that appears at the top for all stopped environments. + 1. Finally, confirm your chosen environment in the modal that appears to delete it. + +##### Delete environments through the API + +Environments can also be deleted by using the [Environments API](../../api/environments.md#delete-an-environment). + +### Grouping similar environments + +> [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/7015) in GitLab 8.14. + +As documented in [Configuring dynamic environments](#configuring-dynamic-environments), you can +prepend environment name with a word, followed by a `/`, and finally the branch +name, which is automatically defined by the `CI_COMMIT_REF_NAME` variable. + +In short, environments that are named like `type/foo` are all presented under the same +group, named `type`. + +In our [minimal example](#example-configuration), we named the environments `review/$CI_COMMIT_REF_NAME` +where `$CI_COMMIT_REF_NAME` is the branch name. Here is a snippet of the example: + +```yaml +deploy_review: + stage: deploy + script: + - echo "Deploy a review app" + environment: + name: review/$CI_COMMIT_REF_NAME +``` + +In this case, if you visit the **Environments** page and the branches +exist, you should see something like: + + + +### Monitoring environments + +If you have enabled [Prometheus for monitoring system and response metrics](../../user/project/integrations/prometheus.md), +you can monitor the behavior of your app running in each environment. For the monitoring +dashboard to appear, you need to Configure Prometheus to collect at least one +[supported metric](../../user/project/integrations/prometheus_library/index.md). + +NOTE: **Note:** +Since GitLab 9.2, all deployments to an environment are shown directly on the monitoring dashboard. + +Once configured, GitLab will attempt to retrieve [supported performance metrics](../../user/project/integrations/prometheus_library/index.md) +for any environment that has had a successful deployment. If monitoring data was +successfully retrieved, a **Monitoring** button will appear for each environment. + + + +Clicking on the **Monitoring** button will display a new page showing up to the last +8 hours of performance data. It may take a minute or two for data to appear +after initial deployment. + +All deployments to an environment are shown directly on the monitoring dashboard, +which allows easy correlation between any changes in performance and new +versions of the app, all without leaving GitLab. + + + +#### Linking to external dashboard + +Add a [button to the Monitoring dashboard](../../user/project/operations/linking_to_an_external_dashboard.md) linking directly to your existing external dashboards. + +#### Embedding metrics in GitLab Flavored Markdown + +Metric charts can be embedded within GitLab Flavored Markdown. See [Embedding Metrics within GitLab Flavored Markdown](../../user/project/integrations/prometheus.md#embedding-metric-charts-within-gitlab-flavored-markdown) for more details. + +### Web terminals + +> Web terminals were added in GitLab 8.15 and are only available to project Maintainers and Owners. + +If you deploy to your environments with the help of a deployment service (for example, +the [Kubernetes integration](../../user/project/clusters/index.md)), GitLab can open +a terminal session to your environment. + +This is a powerful feature that allows you to debug issues without leaving the comfort +of your web browser. To enable it, just follow the instructions given in the service integration +documentation. + +Once enabled, your environments will gain a "terminal" button: + + + +You can also access the terminal button from the page for a specific environment: + + + +Wherever you find it, clicking the button will take you to a separate page to +establish the terminal session: + + + +This works just like any other terminal. You'll be in the container created +by your deployment so you can: + +- Run shell commands and get responses in real time. +- Check the logs. +- Try out configuration or code tweaks etc. + +You can open multiple terminals to the same environment, they each get their own shell +session and even a multiplexer like `screen` or `tmux`. + +NOTE: **Note:** +Container-based deployments often lack basic tools (like an editor), and may +be stopped or restarted at any time. If this happens, you will lose all your +changes. Treat this as a debugging tool, not a comprehensive online IDE. + +### Check out deployments locally + +Since GitLab 8.13, a reference in the Git repository is saved for each deployment, so +knowing the state of your current environments is only a `git fetch` away. + +In your Git configuration, append the `[remote "<your-remote>"]` block with an extra +fetch line: + +```plaintext +fetch = +refs/environments/*:refs/remotes/origin/environments/* +``` + +### Scoping environments with specs + +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/2112) in [GitLab Premium](https://about.gitlab.com/pricing/) 9.4. +> - [Scoping for environment variables was moved to Core](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/30779) to Core in GitLab 12.2. + +You can limit the environment scope of a variable by +defining which environments it can be available for. + +Wildcards can be used, and the default environment scope is `*`, which means +any jobs will have this variable, not matter if an environment is defined or +not. + +For example, if the environment scope is `production`, then only the jobs +having the environment `production` defined would have this specific variable. +Wildcards (`*`) can be used along with the environment name, therefore if the +environment scope is `review/*` then any jobs with environment names starting +with `review/` would have that particular variable. + +Some GitLab features can behave differently for each environment. +For example, you can +[create a secret variable to be injected only into a production environment](../variables/README.md#limit-the-environment-scopes-of-environment-variables). + +In most cases, these features use the _environment specs_ mechanism, which offers +an efficient way to implement scoping within each environment group. + +Let's say there are four environments: + +- `production` +- `staging` +- `review/feature-1` +- `review/feature-2` + +Each environment can be matched with the following environment spec: + +| Environment Spec | `production` | `staging` | `review/feature-1` | `review/feature-2` | +|:-----------------|:-------------|:----------|:-------------------|:-------------------| +| * | Matched | Matched | Matched | Matched | +| production | Matched | | | | +| staging | | Matched | | | +| review/* | | | Matched | Matched | +| review/feature-1 | | | Matched | | + +As you can see, you can use specific matching for selecting a particular environment, +and also use wildcard matching (`*`) for selecting a particular environment group, +such as [Review Apps](../review_apps/index.md) (`review/*`). + +NOTE: **Note:** +The most _specific_ spec takes precedence over the other wildcard matching. +In this case, `review/feature-1` spec takes precedence over `review/*` and `*` specs. + +### Environments Dashboard **(PREMIUM)** + +See [Environments Dashboard](../environments/environments_dashboard.md) for a summary of each +environment's operational health. + +## Limitations + +In the `environment: name`, you are limited to only the [predefined environment variables](../variables/predefined_variables.md). +Re-using variables defined inside `script` as part of the environment name will not work. + +## Further reading + +Below are some links you may find interesting: + +- [The `.gitlab-ci.yml` definition of environments](../yaml/README.md#environment) +- [A blog post on Deployments & Environments](https://about.gitlab.com/blog/2016/08/26/ci-deployment-and-environments/) +- [Review Apps - Use dynamic environments to deploy your code for every branch](../review_apps/index.md) +- [Deploy Boards for your applications running on Kubernetes](../../user/project/deploy_boards.md) **(PREMIUM)** + +<!-- ## Troubleshooting + +Include any troubleshooting steps that you can foresee. If you know beforehand what issues +one might have when setting this up, or when something is changed, or on upgrading, it's +important to describe those, too. Think of things that may go wrong and include them here. +This is important to minimize requests for support, and to avoid doc comments with +questions that you know someone might ask. + +Each scenario can be a third-level heading, e.g. `### Getting error message X`. +If you have none to add when creating a doc, leave this section in place +but commented out to help encourage others to add to it in the future. --> diff --git a/doc/ci/environments/protected_environments.md b/doc/ci/environments/protected_environments.md index fb6cf2a710f..43ac42ea0c7 100644 --- a/doc/ci/environments/protected_environments.md +++ b/doc/ci/environments/protected_environments.md @@ -1,4 +1,7 @@ --- +stage: Release +group: Release Management +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers type: concepts, howto --- @@ -8,7 +11,7 @@ type: concepts, howto ## Overview -[Environments](../environments.md) can be used for different reasons: +[Environments](../environments/index.md) can be used for different reasons: - Some of them are just for testing. - Others are for production. diff --git a/doc/ci/examples/authenticating-with-hashicorp-vault/index.md b/doc/ci/examples/authenticating-with-hashicorp-vault/index.md index 2986afe8284..089babc8945 100644 --- a/doc/ci/examples/authenticating-with-hashicorp-vault/index.md +++ b/doc/ci/examples/authenticating-with-hashicorp-vault/index.md @@ -20,7 +20,7 @@ You will need to replace the `vault.example.com` URL below with the URL of your ## How it works -Each job has JSON Web Token (JWT) provided as environment variable named `CI_JOB_JWT`. This JWT can be used to authenticate with Vault using the [JWT Auth](https://www.vaultproject.io/docs/auth/jwt/#jwt-authentication) method. +Each job has JSON Web Token (JWT) provided as environment variable named `CI_JOB_JWT`. This JWT can be used to authenticate with Vault using the [JWT Auth](https://www.vaultproject.io/docs/auth/jwt#jwt-authentication) method. The JWT's payload looks like this: @@ -51,7 +51,7 @@ The JWT is encoded by using RS256 and signed with your GitLab instance's OpenID You can use this JWT and your instance's JWKS endpoint (`https://gitlab.example.com/-/jwks`) to authenticate with a Vault server that is configured to allow the JWT Authentication method for authentication. -When configuring roles in Vault, you can use [bound_claims](https://www.vaultproject.io/docs/auth/jwt/#bound-claims) to match against the JWT's claims and restrict which secrets each CI job has access to. +When configuring roles in Vault, you can use [bound_claims](https://www.vaultproject.io/docs/auth/jwt#bound-claims) to match against the JWT's claims and restrict which secrets each CI job has access to. To communicate with Vault, you can use either its CLI client or perform API requests (using `curl` or another client). @@ -70,7 +70,7 @@ $ vault kv get -field=password secret/myproject/production/db real-pa$$w0rd ``` -To configure your Vault server, start by enabling the [JWT Auth](https://www.vaultproject.io/docs/auth/jwt/) method: +To configure your Vault server, start by enabling the [JWT Auth](https://www.vaultproject.io/docs/auth/jwt) method: ```shell $ vault auth enable jwt diff --git a/doc/ci/examples/devops_and_game_dev_with_gitlab_ci_cd/index.md b/doc/ci/examples/devops_and_game_dev_with_gitlab_ci_cd/index.md index 848808f65ea..610220a6bff 100644 --- a/doc/ci/examples/devops_and_game_dev_with_gitlab_ci_cd/index.md +++ b/doc/ci/examples/devops_and_game_dev_with_gitlab_ci_cd/index.md @@ -269,7 +269,7 @@ we know we will need to access everything in the `built` folder, given by GitLab We'll also cache `node_modules` to avoid having to do a full re-pull of those dependencies: just pack them up in the cache. Here is the full `build` job: -```yml +```yaml build: stage: build script: @@ -296,7 +296,7 @@ the previous job. Lastly, by convention, we let GitLab CI/CD know this needs to the `build` job by giving it a `test` [stage](../../../ci/yaml/README.md#stages). Following the YAML structure, the `test` job should look like this: -```yml +```yaml test: stage: test script: @@ -318,7 +318,7 @@ we've added test artifacts and a test stage to our GitLab CI/CD pipeline using ` allowing us to run our tests by every push. Our entire `.gitlab-ci.yml` file should now look like this: -```yml +```yaml image: node:10 build: @@ -417,7 +417,7 @@ credentials, which will be the same two credentials (Key ID and Secret). It's a fully understand [IAM Best Practices in AWS](https://docs.aws.amazon.com/IAM/latest/UserGuide/best-practices.html). We need to add these credentials to GitLab: 1. Log into your AWS account and go to the [Security Credentials page](https://console.aws.amazon.com/iam/home#/security_credential) -1. Click the **Access Keys** section and **Create New Access Key**. Create the key and keep the id and secret around, you'll need them later +1. Click the **Access Keys** section and **Create New Access Key**. Create the key and keep the ID and secret around, you'll need them later  @@ -426,7 +426,7 @@ fully understand [IAM Best Practices in AWS](https://docs.aws.amazon.com/IAM/lat  -1. Add a key named `AWS_KEY_ID` and copy the key id from Step 2 into the **Value** textbox +1. Add a key named `AWS_KEY_ID` and copy the key ID from Step 2 into the **Value** textbox 1. Add a key named `AWS_KEY_SECRET` and copy the key secret from Step 2 into the **Value** textbox ### Deploy your game with GitLab CI/CD @@ -440,7 +440,7 @@ we add directives to ensure deployment `only` happens on pushes to `master`. Thi single branch still runs through CI, and only merging (or committing directly) to master will trigger the `deploy` job of our pipeline. Put these together to get the following: -```yml +```yaml deploy: stage: deploy variables: @@ -459,7 +459,7 @@ deploy: Be sure to update the region and S3 URL in that last script command to fit your setup. Our final configuration file `.gitlab-ci.yml` looks like: -```yml +```yaml image: node:10 build: @@ -516,7 +516,7 @@ Errors can be easily debugged through GitLab's build logs, and within minutes of you can see the changes live on your game. Setting up Continuous Integration and Continuous Deployment from the start with Dark Nova enables -rapid but stable development. We can easily test changes in a separate [environment](../../environments.md), +rapid but stable development. We can easily test changes in a separate [environment](../../environments/index.md), or multiple environments if needed. Balancing and updating a multiplayer game can be ongoing and tedious, but having faith in a stable deployment with GitLab CI/CD allows a lot of breathing room in quickly getting changes to players. diff --git a/doc/ci/examples/laravel_with_gitlab_and_envoy/index.md b/doc/ci/examples/laravel_with_gitlab_and_envoy/index.md index 07b054dd2cb..0331fa70f06 100644 --- a/doc/ci/examples/laravel_with_gitlab_and_envoy/index.md +++ b/doc/ci/examples/laravel_with_gitlab_and_envoy/index.md @@ -376,7 +376,7 @@ You might want to create another Envoy task to do that for you. We also create the `.env` file in the same path to set up our production environment variables for Laravel. These are persistent data and will be shared to every new release. -Now, we would need to deploy our app by running `envoy run deploy`, but it won't be necessary since GitLab can handle that for us with CI's [environments](../../environments.md), which will be described [later](#setting-up-gitlab-cicd) in this tutorial. +Now, we would need to deploy our app by running `envoy run deploy`, but it won't be necessary since GitLab can handle that for us with CI's [environments](../../environments/index.md), which will be described [later](#setting-up-gitlab-cicd) in this tutorial. Now it's time to commit [Envoy.blade.php](https://gitlab.com/mehranrasulian/laravel-sample/blob/master/Envoy.blade.php) and push it to the `master` branch. To keep things simple, we commit directly to `master`, without using [feature-branches](../../../topics/gitlab_flow.md#github-flow-as-a-simpler-alternative) since collaboration is beyond the scope of this tutorial. diff --git a/doc/ci/examples/test-and-deploy-python-application-to-heroku.md b/doc/ci/examples/test-and-deploy-python-application-to-heroku.md index 6d05c37390a..2c626cb458a 100644 --- a/doc/ci/examples/test-and-deploy-python-application-to-heroku.md +++ b/doc/ci/examples/test-and-deploy-python-application-to-heroku.md @@ -76,14 +76,21 @@ To build this project you also need to have [GitLab Runner](https://docs.gitlab. You can use public runners available on `gitlab.com` or you can register your own: ```shell +cat > /tmp/test-config.template.toml << EOF +[[runners]] +[runners.docker] +[[runners.docker.services]] +name = "postgres:latest" +EOF + gitlab-runner register \ --non-interactive \ --url "https://gitlab.com/" \ --registration-token "PROJECT_REGISTRATION_TOKEN" \ --description "python-3.5" \ --executor "docker" \ - --docker-image python:3.5 \ - --docker-services postgres:latest + --template-config /tmp/test-config.template.toml \ + --docker-image python:3.5 ``` With the command above, you create a runner that uses the [`python:3.5`](https://hub.docker.com/_/python) image and uses a [PostgreSQL](https://hub.docker.com/_/postgres) database. diff --git a/doc/ci/examples/test-and-deploy-ruby-application-to-heroku.md b/doc/ci/examples/test-and-deploy-ruby-application-to-heroku.md index e480f4565ce..5df407f19fe 100644 --- a/doc/ci/examples/test-and-deploy-ruby-application-to-heroku.md +++ b/doc/ci/examples/test-and-deploy-ruby-application-to-heroku.md @@ -64,7 +64,19 @@ You can do this through the [Heroku Dashboard](https://dashboard.heroku.com/). First install [Docker Engine](https://docs.docker.com/installation/). To build this project you also need to have [GitLab Runner](https://docs.gitlab.com/runner/). -You can use public runners available on `gitlab.com` or register your own: +You can use public runners available on `gitlab.com` or register your own. Start by +creating a template configuration file in order to pass complex configuration: + +```shell +cat > /tmp/test-config.template.toml << EOF +[[runners]] +[runners.docker] +[[runners.docker.services]] +name = "postgres:latest" +EOF +``` + +Finally, register the runner, passing the newly-created template configuration file: ```shell gitlab-runner register \ @@ -73,8 +85,8 @@ gitlab-runner register \ --registration-token "PROJECT_REGISTRATION_TOKEN" \ --description "ruby:2.6" \ --executor "docker" \ - --docker-image ruby:2.6 \ - --docker-services latest + --template-config /tmp/test-config.template.toml \ + --docker-image ruby:2.6 ``` With the command above, you create a Runner that uses the [`ruby:2.6`](https://hub.docker.com/_/ruby) image and uses a [PostgreSQL](https://hub.docker.com/_/postgres) database. diff --git a/doc/ci/examples/test-scala-application.md b/doc/ci/examples/test-scala-application.md index 92fb69286cf..1831417e48e 100644 --- a/doc/ci/examples/test-scala-application.md +++ b/doc/ci/examples/test-scala-application.md @@ -14,7 +14,7 @@ The following `.gitlab-ci.yml` should be added in the root of your repository to trigger CI: ``` yaml -image: java:8 +image: openjdk:8 stages: - test @@ -74,5 +74,5 @@ in the `.gitlab-ci.yml` file with your application's name. ## Heroku API key You can look up your Heroku API key in your -[account](https://dashboard.heroku.com/account). Add a [protected variable](../variables/README.md#protected-environment-variables) with +[account](https://dashboard.heroku.com/account). Add a [protected variable](../variables/README.md#protect-a-custom-variable) with this value in **Project ➔ Variables** with key `HEROKU_API_KEY`. diff --git a/doc/ci/examples/test_phoenix_app_with_gitlab_ci_cd/index.md b/doc/ci/examples/test_phoenix_app_with_gitlab_ci_cd/index.md index 6d92c86c819..cd1ad923249 100644 --- a/doc/ci/examples/test_phoenix_app_with_gitlab_ci_cd/index.md +++ b/doc/ci/examples/test_phoenix_app_with_gitlab_ci_cd/index.md @@ -375,7 +375,7 @@ see if our latest code is running without errors. When we finish this edition, GitLab will start another build and show a **build running** badge. It is expected, after all we just configured GitLab CI/CD to do this for every push! But you may think "Why run build and tests for simple things like editing README.md?" and it is a good question. -For changes that don't affect your application, you can add the keyword [`[ci skip]`](../../yaml/README.md#skipping-jobs) +For changes that don't affect your application, you can add the keyword [`[ci skip]`](../../yaml/README.md#skip-pipeline) to commit message and the build related to that commit will be skipped. In the end, we finally got our pretty green build succeeded badge! By outputting the result on the diff --git a/doc/ci/img/metrics_reports.png b/doc/ci/img/metrics_reports.png Binary files differdeleted file mode 100644 index ffd9f6830a2..00000000000 --- a/doc/ci/img/metrics_reports.png +++ /dev/null diff --git a/doc/ci/img/metrics_reports_v13_0.png b/doc/ci/img/metrics_reports_v13_0.png Binary files differnew file mode 100644 index 00000000000..1597031db0b --- /dev/null +++ b/doc/ci/img/metrics_reports_v13_0.png diff --git a/doc/ci/introduction/index.md b/doc/ci/introduction/index.md index b16cde54b93..b7f1837d83e 100644 --- a/doc/ci/introduction/index.md +++ b/doc/ci/introduction/index.md @@ -76,6 +76,9 @@ to apply all the continuous methods (Continuous Integration, Delivery, and Deployment) to your software with no third-party application or integration needed. +<i class="fa fa-youtube-play youtube" aria-hidden="true"></i> +For an overview, see [Introduction to GitLab CI](https://www.youtube.com/watch?v=l5705U8s_nQ&t=397) from a recent GitLab meetup. + ### How GitLab CI/CD works To use GitLab CI/CD, all you need is an application codebase hosted in a @@ -105,7 +108,7 @@ The scripts are grouped into **jobs**, and together they compose a **pipeline**. A minimalist example of `.gitlab-ci.yml` file could contain: -```yml +```yaml before_script: - apt-get install rubygems ruby-dev -y @@ -133,7 +136,7 @@ displayed by GitLab:  At the end, if anything goes wrong, you can easily -[roll back](../environments.md#retrying-and-rolling-back) all the changes: +[roll back](../environments/index.md#retrying-and-rolling-back) all the changes:  @@ -204,7 +207,7 @@ according to each stage (Verify, Package, Release). With GitLab CI/CD you can also: - Easily set up your app's entire lifecycle with [Auto DevOps](../../topics/autodevops/index.md). -- Deploy your app to different [environments](../environments.md). +- Deploy your app to different [environments](../environments/index.md). - Install your own [GitLab Runner](https://docs.gitlab.com/runner/). - [Schedule pipelines](../pipelines/schedules.md). - Check for app vulnerabilities with [Security Test reports](../../user/application_security/index.md). **(ULTIMATE)** @@ -212,7 +215,7 @@ With GitLab CI/CD you can also: To see all CI/CD features, navigate back to the [CI/CD index](../README.md). <i class="fa fa-youtube-play youtube" aria-hidden="true"></i> -Watch the video [GitLab CI Live Demo](https://www.youtube.com/watch?v=pBe4t1CD8Fc) with a deeper overview of GitLab CI/CD. +Watch the video [GitLab CI Live Demo](https://youtu.be/l5705U8s_nQ?t=369) with a deeper overview of GitLab CI/CD. ### Setting up GitLab CI/CD for the first time diff --git a/doc/ci/jenkins/index.md b/doc/ci/jenkins/index.md index 551e32ac816..c4346005138 100644 --- a/doc/ci/jenkins/index.md +++ b/doc/ci/jenkins/index.md @@ -13,6 +13,10 @@ First of all, our [Quick Start Guide](../quick_start/README.md) contains a good You may also be interested in [Auto DevOps](../../topics/autodevops/index.md) which can potentially be used to build, test, and deploy your applications with little to no configuration needed at all. +For an example of how to convert a Jenkins pipeline into a GitLab CI/CD pipeline, +or how to use Auto DevOps to test your code automatically, watch the +[Migrating from Jenkins to GitLab](https://www.youtube.com/watch?v=RlEVGOpYF5Y) video. + For advanced CI/CD teams, [templates](#templates) can enable the reuse of pipeline configurations. Otherwise, read on for important information that will help you get the ball rolling. Welcome @@ -40,6 +44,15 @@ things we have found that helps this: of the improvements that GitLab offers, and this requires (eventually) updating your implementation as part of the transition. +## JenkinsFile Wrapper + +We are building a [JenkinsFile Wrapper](https://gitlab.com/gitlab-org/jfr-container-builder/) which will allow +you to run a complete Jenkins instance inside of a GitLab job, including plugins. This can help ease the process +of transition, by letting you delay the migration of less urgent pipelines for a period of time. + +If you are interested, join our [public testing issue](https://gitlab.com/gitlab-org/gitlab/-/issues/215675) to +If you are interested, you might be able to [help GitLab test the wrapper](https://gitlab.com/gitlab-org/gitlab/-/issues/215675). + ## Important product differences There are some high level differences between the products worth mentioning: @@ -113,7 +126,7 @@ There are some important differences in the way Runners work in comparison to ag If you are using `gitlab.com`, you can take advantage of our [shared Runner fleet](../../user/gitlab_com/index.md#shared-runners) to run jobs without provisioning your own Runners. We are investigating making them -[available for self-managed instances](https://gitlab.com/gitlab-org/customers-gitlab-com/issues/414) +[available for self-managed instances](https://gitlab.com/groups/gitlab-org/-/epics/835) as well. ## Groovy vs. YAML @@ -282,7 +295,7 @@ my_job: In GitLab, we use the [`variables` keyword](../yaml/README.md#variables) to define different variables at runtime. These can also be set up through the GitLab UI, under CI/CD settings. See also our [general documentation on variables](../variables/README.md), -including the section on [protected variables](../variables/README.md#protected-environment-variables) which can be used +including the section on [protected variables](../variables/README.md#protect-a-custom-variable) which can be used to limit access to certain variables to certain environments or runners: ```yaml diff --git a/doc/ci/junit_test_reports.md b/doc/ci/junit_test_reports.md index c79db6dfbea..a77044e849d 100644 --- a/doc/ci/junit_test_reports.md +++ b/doc/ci/junit_test_reports.md @@ -18,6 +18,8 @@ You can configure your job to use JUnit test reports, and GitLab will display a report on the merge request so that it's easier and faster to identify the failure without having to check the entire log. +If you don't use Merge Requests but still want to see the JUnit output without searching through job logs, the full [JUnit test reports](#viewing-junit-test-reports-on-gitlab) are available in the pipeline detail view. + ## Use cases Consider the following workflow: @@ -66,7 +68,7 @@ For a list of supported languages on JUnit tests, check the [Wikipedia article](https://en.wikipedia.org/wiki/JUnit#Ports). To enable the JUnit reports in merge requests, you need to add -[`artifacts:reports:junit`](yaml/README.md#artifactsreportsjunit) +[`artifacts:reports:junit`](pipelines/job_artifacts.md#artifactsreportsjunit) in `.gitlab-ci.yml`, and specify the path(s) of the generated test reports. In the following examples, the job in the `test` stage runs and GitLab @@ -235,15 +237,44 @@ You can view all the known test suites and click on each of these to see further details, including the cases that makeup the suite. Cases are ordered by status, with failed showing at the top, skipped next and successful cases last. +You can also retrieve the reports via the [GitLab API](../api/pipelines.md#get-a-pipelines-test-report). + ### Enabling the feature This feature comes with the `:junit_pipeline_view` feature flag disabled by default. This feature is disabled due to some performance issues with very large data sets. When [the performance is improved](https://gitlab.com/groups/gitlab-org/-/epics/2854), the feature will be enabled by default. -To enable this feature, ask a GitLab administrator with Rails console access to run the +To enable this feature, ask a GitLab administrator with [Rails console access](../administration/feature_flags.md#how-to-enable-and-disable-features-behind-flags) to run the following command: ```ruby Feature.enable(:junit_pipeline_view) ``` + +## Viewing JUnit screenshots on GitLab + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/202114) in GitLab 13.0. + +If JUnit XML files contain an `attachment` tag, GitLab parses the attachment. + +Upload your screenshots as [artifacts](pipelines/job_artifacts.md#artifactsreportsjunit) to GitLab. The `attachment` tag **must** contain the absolute path to the screenshots you uploaded. + +```xml +<testcase time="1.00" name="Test"> + <system-out>[[ATTACHMENT|/absolute/path/to/some/file]]</system-out> +</testcase> +``` + +When [this issue](https://gitlab.com/gitlab-org/gitlab/-/issues/6061) is complete, the attached file will be visible on the pipeline details page. + +### Enabling the feature + +This feature comes with the `:junit_pipeline_screenshots_view` feature flag disabled by default. + +To enable this feature, ask a GitLab administrator with [Rails console access](../administration/feature_flags.md#how-to-enable-and-disable-features-behind-flags) to run the +following command: + +```ruby +Feature.enable(:junit_pipeline_screenshots_view) +``` diff --git a/doc/ci/large_repositories/index.md b/doc/ci/large_repositories/index.md index 6ac3fa2c92d..b4059fc252b 100644 --- a/doc/ci/large_repositories/index.md +++ b/doc/ci/large_repositories/index.md @@ -130,7 +130,7 @@ other using `docker` executor. ### `shell` executor example -Let's assume that you have the following [config.toml](https://docs.gitlab.com/runner/configuration/advanced-configuration.html). +Let's assume that you have the following [`config.toml`](https://docs.gitlab.com/runner/configuration/advanced-configuration.html). ```toml concurrent = 4 @@ -155,7 +155,7 @@ This `config.toml`: ### `docker` executor example -Let's assume that you have the following [config.toml](https://docs.gitlab.com/runner/configuration/advanced-configuration.html). +Let's assume that you have the following [`config.toml`](https://docs.gitlab.com/runner/configuration/advanced-configuration.html). ```toml concurrent = 4 @@ -216,7 +216,7 @@ but this brings administrative overhead as the `.gitlab-ci.yml` needs to be upda In such cases, it might be desirable to keep the `.gitlab-ci.yml` clone path agnostic, but make it a configuration of Runner. -We can extend our [config.toml](https://docs.gitlab.com/runner/configuration/advanced-configuration.html) +We can extend our [`config.toml`](https://docs.gitlab.com/runner/configuration/advanced-configuration.html) with the following specification that will be used by Runner if `.gitlab-ci.yml` will not override it: ```toml diff --git a/doc/ci/merge_request_pipelines/img/merge_request_pipelines_doubled_MR_v12_09.png b/doc/ci/merge_request_pipelines/img/merge_request_pipelines_doubled_MR_v12_09.png Binary files differnew file mode 100644 index 00000000000..3e4c72b9279 --- /dev/null +++ b/doc/ci/merge_request_pipelines/img/merge_request_pipelines_doubled_MR_v12_09.png diff --git a/doc/ci/merge_request_pipelines/img/merge_request_pipelines_doubled_branch_v12_09.png b/doc/ci/merge_request_pipelines/img/merge_request_pipelines_doubled_branch_v12_09.png Binary files differnew file mode 100644 index 00000000000..dd70c3f8c20 --- /dev/null +++ b/doc/ci/merge_request_pipelines/img/merge_request_pipelines_doubled_branch_v12_09.png diff --git a/doc/ci/merge_request_pipelines/index.md b/doc/ci/merge_request_pipelines/index.md index b57340347d2..a724bf416b6 100644 --- a/doc/ci/merge_request_pipelines/index.md +++ b/doc/ci/merge_request_pipelines/index.md @@ -13,93 +13,47 @@ changes are pushed to a branch. If you want the pipeline to run jobs **only** when merge requests are created or updated, you can use *pipelines for merge requests*. -In the UI, these pipelines are labeled as `detached`. +In the UI, these pipelines are labeled as `detached`. Otherwise, these pipelines appear the same +as other pipelines. - - -A few notes: - -- Pipelines for merge requests are incompatible with - [CI/CD for external repositories](../ci_cd_for_external_repos/index.md). -- [Since GitLab 11.10](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/25504), pipelines for merge requests require GitLab Runner 11.9. -- If you use this feature with [merge when pipeline succeeds](../../user/project/merge_requests/merge_when_pipeline_succeeds.md), - pipelines for merge requests take precedence over the other regular pipelines. - -## Configuring pipelines for merge requests - -To configure pipelines for merge requests, configure your [CI/CD configuration file](../yaml/README.md). -There are a few different ways to do this. - -### Enable pipelines for merge requests for all jobs - -The recommended method for enabling pipelines for merge requests for all jobs in -a pipeline is to use [`workflow:rules`](../yaml/README.md#workflowrules). - -In this example, the pipeline always runs for all merge requests, as well as for all changes -to the master branch: - -```yaml -workflow: - rules: - - if: $CI_MERGE_REQUEST_ID # Execute jobs in merge request context - - if: $CI_COMMIT_BRANCH == 'master' # Execute jobs when a new commit is pushed to master branch +Any user who has developer [permissions](../../user/permissions.md) +can run a pipeline for merge requests. -build: - stage: build - script: ./build - -test: - stage: test - script: ./test + -deploy: - stage: deploy - script: ./deploy -``` +NOTE: **Note**: +If you use this feature with [merge when pipeline succeeds](../../user/project/merge_requests/merge_when_pipeline_succeeds.md), +pipelines for merge requests take precedence over the other regular pipelines. -### Enable pipelines for merge requests for specific jobs +## Prerequisites -To enable pipelines for merge requests for specific jobs, you can use -[`rules`](../yaml/README.md#rules). +To enable pipelines for merge requests: -In the following example: +- You must have maintainer [permissions](../../user/permissions.md). +- Your repository must be a GitLab repository, not an + [external repository](../ci_cd_for_external_repos/index.md). +- [In GitLab 11.10 and later](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/25504), + you must be using GitLab Runner 11.9. -- The `build` job runs for all changes to the `master` branch, as well as for all merge requests. -- The `test` job runs for all merge requests. -- The `deploy` job runs for all changes to the `master` branch, but does *not* run - for merge requests. +## Configuring pipelines for merge requests -```yaml -build: - stage: build - script: ./build - rules: - - if: $CI_COMMIT_BRANCH == 'master' # Execute jobs when a new commit is pushed to master branch - - if: $CI_MERGE_REQUEST_ID # Execute jobs in merge request context +To configure pipelines for merge requests you need to configure your [CI/CD configuration file](../yaml/README.md). +There are a few different ways to do this: -test: - stage: test - script: ./test - rules: - - if: $CI_MERGE_REQUEST_ID # Execute jobs in merge request context +### Use `rules` to run pipelines for merge requests -deploy: - stage: deploy - script: ./deploy - rules: - - if: $CI_COMMIT_BRANCH == 'master' # Execute jobs when a new commit is pushed to master branch -``` +When using `rules`, which is the preferred method, we recommend starting with one +of the [`workflow:rules` templates](../yaml/README.md#workflowrules-templates) to ensure +your basic configuration is correct. Instructions on how to do this, as well as how +to customize, are available at that link. ### Use `only` or `except` to run pipelines for merge requests -NOTE: **Note**: -The [`only` / `except`](../yaml/README.md#onlyexcept-basic) keywords are going to be deprecated -and you should not use them. - -To enable pipelines for merge requests, you can use `only / except`. When you use this method, -you have to specify `only: - merge_requests` for each job. +If you want to continue using `only/except`, this is possible but please review the drawbacks +below. -In this example, the pipeline contains a `test` job that is configured to run on merge requests. +When you use this method, you have to specify `only: - merge_requests` for each job. In this +example, the pipeline contains a `test` job that is configured to run on merge requests. The `build` and `deploy` jobs don't have the `only: - merge_requests` parameter, so they will not run on merge requests. @@ -245,14 +199,24 @@ to integrate your job with [GitLab Merge Request API](../../api/merge_requests.m You can find the list of available variables in [the reference sheet](../variables/predefined_variables.md). The variable names begin with the `CI_MERGE_REQUEST_` prefix. -<!-- ## Troubleshooting +## Troubleshooting + +### Two pipelines created when pushing to a merge request + +If you are experiencing duplicated pipelines when using `rules`, take a look at +the [key details when using `rules`](../yaml/README.md#key-details-when-using-rules), +which will help you get your starting configuration correct. + +If you are seeing two pipelines when using `only/except`, please see the caveats +related to using `only/except` above (or, consider moving to `rules`). + +### Two pipelines created when pushing an invalid CI configuration file + +Pushing to a branch with an invalid CI configuration file can trigger +the creation of two types of failed pipelines. One pipeline is a failed merge request +pipeline, and the other is a failed branch pipeline, but both are caused by the same +invalid configuration. -Include any troubleshooting steps that you can foresee. If you know beforehand what issues -one might have when setting this up, or when something is changed, or on upgrading, it's -important to describe those, too. Think of things that may go wrong and include them here. -This is important to minimize requests for support, and to avoid doc comments with -questions that you know someone might ask. +In rare cases, duplicate pipelines are created. -Each scenario can be a third-level heading, e.g. `### Getting error message X`. -If you have none to add when creating a doc, leave this section in place -but commented out to help encourage others to add to it in the future. --> +See [this issue](https://gitlab.com/gitlab-org/gitlab/-/issues/201845) for details. diff --git a/doc/ci/merge_request_pipelines/pipelines_for_merged_results/index.md b/doc/ci/merge_request_pipelines/pipelines_for_merged_results/index.md index fb5c7830ac2..c35a5d0a07e 100644 --- a/doc/ci/merge_request_pipelines/pipelines_for_merged_results/index.md +++ b/doc/ci/merge_request_pipelines/pipelines_for_merged_results/index.md @@ -34,14 +34,18 @@ In these cases, the pipeline runs as a [pipeline for merge requests](../index.md and is labeled as `detached`. If these cases no longer exist, new pipelines will again run against the merged results. -## Requirements and limitations +Any user who has developer [permissions](../../../user/permissions.md) can run a +pipeline for merged results. -Pipelines for merged results have the following requirements and limitations: +## Prerequisites -- Pipelines for merged results require [GitLab Runner](https://gitlab.com/gitlab-org/gitlab-runner) 11.9 or newer. -- Forking/cross-repo workflows are not currently supported. To follow progress, +To enable pipelines for merge results: + +- You must have maintainer [permissions](../../../user/permissions.md). +- You must be using [GitLab Runner](https://gitlab.com/gitlab-org/gitlab-runner) 11.9 or later. +- You must not be forking or using cross-repo workflows. To follow progress, see [#11934](https://gitlab.com/gitlab-org/gitlab/issues/11934). -- This feature is not available for +- You must not be using [fast forward merges](../../../user/project/merge_requests/fast_forward_merge.md) yet. To follow progress, see [#58226](https://gitlab.com/gitlab-org/gitlab/-/issues/26996). @@ -93,7 +97,6 @@ canceled. Can be caused by some disabled feature flags. Please make sure that the following feature flags are enabled on your GitLab instance: -- `:ci_use_merge_request_ref` - `:merge_ref_auto_sync` To check and set these feature flag values, please ask an administrator to: @@ -107,14 +110,12 @@ To check and set these feature flag values, please ask an administrator to: 1. Check if the flags are enabled or not: ```ruby - Feature.enabled?(:ci_use_merge_request_ref) Feature.enabled?(:merge_ref_auto_sync) ``` 1. If needed, enable the feature flags: ```ruby - Feature.enable(:ci_use_merge_request_ref) Feature.enable(:merge_ref_auto_sync) ``` diff --git a/doc/ci/merge_request_pipelines/pipelines_for_merged_results/merge_trains/index.md b/doc/ci/merge_request_pipelines/pipelines_for_merged_results/merge_trains/index.md index 641192afea1..d921b75aa44 100644 --- a/doc/ci/merge_request_pipelines/pipelines_for_merged_results/merge_trains/index.md +++ b/doc/ci/merge_request_pipelines/pipelines_for_merged_results/merge_trains/index.md @@ -30,6 +30,14 @@ If the pipeline for the merge request at the front of the train completes succes the changes are merged into the target branch, and the other pipelines continue to run. +To add a merge request to a merge train, you need [permissions](../../../../user/permissions.md) to push to the target branch. + +NOTE: **Note:** +Each merge train can run a maximum of **twenty** pipelines in parallel. +If more than twenty merge requests are added to the merge train, the merge requests +will be queued until a slot in the merge train is free. There is no limit to the +number of merge requests that can be queued. + ## Merge train example Three merge requests (`A`, `B` and `C`) are added to a merge train in order, which @@ -55,16 +63,13 @@ Watch this video for a demonstration on [how parallel execution of Merge Trains can prevent commits from breaking the default branch](https://www.youtube.com/watch?v=D4qCqXgZkHQ). -## Requirements and limitations +## Prerequisites -Merge trains have the following requirements and limitations: +To enable merge trains: -- Merge trains require [GitLab Runner](https://gitlab.com/gitlab-org/gitlab-runner) 11.9 or newer. -- GitLab 12.0 and later requires [Redis](https://redis.io/) 3.2 or higher. -- Each merge train can run a maximum of **twenty** pipelines in parallel. - If more than twenty merge requests are added to the merge train, the merge requests - will be queued until a slot in the merge train is free. There is no limit to the - number of merge requests that can be queued. +- You must have maintainer [permissions](../../../../user/permissions.md). +- You must be using [GitLab Runner](https://gitlab.com/gitlab-org/gitlab-runner) 11.9 or later. +- In GitLab 12.0 and later, you need [Redis](https://redis.io/) 3.2 or later. ## Enable merge trains diff --git a/doc/ci/metrics_reports.md b/doc/ci/metrics_reports.md index 871f8c55e83..f353aa2670f 100644 --- a/doc/ci/metrics_reports.md +++ b/doc/ci/metrics_reports.md @@ -12,7 +12,7 @@ GitLab provides a lot of great reporting tools for [merge requests](../user/proj You can configure your job to use custom Metrics Reports, and GitLab will display a report on the merge request so that it's easier and faster to identify changes without having to check the entire log. - + ## Use cases @@ -34,7 +34,7 @@ All values are considered strings and string compare is used to find differences ## How to set it up -Add a job that creates a [metrics report](yaml/README.md#artifactsreportsmetrics-premium) (default filename: `metrics.txt`). The file should conform to the [OpenMetrics](https://openmetrics.io/) format. +Add a job that creates a [metrics report](pipelines/job_artifacts.md#artifactsreportsmetrics-premium) (default filename: `metrics.txt`). The file should conform to the [OpenMetrics](https://openmetrics.io/) format. For example: diff --git a/doc/ci/parent_child_pipelines.md b/doc/ci/parent_child_pipelines.md index 2bc897901fa..e28adc2bc01 100644 --- a/doc/ci/parent_child_pipelines.md +++ b/doc/ci/parent_child_pipelines.md @@ -43,6 +43,9 @@ Child pipelines work well with other GitLab CI/CD features: All of this will work with the [`include:`](yaml/README.md#include) feature so you can compose the child pipeline configuration. +<i class="fa fa-youtube-play youtube" aria-hidden="true"></i> +For an overview, see [Parent-Child Pipelines feature demo](https://youtu.be/n8KpBSqZNbk). + ## Examples The simplest case is [triggering a child pipeline](yaml/README.md#trigger) using a @@ -136,6 +139,9 @@ your own script to generate a YAML file, which is then [used to trigger a child This technique can be very powerful in generating pipelines targeting content that changed or to build a matrix of targets and architectures. +<i class="fa fa-youtube-play youtube" aria-hidden="true"></i> +For an overview, see [Create child pipelines using dynamically generated configurations](https://youtu.be/nMdfus2JWHM). + In GitLab 12.9, the child pipeline could fail to be created in certain cases, causing the parent pipeline to fail. This is [resolved in GitLab 12.10](https://gitlab.com/gitlab-org/gitlab/-/issues/209070). diff --git a/doc/ci/pipelines/img/pipelines_index.png b/doc/ci/pipelines/img/pipelines_index.png Binary files differdeleted file mode 100644 index e168e7e23df..00000000000 --- a/doc/ci/pipelines/img/pipelines_index.png +++ /dev/null diff --git a/doc/ci/pipelines/img/pipelines_index_v13_0.png b/doc/ci/pipelines/img/pipelines_index_v13_0.png Binary files differnew file mode 100644 index 00000000000..1dc5ea5d55c --- /dev/null +++ b/doc/ci/pipelines/img/pipelines_index_v13_0.png diff --git a/doc/ci/pipelines/index.md b/doc/ci/pipelines/index.md index 27e968c0a40..731cb196eaa 100644 --- a/doc/ci/pipelines/index.md +++ b/doc/ci/pipelines/index.md @@ -74,7 +74,7 @@ You can also configure specific aspects of your pipelines through the GitLab UI. - [Pipeline settings](settings.md) for each project. - [Pipeline schedules](schedules.md). -- [Custom CI/CD variables](../variables/README.md#creating-a-custom-environment-variable). +- [Custom CI/CD variables](../variables/README.md#custom-environment-variables). ### View pipelines @@ -82,7 +82,7 @@ You can find the current and historical pipeline runs under your project's **CI/CD > Pipelines** page. You can also access pipelines for a merge request by navigating to its **Pipelines** tab. - + Clicking a pipeline will bring you to the **Pipeline Details** page and show the jobs that were run for that pipeline. From here you can cancel a running pipeline, @@ -93,6 +93,12 @@ latest pipeline for the last commit of a given branch is available at `/project/ Also, `/project/pipelines/latest` will redirect you to the latest pipeline for the last commit on the project's default branch. +[Starting in GitLab 13.0](https://gitlab.com/gitlab-org/gitlab/-/issues/215367), +you can filter the pipeline list by: + +- Trigger author +- Branch name + ### Run a pipeline manually Pipelines can be manually executed, with predefined or manually-specified [variables](../variables/README.md). @@ -153,7 +159,7 @@ You can do this straight from the pipeline graph. Just click the play button to execute that particular job. For example, your pipeline might start automatically, but it requires manual action to -[deploy to production](../environments.md#configuring-manual-deployments). In the example below, the `production` +[deploy to production](../environments/index.md#configuring-manual-deployments). In the example below, the `production` stage has a job with a manual action.  @@ -216,7 +222,7 @@ In the example: Visually, it can be viewed as: -```text +```plaintext 0 1 2 3 4 5 6 7 AAAAAAA BBBBBBB @@ -225,7 +231,7 @@ Visually, it can be viewed as: The union of A, B, and C is (1, 4) and (6, 7). Therefore, the total running time is: -```text +```plaintext (4 - 1) + (7 - 6) => 4 ``` @@ -343,7 +349,7 @@ build ruby 2/3: stage: build script: - echo "ruby2" - + build ruby 3/3: stage: build script: @@ -367,10 +373,14 @@ evaluates the job names: `\d+[\s:\/\\]+\d+\s*`. When running manual jobs you can supply additional job specific variables. You can do this from the job page of the manual job you want to run with -additional variables. +additional variables. To access this page, click on the **name** of the manual job in +the pipeline view, *not* the play (**{play}**) button. -This is useful when you want to alter the execution of a job by using -environment variables. +This is useful when you want to alter the execution of a job that uses +[custom environment variables](../variables/README.md#custom-environment-variables). +Adding a variable name (key) and value here will override the value defined in +[the UI or `.gitlab-ci.yml`](../variables/README.md#custom-environment-variables), +for a single run of the manual job.  @@ -387,7 +397,7 @@ For example, if you start rolling out new code and: - Users do not experience trouble, GitLab can automatically complete the deployment from 0% to 100%. - Users experience trouble with the new code, you can stop the timed incremental rollout by canceling the pipeline - and [rolling](../environments.md#retrying-and-rolling-back) back to the last stable version. + and [rolling](../environments/index.md#retrying-and-rolling-back) back to the last stable version.  @@ -545,15 +555,3 @@ To illustrate its life cycle: even if the commit history of the `example` branch has been overwritten by force-push. 1. GitLab Runner fetches the persistent pipeline ref and gets source code from the checkout-SHA. 1. When the pipeline finished, its persistent ref is cleaned up in a background process. - -NOTE: **NOTE**: At this moment, this feature is on by default and can be manually disabled -by disabling `depend_on_persistent_pipeline_ref` feature flag. If you're interested in -manually disabling this behavior, please ask the administrator -to execute the following commands in rails console. - -```shell -> sudo gitlab-rails console # Login to Rails console of GitLab instance. -> project = Project.find_by_full_path('namespace/project-name') # Get the project instance. -> Feature.disable(:depend_on_persistent_pipeline_ref, project) # Disable the feature flag for specific project -> Feature.disable(:depend_on_persistent_pipeline_ref) # Disable the feature flag system-wide -``` diff --git a/doc/ci/pipelines/job_artifacts.md b/doc/ci/pipelines/job_artifacts.md index ed791ea9c4a..d4774016d9c 100644 --- a/doc/ci/pipelines/job_artifacts.md +++ b/doc/ci/pipelines/job_artifacts.md @@ -6,9 +6,9 @@ type: reference, howto # Job artifacts > - Introduced in GitLab 8.2 and GitLab Runner 0.7.0. -> - Starting with GitLab 8.4 and GitLab Runner 1.0, the artifacts archive format changed to `ZIP`, and it is now possible to browse its contents, with the added ability of downloading the files separately. +> - Starting with GitLab 8.4 and GitLab Runner 1.0, the artifacts archive format changed to `ZIP`, and it's now possible to browse its contents, with the added ability of downloading the files separately. > - In GitLab 8.17, builds were renamed to jobs. -> - The artifacts browser will be available only for new artifacts that are sent to GitLab using GitLab Runner version 1.0 and up. It will not be possible to browse old artifacts already uploaded to GitLab. +> - The artifacts browser will be available only for new artifacts that are sent to GitLab using GitLab Runner version 1.0 and up. It won't be possible to browse old artifacts already uploaded to GitLab. Job artifacts are a list of files and directories created by a job once it finishes. This feature is [enabled by default](../../administration/job_artifacts.md) in all @@ -34,7 +34,7 @@ pdf: expire_in: 1 week ``` -A job named `pdf` calls the `xelatex` command in order to build a pdf file from +A job named `pdf` calls the `xelatex` command in order to build a PDF file from the latex source file `mycv.tex`. We then define the `artifacts` paths which in turn are defined with the `paths` keyword. All paths to files and directories are relative to the repository that was cloned during the build. @@ -42,28 +42,229 @@ are relative to the repository that was cloned during the build. The artifacts will be uploaded when the job succeeds by default, but can be set to upload when the job fails, or always, if the [`artifacts:when`](../yaml/README.md#artifactswhen) parameter is used. These uploaded artifacts will be kept in GitLab for 1 week as defined -by the `expire_in` definition. You have the option to keep the artifacts from expiring +by the `expire_in` definition. You can keep the artifacts from expiring via the [web interface](#browsing-artifacts). If the expiry time is not defined, it defaults to the [instance wide setting](../../user/admin_area/settings/continuous_integration.md#default-artifacts-expiration-core-only). For more examples on artifacts, follow the [artifacts reference in `.gitlab-ci.yml`](../yaml/README.md#artifacts). +### `artifacts:reports` + +> - [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/20390) in GitLab 11.2. +> - Requires GitLab Runner 11.2 and above. + +The `artifacts:reports` keyword is used for collecting test reports, code quality +reports, and security reports from jobs. It also exposes these reports in GitLab's +UI (merge requests, pipeline views, and security dashboards). + +NOTE: **Note:** +The test reports are collected regardless of the job results (success or failure). +You can use [`artifacts:expire_in`](../yaml/README.md#artifactsexpire_in) to set up an expiration +date for their artifacts. + +NOTE: **Note:** +If you also want the ability to browse the report output files, include the +[`artifacts:paths`](../yaml/README.md#artifactspaths) keyword. + +#### `artifacts:reports:junit` + +> - [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/20390) in GitLab 11.2. +> - Requires GitLab Runner 11.2 and above. + +The `junit` report collects [JUnit XML files](https://www.ibm.com/support/knowledgecenter/en/SSQ2R2_14.1.0/com.ibm.rsar.analysis.codereview.cobol.doc/topics/cac_useresults_junit.html) +as artifacts. Although JUnit was originally developed in Java, there are many +[third party ports](https://en.wikipedia.org/wiki/JUnit#Ports) for other +languages like JavaScript, Python, Ruby, and so on. + +See [JUnit test reports](../junit_test_reports.md) for more details and examples. +Below is an example of collecting a JUnit XML file from Ruby's RSpec test tool: + +```yaml +rspec: + stage: test + script: + - bundle install + - rspec --format RspecJunitFormatter --out rspec.xml + artifacts: + reports: + junit: rspec.xml +``` + +The collected JUnit reports will be uploaded to GitLab as an artifact and will +be automatically shown in merge requests. + +NOTE: **Note:** +In case the JUnit tool you use exports to multiple XML files, you can specify +multiple test report paths within a single job and they will be automatically +concatenated into a single file. Use a filename pattern (`junit: rspec-*.xml`), +an array of filenames (`junit: [rspec-1.xml, rspec-2.xml, rspec-3.xml]`), or a +combination thereof (`junit: [rspec.xml, test-results/TEST-*.xml]`). + +#### `artifacts:reports:dotenv` + +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/issues/17066) in GitLab 12.9. +> - Requires GitLab Runner 11.5 and later. + +The `dotenv` report collects a set of environment variables as artifacts. + +The collected variables are registered as runtime-created variables of the job, +which is useful to [set dynamic environment URLs after a job finishes](../environments/index.md#set-dynamic-environment-urls-after-a-job-finishes). + +There are a couple of limitations on top of the [original dotenv rules](https://github.com/motdotla/dotenv#rules). + +- The variable key can contain only letters, digits and underscore ('_'). +- The size of the dotenv file must be smaller than 5 kilobytes. +- The number of variables must be less than 10. +- It does not support variable substitution in the dotenv file itself. +- It does not support empty lines and comments (`#`) in dotenv file. +- It does not support quote escape, spaces in a quote, a new line expansion in a quote, in dotenv file. + +#### `artifacts:reports:cobertura` + +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/issues/3708) in GitLab 12.9. +> - Requires [GitLab Runner](https://docs.gitlab.com/runner/) 11.5 and above. + +The `cobertura` report collects [Cobertura coverage XML files](../../user/project/merge_requests/test_coverage_visualization.md). +The collected Cobertura coverage reports will be uploaded to GitLab as an artifact +and will be automatically shown in merge requests. + +Cobertura was originally developed for Java, but there are many +third party ports for other languages like JavaScript, Python, Ruby, and so on. + +#### `artifacts:reports:terraform` + +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/207528) in GitLab 13.0. +> - Requires [GitLab Runner](https://docs.gitlab.com/runner/) 11.5 and above. + +The `terraform` report obtains a Terraform `tfplan.json` file. The collected Terraform +plan report will be uploaded to GitLab as an artifact and will be automatically shown +in merge requests. + +#### `artifacts:reports:codequality` **(STARTER)** + +> - Introduced in GitLab 11.5. +> - Requires GitLab Runner 11.5 and above. + +The `codequality` report collects [CodeQuality issues](../../user/project/merge_requests/code_quality.md) +as artifacts. + +The collected Code Quality report will be uploaded to GitLab as an artifact and will +be summarized in merge requests. + +#### `artifacts:reports:sast` **(ULTIMATE)** + +> - Introduced in GitLab 11.5. +> - Requires GitLab Runner 11.5 and above. + +The `sast` report collects [SAST vulnerabilities](../../user/application_security/sast/index.md) +as artifacts. + +The collected SAST report will be uploaded to GitLab as an artifact and will be summarized +in the merge requests and pipeline view. It's also used to provide data for security +dashboards. + +#### `artifacts:reports:dependency_scanning` **(ULTIMATE)** + +> - Introduced in GitLab 11.5. +> - Requires GitLab Runner 11.5 and above. + +The `dependency_scanning` report collects [Dependency Scanning vulnerabilities](../../user/application_security/dependency_scanning/index.md) +as artifacts. + +The collected Dependency Scanning report will be uploaded to GitLab as an artifact and will +be summarized in the merge requests and pipeline view. It's also used to provide data for security +dashboards. + +#### `artifacts:reports:container_scanning` **(ULTIMATE)** + +> - Introduced in GitLab 11.5. +> - Requires GitLab Runner 11.5 and above. + +The `container_scanning` report collects [Container Scanning vulnerabilities](../../user/application_security/container_scanning/index.md) +as artifacts. + +The collected Container Scanning report will be uploaded to GitLab as an artifact and will +be summarized in the merge requests and pipeline view. It's also used to provide data for security +dashboards. + +#### `artifacts:reports:dast` **(ULTIMATE)** + +> - Introduced in GitLab 11.5. +> - Requires GitLab Runner 11.5 and above. + +The `dast` report collects [DAST vulnerabilities](../../user/application_security/dast/index.md) +as artifacts. + +The collected DAST report will be uploaded to GitLab as an artifact and will +be summarized in the merge requests and pipeline view. It's also used to provide data for security +dashboards. + +#### `artifacts:reports:license_management` **(ULTIMATE)** + +> - Introduced in GitLab 11.5. +> - Requires GitLab Runner 11.5 and above. + +CAUTION: **Warning:** +This artifact is still valid but is **deprecated** in favor of the +[artifacts:reports:license_scanning](../pipelines/job_artifacts.md#artifactsreportslicense_scanning-ultimate) +introduced in GitLab 12.8. + +The `license_management` report collects [Licenses](../../user/compliance/license_compliance/index.md) +as artifacts. + +The collected License Compliance report will be uploaded to GitLab as an artifact and will +be summarized in the merge requests and pipeline view. It's also used to provide data for security +dashboards. + +#### `artifacts:reports:license_scanning` **(ULTIMATE)** + +> - Introduced in GitLab 12.8. +> - Requires GitLab Runner 11.5 and above. + +The `license_scanning` report collects [Licenses](../../user/compliance/license_compliance/index.md) +as artifacts. + +The License Compliance report will be uploaded to GitLab as an artifact and will +be automatically shown in merge requests, pipeline view and provide data for security +dashboards. + +#### `artifacts:reports:performance` **(PREMIUM)** + +> - Introduced in GitLab 11.5. +> - Requires GitLab Runner 11.5 and above. + +The `performance` report collects [Performance metrics](../../user/project/merge_requests/browser_performance_testing.md) +as artifacts. + +The collected Performance report will be uploaded to GitLab as an artifact and will +be automatically shown in merge requests. + +#### `artifacts:reports:metrics` **(PREMIUM)** + +> Introduced in GitLab 11.10. + +The `metrics` report collects [Metrics](../metrics_reports.md) +as artifacts. + +The collected Metrics report will be uploaded to GitLab as an artifact and will +be automatically shown in merge requests. + ## Browsing artifacts -> - From GitLab 9.2, PDFs, images, videos and other formats can be previewed directly in the job artifacts browser without the need to download them. +> - From GitLab 9.2, PDFs, images, videos, and other formats can be previewed directly in the job artifacts browser without the need to download them. > - Introduced in [GitLab 10.1](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/14399), HTML files in a public project can be previewed directly in a new tab without the need to download them when [GitLab Pages](../../administration/pages/index.md) is enabled. The same applies for textual formats (currently supported extensions: `.txt`, `.json`, and `.log`). > - Introduced in [GitLab 12.4](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/16675), artifacts in private projects can be previewed when [GitLab Pages access control](../../administration/pages/index.md#access-control) is enabled. After a job finishes, if you visit the job's specific page, there are three buttons. You can download the artifacts archive or browse its contents, whereas -the **Keep** button appears only if you have set an [expiry date](../yaml/README.md#artifactsexpire_in) to the +the **Keep** button appears only if you've set an [expiry date](../yaml/README.md#artifactsexpire_in) to the artifacts in case you changed your mind and want to keep them.  The archive browser shows the name and the actual file size of each file in the -archive. If your artifacts contained directories, then you are also able to +archive. If your artifacts contained directories, then you're also able to browse inside them. Below you can see what browsing looks like. In this case we have browsed inside @@ -75,20 +276,20 @@ one HTML file that you can view directly online when ## Downloading artifacts -If you need to download the whole archive, there are buttons in various places +If you need to download an artifact or the whole archive, there are buttons in various places in the GitLab UI to do this: 1. While on the pipelines page, you can see the download icon for each job's - artifacts archive in the right corner: + artifacts and archive in the right corner:  1. While on the **Jobs** page, you can see the download icon for each job's - artifacts archive in the right corner: + artifacts and archive in the right corner:  -1. While inside a specific job, you are presented with a download button +1. While inside a specific job, you're presented with a download button along with the one that browses the archive:  @@ -100,7 +301,7 @@ in the GitLab UI to do this: ## Downloading the latest artifacts -It is possible to download the latest artifacts of a job via a well known URL +It's possible to download the latest artifacts of a job via a well known URL so you can use it for scripting purposes. NOTE: **Note:** @@ -151,7 +352,7 @@ For example: https://gitlab.com/gitlab-org/gitlab/-/jobs/artifacts/master/browse?job=coverage ``` -There is also a URL to specific files, including html files that +There is also a URL to specific files, including HTML files that are shown in [GitLab Pages](../../administration/pages/index.md): ```plaintext diff --git a/doc/ci/pipelines/schedules.md b/doc/ci/pipelines/schedules.md index 0ca794c5411..0c0a613c628 100644 --- a/doc/ci/pipelines/schedules.md +++ b/doc/ci/pipelines/schedules.md @@ -21,6 +21,15 @@ Pipeline schedules can be used to also run [pipelines](index.md) at specific int In addition to using the GitLab UI, pipeline schedules can be maintained using the [Pipeline schedules API](../../api/pipeline_schedules.md). +## Prerequisites + +In order for a scheduled pipeline to be created successfully: + +- The schedule owner must have [permissions](../../user/permissions.md) to merge into the target branch. +- The pipeline configuration must be valid. + +Otherwise the pipeline is not created. + ## Configuring pipeline schedules To schedule a pipeline for project: diff --git a/doc/ci/pipelines/settings.md b/doc/ci/pipelines/settings.md index 3221023f73a..0a859b5b68f 100644 --- a/doc/ci/pipelines/settings.md +++ b/doc/ci/pipelines/settings.md @@ -130,6 +130,16 @@ in the jobs table. A few examples of known coverage tools for a variety of languages can be found in the pipelines settings page. +### Download test coverage history + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/209121) in GitLab 12.10. + +If you want to see the evolution of your project code coverage over time, +you can download a CSV file with this data. From your project: + +1. Go to **{chart}** **Project Analytics > Repository**. +1. Click **Download raw data (.csv)** + ### Removing color codes Some test coverage tools output with ANSI color codes that won't be @@ -162,6 +172,9 @@ This also determines the visibility of these related features: - Job artifacts - The [pipeline security dashboard](../../user/application_security/security_dashboard/index.md#pipeline-security) **(ULTIMATE)** +NOTE: **Note:** +Currently, job logs and artifacts are [not yet visible for guest users and non-project members](https://gitlab.com/gitlab-org/gitlab/-/issues/25649). + If **Public pipelines** is enabled (default): - For **public** projects, anyone can view the pipelines and related features. @@ -237,7 +250,7 @@ Depending on the status of your job, a badge can have the following values: You can access a pipeline status badge image using the following link: -```text +```plaintext https://example.gitlab.com/<namespace>/<project>/badges/<branch>/pipeline.svg ``` @@ -249,7 +262,7 @@ pipeline can have the test coverage percentage value defined. The test coverage badge can be accessed using following link: -```text +```plaintext https://example.gitlab.com/<namespace>/<project>/badges/<branch>/coverage.svg ``` @@ -268,7 +281,7 @@ Pipeline badges can be rendered in different styles by adding the `style=style_n #### Flat (default) -```text +```plaintext https://example.gitlab.com/<namespace>/<project>/badges/<branch>/coverage.svg?style=flat ``` @@ -278,7 +291,7 @@ https://example.gitlab.com/<namespace>/<project>/badges/<branch>/coverage.svg?st > [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/issues/30120) in GitLab 11.8. -```text +```plaintext https://example.gitlab.com/<namespace>/<project>/badges/<branch>/coverage.svg?style=flat-square ``` diff --git a/doc/ci/review_apps/index.md b/doc/ci/review_apps/index.md index c3bdd524bff..ad0a4270f43 100644 --- a/doc/ci/review_apps/index.md +++ b/doc/ci/review_apps/index.md @@ -1,4 +1,7 @@ --- +stage: Release +group: Progressive Delivery +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers type: reference --- @@ -29,7 +32,7 @@ In the above example: ## How Review Apps work -A Review App is a mapping of a branch with an [environment](../environments.md). +A Review App is a mapping of a branch with an [environment](../environments/index.md). Access to the Review App is made available as a link on the [merge request](../../user/project/merge_requests.md) relevant to the branch. The following is an example of a merge request with an environment set dynamically. @@ -49,7 +52,7 @@ After adding Review Apps to your workflow, you follow the branched Git flow. Tha ## Configuring Review Apps -Review Apps are built on [dynamic environments](../environments.md#configuring-dynamic-environments), which allow you to dynamically create a new environment for each branch. +Review Apps are built on [dynamic environments](../environments/index.md#configuring-dynamic-environments), which allow you to dynamically create a new environment for each branch. The process of configuring Review Apps is as follows: @@ -58,7 +61,7 @@ The process of configuring Review Apps is as follows: 1. Set up a job in `.gitlab-ci.yml` that uses the [predefined CI environment variable](../variables/README.md) `${CI_COMMIT_REF_NAME}` to create dynamic environments and restrict it to run only on branches. Alternatively, you can get a YML template for this job by [enabling review apps](#enable-review-apps-button) for your project. -1. Optionally, set a job that [manually stops](../environments.md#stopping-an-environment) the Review Apps. +1. Optionally, set a job that [manually stops](../environments/index.md#stopping-an-environment) the Review Apps. ### Enable Review Apps button @@ -82,7 +85,7 @@ you can copy and paste into `.gitlab-ci.yml` as a starting point. To do so: ## Review Apps auto-stop -See how to [configure Review Apps environments to expire and auto-stop](../environments.md#environments-auto-stop) +See how to [configure Review Apps environments to expire and auto-stop](../environments/index.md#environments-auto-stop) after a given period of time. ## Review Apps examples @@ -100,7 +103,7 @@ See also the video [Demo: Cloud Native Development with GitLab](https://www.yout > Introduced in GitLab 8.17. In GitLab 11.5, the file links are available in the merge request widget. Route Maps allows you to go directly from source files -to public pages on the [environment](../environments.md) defined for +to public pages on the [environment](../environments/index.md) defined for Review Apps. Once set up, the review app link in the merge request @@ -205,7 +208,7 @@ if [route maps](#route-maps) are configured in the project.  -The provided script should be added to the `<head>` of you application and +The provided script should be added to the `<head>` of your application and consists of some project and merge request specific values. Here's what it looks like: @@ -267,7 +270,7 @@ to your review app. ​After determining the ID for the merge request to link to a visual review app, you can supply the ID by either:​​ -- Hardcoding it in the script tag via the data attribute `data-merge-request-id` of the app. +- Hard-coding it in the script tag via the data attribute `data-merge-request-id` of the app. - Dynamically adding the `data-merge-request-id` value during the build of the app. - Supplying it manually through the visual review form in the app. @@ -278,7 +281,7 @@ can supply the ID by either:​​ To enable visual reviews for private and internal projects, set the [`data-require-auth` variable](#configuring-visual-reviews) to `true`. When enabled, the user must enter a [personal access token](../../user/profile/personal_access_tokens.md) -with `read_api` scope before submitting feedback. +with `api` scope before submitting feedback. ### Using Visual Reviews @@ -301,4 +304,4 @@ automatically in the respective merge request. ## Limitations -Review App limitations are the same as [environments limitations](../environments.md#limitations). +Review App limitations are the same as [environments limitations](../environments/index.md#limitations). diff --git a/doc/ci/services/mysql.md b/doc/ci/services/mysql.md index e21b62e93cf..dcfd863709e 100644 --- a/doc/ci/services/mysql.md +++ b/doc/ci/services/mysql.md @@ -27,7 +27,7 @@ variables: NOTE: **Note:** The `MYSQL_DATABASE` and `MYSQL_ROOT_PASSWORD` variables can't be set in the GitLab UI. -To set them, assign them to a variable [in the UI](../variables/README.md#via-the-ui), +To set them, assign them to a variable [in the UI](../variables/README.md#create-a-custom-variable-in-the-ui), and then assign that variable to the `MYSQL_DATABASE` and `MYSQL_ROOT_PASSWORD` variables in your `.gitlab-ci.yml`. diff --git a/doc/ci/services/postgres.md b/doc/ci/services/postgres.md index cf34c28497e..2f92bd969ff 100644 --- a/doc/ci/services/postgres.md +++ b/doc/ci/services/postgres.md @@ -29,7 +29,7 @@ variables: NOTE: **Note:** The `POSTGRES_DB`, `POSTGRES_USER`, `POSTGRES_PASSWORD` and `POSTGRES_HOST_AUTH_METHOD` variables can't be set in the GitLab UI. To set them, assign them to a variable -[in the UI](../variables/README.md#via-the-ui), and then assign that +[in the UI](../variables/README.md#create-a-custom-variable-in-the-ui), and then assign that variable to the `POSTGRES_DB`, `POSTGRES_USER`, `POSTGRES_PASSWORD` and `POSTGRES_HOST_AUTH_METHOD` variables in your `.gitlab-ci.yml`. diff --git a/doc/ci/variables/README.md b/doc/ci/variables/README.md index 99fbc2134a4..3e31a2169e2 100644 --- a/doc/ci/variables/README.md +++ b/doc/ci/variables/README.md @@ -4,149 +4,41 @@ type: reference # GitLab CI/CD environment variables -After a brief overview of the use of environment -variables, this document teaches you how to use GitLab CI/CD's -variables, presents the full reference for predefined variables, -and dives into more advanced applications. - -## Overview - -An environment variable is a dynamic-named value that can -affect the way running processes will behave on an operating +An environment variable is a dynamically-named value that can +affect the way running processes behave on an operating system. -They are part of the environment in which a process runs. +Environment variables are part of the environment in which a process runs. For example, a running process can query the value of the `TEMP` environment variable to discover a suitable location to store temporary files, or to define a `URL` for a database that can be reused in different scripts. -Variables are useful for customizing your jobs in GitLab -CI/CD's pipelines. Using variables means no hardcoded values. +Variables are useful for customizing your jobs in GitLab CI/CD. +When you use variables, you don't have to hard-code values. -### Predefined environment variables +## Predefined environment variables GitLab CI/CD has a [default set of predefined variables](predefined_variables.md) -which can be used without any specification needed. -You can call issues numbers, user names, branch names, +that you can use without any additional specification. +You can call issue numbers, user names, branch names, pipeline and commit IDs, and much more. -Predefined environment variables are the ones that GitLab -provides out of the box for the local environment of the Runner. - -GitLab reads the `.gitlab-ci.yml` file, sends the information -to the Runner (which runs the script commands), under which -the variables are exposed. - -For example, two jobs under the same pipeline can share the same -`CI_PIPELINE_ID` variable, but each one has its own `CI_JOB_ID` -variable. - -NOTE: **Note:** -Find here the full [**predefined variables reference table**](predefined_variables.md). - -### Custom environment variables - -When your use case requires a specific variable, you can -[set them up easily from the UI](#creating-a-custom-environment-variable) -or directly in the `.gitlab-ci.yml` file and reuse them as you wish. - -That can be very powerful as it can be used for scripting without -the need to specify the value itself. - -#### Types of variables - -> [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/issues/46806) in GitLab 11.11. - -There are two types of variables supported by GitLab: - -- [Variable type](#variable-type): The Runner will create an environment variable named the same as the - variable key and set its value to the variable value. -- [File type](#file-type): The Runner will write the variable value to a temporary file and set the - path to this file as the value of an environment variable, named the same as the variable key. - -##### Variable type - -Many tools (like [AWS CLI](https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-envvars.html) -and [kubectl](https://kubernetes.io/docs/concepts/configuration/organize-cluster-access-kubeconfig/#the-kubeconfig-environment-variable)) -provide the ability to customise configuration using files by either providing the -file path as a command line argument or an environment variable. In the past, the -common pattern was to read the value of a CI variable, save it in a file, and then -use the newly created file in your script: - -```shell -# Read certificate stored in $KUBE_CA_PEM variable and save it in a new file -echo "$KUBE_CA_PEM" > "$(pwd)/kube.ca.pem" -# Pass the newly created file to kubectl -kubectl config set-cluster e2e --server="$KUBE_URL" --certificate-authority="$(pwd)/kube.ca.pem" -``` - -There are [some predefined variables](#custom-variables-validated-by-gitlab) of this type, which may be further validated. They will appear when you add or update a variable. - -##### File type - -The example above can now be simplified by creating a "File" type variable, and using -it directly. For example, let's say we have the following variables: - - - -We can then call them from `.gitlab-ci.yml` like this: - -```shell -kubectl config set-cluster e2e --server="$KUBE_URL" --certificate-authority="$KUBE_CA_PEM" -``` - -Variable types can be set via the [UI](#via-the-ui) or the [API](../../api/project_level_variables.md#create-variable), but not in `.gitlab-ci.yml`. - -#### Masked variables - -> [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/issues/13784) in GitLab 11.10 - -Variables can be created as masked variables. -This means that the value of the variable will be hidden in job logs, -though it must match certain requirements to do so: - -- The value must be in a single line. -- The value must only consist of characters from the Base64 alphabet (RFC4648). - - [In GitLab 12.2](https://gitlab.com/gitlab-org/gitlab-foss/issues/63043) - and newer, `@` and `:` are also valid values. -- The value must be at least 8 characters long. -- The value must not use variables. - -If the value does not meet the requirements above, then the CI variable will fail to save. -In order to save, either alter the value to meet the masking requirements -or disable **Masked** for the variable. - -#### Custom variables validated by GitLab - -Some variables are listed in the UI so you can choose them more quickly. -GitLab validates the values of these variables to ensure they are in the correct format. - -| Variable | Allowed Values | Introduced in | -|-------------------------|----------------------------------------------------|---------------| -| `AWS_ACCESS_KEY_ID` | 20 characters: letters, digits | 12.10 | -| `AWS_DEFAULT_REGION` | Any | 12.10 | -| `AWS_SECRET_ACCESS_KEY` | 40 characters: letters, digits, special characters | 12.10 | +Predefined environment variables are provided by GitLab +for the local environment of the Runner. -NOTE: **Note:** -When you store credentials, there are security implications. If you are using AWS keys, -for example, follow their [best practices](https://docs.aws.amazon.com/general/latest/gr/aws-access-keys-best-practices.html). +GitLab reads the `.gitlab-ci.yml` file and sends the information +to the Runner, where the variables are exposed. The Runner then runs the script commands. -## Getting started +### Use predefined environment variables -To get started with environment variables in the scope of GitLab -CI/CD, let's go over a few examples. +You can choose one of the existing predefined variables +to be output by the Runner. -### Using predefined environment variables +This example shows how to output a job's stage by using the predefined variable `CI_JOB_STAGE`. -To get started, choose one of the existing -[predefined variables](predefined_variables.md) -to be output by the Runner. For example, let's say that you want -a given job you're running through your script to output the -stage that job is running for. In your `.gitlab-ci.yml` file, -call the variable from your script according to the [syntaxes](#syntax-of-environment-variables-in-job-scripts) available. To -output the job stage, use the predefined variable `CI_JOB_STAGE`: +In your `.gitlab-ci.yml` file, call the variable from your script. Ensure +you use the correct [syntax](#syntax-of-environment-variables-in-job-scripts). ```yaml test_variable: @@ -155,14 +47,14 @@ test_variable: - echo $CI_JOB_STAGE ``` -For this case, the Runner will output the `stage` for the +In this case, the Runner outputs the `stage` for the job `test_variable`, which is `test`:  As another example, let's say you're using your own GitLab -instance you want to know what domain your GitLab Pages are -served under. You can easily call it with the predefined +instance and you want to know what domain your GitLab Pages are +served under. You can call it by using the predefined variable `$CI_PAGES_DOMAIN` in your script: ```yaml @@ -176,47 +68,54 @@ For GitLab.com users, the output will be `gitlab.io`. For your private instance, the output will be whatever your sysadmin has defined. -### Creating a custom environment variable +## Custom environment variables -Assume you have something you want to repeat through your scripts -in GitLab CI/CD's configuration file. To keep this example simple, -let's say you want to output `HELLO WORLD` for a `TEST` variable. +When you need a specific custom environment variable, you can +[set it up in the UI](#create-a-custom-variable-in-the-ui), in [the API](../../api/project_level_variables.md), +or directly [in the `.gitlab-ci.yml` file](#create-a-custom-variable-in-gitlab-ciyml). -You can either set the variable directly in the `.gitlab-ci.yml` -file or through the UI. +The variables are used by the Runner any time the pipeline runs. +You can also [override variable values manually for a specific pipeline](../pipelines/index.md#specifying-variables-when-running-manual-jobs). -NOTE: **Note:** -It is possible to [specify variables when running manual jobs](../pipelines/index.md#specifying-variables-when-running-manual-jobs). +There are two types of variables: **Variable** and **File**. You cannot set types in +the `.gitlab-ci.yml` file, but you can set them in the UI and API. -#### Via `.gitlab-ci.yml` +### Create a custom variable in `.gitlab-ci.yml` -To create a new custom `env_var` variable via [`.gitlab-ci.yml`](../yaml/README.md#variables), define their variable/value pair under -`variables`: +To create a custom `env_var` variable in the [`.gitlab-ci.yml`](../yaml/README.md#variables) file, +define the variable/value pair under `variables`: ```yaml variables: TEST: "HELLO WORLD" ``` -For a deeper look into them, see [`.gitlab-ci.yml` defined variables](#gitlab-ciyml-defined-variables). +You can then call its value in your script: + +```yaml + script: + - echo "$TEST" +``` + +For more details, see [`.gitlab-ci.yml` defined variables](#gitlab-ciyml-defined-variables). -#### Via the UI +### Create a custom variable in the UI From within the UI, you can add or update custom environment variables: 1. Go to your project's **Settings > CI/CD** and expand the **Variables** section. -1. Click the **Add variable** button. In the **Add variable** modal, fill in the details: +1. Click the **Add Variable** button. In the **Add variable** modal, fill in the details: - **Key**: Must be one line, with no spaces, using only letters, numbers, `-` or `_`. - **Value**: No limitations. - **Type**: `File` or `Variable`. - **Environment scope**: `All`, or specific environments. - **Protect variable** (Optional): If selected, the variable will only be available in pipelines that run on protected branches or tags. - - **Mask variable** (Optional): If selected, the variable's **Value** will be masked in job logs. The variable will fail to save if the value does not meet the [masking requirements](#masked-variables). + - **Mask variable** (Optional): If selected, the variable's **Value** will be masked in job logs. The variable fails to save if the value does not meet the [masking requirements](#masked-variable-requirements). -After a variable is created, you can update any of the details by clicking on the **{pencil}** **Edit** button. +After a variable is created, you can update any of the details by clicking the **{pencil}** **Edit** button. -Once you've set the variables, call them from the `.gitlab-ci.yml` file: +After you set a variable, call it from the `.gitlab-ci.yml` file: ```yaml test_variable: @@ -232,7 +131,110 @@ The output will be:  -### Syntax of environment variables in job scripts +### Custom environment variables of type Variable + +> [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/issues/46806) in GitLab 11.11. + +For variables with the type **Variable**, the Runner creates an environment variable +that uses the key for the name and the value for the value. + +There are [some predefined variables](#custom-variables-validated-by-gitlab) of this type, +which may be further validated. They appear when you add or update a variable in the UI. + +### Custom environment variables of type File + +> [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/issues/46806) in GitLab 11.11. + +For variables with the type **File**, the Runner creates an environment variable that uses the key for the name. +For the value, the Runner writes the variable value to a temporary file and uses this path. + +You can use tools like [the AWS CLI](https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-envvars.html) +and [kubectl](https://kubernetes.io/docs/concepts/configuration/organize-cluster-access-kubeconfig/#the-kubeconfig-environment-variable) +to customize your configuration by using **File** type variables. + +In the past, a common pattern was to read the value of a CI variable, save it in a file, and then +use the newly created file in your script: + +```shell +# Read certificate stored in $KUBE_CA_PEM variable and save it in a new file +echo "$KUBE_CA_PEM" > "$(pwd)/kube.ca.pem" +# Pass the newly created file to kubectl +kubectl config set-cluster e2e --server="$KUBE_URL" --certificate-authority="$(pwd)/kube.ca.pem" +``` + +Instead of this, you can use a **File** type variable. For example, if you have the following variables: + +- A variable of type **Variable**: `KUBE_URL` with the value `https://example.com`. +- A variable of type **File**: `KUBE_CA_PEM` with a certificate as the value. + +You can call them from `.gitlab-ci.yml`, like this: + +```shell +kubectl config set-cluster e2e --server="$KUBE_URL" --certificate-authority="$KUBE_CA_PEM" +``` + +### Mask a custom variable + +> [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/issues/13784) in GitLab 11.10 + +Variables can be masked so that the value of the variable will be hidden in job logs. + +To mask a variable: + +1. Go to **Settings > CI/CD**. +1. Expand the **Variables** section. +1. Next to the variable you want to protect, click **Edit**. +1. Select the **Mask variable** check box. +1. Click **Update variable**. + +#### Masked variable requirements + +The value of the variable must: + +- Be in a single line. +- Be at least 8 characters long. +- Not be a predefined or custom environment variable. +- Consist only of characters from the Base64 alphabet (RFC4648). + [In GitLab 12.2](https://gitlab.com/gitlab-org/gitlab-foss/issues/63043) + and newer, `@` and `:` are also valid values. + +You can't mask variables that don't meet these requirements. + +### Protect a custom variable + +> Introduced in GitLab 9.3. + +Variables can be protected. When a variable is +protected, it is securely passed to pipelines running on +[protected branches](../../user/project/protected_branches.md) or [protected tags](../../user/project/protected_tags.md) only. The other pipelines do not get +the protected variable. + +To protect a variable: + +1. Go to **Settings > CI/CD**. +1. Expand the **Variables** section. +1. Next to the variable you want to protect, click **Edit**. +1. Select the **Protect variable** check box. +1. Click **Update variable**. + +The variable is available for all subsequent pipelines. + +### Custom variables validated by GitLab + +Some variables are listed in the UI so you can choose them more quickly. +GitLab validates the values of these variables to ensure they are in the correct format. + +| Variable | Allowed Values | Introduced in | +|-------------------------|----------------------------------------------------|---------------| +| `AWS_ACCESS_KEY_ID` | 20 characters: letters, digits | 12.10 | +| `AWS_DEFAULT_REGION` | Any | 12.10 | +| `AWS_SECRET_ACCESS_KEY` | 40 characters: letters, digits, special characters | 12.10 | + +NOTE: **Note:** +When you store credentials, there are security implications. If you are using AWS keys, +for example, follow their [best practices](https://docs.aws.amazon.com/general/latest/gr/aws-access-keys-best-practices.html). + +## Syntax of environment variables in job scripts All variables are set as environment variables in the build environment, and they are accessible with normal methods that are used to access such variables. @@ -329,14 +331,14 @@ export GITLAB_USER_EMAIL="user@example.com" export GITLAB_USER_ID="42" ``` -### `.gitlab-ci.yml` defined variables +## `.gitlab-ci.yml` defined variables NOTE: **Note:** This feature requires GitLab Runner 0.5.0 or higher and GitLab 7.14 or higher. -GitLab CI/CD allows you to add to `.gitlab-ci.yml` variables that are set in the -build environment. The variables are hence saved in the repository, and they -are meant to store non-sensitive project configuration. For example, `RAILS_ENV` or +You can add variables that are set in the build environment to `.gitlab-ci.yml`. +These variables are saved in the repository, and they +are meant to store non-sensitive project configuration, like `RAILS_ENV` or `DATABASE_URL`. For example, if you set the variable below globally (not inside a job), it will @@ -348,7 +350,7 @@ variables: ``` The YAML-defined variables are also set to all created -[service containers](../docker/using_docker_images.md), thus allowing to fine +[service containers](../docker/using_docker_images.md), so that you can fine tune them. Variables can be defined at a global level, but also at a job level. To turn off @@ -369,11 +371,11 @@ script: - 'eval $LS_CMD' # will execute 'ls -al $TMP_DIR' ``` -### Group-level environment variables +## Group-level environment variables > Introduced in GitLab 9.4. -GitLab CI/CD allows you to define per-project or per-group variables +You can define per-project or per-group variables that are set in the pipeline environment. Group-level variables are stored out of the repository (not in `.gitlab-ci.yml`) and are securely passed to GitLab Runner making them available during a pipeline run. It's the **recommended method** to @@ -382,7 +384,7 @@ use for storing things like passwords, SSH keys, and credentials. Group-level variables can be added by: 1. Navigating to your group's **Settings > CI/CD** page. -1. Inputing variable types, keys, and values in the **Variables** section. +1. Inputting variable types, keys, and values in the **Variables** section. Any variables of [subgroups](../../user/group/subgroups/index.md) will be inherited recursively. Once you set them, they will be available for all subsequent pipelines. Any group-level user defined variables can be viewed in projects by: @@ -392,6 +394,73 @@ Once you set them, they will be available for all subsequent pipelines. Any grou  +### Inherit environment variables + +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/22638) in GitLab 13.0. +> - It's deployed behind a feature flag (`ci_dependency_variables`), disabled by default. + +You can inherit environment variables from dependent jobs. + +This feature makes use of the [`artifacts:reports:dotenv`](../pipelines/job_artifacts.md#artifactsreportsdotenv) report feature. + +Example with [`dependencies`](../yaml/README.md#dependencies) keyword. + +```yaml +build: + stage: build + script: + - echo "BUILD_VERSION=hello" >> build.env + artifacts: + reports: + dotenv: build.env + +deploy: + stage: deploy + script: + - echo $BUILD_VERSION # => hello + dependencies: + - build +``` + +Example with the [`needs`](../yaml/README.md#artifact-downloads-with-needs) keyword: + +```yaml +build: + stage: build + script: + - echo "BUILD_VERSION=hello" >> build.env + artifacts: + reports: + dotenv: build.env + +deploy: + stage: deploy + script: + - echo $BUILD_VERSION # => hello + needs: + - job: build + artifacts: true +``` + +### Enable inherited environment variables **(CORE ONLY)** + +The Inherited Environment Variables feature is under development and not ready for production use. It is +deployed behind a feature flag that is **disabled by default**. +[GitLab administrators with access to the GitLab Rails console](../../administration/feature_flags.md) +can enable it for your instance. + +To enable it: + +```ruby +Feature.enable(:ci_dependency_variables) +``` + +To disable it: + +```ruby +Feature.disable(:ci_dependency_variables) +``` + ## Priority of environment variables Variables of different types can take precedence over other @@ -400,8 +469,9 @@ variables, depending on where they are defined. The order of precedence for variables is (from highest to lowest): 1. [Trigger variables](../triggers/README.md#making-use-of-trigger-variables) or [scheduled pipeline variables](../pipelines/schedules.md#using-variables). -1. Project-level [variables](#creating-a-custom-environment-variable) or [protected variables](#protected-environment-variables). -1. Group-level [variables](#group-level-environment-variables) or [protected variables](#protected-environment-variables). +1. Project-level [variables](#custom-environment-variables) or [protected variables](#protect-a-custom-variable). +1. Group-level [variables](#group-level-environment-variables) or [protected variables](#protect-a-custom-variable). +1. [Inherited environment variables](#inherit-environment-variables). 1. YAML-defined [job-level variables](../yaml/README.md#variables). 1. YAML-defined [global variables](../yaml/README.md#variables). 1. [Deployment variables](#deployment-environment-variables). @@ -426,27 +496,12 @@ Click [here](where_variables_can_be_used.md) for a section that describes where ## Advanced use -### Protected environment variables - -> Introduced in GitLab 9.3. - -Variables can be protected. Whenever a variable is -protected, it would only be securely passed to pipelines running on the -[protected branches](../../user/project/protected_branches.md) or [protected tags](../../user/project/protected_tags.md). The other pipelines would not get any -protected variables. - -Protected variables can be added by going to your project's -**Settings > CI/CD**, then finding the section called -**Variables**, and check "Protected". - -Once you set them, they will be available for all subsequent pipelines. - -### Limiting environment scopes of environment variables +### Limit the environment scopes of environment variables You can limit the environment scope of a variable by -[defining which environments](../environments.md) it can be available for. +[defining which environments](../environments/index.md) it can be available for. -To learn more about scoping environments, see [Scoping environments with specs](../environments.md#scoping-environments-with-specs). +To learn more about scoping environments, see [Scoping environments with specs](../environments/index.md#scoping-environments-with-specs). ### Deployment environment variables @@ -455,7 +510,7 @@ To learn more about scoping environments, see [Scoping environments with specs]( [Integrations](../../user/project/integrations/overview.md) that are responsible for deployment configuration may define their own variables that are set in the build environment. These variables are only defined for -[deployment jobs](../environments.md). Please consult the documentation of +[deployment jobs](../environments/index.md). Please consult the documentation of the integrations that you are using to learn which variables they define. An example integration that defines deployment variables is the @@ -478,22 +533,23 @@ CAUTION: **Caution:** Variables with multiline values are not currently supported due to limitations with the current Auto DevOps scripting environment. -### Environment variables triggered manually +### Override a variable by manually running a pipeline > [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/issues/44059) in GitLab 10.8. -[Manually triggered pipelines](../pipelines/index.md#run-a-pipeline-manually) allow you to override the value of a current variable. +You can override the value of a current variable by +[running a pipeline manually](../pipelines/index.md#run-a-pipeline-manually). + +For instance, suppose you added a custom variable named `$TEST` +and you want to override it in a manual pipeline. -For instance, suppose you added a -[custom variable `$TEST`](#creating-a-custom-environment-variable) -as exemplified above and you want to override it in a manual pipeline. Navigate to your project's **CI/CD > Pipelines** and click **Run pipeline**. -Choose the branch you want to run the pipeline for, then add a new variable through the UI: +Choose the branch you want to run the pipeline for, then add a variable and its value in the UI:  -The Runner will override the value previously set and use the custom -value you set for this specific pipeline: +The Runner overrides the value previously set and uses the custom +value for this specific pipeline.  @@ -502,10 +558,10 @@ value you set for this specific pipeline: > - [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/issues/37397) in GitLab 10.7 for [the `only` and `except` CI keywords](../yaml/README.md#onlyexcept-advanced) > - [Expanded](https://gitlab.com/gitlab-org/gitlab/issues/27863) in GitLab 12.3 with [the `rules` keyword](../yaml/README.md#rules) -Variable expressions can be used to limit what jobs are going to be created -within a pipeline after pushing changes to GitLab. +Use variable expressions to limit which jobs are created +within a pipeline after changes are pushed to GitLab. -In `.gitlab-ci.yml`, they work with both +In `.gitlab-ci.yml`, variable expressions work with both: - [`rules`](../yaml/README.md#rules), which is the recommended approach, and - [`only` and `except`](../yaml/README.md#onlyexcept-basic), which are candidates for deprecation. @@ -523,15 +579,15 @@ deploy: - $STAGING ``` -Each expression provided is going to be evaluated before creating a pipeline. +Each expression provided is evaluated before a pipeline is created. -If any of the conditions in `variables` evaluates to truth when using `only`, -a new job is going to be created. If any of the expressions evaluates to truth -when `except` is being used, a job is not going to be created. +If any of the conditions in `variables` evaluates to true when using `only`, +a new job is created. If any of the expressions evaluates to true +when `except` is being used, a job is not created. -This follows usual rules for [`only` / `except` policies](../yaml/README.md#onlyexcept-advanced). +This follows the usual rules for [`only` / `except` policies](../yaml/README.md#onlyexcept-advanced). -### Supported syntax +### Syntax of environment variable expressions Below you can find supported syntax reference: @@ -679,7 +735,7 @@ If a job isn't working as expected, this can make the problem difficult to investigate; in these cases, you can enable debug tracing in `.gitlab-ci.yml`. Available on GitLab Runner v1.7+, this feature enables the shell's execution log, resulting in a verbose job log listing all commands that were run, -variables that were set, etc. +variables that were set, and so on. Before enabling this, you should ensure jobs are visible to [team members only](../../user/permissions.md#project-features). You should @@ -864,3 +920,10 @@ if [[ -d "/builds/gitlab-examples/ci-debug-trace/.git" ]]; then ... ``` + +## Video walkthrough of a working example + +The [Managing the Complex Configuration Data Management Monster Using GitLab](https://www.youtube.com/watch?v=v4ZOJ96hAck) video is a walkthrough of the [Complex Config Data Monorepo](https://gitlab.com/guided-explorations/config-data-top-scope/config-data-subscope/config-data-monorepo) working example project. It explains how multiple levels of group CI/CD variables can be combined with environment-scoped project variables for complex configuration of application builds or deployments. + +The example can be copied to your own group or instance for testing. More details +on what other GitLab CI patterns are demonstrated are available at the project page. diff --git a/doc/ci/variables/img/ci_job_stage_output_example.png b/doc/ci/variables/img/ci_job_stage_output_example.png Binary files differindex 056238d5693..e333da57121 100644 --- a/doc/ci/variables/img/ci_job_stage_output_example.png +++ b/doc/ci/variables/img/ci_job_stage_output_example.png diff --git a/doc/ci/variables/img/inherited_group_variables_v12_5.png b/doc/ci/variables/img/inherited_group_variables_v12_5.png Binary files differindex fd41859605f..a13ba711083 100644 --- a/doc/ci/variables/img/inherited_group_variables_v12_5.png +++ b/doc/ci/variables/img/inherited_group_variables_v12_5.png diff --git a/doc/ci/variables/img/override_value_via_manual_pipeline_output.png b/doc/ci/variables/img/override_value_via_manual_pipeline_output.png Binary files differindex 02369d57fb8..8a13bb3849e 100644 --- a/doc/ci/variables/img/override_value_via_manual_pipeline_output.png +++ b/doc/ci/variables/img/override_value_via_manual_pipeline_output.png diff --git a/doc/ci/variables/img/override_variable_manual_pipeline.png b/doc/ci/variables/img/override_variable_manual_pipeline.png Binary files differindex c77c5cb7764..de768105aec 100644 --- a/doc/ci/variables/img/override_variable_manual_pipeline.png +++ b/doc/ci/variables/img/override_variable_manual_pipeline.png diff --git a/doc/ci/variables/img/variable_types_usage_example.png b/doc/ci/variables/img/variable_types_usage_example.png Binary files differdeleted file mode 100644 index c2ae32fd048..00000000000 --- a/doc/ci/variables/img/variable_types_usage_example.png +++ /dev/null diff --git a/doc/ci/variables/predefined_variables.md b/doc/ci/variables/predefined_variables.md index f53fd371c10..d4d3a13bb2a 100644 --- a/doc/ci/variables/predefined_variables.md +++ b/doc/ci/variables/predefined_variables.md @@ -56,7 +56,7 @@ future GitLab releases.** | `CI_EXTERNAL_PULL_REQUEST_SOURCE_BRANCH_SHA` | 12.3 | all | The HEAD SHA of the source branch of the pull request if [the pipelines are for external pull requests](../ci_cd_for_external_repos/index.md#pipelines-for-external-pull-requests). Available only if `only: [external_pull_requests]` or [`rules`](../yaml/README.md#rules) syntax is used and the pull request is open. | | `CI_EXTERNAL_PULL_REQUEST_TARGET_BRANCH_NAME` | 12.3 | all | The target branch name of the pull request if [the pipelines are for external pull requests](../ci_cd_for_external_repos/index.md#pipelines-for-external-pull-requests). Available only if `only: [external_pull_requests]` or [`rules`](../yaml/README.md#rules) syntax is used and the pull request is open. | | `CI_EXTERNAL_PULL_REQUEST_TARGET_BRANCH_SHA` | 12.3 | all | The HEAD SHA of the target branch of the pull request if [the pipelines are for external pull requests](../ci_cd_for_external_repos/index.md#pipelines-for-external-pull-requests). Available only if `only: [external_pull_requests]` or [`rules`](../yaml/README.md#rules) syntax is used and the pull request is open. | -| `CI_JOB_ID` | 9.0 | all | The unique id of the current job that GitLab CI/CD uses internally | +| `CI_JOB_ID` | 9.0 | all | The unique ID of the current job that GitLab CI/CD uses internally | | `CI_JOB_IMAGE` | 12.9 | 12.9 | The name of the image running the CI job | | `CI_JOB_MANUAL` | 8.12 | all | The flag to indicate that job was manually started | | `CI_JOB_NAME` | 9.0 | 0.5 | The name of the job as defined in `.gitlab-ci.yml` | @@ -64,6 +64,7 @@ future GitLab releases.** | `CI_JOB_TOKEN` | 9.0 | 1.2 | Token used for authenticating with the [GitLab Container Registry](../../user/packages/container_registry/index.md) and downloading [dependent repositories](../../user/project/new_ci_build_permissions_model.md#dependent-repositories) | | `CI_JOB_JWT` | 12.10 | all | RS256 JSON web token that can be used for authenticating with third party systems that support JWT authentication, for example [HashiCorp's Vault](../examples/authenticating-with-hashicorp-vault). | | `CI_JOB_URL` | 11.1 | 0.5 | Job details URL | +| `CI_KUBERNETES_ACTIVE` | 13.0 | all | Included with the value `true` only if the pipeline has a Kubernetes cluster available for deployments. Not included if no cluster is availble. Can be used as an alternative to [`only:kubernetes`/`except:kubernetes`](../yaml/README.md#onlykubernetesexceptkubernetes) with [`rules:if`](../yaml/README.md#rulesif) | | `CI_MERGE_REQUEST_ASSIGNEES` | 11.9 | all | Comma-separated list of username(s) of assignee(s) for the merge request if [the pipelines are for merge requests](../merge_request_pipelines/index.md). Available only if `only: [merge_requests]` or [`rules`](../yaml/README.md#rules) syntax is used and the merge request is created. | | `CI_MERGE_REQUEST_CHANGED_PAGE_PATHS` | 12.9 | all | Comma-separated list of paths of changed pages in a deployed [Review App](../review_apps/index.md) for a [Merge Request](../merge_request_pipelines/index.md). A [Route Map](../review_apps/index.md#route-maps) must be configured. | | `CI_MERGE_REQUEST_CHANGED_PAGE_URLS` | 12.9 | all | Comma-separated list of URLs of changed pages in a deployed [Review App](../review_apps/index.md) for a [Merge Request](../merge_request_pipelines/index.md). A [Route Map](../review_apps/index.md#route-maps) must be configured. | @@ -88,13 +89,13 @@ future GitLab releases.** | `CI_NODE_TOTAL` | 11.5 | all | Total number of instances of this job running in parallel. If the job is not parallelized, this variable is set to `1`. | | `CI_PAGES_DOMAIN` | 11.8 | all | The configured domain that hosts GitLab Pages. | | `CI_PAGES_URL` | 11.8 | all | URL to GitLab Pages-built pages. Always belongs to a subdomain of `CI_PAGES_DOMAIN`. | -| `CI_PIPELINE_ID` | 8.10 | all | The unique id of the current pipeline that GitLab CI/CD uses internally | -| `CI_PIPELINE_IID` | 11.0 | all | The unique id of the current pipeline scoped to project | +| `CI_PIPELINE_ID` | 8.10 | all | The unique ID of the current pipeline that GitLab CI/CD uses internally | +| `CI_PIPELINE_IID` | 11.0 | all | The unique ID of the current pipeline scoped to project | | `CI_PIPELINE_SOURCE` | 10.0 | all | Indicates how the pipeline was triggered. Possible options are: `push`, `web`, `trigger`, `schedule`, `api`, `pipeline`, `parent_pipeline`, `external`, `chat`, `merge_request_event`, and `external_pull_request_event`. For pipelines created before GitLab 9.5, this will show as `unknown` | | `CI_PIPELINE_TRIGGERED` | all | all | The flag to indicate that job was [triggered](../triggers/README.md) | | `CI_PIPELINE_URL` | 11.1 | 0.5 | Pipeline details URL | | `CI_PROJECT_DIR` | all | all | The full path where the repository is cloned and where the job is run. If the GitLab Runner `builds_dir` parameter is set, this variable is set relative to the value of `builds_dir`. For more information, see [Advanced configuration](https://docs.gitlab.com/runner/configuration/advanced-configuration.html#the-runners-section) for GitLab Runner. | -| `CI_PROJECT_ID` | all | all | The unique id of the current project that GitLab CI/CD uses internally | +| `CI_PROJECT_ID` | all | all | The unique ID of the current project that GitLab CI/CD uses internally | | `CI_PROJECT_NAME` | 8.10 | 0.5 | The name of the directory for the project that is currently being built. For example, if the project URL is `gitlab.example.com/group-name/project-1`, the `CI_PROJECT_NAME` would be `project-1`. | | `CI_PROJECT_NAMESPACE` | 8.10 | 0.5 | The project namespace (username or groupname) that is currently being built | | `CI_PROJECT_PATH` | 8.10 | 0.5 | The namespace with project name | @@ -110,7 +111,7 @@ future GitLab releases.** | `CI_REPOSITORY_URL` | 9.0 | all | The URL to clone the Git repository | | `CI_RUNNER_DESCRIPTION` | 8.10 | 0.5 | The description of the runner as saved in GitLab | | `CI_RUNNER_EXECUTABLE_ARCH` | all | 10.6 | The OS/architecture of the GitLab Runner executable (note that this is not necessarily the same as the environment of the executor) | -| `CI_RUNNER_ID` | 8.10 | 0.5 | The unique id of runner being used | +| `CI_RUNNER_ID` | 8.10 | 0.5 | The unique ID of runner being used | | `CI_RUNNER_REVISION` | all | 10.6 | GitLab Runner revision that is executing the current job | | `CI_RUNNER_SHORT_TOKEN` | all | 12.3 | First eight characters of GitLab Runner's token used to authenticate new job requests. Used as Runner's unique ID | | `CI_RUNNER_TAGS` | 8.10 | 0.5 | The defined runner tags | @@ -131,7 +132,7 @@ future GitLab releases.** | `GITLAB_CI` | all | all | Mark that job is executed in GitLab CI/CD environment | | `GITLAB_FEATURES` | 10.6 | all | The comma separated list of licensed features available for your instance and plan | | `GITLAB_USER_EMAIL` | 8.12 | all | The email of the user who started the job | -| `GITLAB_USER_ID` | 8.12 | all | The id of the user who started the job | +| `GITLAB_USER_ID` | 8.12 | all | The ID of the user who started the job | | `GITLAB_USER_LOGIN` | 10.0 | all | The login username of the user who started the job | | `GITLAB_USER_NAME` | 10.0 | all | The real name of the user who started the job | | `RESTORE_CACHE_ATTEMPTS` | 8.15 | 1.9 | Number of attempts to restore the cache running a job | diff --git a/doc/ci/yaml/README.md b/doc/ci/yaml/README.md index 31459735101..c40580cbfb7 100644 --- a/doc/ci/yaml/README.md +++ b/doc/ci/yaml/README.md @@ -65,7 +65,7 @@ project namespace. For example, `https://gitlab.example.com/gitlab-org/project-1 ### Unavailable names for jobs Each job must have a unique name, but there are a few **reserved `keywords` that -cannot be used as job names**: +can't be used as job names**: - `image` - `services` @@ -90,41 +90,44 @@ A job is defined as a list of parameters that define the job's behavior. The following table lists available parameters for jobs: -| Keyword | Description | -|:---------------------------------------------------|:----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| -| [`script`](#script) | Shell script which is executed by Runner. | -| [`image`](#image) | Use docker images. Also available: `image:name` and `image:entrypoint`. | -| [`services`](#services) | Use docker services images. Also available: `services:name`, `services:alias`, `services:entrypoint`, and `services:command`. | -| [`before_script`](#before_script-and-after_script) | Override a set of commands that are executed before job. | -| [`after_script`](#before_script-and-after_script) | Override a set of commands that are executed after job. | -| [`stages`](#stages) | Define stages in a pipeline. | -| [`stage`](#stage) | Defines a job stage (default: `test`). | -| [`only`](#onlyexcept-basic) | Limit when jobs are created. Also available: [`only:refs`, `only:kubernetes`, `only:variables`, and `only:changes`](#onlyexcept-advanced). | -| [`except`](#onlyexcept-basic) | Limit when jobs are not created. Also available: [`except:refs`, `except:kubernetes`, `except:variables`, and `except:changes`](#onlyexcept-advanced). | -| [`rules`](#rules) | List of conditions to evaluate and determine selected attributes of a job, and whether or not it is created. May not be used alongside `only`/`except`. | -| [`tags`](#tags) | List of tags which are used to select Runner. | -| [`allow_failure`](#allow_failure) | Allow job to fail. Failed job doesn't contribute to commit status. | -| [`when`](#when) | When to run job. Also available: `when:manual` and `when:delayed`. | -| [`environment`](#environment) | Name of an environment to which the job deploys. Also available: `environment:name`, `environment:url`, `environment:on_stop`, `environment:auto_stop_in` and `environment:action`. | -| [`cache`](#cache) | List of files that should be cached between subsequent runs. Also available: `cache:paths`, `cache:key`, `cache:untracked`, and `cache:policy`. | -| [`artifacts`](#artifacts) | List of files and directories to attach to a job on success. Also available: `artifacts:paths`, `artifacts:expose_as`, `artifacts:name`, `artifacts:untracked`, `artifacts:when`, `artifacts:expire_in`, `artifacts:reports`, `artifacts:reports:junit`, and `artifacts:reports:cobertura`.<br><br>In GitLab [Enterprise Edition](https://about.gitlab.com/pricing/), these are available: `artifacts:reports:codequality`, `artifacts:reports:sast`, `artifacts:reports:dependency_scanning`, `artifacts:reports:container_scanning`, `artifacts:reports:dast`, `artifacts:reports:license_management`, `artifacts:reports:performance` and `artifacts:reports:metrics`. | -| [`dependencies`](#dependencies) | Restrict which artifacts are passed to a specific job by providing a list of jobs to fetch artifacts from. | -| [`coverage`](#coverage) | Code coverage settings for a given job. | -| [`retry`](#retry) | When and how many times a job can be auto-retried in case of a failure. | -| [`timeout`](#timeout) | Define a custom job-level timeout that takes precedence over the project-wide setting. | -| [`parallel`](#parallel) | How many instances of a job should be run in parallel. | -| [`trigger`](#trigger) | Defines a downstream pipeline trigger. | -| [`include`](#include) | Allows this job to include external YAML files. Also available: `include:local`, `include:file`, `include:template`, and `include:remote`. | -| [`extends`](#extends) | Configuration entries that this job is going to inherit from. | -| [`pages`](#pages) | Upload the result of a job to use with GitLab Pages. | -| [`variables`](#variables) | Define job variables on a job level. | -| [`interruptible`](#interruptible) | Defines if a job can be canceled when made redundant by a newer run. | -| [`resource_group`](#resource_group) | Limit job concurrency. | +| Keyword | Description | +|:---------------------------------------------------|:------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| +| [`script`](#script) | Shell script which is executed by Runner. | +| [`image`](#image) | Use docker images. Also available: `image:name` and `image:entrypoint`. | +| [`services`](#services) | Use docker services images. Also available: `services:name`, `services:alias`, `services:entrypoint`, and `services:command`. | +| [`before_script`](#before_script-and-after_script) | Override a set of commands that are executed before job. | +| [`after_script`](#before_script-and-after_script) | Override a set of commands that are executed after job. | +| [`stage`](#stage) | Defines a job stage (default: `test`). | +| [`only`](#onlyexcept-basic) | Limit when jobs are created. Also available: [`only:refs`, `only:kubernetes`, `only:variables`, and `only:changes`](#onlyexcept-advanced). | +| [`except`](#onlyexcept-basic) | Limit when jobs are not created. Also available: [`except:refs`, `except:kubernetes`, `except:variables`, and `except:changes`](#onlyexcept-advanced). | +| [`rules`](#rules) | List of conditions to evaluate and determine selected attributes of a job, and whether or not it's created. May not be used alongside `only`/`except`. | +| [`tags`](#tags) | List of tags which are used to select Runner. | +| [`allow_failure`](#allow_failure) | Allow job to fail. Failed job does not contribute to commit status. | +| [`when`](#when) | When to run job. Also available: `when:manual` and `when:delayed`. | +| [`environment`](#environment) | Name of an environment to which the job deploys. Also available: `environment:name`, `environment:url`, `environment:on_stop`, `environment:auto_stop_in` and `environment:action`. | +| [`cache`](#cache) | List of files that should be cached between subsequent runs. Also available: `cache:paths`, `cache:key`, `cache:untracked`, and `cache:policy`. | +| [`artifacts`](#artifacts) | List of files and directories to attach to a job on success. Also available: `artifacts:paths`, `artifacts:expose_as`, `artifacts:name`, `artifacts:untracked`, `artifacts:when`, `artifacts:expire_in`, `artifacts:reports`, `artifacts:reports:junit`, `artifacts:reports:cobertura`, and `artifacts:reports:terraform`.<br><br>In GitLab [Enterprise Edition](https://about.gitlab.com/pricing/), these are available: `artifacts:reports:codequality`, `artifacts:reports:sast`, `artifacts:reports:dependency_scanning`, `artifacts:reports:container_scanning`, `artifacts:reports:dast`, `artifacts:reports:license_scanning`, `artifacts:reports:license_management` (removed in GitLab 13.0),`artifacts:reports:performance` and `artifacts:reports:metrics`. | +| [`dependencies`](#dependencies) | Restrict which artifacts are passed to a specific job by providing a list of jobs to fetch artifacts from. | +| [`coverage`](#coverage) | Code coverage settings for a given job. | +| [`retry`](#retry) | When and how many times a job can be auto-retried in case of a failure. | +| [`timeout`](#timeout) | Define a custom job-level timeout that takes precedence over the project-wide setting. | +| [`parallel`](#parallel) | How many instances of a job should be run in parallel. | +| [`trigger`](#trigger) | Defines a downstream pipeline trigger. | +| [`include`](#include) | Allows this job to include external YAML files. Also available: `include:local`, `include:file`, `include:template`, and `include:remote`. | +| [`extends`](#extends) | Configuration entries that this job is going to inherit from. | +| [`pages`](#pages) | Upload the result of a job to use with GitLab Pages. | +| [`variables`](#variables) | Define job variables on a job level. | +| [`interruptible`](#interruptible) | Defines if a job can be canceled when made redundant by a newer run. | +| [`resource_group`](#resource_group) | Limit job concurrency. | NOTE: **Note:** Parameters `types` and `type` are [deprecated](#deprecated-parameters). -## Setting default parameters +## Global parameters + +Some parameters must be defined at a global level, affecting all jobs in the pipeline. + +### Global defaults Some parameters can be set globally as the default for all jobs using the `default:` keyword. Default parameters can then be overridden by job-specific @@ -158,7 +161,7 @@ rspec 2.6: script: bundle exec rspec ``` -### `inherit` +#### `inherit` > [Introduced](https://gitlab.com/gitlab-org/gitlab/issues/207484) in GitLab 12.9. @@ -198,13 +201,13 @@ In the example below: - **will** inherit: Nothing. - `rspec`: - **will** inherit: the default `image` and the `WEBHOOK_URL` variable. - - **will not** inherit: the default `before_script` and the `DOMAIN` variable. + - will **not** inherit: the default `before_script` and the `DOMAIN` variable. - `capybara`: - **will** inherit: the default `before_script` and `image`. - - **will not** inherit: the `DOMAIN` and `WEBHOOK_URL` variables. + - will **not** inherit: the `DOMAIN` and `WEBHOOK_URL` variables. - `karma`: - **will** inherit: the default `image` and `before_script`, and the `DOMAIN` variable. - - **will not** inherit: `WEBHOOK_URL` variable. + - will **not** inherit: `WEBHOOK_URL` variable. ```yaml default: @@ -240,49 +243,274 @@ karma: script: karma ``` -## Parameter details +### `stages` -The following are detailed explanations for parameters used to configure CI/CD pipelines. +`stages` is used to define stages that can be used by jobs and is defined +globally. -### `script` +The specification of `stages` allows for having flexible multi stage pipelines. +The ordering of elements in `stages` defines the ordering of jobs' execution: -`script` is the only required keyword that a job needs. It's a shell script -which is executed by the Runner. For example: +1. Jobs of the same stage are run in parallel. +1. Jobs of the next stage are run after the jobs from the previous stage + complete successfully. + +Let's consider the following example, which defines 3 stages: ```yaml -job: - script: "bundle exec rspec" +stages: + - build + - test + - deploy ``` -[YAML anchors for scripts](#yaml-anchors-for-script) are available. +1. First, all jobs of `build` are executed in parallel. +1. If all jobs of `build` succeed, the `test` jobs are executed in parallel. +1. If all jobs of `test` succeed, the `deploy` jobs are executed in parallel. +1. If all jobs of `deploy` succeed, the commit is marked as `passed`. +1. If any of the previous jobs fails, the commit is marked as `failed` and no + jobs of further stage are executed. -This parameter can also contain several commands using an array: +There are also two edge cases worth mentioning: + +1. If no `stages` are defined in `.gitlab-ci.yml`, then the `build`, + `test` and `deploy` are allowed to be used as job's stage by default. +1. If a job does not specify a `stage`, the job is assigned the `test` stage. + +### `workflow:rules` + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/issues/29654) in GitLab 12.5 + +The top-level `workflow:` key applies to the entirety of a pipeline, and will +determine whether or not a pipeline is created. It currently accepts a single +`rules:` key that operates similarly to [`rules:` defined within jobs](#rules), +enabling dynamic configuration of the pipeline. + +#### `workflow:rules` templates + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/217732) in GitLab 13.0. + +We provide pre-made templates for use with your pipelines that set up `workflow: rules` +for common scenarios. Usage of these will make things easier and prevent duplicate pipelines from running. + +The [`Branch-Pipelines` template](https://gitlab.com/gitlab-org/gitlab/-/tree/master/lib/gitlab/ci/templates/Workflows/Branch-Pipelines.gitlab-ci.yml) +makes your pipelines run for branches and tags. + +Branch pipeline status will be displayed within merge requests that use that branch +as a source, but this pipeline type does not support any features offered by +[Merge Request Pipelines](../merge_request_pipelines/) like +[Pipelines for Merge Results](../merge_request_pipelines/#pipelines-for-merged-results-premium) +or [Merge Trains](../merge_request_pipelines/pipelines_for_merged_results/merge_trains/). +Use this template if you are intentionally avoiding those features. + +It is [included](#include) as follows: ```yaml -job: - script: - - uname -a - - bundle exec rspec +include: + - template: 'Workflows/Branch-Pipelines.gitlab-ci.yml' +``` + +The [`MergeRequest-Pipelines` include](https://gitlab.com/gitlab-org/gitlab/-/tree/master/lib/gitlab/ci/templates/Workflows/MergeRequest-Pipelines.gitlab-ci.yml) sets your pipelines to run for the default branch (usually `master`), tags, and +The [`MergeRequest-Pipelines` template](https://gitlab.com/gitlab-org/gitlab/-/tree/master/lib/gitlab/ci/templates/Workflows/MergeRequest-Pipelines.gitlab-ci.yml) +makes your pipelines run for the default branch (usually `master`), tags, and +all types of merge request pipelines. Use this template if you use any of the +the [Pipelines for Merge Requests features](../merge_request_pipelines/), as mentioned +above. + +It is [included](#include) as follows: + +```yaml +include: + - template: 'Workflows/MergeRequest-Pipelines.gitlab-ci.yml' +``` + +If you prefer to define your own rules, the configuration options currently available are:​ + +- [`if`](#rulesif): Define a rule. +- [`when`](#when): May be set to `always` or `never` only. If not provided, the default value is `always`​. + +The list of `if` rules is evaluated until a single one is matched. If none +match, the last `when` will be used: + +```yaml +workflow: + rules: + - if: $CI_COMMIT_REF_NAME =~ /-wip$/ + when: never + - if: $CI_COMMIT_TAG + when: never + - when: always ``` +### `include` + +> - Introduced in [GitLab Premium](https://about.gitlab.com/pricing/) 10.5. +> - Available for Starter, Premium and Ultimate since 10.6. +> - [Moved](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/42861) to GitLab Core in 11.4. + +Using the `include` keyword allows the inclusion of external YAML files. This helps +to break down the CI/CD configuration into multiple files and increases readability for long configuration files. +It's also possible to have template files stored in a central repository and projects include their +configuration files. This helps avoid duplicated configuration, for example, global default variables for all projects. + +`include` requires the external YAML file to have the extensions `.yml` or `.yaml`, +otherwise the external file won't be included. + +`include` supports the following inclusion methods: + +| Method | Description | +|:--------------------------------|:------------------------------------------------------------------| +| [`local`](#includelocal) | Include a file from the local project repository. | +| [`file`](#includefile) | Include a file from a different project repository. | +| [`remote`](#includeremote) | Include a file from a remote URL. Must be publicly accessible. | +| [`template`](#includetemplate) | Include templates which are provided by GitLab. | + NOTE: **Note:** -Sometimes, `script` commands will need to be wrapped in single or double quotes. -For example, commands that contain a colon (`:`) need to be wrapped in quotes so -that the YAML parser knows to interpret the whole thing as a string rather than -a "key: value" pair. Be careful when using special characters: -`:`, `{`, `}`, `[`, `]`, `,`, `&`, `*`, `#`, `?`, `|`, `-`, `<`, `>`, `=`, `!`, `%`, `@`, `` ` ``. +`.gitlab-ci.yml` configuration included by all methods is evaluated at pipeline creation. +The configuration is a snapshot in time and persisted in the database. Any changes to +referenced `.gitlab-ci.yml` configuration won't be reflected in GitLab until the next pipeline is created. -If any of the script commands return an exit code different from zero, the job -will fail and further commands will not be executed. This behavior can be avoided by -storing the exit code in a variable: +The files defined in `include` are: + +- Deep merged with those in `.gitlab-ci.yml`. +- Always evaluated first and merged with the content of `.gitlab-ci.yml`, + regardless of the position of the `include` keyword. + +TIP: **Tip:** +Use merging to customize and override included CI/CD configurations with local +definitions. + +NOTE: **Note:** +Using YAML aliases across different YAML files sourced by `include` is not +supported. You must only refer to aliases in the same file. Instead +of using YAML anchors, you can use the [`extends` keyword](#extends). + +#### `include:local` + +`include:local` includes a file from the same repository as `.gitlab-ci.yml`. +It's referenced using full paths relative to the root directory (`/`). + +You can only use files that are currently tracked by Git on the same branch +your configuration file is on. In other words, when using a `include:local`, make +sure that both `.gitlab-ci.yml` and the local file are on the same branch. + +All [nested includes](#nested-includes) will be executed in the scope of the same project, +so it's possible to use local, project, remote, or template includes. + +NOTE: **Note:** +Including local files through Git submodules paths is not supported. + +Example: ```yaml -job: - script: - - false || exit_code=$? - - if [ $exit_code -ne 0 ]; then echo "Previous command failed"; fi; +include: + - local: '/templates/.gitlab-ci-template.yml' +``` + +TIP: **Tip:** +Local includes can be used as a replacement for symbolic links which are not followed. + +This can be defined as a short local include: + +```yaml +include: '.gitlab-ci-production.yml' +``` + +#### `include:file` + +> [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/issues/53903) in GitLab 11.7. + +To include files from another private project under the same GitLab instance, +use `include:file`. This file is referenced using full paths relative to the +root directory (`/`). For example: + +```yaml +include: + - project: 'my-group/my-project' + file: '/templates/.gitlab-ci-template.yml' ``` +You can also specify `ref`, with the default being the `HEAD` of the project: + +```yaml +include: + - project: 'my-group/my-project' + ref: master + file: '/templates/.gitlab-ci-template.yml' + + - project: 'my-group/my-project' + ref: v1.0.0 + file: '/templates/.gitlab-ci-template.yml' + + - project: 'my-group/my-project' + ref: 787123b47f14b552955ca2786bc9542ae66fee5b # Git SHA + file: '/templates/.gitlab-ci-template.yml' +``` + +All [nested includes](#nested-includes) will be executed in the scope of the target project, +so it's possible to use local (relative to target project), project, remote +or template includes. + +#### `include:remote` + +`include:remote` can be used to include a file from a different location, +using HTTP/HTTPS, referenced by using the full URL. The remote file must be +publicly accessible through a simple GET request as authentication schemas +in the remote URL are not supported. For example: + +```yaml +include: + - remote: 'https://gitlab.com/awesome-project/raw/master/.gitlab-ci-template.yml' +``` + +All [nested includes](#nested-includes) will be executed without context as public user, so only another remote +or public project, or template, is allowed. + +#### `include:template` + +> [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/issues/53445) in GitLab 11.7. + +`include:template` can be used to include `.gitlab-ci.yml` templates that are +[shipped with GitLab](https://gitlab.com/gitlab-org/gitlab/tree/master/lib/gitlab/ci/templates). + +For example: + +```yaml +# File sourced from GitLab's template collection +include: + - template: Auto-DevOps.gitlab-ci.yml +``` + +Multiple `include:template` files: + +```yaml +include: + - template: Android-Fastlane.gitlab-ci.yml + - template: Auto-DevOps.gitlab-ci.yml +``` + +All [nested includes](#nested-includes) will be executed only with the permission of the user, +so it's possible to use project, remote or template includes. + +#### Nested includes + +> [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/issues/56836) in GitLab 11.9. + +Nested includes allow you to compose a set of includes. + +A total of 100 includes is allowed, but duplicate includes are considered a configuration error. + +Since [GitLab 12.4](https://gitlab.com/gitlab-org/gitlab/issues/28212), the time limit +for resolving all files is 30 seconds. + +#### Additional `includes` examples + +There is a list of [additional `includes` examples](includes.md) available. + +## Parameter details + +The following are detailed explanations for parameters used to configure CI/CD pipelines. + ### `image` Used to specify [a Docker image](../docker/using_docker_images.md#what-is-an-image) to use for the job. @@ -304,7 +532,7 @@ An [extended docker configuration option](../docker/using_docker_images.md#exten For more information, see [Available settings for `image`](../docker/using_docker_images.md#available-settings-for-image). -### `services` +#### `services` Used to specify a [service Docker image](../docker/using_docker_images.md#what-is-a-service), linked to a base image specified in [`image`](#image). @@ -314,31 +542,70 @@ For: - Detailed usage information, refer to [Docker integration](../docker/README.md) documentation. - For example services, see [GitLab CI/CD Services](../services/README.md). -#### `services:name` +##### `services:name` An [extended docker configuration option](../docker/using_docker_images.md#extended-docker-configuration-options). For more information, see [Available settings for `services`](../docker/using_docker_images.md#available-settings-for-services). -#### `services:alias` +##### `services:alias` An [extended docker configuration option](../docker/using_docker_images.md#extended-docker-configuration-options). For more information, see [Available settings for `services`](../docker/using_docker_images.md#available-settings-for-services). -#### `services:entrypoint` +##### `services:entrypoint` An [extended docker configuration option](../docker/using_docker_images.md#extended-docker-configuration-options). For more information, see [Available settings for `services`](../docker/using_docker_images.md#available-settings-for-services). -#### `services:command` +##### `services:command` An [extended docker configuration option](../docker/using_docker_images.md#extended-docker-configuration-options). For more information, see [Available settings for `services`](../docker/using_docker_images.md#available-settings-for-services). -### `before_script` and `after_script` +### `script` + +`script` is the only required keyword that a job needs. It's a shell script +which is executed by the Runner. For example: + +```yaml +job: + script: "bundle exec rspec" +``` + +[YAML anchors for scripts](#yaml-anchors-for-script) are available. + +This parameter can also contain several commands using an array: + +```yaml +job: + script: + - uname -a + - bundle exec rspec +``` + +NOTE: **Note:** +Sometimes, `script` commands will need to be wrapped in single or double quotes. +For example, commands that contain a colon (`:`) need to be wrapped in quotes so +that the YAML parser knows to interpret the whole thing as a string rather than +a "key: value" pair. Be careful when using special characters: +`:`, `{`, `}`, `[`, `]`, `,`, `&`, `*`, `#`, `?`, `|`, `-`, `<`, `>`, `=`, `!`, `%`, `@`, `` ` ``. + +If any of the script commands return an exit code different from zero, the job +will fail and further commands won't be executed. This behavior can be avoided by +storing the exit code in a variable: + +```yaml +job: + script: + - false || exit_code=$? + - if [ $exit_code -ne 0 ]; then echo "Previous command failed"; fi; +``` + +#### `before_script` and `after_script` > Introduced in GitLab 8.7 and requires GitLab Runner v1.2. @@ -362,7 +629,7 @@ Scripts specified in `after_script` are executed in a new shell, separate from a software installed by a `before_script` or `script` script. - Have a separate timeout, which is hard coded to 5 minutes. See [related issue](https://gitlab.com/gitlab-org/gitlab-runner/issues/2716) for details. -- Do not affect the job's exit code. If the `script` section succeeds and the +- Don't affect the job's exit code. If the `script` section succeeds and the `after_script` times out or fails, the job will exit with code `0` (`Job Succeeded`). It's possible to overwrite a globally defined `before_script` or `after_script` @@ -384,39 +651,53 @@ job: [YAML anchors for `before_script` and `after_script`](#yaml-anchors-for-before_script-and-after_script) are available. -### `stages` - -`stages` is used to define stages that can be used by jobs and is defined -globally. - -The specification of `stages` allows for having flexible multi stage pipelines. -The ordering of elements in `stages` defines the ordering of jobs' execution: - -1. Jobs of the same stage are run in parallel. -1. Jobs of the next stage are run after the jobs from the previous stage - complete successfully. +### `stage` -Let's consider the following example, which defines 3 stages: +`stage` is defined per-job and relies on [`stages`](#stages) which is defined +globally. It allows to group jobs into different stages, and jobs of the same +`stage` are executed in parallel (subject to [certain conditions](#using-your-own-runners)). For example: ```yaml stages: - build - test - deploy + +job 0: + stage: .pre + script: make something useful before build stage + +job 1: + stage: build + script: make build dependencies + +job 2: + stage: build + script: make build artifacts + +job 3: + stage: test + script: make test + +job 4: + stage: deploy + script: make deploy + +job 5: + stage: .post + script: make something useful at the end of pipeline ``` -1. First, all jobs of `build` are executed in parallel. -1. If all jobs of `build` succeed, the `test` jobs are executed in parallel. -1. If all jobs of `test` succeed, the `deploy` jobs are executed in parallel. -1. If all jobs of `deploy` succeed, the commit is marked as `passed`. -1. If any of the previous jobs fails, the commit is marked as `failed` and no - jobs of further stage are executed. +#### Using your own Runners -There are also two edge cases worth mentioning: +When using your own Runners, GitLab Runner runs only one job at a time by default (see the +`concurrent` flag in [Runner global settings](https://docs.gitlab.com/runner/configuration/advanced-configuration.html#the-global-section) +for more information). -1. If no `stages` are defined in `.gitlab-ci.yml`, then the `build`, - `test` and `deploy` are allowed to be used as job's stage by default. -1. If a job doesn't specify a `stage`, the job is assigned the `test` stage. +Jobs will run on your own Runners in parallel only if: + +- Run on different Runners. +- The Runner's `concurrent` setting has been changed. #### `.pre` and `.post` @@ -429,12 +710,12 @@ The following stages are available to every pipeline: User-defined stages are executed after `.pre` and before `.post`. -The order of `.pre` and `.post` cannot be changed, even if defined out of order in `.gitlab-ci.yml`. +The order of `.pre` and `.post` can't be changed, even if defined out of order in `.gitlab-ci.yml`. For example, the following are equivalent configuration: - Configured in order: - ```yml + ```yaml stages: - .pre - a @@ -444,7 +725,7 @@ For example, the following are equivalent configuration: - Configured out of order: - ```yml + ```yaml stages: - a - .pre @@ -454,69 +735,390 @@ For example, the following are equivalent configuration: - Not explicitly configured: - ```yml + ```yaml stages: - a - b ``` NOTE: **Note:** -A pipeline will not be created if it only contains jobs in `.pre` or `.post` stages. +A pipeline won't be created if it only contains jobs in `.pre` or `.post` stages. -### `stage` +### `extends` -`stage` is defined per-job and relies on [`stages`](#stages) which is defined -globally. It allows to group jobs into different stages, and jobs of the same -`stage` are executed in parallel (subject to [certain conditions](#using-your-own-runners)). For example: +> Introduced in GitLab 11.3. + +`extends` defines entry names that a job that uses `extends` is going to +inherit from. + +It's an alternative to using [YAML anchors](#anchors) and is a little +more flexible and readable: ```yaml -stages: - - build - - test - - deploy +.tests: + script: rake test + stage: test + only: + refs: + - branches -job 0: - stage: .pre - script: make something useful before build stage +rspec: + extends: .tests + script: rake rspec + only: + variables: + - $RSPEC +``` -job 1: - stage: build - script: make build dependencies +In the example above, the `rspec` job inherits from the `.tests` template job. +GitLab will perform a reverse deep merge based on the keys. GitLab will: -job 2: - stage: build - script: make build artifacts +- Merge the `rspec` contents into `.tests` recursively. +- Not merge the values of the keys. -job 3: +This results in the following `rspec` job: + +```yaml +rspec: + script: rake rspec stage: test - script: make test + only: + refs: + - branches + variables: + - $RSPEC +``` -job 4: - stage: deploy - script: make deploy +NOTE: **Note:** +Note that `script: rake test` has been overwritten by `script: rake rspec`. -job 5: - stage: .post - script: make something useful at the end of pipeline +If you do want to include the `rake test`, see [`before_script` and `after_script`](#before_script-and-after_script). + +`.tests` in this example is a [hidden job](#hide-jobs), but it's +possible to inherit from regular jobs as well. + +`extends` supports multi-level inheritance, however it's not recommended to +use more than three levels. The maximum nesting level that is supported is 10. +The following example has two levels of inheritance: + +```yaml +.tests: + only: + - pushes + +.rspec: + extends: .tests + script: rake rspec + +rspec 1: + variables: + RSPEC_SUITE: '1' + extends: .rspec + +rspec 2: + variables: + RSPEC_SUITE: '2' + extends: .rspec + +spinach: + extends: .tests + script: rake spinach ``` -#### Using your own Runners +In GitLab 12.0 and later, it's also possible to use multiple parents for +`extends`. The algorithm used for merge is "closest scope wins", so +keys from the last member will always shadow anything defined on other +levels. For example: -When using your own Runners, GitLab Runner runs only one job at a time by default (see the -`concurrent` flag in [Runner global settings](https://docs.gitlab.com/runner/configuration/advanced-configuration.html#the-global-section) -for more information). +```yaml +.only-important: + only: + - master + - stable + tags: + - production -Jobs will run on your own Runners in parallel only if: +.in-docker: + tags: + - docker + image: alpine -- Run on different Runners. -- The Runner's `concurrent` setting has been changed. +rspec: + extends: + - .only-important + - .in-docker + script: + - rake rspec +``` + +This results in the following `rspec` job: + +```yaml +rspec: + only: + - master + - stable + tags: + - docker + image: alpine + script: + - rake rspec +``` + +#### Using `extends` and `include` together + +`extends` works across configuration files combined with `include`. + +For example, if you have a local `included.yml` file: + +```yaml +.template: + script: + - echo Hello! +``` + +Then, in `.gitlab-ci.yml` you can use it like this: + +```yaml +include: included.yml + +useTemplate: + image: alpine + extends: .template +``` + +This will run a job called `useTemplate` that runs `echo Hello!` as defined in +the `.template` job, and uses the `alpine` Docker image as defined in the local job. + +### `rules` + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/27863) in GitLab 12.3. + +`rules` allows for a list of individual rule objects to be evaluated +*in order*, until one matches and dynamically provides attributes to the job. + +CAUTION: **Caution:** +`rules` can't be used in combination with `only/except` as it is a replacement for that functionality. If you attempt to do this, the linter will return a +`key may not be used with rules` error. + +#### Key details when using `rules` + +A very important difference between `rules` and `only/except`, is that jobs defined +with `rules` trigger merge request pipelines by default, but `only/except` jobs do not. +This may be surprising if migrating from `only` and `except`, so new users of `rules` +can use one of the [`workflow: rules` templates](#workflowrules-templates) to get started. +This will ensure that the behavior is more stable as you start adding additional `rules` +blocks, and will avoid issues like creating a duplicate, merge request (detached) pipeline. + +We don't recomment mixing `only/except` jobs with `rules` jobs in the same pipeline. +It may not cause YAML errors, but debugging the exact execution behavior can be complex +due to the different default behaviors of `only/except` and `rules`. + +### Rules clauses + +Available rule clauses include: + +- [`if`](#rulesif) (similar to [`only:variables`](#onlyvariablesexceptvariables)) +- [`changes`](#ruleschanges) (same as [`only:changes`](#onlychangesexceptchanges)) +- [`exists`](#rulesexists) + +For example, using `if`. This configuration specifies that `job` should be built +and run for every pipeline on merge requests targeting `master`, regardless of +the status of other builds: + +```yaml +job: + script: "echo Hello, Rules!" + rules: + - if: '$CI_MERGE_REQUEST_TARGET_BRANCH_NAME == "master"' + when: always + - if: '$VAR =~ /pattern/' + when: manual + - when: on_success +``` + +In this example, if the first rule: + +- Matches, the job will be given the `when:always` attribute. +- Does not match, the second and third rules will be evaluated sequentially + until a match is found. That is, the job will be given either the: + - `when: manual` attribute if the second rule matches. **The stage won't complete until this manual job is triggered and completes successfully.** + - `when: on_success` attribute if the second rule does not match. The third + rule will always match when reached because it has no conditional clauses. + +#### `rules:if` + +`rules:if` differs slightly from `only:variables` by accepting only a single +expression string, rather than an array of them. Any set of expressions to be +evaluated should be conjoined into a single expression using `&&` or `||`, and use +the [variable matching syntax](../variables/README.md#syntax-of-environment-variable-expressions). + +For example: + +```yaml +job: + script: "echo Hello, Rules!" + rules: + - if: '$CI_MERGE_REQUEST_SOURCE_BRANCH_NAME =~ /^feature/ && $CI_MERGE_REQUEST_TARGET_BRANCH_NAME == "master"' # This rule will be evaluated + when: always + - if: '$CI_MERGE_REQUEST_SOURCE_BRANCH_NAME =~ /^feature/' # This rule will only be evaluated if the target branch is not "master" + when: manual + - if: '$CI_MERGE_REQUEST_SOURCE_BRANCH_NAME' # If neither of the first two match but the simple presence does, we set to "on_success" by default +``` + +If none of the provided rules match, the job will be set to `when:never`, and +not included in the pipeline. If `rules:when` is not included in the configuration +at all, the behavior defaults to `job:when`, which continues to default to +`on_success`. + +#### `rules:changes` + +`rules: changes` works exactly the same way as `only: changes` and `except: changes`, +accepting an array of paths. Similarly, it will always return true if there is no +Git push event. See [`only/except: changes`](#onlychangesexceptchanges) for more information. + +For example: + +```yaml +docker build: + script: docker build -t my-image:$CI_COMMIT_REF_SLUG . + rules: + - changes: # Will include the job and set to when:manual if any of the follow paths match a modified file. + - Dockerfile + when: manual + - if: '$VAR == "string value"' + when: manual # Will include the job and set to when:manual if the expression evaluates to true, after the `changes:` rule fails to match. + - when: on_success # If neither of the first rules match, set to on_success +``` + +In this example, a job either set to: + +- Run manually if `Dockerfile` has changed OR `$VAR == "string value"`. +- `when:on_success` by the last rule, where no earlier clauses evaluate to true. + +#### `rules:exists` + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/24021) in GitLab 12.4. + +`exists` accepts an array of paths and will match if any of these paths exist +as files in the repository. + +For example: + +```yaml +job: + script: docker build -t my-image:$CI_COMMIT_REF_SLUG . + rules: + - exists: + - Dockerfile +``` + +You can also use glob patterns to match multiple files in any directory within +the repository. + +For example: + +```yaml +job: + script: bundle exec rspec + rules: + - exists: + - spec/**.rb +``` + +NOTE: **Note:** +For performance reasons, using `exists` with patterns is limited to 10000 +checks. After the 10000th check, rules with patterned globs will always match. + +#### `rules:allow_failure` + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/issues/30235) in GitLab 12.8. + +You can use [`allow_failure: true`](#allow_failure) within `rules:` to allow a job to fail, or a manual job to +wait for action, without stopping the pipeline itself. All jobs using `rules:` default to `allow_failure: false` +if `allow_failure:` is not defined. + +```yaml +job: + script: "echo Hello, Rules!" + rules: + - if: '$CI_MERGE_REQUEST_TARGET_BRANCH_NAME == "master"' + when: manual + allow_failure: true +``` + +In this example, if the first rule matches, then the job will have `when: manual` and `allow_failure: true`. + +#### Complex rule clauses + +To conjoin `if`, `changes`, and `exists` clauses with an AND, use them in the +same rule. + +In the following example: + +- We run the job manually if `Dockerfile` or any file in `docker/scripts/` + has changed AND `$VAR == "string value"`. +- Otherwise, the job won't be included in the pipeline. + +```yaml +docker build: + script: docker build -t my-image:$CI_COMMIT_REF_SLUG . + rules: + - if: '$VAR == "string value"' + changes: # Will include the job and set to when:manual if any of the follow paths match a modified file. + - Dockerfile + - docker/scripts/* + when: manual + # - when: never would be redundant here, this is implied any time rules are listed. +``` + +The only clauses currently available are: + +- `if` +- `changes` +- `exists` + +Keywords such as `branches` or `refs` that are currently available for +`only`/`except` are not yet available in `rules` as they are being individually +considered for their usage and behavior in this context. Future keyword improvements +are being discussed in our [epic for improving `rules`](https://gitlab.com/groups/gitlab-org/-/epics/2783), +where anyone can add suggestions or requests. + +#### Permitted attributes + +The only job attributes currently set by `rules` are: + +- `when`. +- `start_in`, if `when` is set to `delayed`. +- `allow_failure`. + +A job will be included in a pipeline if `when` is evaluated to any value +except `never`. + +Delayed jobs require a `start_in` value, so rule objects do as well. For +example: + +```yaml +docker build: + script: docker build -t my-image:$CI_COMMIT_REF_SLUG . + rules: + - changes: # Will include the job and delay 3 hours when the Dockerfile has changed + - Dockerfile + when: delayed + start_in: '3 hours' + - when: on_success # Otherwise include the job and set to run normally +``` + +Additional job configuration may be added to rules in the future. If something +useful is not available, please +[open an issue](https://gitlab.com/gitlab-org/gitlab/issues). ### `only`/`except` (basic) NOTE: **Note:** -The [`rules`](#rules) syntax is now the preferred method of setting job policies. -`only` and `except` are [candidates for deprecation](https://gitlab.com/gitlab-org/gitlab/issues/27449), -and may be removed in the future. +The [`rules`](#rules) syntax is an improved, more powerful solution for defining +when jobs should run or not. Consider using `rules` instead of `only/except` to get +the most out of your pipelines. `only` and `except` are two parameters that set a job policy to limit when jobs are created: @@ -604,7 +1206,7 @@ The above example will run `job` for all branches on `gitlab-org/gitlab`, except `master` and those with names prefixed with `release/`. If a job does not have an `only` rule, `only: ['branches', 'tags']` is set by -default. If it doesn't have an `except` rule, it is empty. +default. If it does not have an `except` rule, it's empty. For example, @@ -643,7 +1245,7 @@ matching only a substring of the tag name or branch name. For example, `/^issue-.*$/` is equivalent to `/^issue-/`, while just `/issue/` would also match a branch called `severe-issues`. -### Supported `only`/`except` regexp syntax +#### Supported `only`/`except` regexp syntax CAUTION: **Warning:** This is a breaking change that was introduced with GitLab 11.9.4. @@ -667,7 +1269,7 @@ Feature.enable(:allow_unsafe_ruby_regexp) ### `only`/`except` (advanced) CAUTION: **Warning:** -This is an _alpha_ feature, and it is subject to change at any time without +This is an _alpha_ feature, and is subject to change at any time without prior notice! GitLab supports both simple and complex strategies, so it's possible to use an @@ -719,7 +1321,7 @@ This means the keys are treated as if joined by an OR. This relationship could b In the example below, the `test` job will **not** be created when **any** of the following are true: - The pipeline runs for the `master`. -- There are changes to the `README.md` file in the root directory of the repo. +- There are changes to the `README.md` file in the root directory of the repository. ```yaml test: @@ -813,7 +1415,7 @@ This means the `only:changes` policy is useful for pipelines where: If there is no Git push event, such as for pipelines with [sources other than the three above](../variables/predefined_variables.md#variables-reference), -`changes` cannot determine if a given file is new or old, and will always +`changes` can't determine if a given file is new or old, and will always return true. A basic example of using `only: changes`: @@ -840,10 +1442,10 @@ commits contains changes to any of the following: CAUTION: **Warning:** If using `only:changes` with [only allow merge requests to be merged if the pipeline succeeds](../../user/project/merge_requests/merge_when_pipeline_succeeds.md#only-allow-merge-requests-to-be-merged-if-the-pipeline-succeeds), -undesired behavior could result if you do not [also use `only:merge_requests`](#using-onlychanges-with-pipelines-for-merge-requests). +undesired behavior could result if you don't [also use `only:merge_requests`](#using-onlychanges-with-pipelines-for-merge-requests). You can also use glob patterns to match multiple files in either the root directory -of the repo, or in _any_ directory within the repo, but they must be wrapped +of the repository, or in _any_ directory within the repository, but they must be wrapped in double quotes or GitLab will fail to parse the `.gitlab-ci.yml`. For example: ```yaml @@ -856,7 +1458,7 @@ test: ``` The following example will skip the `build` job if a change is detected in any file -in the root directory of the repo with a `.md` extension: +in the root directory of the repository with a `.md` extension: ```yaml build: @@ -877,7 +1479,7 @@ There are some points to be aware of when ##### Using `only:changes` with pipelines for merge requests With [pipelines for merge requests](../merge_request_pipelines/index.md), -it is possible to define a job to be created based on files modified +it's possible to define a job to be created based on files modified in a merge request. In order to deduce the correct base SHA of the source branch, we recommend combining @@ -919,7 +1521,7 @@ docker build service one: In the example above, a pipeline could fail due to changes to a file in `service-one/**/*`. A later commit could then be pushed that does not include any changes to this file, -but includes changes to the `Dockerfile`, and this pipeline could pass because it is only +but includes changes to the `Dockerfile`, and this pipeline could pass because it's only testing the changes to the `Dockerfile`. GitLab checks the **most recent pipeline**, that **passed**, and will show the merge request as mergeable, despite the earlier failed pipeline caused by a change that was not yet corrected. @@ -944,279 +1546,189 @@ This could result in some unexpected behavior, including: All files are considered to have "changed" when a scheduled pipeline runs. -### `rules` - -> [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/29011) in GitLab 12.3. +### `needs` -`rules` allows for a list of individual rule objects to be evaluated -*in order*, until one matches and dynamically provides attributes to the job. -Note that `rules` cannot be used in combination with `only/except` since it is intended -to replace that functionality. If you attempt to do this the linter will return a -`key may not be used with rules` error. +> - [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/issues/47063) in GitLab 12.2. +> - In GitLab 12.3, maximum number of jobs in `needs` array raised from five to 50. +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/issues/30631) in GitLab 12.8, `needs: []` lets jobs start immediately. -Available rule clauses include: +The `needs:` keyword enables executing jobs out-of-order, allowing you to implement +a [directed acyclic graph](../directed_acyclic_graph/index.md) in your `.gitlab-ci.yml`. -- [`if`](#rulesif) (similar to [`only:variables`](#onlyvariablesexceptvariables)) -- [`changes`](#ruleschanges) (same as [`only:changes`](#onlychangesexceptchanges)) -- [`exists`](#rulesexists) +This lets you run some jobs without waiting for other ones, disregarding stage ordering +so you can have multiple stages running concurrently. -For example, using `if`. This configuration specifies that `job` should be built -and run for every pipeline on merge requests targeting `master`, regardless of -the status of other builds: +Let's consider the following example: ```yaml -job: - script: "echo Hello, Rules!" - rules: - - if: '$CI_MERGE_REQUEST_TARGET_BRANCH_NAME == "master"' - when: always - - if: '$VAR =~ /pattern/' - when: manual - - when: on_success -``` - -In this example, if the first rule: - -- Matches, the job will be given the `when:always` attribute. -- Does not match, the second and third rules will be evaluated sequentially - until a match is found. That is, the job will be given either the: - - `when: manual` attribute if the second rule matches. **The stage will not complete until this manual job is triggered and completes successfully.** - - `when: on_success` attribute if the second rule does not match. The third - rule will always match when reached because it has no conditional clauses. - -#### `rules:if` - -`rules:if` differs slightly from `only:variables` by accepting only a single -expression string, rather than an array of them. Any set of expressions to be -evaluated should be conjoined into a single expression using `&&` or `||`, and use -the [variable matching syntax](../variables/README.md#supported-syntax). +linux:build: + stage: build -For example: +mac:build: + stage: build -```yaml -job: - script: "echo Hello, Rules!" - rules: - - if: '$CI_MERGE_REQUEST_SOURCE_BRANCH_NAME =~ /^feature/ && $CI_MERGE_REQUEST_TARGET_BRANCH_NAME == "master"' # This rule will be evaluated - when: always - - if: '$CI_MERGE_REQUEST_SOURCE_BRANCH_NAME =~ /^feature/' # This rule will only be evaluated if the target branch is not "master" - when: manual - - if: '$CI_MERGE_REQUEST_SOURCE_BRANCH_NAME' # If neither of the first two match but the simple presence does, we set to "on_success" by default -``` +lint: + stage: test + needs: [] -If none of the provided rules match, the job will be set to `when:never`, and -not included in the pipeline. If `rules:when` is not included in the configuration -at all, the behavior defaults to `job:when`, which continues to default to -`on_success`. +linux:rspec: + stage: test + needs: ["linux:build"] -#### `rules:changes` +linux:rubocop: + stage: test + needs: ["linux:build"] -`rules: changes` works exactly the same way as `only: changes` and `except: changes`, -accepting an array of paths. Similarly, it will always return true if there is no -Git push event. See [`only/except: changes`](#onlychangesexceptchanges) for more information. +mac:rspec: + stage: test + needs: ["mac:build"] -For example: +mac:rubocop: + stage: test + needs: ["mac:build"] -```yaml -docker build: - script: docker build -t my-image:$CI_COMMIT_REF_SLUG . - rules: - - changes: # Will include the job and set to when:manual if any of the follow paths match a modified file. - - Dockerfile - when: manual - - if: '$VAR == "string value"' - when: manual # Will include the job and set to when:manual if the expression evaluates to true, after the `changes:` rule fails to match. - - when: on_success # If neither of the first rules match, set to on_success +production: + stage: deploy ``` -In this example, a job either set to: +This example creates four paths of execution: -- Run manually if `Dockerfile` has changed OR `$VAR == "string value"`. -- `when:on_success` by the last rule, where no earlier clauses evaluate to true. +- Linter: the `lint` job will run immediately without waiting for the `build` stage to complete because it has no needs (`needs: []`). -#### `rules:exists` +- Linux path: the `linux:rspec` and `linux:rubocop` jobs will be run as soon + as the `linux:build` job finishes without waiting for `mac:build` to finish. -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/16574) in GitLab 12.4. +- macOS path: the `mac:rspec` and `mac:rubocop` jobs will be run as soon + as the `mac:build` job finishes, without waiting for `linux:build` to finish. -`exists` accepts an array of paths and will match if any of these paths exist -as files in the repository. +- The `production` job will be executed as soon as all previous jobs + finish; in this case: `linux:build`, `linux:rspec`, `linux:rubocop`, + `mac:build`, `mac:rspec`, `mac:rubocop`. -For example: +#### Requirements and limitations -```yaml -job: - script: docker build -t my-image:$CI_COMMIT_REF_SLUG . - rules: - - exists: - - Dockerfile -``` +- If `needs:` is set to point to a job that is not instantiated + because of `only/except` rules or otherwise does not exist, the + pipeline will be created with YAML error. +- The maximum number of jobs that a single job can need in the `needs:` array is limited: + - For GitLab.com, the limit is ten. For more information, see our + [infrastructure issue](https://gitlab.com/gitlab-com/gl-infra/infrastructure/issues/7541). + - For self-managed instances, the limit is: + - 10, if the `ci_dag_limit_needs` feature flag is enabled (default). + - 50, if the `ci_dag_limit_needs` feature flag is disabled. +- If `needs:` refers to a job that is marked as `parallel:`. + the current job will depend on all parallel jobs created. +- `needs:` is similar to `dependencies:` in that it needs to use jobs from prior stages, + meaning it's impossible to create circular dependencies. Depending on jobs in the + current stage is not possible either, but support [is planned](https://gitlab.com/gitlab-org/gitlab/issues/30632). +- Related to the above, stages must be explicitly defined for all jobs + that have the keyword `needs:` or are referred to by one. -You can also use glob patterns to match multiple files in any directory within -the repository. +##### Changing the `needs:` job limit -For example: +The maximum number of jobs that can be defined within `needs:` defaults to 10, but +can be changed to 50 via a feature flag. To change the limit to 50, +[start a Rails console session](../../administration/troubleshooting/debug.md#starting-a-rails-console-session) +and run: -```yaml -job: - script: bundle exec rspec - rules: - - exists: - - spec/**.rb +```ruby +Feature::disable(:ci_dag_limit_needs) ``` -NOTE: **Note:** -For performance reasons, using `exists` with patterns is limited to 10000 -checks. After the 10000th check, rules with patterned globs will always match. - -#### `rules:allow_failure` - -> [Introduced](https://gitlab.com/gitlab-org/gitlab/issues/30235) in GitLab 12.8. - -You can use [`allow_failure: true`](#allow_failure) within `rules:` to allow a job to fail, or a manual job to -wait for action, without stopping the pipeline itself. All jobs using `rules:` default to `allow_failure: false` -if `allow_failure:` is not defined. +To set it back to 10, run the opposite command: -```yaml -job: - script: "echo Hello, Rules!" - rules: - - if: '$CI_MERGE_REQUEST_TARGET_BRANCH_NAME == "master"' - when: manual - allow_failure: true +```ruby +Feature::enable(:ci_dag_limit_needs) ``` -In this example, if the first rule matches, then the job will have `when: manual` and `allow_failure: true`. - -#### Exclude jobs with `rules:` from certain pipelines - -Jobs with `rules:` can cause two pipelines to be created unexpectedly: - -- One pipeline from pushing a commit to a branch. -- A second ["detached" pipeline for a merge request](../merge_request_pipelines/index.md). - -`only` and `except` jobs do not trigger merge request pipelines by default, but this -is not the case for jobs with `rules:`, which may be surprising if migrating from `only` -and `except` to `rules:`. - -If you are using `rules:` and you see two pipelines for commits to branches that have -a merge request, you have two options: - -- Individually exclude each job that uses `rules:` from merge request pipelines. The - example below will cause the job to **not** run in *pipelines for merge requests*, - but it **will** run in pipelines for *new tags and pipelines running on branch refs*: +#### Artifact downloads with `needs` - ```yaml - job: - rules: - - if: $CI_MERGE_REQUEST_ID - when: never - - when: manual - script: - - echo hello - ``` +> [Introduced](https://gitlab.com/gitlab-org/gitlab/issues/14311) in GitLab v12.6. -- Add a global [`workflow: rules`](#workflowrules) to allow pipelines in only certain - situations. The example below will only run pipelines for merge requests, new tags and - changes to master. It will **not** run any pipelines *on any branch except master*, but - it will run **detached merge request pipelines** for any merge request, targeting any branch: +When using `needs`, artifact downloads are controlled with `artifacts: true` or `artifacts: false`. +The `dependencies` keyword should not be used with `needs`, as this is deprecated since GitLab 12.6. - ```yaml - workflow: - rules: - - if: $CI_MERGE_REQUEST_ID - - if: $CI_COMMIT_TAG - - if: $CI_COMMIT_BRANCH == "master" - ``` +In the example below, the `rspec` job will download the `build_job` artifacts, while the +`rubocop` job won't: -#### Complex rule clauses +```yaml +build_job: + stage: build + artifacts: + paths: + - binaries/ -To conjoin `if`, `changes`, and `exists` clauses with an AND, use them in the -same rule. +rspec: + stage: test + needs: + - job: build_job + artifacts: true -In the following example: +rubocop: + stage: test + needs: + - job: build_job + artifacts: false +``` -- We run the job manually if `Dockerfile` or any file in `docker/scripts/` - has changed AND `$VAR == "string value"`. -- Otherwise, the job will not be included in the pipeline. +Additionally, in the three syntax examples below, the `rspec` job will download the artifacts +from all three `build_jobs`, as `artifacts` is true for `build_job_1`, and will +**default** to true for both `build_job_2` and `build_job_3`. ```yaml -docker build: - script: docker build -t my-image:$CI_COMMIT_REF_SLUG . - rules: - - if: '$VAR == "string value"' - changes: # Will include the job and set to when:manual if any of the follow paths match a modified file. - - Dockerfile - - docker/scripts/* - when: manual - # - when: never would be redundant here, this is implied any time rules are listed. +rspec: + needs: + - job: build_job_1 + artifacts: true + - job: build_job_2 + - build_job_3 ``` -The only clauses currently available are: - -- `if` -- `changes` -- `exists` - -Keywords such as `branches` or `refs` that are currently available for -`only`/`except` are not yet available in `rules` as they are being individually -considered for their usage and behavior in this context. - -#### Permitted attributes - -The only job attributes currently set by `rules` are: - -- `when`. -- `start_in`, if `when` is set to `delayed`. -- `allow_failure`. +#### Cross project artifact downloads with `needs` **(PREMIUM)** -A job will be included in a pipeline if `when` is evaluated to any value -except `never`. +> [Introduced](https://gitlab.com/gitlab-org/gitlab/issues/14311) in GitLab v12.7. -Delayed jobs require a `start_in` value, so rule objects do as well. For -example: +`needs` can be used to download artifacts from up to five jobs in pipelines on +[other refs in the same project](#artifact-downloads-between-pipelines-in-the-same-project), +or pipelines in different projects: ```yaml -docker build: - script: docker build -t my-image:$CI_COMMIT_REF_SLUG . - rules: - - changes: # Will include the job and delay 3 hours when the Dockerfile has changed - - Dockerfile - when: delayed - start_in: '3 hours' - - when: on_success # Otherwise include the job and set to run normally +build_job: + stage: build + script: + - ls -lhR + needs: + - project: group/project-name + job: build-1 + ref: master + artifacts: true ``` -Additional job configuration may be added to rules in the future. If something -useful isn't available, please -[open an issue](https://gitlab.com/gitlab-org/gitlab/issues). - -### `workflow:rules` - -> [Introduced](https://gitlab.com/gitlab-org/gitlab/issues/29654) in GitLab 12.5 - -The top-level `workflow:` key applies to the entirety of a pipeline, and will -determine whether or not a pipeline is created. It currently accepts a single -`rules:` key that operates similarly to [`rules:` defined within jobs](#rules), -enabling dynamic configuration of the pipeline. - -The configuration options currently available for `workflow:rules` are:​ +`build_job` will download the artifacts from the latest successful `build-1` job +on the `master` branch in the `group/project-name` project. -- [`if`](#rulesif): Define a rule. -- [`when`](#when): May be set to `always` or `never` only. If not provided, the default value is `always`​. +##### Artifact downloads between pipelines in the same project -The list of `if` rules is evaluated until a single one is matched. If none -match, the last `when` will be used: +`needs` can be used to download artifacts from different pipelines in the current project +by setting the `project` keyword as the current project's name, and specifying a ref. +In the example below, `build_job` will download the artifacts for the latest successful +`build-1` job with the `other-ref` ref: ```yaml -workflow: - rules: - - if: $CI_COMMIT_REF_NAME =~ /-wip$/ - when: never - - if: $CI_COMMIT_TAG - when: never - - when: always +build_job: + stage: build + script: + - ls -lhR + needs: + - project: group/same-project-name + job: build-1 + ref: other-ref + artifacts: true ``` +NOTE: **Note:** +Downloading artifacts from jobs that are run in [`parallel:`](#parallel) is not supported. + ### `tags` `tags` is used to select specific Runners from the list of all Runners that are @@ -1277,7 +1789,7 @@ show the same orange warning. However, the associated commit will be marked "passed", without warnings. In the example below, `job1` and `job2` will run in parallel, but if `job1` -fails, it will not stop the next stage from running, since it's marked with +fails, it won't stop the next stage from running, since it's marked with `allow_failure: true`: ```yaml @@ -1372,21 +1884,21 @@ Manual actions are a special type of job that are not executed automatically, they need to be explicitly started by a user. An example usage of manual actions would be a deployment to a production environment. Manual actions can be started from the pipeline, job, environment, and deployment views. Read more at the -[environments documentation](../environments.md#configuring-manual-deployments). +[environments documentation](../environments/index.md#configuring-manual-deployments). Manual actions can be either optional or blocking. Blocking manual actions will block the execution of the pipeline at the stage this action is defined in. It's possible to resume execution of the pipeline when someone executes a blocking manual action by clicking a _play_ button. -When a pipeline is blocked, it will not be merged if Merge When Pipeline Succeeds +When a pipeline is blocked, it won't be merged if Merge When Pipeline Succeeds is set. Blocked pipelines also do have a special status, called _manual_. When the `when:manual` syntax is used, manual actions are non-blocking by -default. If you want to make manual action blocking, it is necessary to add +default. If you want to make manual action blocking, it's necessary to add `allow_failure: false` to the job's definition in `.gitlab-ci.yml`. Optional manual actions have `allow_failure: true` set by default and their -Statuses do not contribute to the overall pipeline status. So, if a manual +Statuses don't contribute to the overall pipeline status. So, if a manual action fails, the pipeline will eventually succeed. NOTE: **Note:** @@ -1396,7 +1908,7 @@ Manual actions are considered to be write actions, so permissions for [protected branches](../../user/project/protected_branches.md) are used when a user wants to trigger an action. In other words, in order to trigger a manual action assigned to a branch that the pipeline is running for, the user needs to -have the ability to merge to this branch. It is possible to use protected environments +have the ability to merge to this branch. It's possible to use protected environments to more strictly [protect manual deployments](#protecting-manual-jobs-premium) from being run by unauthorized users. @@ -1409,7 +1921,7 @@ being used. It's possible to use [protected environments](../environments/protected_environments.md) to define a precise list of users authorized to run a manual job. By allowing only -users associated with a protected environment to trigger manual jobs, it is possible +users associated with a protected environment to trigger manual jobs, it's possible to implement some special use cases, such as: - More precisely limiting who can deploy to an environment. @@ -1439,13 +1951,13 @@ To do this, you must: who are always able to use protected environments. Additionally, if a manual job is defined as blocking by adding `allow_failure: false`, -the next stages of the pipeline will not run until the manual job is triggered. This +the next stages of the pipeline won't run until the manual job is triggered. This can be used as a way to have a defined list of users allowed to "approve" later pipeline stages by triggering the blocking manual job. #### `when:delayed` -> [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/21767) in GitLab 11.4. +> [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/51352) in GitLab 11.4. Delayed job are for executing scripts after a certain period. This is useful if you want to avoid jobs entering `pending` state immediately. @@ -1459,11 +1971,11 @@ provided. `start_in` key must be less than or equal to one week. Examples of val - `1 day` - `1 week` -When there is a delayed job in a stage, the pipeline will not progress until the delayed job has finished. +When there is a delayed job in a stage, the pipeline won't progress until the delayed job has finished. This means this keyword can also be used for inserting delays between different stages. The timer of a delayed job starts immediately after the previous stage has completed. -Similar to other types of jobs, a delayed job's timer will not start unless the previous stage passed. +Similar to other types of jobs, a delayed job's timer won't start unless the previous stage passed. The following example creates a job named `timed rollout 10%` that is executed 30 minutes after the previous stage has completed: @@ -1475,7 +1987,7 @@ timed rollout 10%: start_in: 30 minutes ``` -You can stop the active timer of a delayed job by clicking the **Unschedule** button. +You can stop the active timer of a delayed job by clicking the **{time-out}** (**Unschedule**) button. This job will never be executed in the future unless you execute the job manually. You can start a delayed job immediately by clicking the **Play** button. @@ -1485,7 +1997,7 @@ GitLab Runner will pick your job soon and start the job. > - Introduced in GitLab 8.9. > - You can read more about environments and find more examples in the -> [documentation about environments](../environments.md). +> [documentation about environments](../environments/index.md). `environment` is used to define that a job deploys to a specific environment. If `environment` is specified and no environment under that name exists, a new @@ -1511,7 +2023,7 @@ deployment to the `production` environment. > `name` keyword. > - The `name` parameter can use any of the defined CI variables, > including predefined, secure variables and `.gitlab-ci.yml` [`variables`](#variables). -> You however cannot use variables defined under `script`. +> You however can't use variables defined under `script`. The `environment` name can contain: @@ -1529,7 +2041,7 @@ Common names are `qa`, `staging`, and `production`, but you can use whatever name works with your workflow. Instead of defining the name of the environment right after the `environment` -keyword, it is also possible to define it as a separate value. For that, use +keyword, it's also possible to define it as a separate value. For that, use the `name` keyword under `environment`: ```yaml @@ -1547,7 +2059,7 @@ deploy to production: > recommended way now is to define it in `.gitlab-ci.yml`. > - The `url` parameter can use any of the defined CI variables, > including predefined, secure variables and `.gitlab-ci.yml` [`variables`](#variables). -> You however cannot use variables defined under `script`. +> You however can't use variables defined under `script`. This is an optional value that when set, it exposes buttons in various places in GitLab which when clicked take you to the defined URL. @@ -1567,7 +2079,7 @@ deploy to production: #### `environment:on_stop` -> - [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/6669) in GitLab 8.13. +> - [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/22191) in GitLab 8.13. > - Starting with GitLab 8.14, when you have an environment that has a stop action > defined, GitLab will automatically trigger a stop action when the associated > branch is deleted. @@ -1580,7 +2092,7 @@ Read the `environment:action` section for an example. #### `environment:action` -> [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/6669) in GitLab 8.13. +> [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/22191) in GitLab 8.13. The `action` keyword is to be used in conjunction with `on_stop` and is defined in the job that is called to close the environment. @@ -1615,7 +2127,7 @@ GitLab's web interface in order to run. Also in the example, `GIT_STRATEGY` is set to `none` so that GitLab Runner won’t try to check out the code after the branch is deleted when the `stop_review_app` -job is [automatically triggered](../environments.md#automatically-stopping-an-environment). +job is [automatically triggered](../environments/index.md#automatically-stopping-an-environment). NOTE: **Note:** The above example overwrites global variables. If your stop environment job depends @@ -1651,7 +2163,7 @@ When `review_app` job is executed and a review app is created, a life period of the environment is set to `1 day`. For more information, see -[the environments auto-stop documentation](../environments.md#environments-auto-stop) +[the environments auto-stop documentation](../environments/index.md#environments-auto-stop) #### `environment:kubernetes` @@ -1677,7 +2189,7 @@ environment, using the `production` [Kubernetes namespace](https://kubernetes.io/docs/concepts/overview/working-with-objects/namespaces/). For more information, see -[Available settings for `kubernetes`](../environments.md#configuring-kubernetes-deployments). +[Available settings for `kubernetes`](../environments/index.md#configuring-kubernetes-deployments). NOTE: **Note:** Kubernetes configuration is not supported for Kubernetes clusters @@ -1687,11 +2199,11 @@ To follow progress on support for GitLab-managed clusters, see the #### Dynamic environments -> - [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/6323) in GitLab 8.12 and GitLab Runner 1.6. -> - The `$CI_ENVIRONMENT_SLUG` was [introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/7983) in GitLab 8.15. +> - [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/21971) in GitLab 8.12 and GitLab Runner 1.6. +> - The `$CI_ENVIRONMENT_SLUG` was [introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/22864) in GitLab 8.15. > - The `name` and `url` parameters can use any of the defined CI variables, > including predefined, secure variables and `.gitlab-ci.yml` [`variables`](#variables). -> You however cannot use variables defined under `script`. +> You however can't use variables defined under `script`. For example: @@ -1735,15 +2247,20 @@ Read how caching works and find out some good practices in the cached between jobs. You can only use paths that are within the local working copy. -If `cache` is defined outside the scope of jobs, it means it is set +If `cache` is defined outside the scope of jobs, it means it's set globally and all jobs will use that definition. #### `cache:paths` Use the `paths` directive to choose which files or directories will be cached. Paths -are relative to the project directory (`$CI_PROJECT_DIR`) and cannot directly link outside it. +are relative to the project directory (`$CI_PROJECT_DIR`) and can't directly link outside it. Wildcards can be used that follow the [glob](https://en.wikipedia.org/wiki/Glob_(programming)) -patterns and [`filepath.Match`](https://golang.org/pkg/path/filepath/#Match). +patterns and: + +- In [GitLab Runner 13.0](https://gitlab.com/gitlab-org/gitlab-runner/-/issues/2620) and later, +[`doublestar.Glob`](https://pkg.go.dev/github.com/bmatcuk/doublestar@v1.2.2?tab=doc#Match). +- In GitLab Runner 12.10 and earlier, +[`filepath.Match`](https://pkg.go.dev/path/filepath/#Match). Cache all files in `binaries` that end in `.apk` and the `.config` file: @@ -1795,7 +2312,7 @@ set, is just literal `default` which means everything is shared between pipelines and jobs by default, starting from GitLab 9.0. NOTE: **Note:** -The `cache:key` variable cannot contain the `/` character, or the equivalent +The `cache:key` variable can't contain the `/` character, or the equivalent URI-encoded `%2F`; a value made only of dots (`.`, `%2E`) is also forbidden. For example, to enable per-branch caching: @@ -1841,7 +2358,7 @@ cache: - node_modules ``` -In this example we are creating a cache for Ruby and Node.js dependencies that +In this example we're creating a cache for Ruby and Node.js dependencies that is tied to current versions of the `Gemfile.lock` and `package.json` files. Whenever one of these files changes, a new cache key is computed and a new cache is created. Any future job runs using the same `Gemfile.lock` and `package.json` with `cache:key:files` will @@ -1912,12 +2429,12 @@ rspec: > Introduced in GitLab 9.4. -The default behaviour of a caching job is to download the files at the start of +The default behavior of a caching job is to download the files at the start of execution, and to re-upload them at the end. This allows any changes made by the job to be persisted for future runs, and is known as the `pull-push` cache policy. -If you know the job doesn't alter the cached files, you can skip the upload step +If you know the job does not alter the cached files, you can skip the upload step by setting `policy: pull` in the job specification. Typically, this would be twinned with an ordinary cache job at an earlier stage to ensure the cache is updated from time to time: @@ -1973,7 +2490,7 @@ be available for download in the GitLab UI. #### `artifacts:paths` -Paths are relative to the project directory (`$CI_PROJECT_DIR`) and cannot directly +Paths are relative to the project directory (`$CI_PROJECT_DIR`) and can't directly link outside it. Wildcards can be used that follow the [glob](https://en.wikipedia.org/wiki/Glob_(programming)) patterns and [`filepath.Match`](https://golang.org/pkg/path/filepath/#Match). @@ -2000,7 +2517,7 @@ job: You may want to create artifacts only for tagged releases to avoid filling the build server storage with temporary build artifacts. -Create artifacts only for tags (`default-job` will not create artifacts): +Create artifacts only for tags (`default-job` won't create artifacts): ```yaml default-job: @@ -2037,7 +2554,7 @@ in the [merge request](../../user/project/merge_requests/index.md) UI. For example, to match a single file: -```yml +```yaml test: script: [ 'echo 1' ] artifacts: @@ -2050,7 +2567,7 @@ that points to `file1.txt`. An example that will match an entire directory: -```yml +```yaml test: script: [ 'echo 1' ] artifacts: @@ -2081,7 +2598,7 @@ The default name is `artifacts`, which becomes `artifacts.zip` when downloaded. NOTE: **Note:** If your branch-name contains forward slashes -(e.g. `feature/my-feature`) it is advised to use `$CI_COMMIT_REF_SLUG` +(for example `feature/my-feature`) it's advised to use `$CI_COMMIT_REF_SLUG` instead of `$CI_COMMIT_REF_NAME` for proper naming of the artifact. To create an archive with a name of the current job: @@ -2212,15 +2729,15 @@ After their expiry, artifacts are deleted hourly by default (via a cron job), and are not accessible anymore. The value of `expire_in` is an elapsed time in seconds, unless a unit is -provided. Examples of parsable values: +provided. Examples of valid values: -- '42' -- '3 mins 4 sec' -- '2 hrs 20 min' -- '2h20min' -- '6 mos 1 day' -- '47 yrs 6 mos and 4d' -- '3 weeks and 2 days' +- `42` +- `3 mins 4 sec` +- `2 hrs 20 min` +- `2h20min` +- `6 mos 1 day` +- `47 yrs 6 mos and 4d` +- `3 weeks and 2 days` To expire artifacts 1 week after being uploaded: @@ -2230,187 +2747,35 @@ job: expire_in: 1 week ``` -#### `artifacts:reports` - -> [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/20390) in GitLab 11.2. Requires GitLab Runner 11.2 and above. - -The `reports` keyword is used for collecting test reports, code quality reports, and security reports from jobs. -It also exposes these reports in GitLab's UI (merge requests, pipeline views, and security dashboards). - -NOTE: **Note:** -The test reports are collected regardless of the job results (success or failure). -You can use [`artifacts:expire_in`](#artifactsexpire_in) to set up an expiration -date for their artifacts. - -NOTE: **Note:** -If you also want the ability to browse the report output files, include the -[`artifacts:paths`](#artifactspaths) keyword. - -##### `artifacts:reports:junit` - -> [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/20390) in GitLab 11.2. Requires GitLab Runner 11.2 and above. - -The `junit` report collects [JUnit XML files](https://www.ibm.com/support/knowledgecenter/en/SSQ2R2_14.1.0/com.ibm.rsar.analysis.codereview.cobol.doc/topics/cac_useresults_junit.html) -as artifacts. Although JUnit was originally developed in Java, there are many -[third party ports](https://en.wikipedia.org/wiki/JUnit#Ports) for other -languages like JavaScript, Python, Ruby, etc. - -See [JUnit test reports](../junit_test_reports.md) for more details and examples. -Below is an example of collecting a JUnit XML file from Ruby's RSpec test tool: - -```yaml -rspec: - stage: test - script: - - bundle install - - rspec --format RspecJunitFormatter --out rspec.xml - artifacts: - reports: - junit: rspec.xml -``` - -The collected JUnit reports will be uploaded to GitLab as an artifact and will -be automatically shown in merge requests. - NOTE: **Note:** -In case the JUnit tool you use exports to multiple XML files, you can specify -multiple test report paths within a single job and they will be automatically -concatenated into a single file. Use a filename pattern (`junit: rspec-*.xml`), -an array of filenames (`junit: [rspec-1.xml, rspec-2.xml, rspec-3.xml]`), or a -combination thereof (`junit: [rspec.xml, test-results/TEST-*.xml]`). - -##### `artifacts:reports:dotenv` - -> [Introduced](https://gitlab.com/gitlab-org/gitlab/issues/17066) in GitLab 12.9. Requires GitLab Runner 11.5 and later. - -The `dotenv` report collects a set of environment variables as artifacts. - -The collected variables are registered as runtime-created variables of the job, -which is useful to [set dynamic environment URLs after a job finishes](../environments.md#set-dynamic-environment-urls-after-a-job-finishes). -It is not available for download through the web interface. - -There are a couple of limitations on top of the [original dotenv rules](https://github.com/motdotla/dotenv#rules). - -- The variable key can contain only letters, digits and underscore ('_'). -- The size of dotenv file must be smaller than 5 kilobytes. -- The number of variables must be less than 10. -- It doesn't support variable substitution in the dotenv file itself. -- It doesn't support empty lines and comments (`#`) in dotenv file. -- It doesn't support quote escape, spaces in a quote, a new line expansion in a quote, in dotenv file. - -##### `artifacts:reports:cobertura` - -> [Introduced](https://gitlab.com/gitlab-org/gitlab/issues/3708) in GitLab 12.9. Requires [GitLab Runner](https://docs.gitlab.com/runner/) 11.5 and above. - -The `cobertura` report collects [Cobertura coverage XML files](../../user/project/merge_requests/test_coverage_visualization.md). -The collected Cobertura coverage reports will be uploaded to GitLab as an artifact -and will be automatically shown in merge requests. - -Cobertura was originally developed for Java, but there are many -third party ports for other languages like JavaScript, Python, Ruby, etc. - -##### `artifacts:reports:codequality` **(STARTER)** - -> Introduced in GitLab 11.5. Requires GitLab Runner 11.5 and above. - -The `codequality` report collects [CodeQuality issues](../../user/project/merge_requests/code_quality.md) -as artifacts. - -The collected Code Quality report will be uploaded to GitLab as an artifact and will -be summarized in merge requests. It is not available for download through the web interface. - -##### `artifacts:reports:sast` **(ULTIMATE)** - -> Introduced in GitLab 11.5. Requires GitLab Runner 11.5 and above. - -The `sast` report collects [SAST vulnerabilities](../../user/application_security/sast/index.md) -as artifacts. - -The collected SAST report will be uploaded to GitLab as an artifact and will -be summarized in the merge requests and pipeline view. It is also used to provide data for security -dashboards. It is not available for download through the web interface. - -##### `artifacts:reports:dependency_scanning` **(ULTIMATE)** - -> Introduced in GitLab 11.5. Requires GitLab Runner 11.5 and above. - -The `dependency_scanning` report collects [Dependency Scanning vulnerabilities](../../user/application_security/dependency_scanning/index.md) -as artifacts. - -The collected Dependency Scanning report will be uploaded to GitLab as an artifact and will -be summarized in the merge requests and pipeline view. It is also used to provide data for security -dashboards. It is not available for download through the web interface. - -##### `artifacts:reports:container_scanning` **(ULTIMATE)** - -> Introduced in GitLab 11.5. Requires GitLab Runner 11.5 and above. - -The `container_scanning` report collects [Container Scanning vulnerabilities](../../user/application_security/container_scanning/index.md) -as artifacts. - -The collected Container Scanning report will be uploaded to GitLab as an artifact and will -be summarized in the merge requests and pipeline view. It is also used to provide data for security -dashboards. It is not available for download through the web interface. - -##### `artifacts:reports:dast` **(ULTIMATE)** - -> Introduced in GitLab 11.5. Requires GitLab Runner 11.5 and above. - -The `dast` report collects [DAST vulnerabilities](../../user/application_security/dast/index.md) -as artifacts. - -The collected DAST report will be uploaded to GitLab as an artifact and will -be summarized in the merge requests and pipeline view. It is also used to provide data for security -dashboards. It is not available for download through the web interface. - -##### `artifacts:reports:license_management` **(ULTIMATE)** - -CAUTION: **Warning:** -This artifact is still valid but was **deprecated** in favor of the -[artifacts:reports:license_scanning](#artifactsreportslicense_scanning-ultimate) -introduced in GitLab 12.8. - -> Introduced in GitLab 11.5. Requires GitLab Runner 11.5 and above. - -The `license_management` report collects [Licenses](../../user/compliance/license_compliance/index.md) -as artifacts. - -The collected License Compliance report will be uploaded to GitLab as an artifact and will -be summarized in the merge requests and pipeline view. It is also used to provide data for security -dashboards. It is not available for download through the web interface. - -##### `artifacts:reports:license_scanning` **(ULTIMATE)** - -> Introduced in GitLab 12.8. Requires GitLab Runner 11.5 and above. +For artifacts created in [GitLab 13.0](https://gitlab.com/gitlab-org/gitlab/-/issues/16267) +and later, the latest artifact for a ref is always kept, regardless of the expiry time. -The `license_scanning` report collects [Licenses](../../user/compliance/license_compliance/index.md) -as artifacts. - -The License Compliance report will be uploaded to GitLab as an artifact and will -be automatically shown in merge requests, pipeline view and provide data for security -dashboards. - -##### `artifacts:reports:performance` **(PREMIUM)** - -> Introduced in GitLab 11.5. Requires GitLab Runner 11.5 and above. - -The `performance` report collects [Performance metrics](../../user/project/merge_requests/browser_performance_testing.md) -as artifacts. - -The collected Performance report will be uploaded to GitLab as an artifact and will -be automatically shown in merge requests. It is not available for download through the web interface. - -##### `artifacts:reports:metrics` **(PREMIUM)** - -> Introduced in GitLab 11.10. - -The `metrics` report collects [Metrics](../../ci/metrics_reports.md) -as artifacts. +#### `artifacts:reports` -The collected Metrics report will be uploaded to GitLab as an artifact and will -be automatically shown in merge requests. It is not available for download through the web interface. +The [`artifacts:reports` keyword](../pipelines/job_artifacts.md#artifactsreports) +is used for collecting test reports, code quality reports, and security reports from jobs. +It also exposes these reports in GitLab's UI (merge requests, pipeline views, and security dashboards). -### `dependencies` +These are the available report types: + +| Parameter | Description | +|--------------------------------------------------------------------------------------------------------------------------------------|-------------| +| [`artifacts:reports:junit`](../pipelines/job_artifacts.md#artifactsreportsjunit) | The `junit` report collects JUnit XML files. | +| [`artifacts:reports:dotenv`](../pipelines/job_artifacts.md#artifactsreportsdotenv) | The `dotenv` report collects a set of environment variables. | +| [`artifacts:reports:cobertura`](../pipelines/job_artifacts.md#artifactsreportscobertura) | The `cobertura` report collects Cobertura coverage XML files. | +| [`artifacts:reports:terraform`](../pipelines/job_artifacts.md#artifactsreportsterraform) | The `terraform` report collects Terraform `tfplan.json` files. | +| [`artifacts:reports:codequality`](../pipelines/job_artifacts.md#artifactsreportscodequality-starter) **(STARTER)** | The `codequality` report collects CodeQuality issues. | +| [`artifacts:reports:sast`](../pipelines/job_artifacts.md#artifactsreportssast-ultimate) **(ULTIMATE)** | The `sast` report collects Static Application Security Testing vulnerabilities. | +| [`artifacts:reports:dependency_scanning`](../pipelines/job_artifacts.md#artifactsreportsdependency_scanning-ultimate) **(ULTIMATE)** | The `dependency_scanning` report collects Dependency Scanning vulnerabilities. | +| [`artifacts:reports:container_scanning`](../pipelines/job_artifacts.md#artifactsreportscontainer_scanning-ultimate) **(ULTIMATE)** | The `container_scanning` report collects Container Scanning vulnerabilities. | +| [`artifacts:reports:dast`](../pipelines/job_artifacts.md#artifactsreportsdast-ultimate) **(ULTIMATE)** | The `dast` report collects Dynamic Application Security Testing vulnerabilities. | +| [`artifacts:reports:license_management`](../pipelines/job_artifacts.md#artifactsreportslicense_management-ultimate) **(ULTIMATE)** | The `license_management` report collects Licenses (*removed from GitLab 13.0*). | +| [`artifacts:reports:license_scanning`](../pipelines/job_artifacts.md#artifactsreportslicense_scanning-ultimate) **(ULTIMATE)** | The `license_scanning` report collects Licenses. | +| [`artifacts:reports:performance`](../pipelines/job_artifacts.md#artifactsreportsperformance-premium) **(PREMIUM)** | The `performance` report collects Performance metrics. | +| [`artifacts:reports:metrics`](../pipelines/job_artifacts.md#artifactsreportsmetrics-premium) **(PREMIUM)** | The `metrics` report collects Metrics. | + +#### `dependencies` > Introduced in GitLab 8.6 and GitLab Runner v1.1.1. @@ -2424,7 +2789,7 @@ You can only define jobs from stages that are executed before the current one. An error will be shown if you define jobs from the current stage or next ones. Defining an empty array will skip downloading any artifacts for that job. The status of the previous job is not considered when using `dependencies`, so -if it failed or it is a manual job that was not run, no error occurs. +if it failed or it's a manual job that was not run, no error occurs. In the following example, we define two jobs with artifacts, `build:osx` and `build:linux`. When the `test:osx` is executed, the artifacts from `build:osx` @@ -2466,7 +2831,7 @@ deploy: script: make deploy ``` -#### When a dependent job will fail +##### When a dependent job will fail > Introduced in GitLab 10.3. @@ -2480,193 +2845,9 @@ You can ask your administrator to [flip this switch](../../administration/job_artifacts.md#validation-for-dependencies) and bring back the old behavior. -### `needs` - -> - [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/issues/47063) in GitLab 12.2. -> - In GitLab 12.3, maximum number of jobs in `needs` array raised from five to 50. -> - [Introduced](https://gitlab.com/gitlab-org/gitlab/issues/30631) in GitLab 12.8, `needs: []` lets jobs start immediately. - -The `needs:` keyword enables executing jobs out-of-order, allowing you to implement -a [directed acyclic graph](../directed_acyclic_graph/index.md) in your `.gitlab-ci.yml`. - -This lets you run some jobs without waiting for other ones, disregarding stage ordering -so you can have multiple stages running concurrently. - -Let's consider the following example: - -```yaml -linux:build: - stage: build - -mac:build: - stage: build - -lint: - stage: test - needs: [] - -linux:rspec: - stage: test - needs: ["linux:build"] - -linux:rubocop: - stage: test - needs: ["linux:build"] - -mac:rspec: - stage: test - needs: ["mac:build"] - -mac:rubocop: - stage: test - needs: ["mac:build"] - -production: - stage: deploy -``` - -This example creates four paths of execution: - -- Linter: the `lint` job will run immediately without waiting for the `build` stage to complete because it has no needs (`needs: []`). - -- Linux path: the `linux:rspec` and `linux:rubocop` jobs will be run as soon - as the `linux:build` job finishes without waiting for `mac:build` to finish. - -- macOS path: the `mac:rspec` and `mac:rubocop` jobs will be run as soon - as the `mac:build` job finishes, without waiting for `linux:build` to finish. - -- The `production` job will be executed as soon as all previous jobs - finish; in this case: `linux:build`, `linux:rspec`, `linux:rubocop`, - `mac:build`, `mac:rspec`, `mac:rubocop`. - -#### Requirements and limitations - -- If `needs:` is set to point to a job that is not instantiated - because of `only/except` rules or otherwise does not exist, the - pipeline will be created with YAML error. -- We are temporarily limiting the maximum number of jobs that a single job can - need in the `needs:` array: - - For GitLab.com, the limit is ten. For more information, see our - [infrastructure issue](https://gitlab.com/gitlab-com/gl-infra/infrastructure/issues/7541). - - For self-managed instances, the limit is: - - 10, if the `ci_dag_limit_needs` feature flag is enabled (default). - - 50, if the `ci_dag_limit_needs` feature flag is disabled. -- If `needs:` refers to a job that is marked as `parallel:`. - the current job will depend on all parallel jobs created. -- `needs:` is similar to `dependencies:` in that it needs to use jobs from prior stages, - meaning it is impossible to create circular dependencies. Depending on jobs in the - current stage is not possible either, but support [is planned](https://gitlab.com/gitlab-org/gitlab/issues/30632). -- Related to the above, stages must be explicitly defined for all jobs - that have the keyword `needs:` or are referred to by one. - -##### Changing the `needs:` job limit - -The maximum number of jobs that can be defined within `needs:` defaults to 10, but -can be changed to 50 via a feature flag. To change the limit to 50, -[start a Rails console session](../../administration/troubleshooting/debug.md#starting-a-rails-console-session) -and run: - -```ruby -Feature::disable(:ci_dag_limit_needs) -``` - -To set it back to 10, run the opposite command: - -```ruby -Feature::enable(:ci_dag_limit_needs) -``` - -#### Artifact downloads with `needs` - -> [Introduced](https://gitlab.com/gitlab-org/gitlab/issues/14311) in GitLab v12.6. - -When using `needs`, artifact downloads are controlled with `artifacts: true` or `artifacts: false`. -The `dependencies` keyword should not be used with `needs`, as this is deprecated since GitLab 12.6. - -In the example below, the `rspec` job will download the `build_job` artifacts, while the -`rubocop` job will not: - -```yaml -build_job: - stage: build - artifacts: - paths: - - binaries/ - -rspec: - stage: test - needs: - - job: build_job - artifacts: true - -rubocop: - stage: test - needs: - - job: build_job - artifacts: false -``` - -Additionally, in the three syntax examples below, the `rspec` job will download the artifacts -from all three `build_jobs`, as `artifacts` is true for `build_job_1`, and will -**default** to true for both `build_job_2` and `build_job_3`. - -```yaml -rspec: - needs: - - job: build_job_1 - artifacts: true - - job: build_job_2 - - build_job_3 -``` - -#### Cross project artifact downloads with `needs` **(PREMIUM)** - -> [Introduced](https://gitlab.com/gitlab-org/gitlab/issues/14311) in GitLab v12.7. - -`needs` can be used to download artifacts from up to five jobs in pipelines on -[other refs in the same project](#artifact-downloads-between-pipelines-in-the-same-project), -or pipelines in different projects: - -```yaml -build_job: - stage: build - script: - - ls -lhR - needs: - - project: group/project-name - job: build-1 - ref: master - artifacts: true -``` - -`build_job` will download the artifacts from the latest successful `build-1` job -on the `master` branch in the `group/project-name` project. - -##### Artifact downloads between pipelines in the same project - -`needs` can be used to download artifacts from different pipelines in the current project -by setting the `project` keyword as the current project's name, and specifying a ref. -In the example below, `build_job` will download the artifacts for the latest successful -`build-1` job with the `other-ref` ref: - -```yaml -build_job: - stage: build - script: - - ls -lhR - needs: - - project: group/same-project-name - job: build-1 - ref: other-ref - artifacts: true -``` - -NOTE: **Note:** -Downloading artifacts from jobs that are run in [`parallel:`](#parallel) is not supported. - ### `coverage` -> [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/7447) in GitLab 8.17. +> [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/20428) in GitLab 8.17. `coverage` allows you to configure how code coverage will be extracted from the job output. @@ -2686,13 +2867,13 @@ job1: ### `retry` -> - [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/12909) in GitLab 9.5. -> - [Behaviour expanded](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/21758) in GitLab 11.5 to control on which failures to retry. +> - [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/3442) in GitLab 9.5. +> - [Behavior expanded](https://gitlab.com/gitlab-org/gitlab-runner/-/issues/3515) in GitLab 11.5 to control on which failures to retry. `retry` allows you to configure how many times a job is going to be retried in case of a failure. -When a job fails and has `retry` configured, it is going to be processed again +When a job fails and has `retry` configured, it's going to be processed again up to the amount of times specified by the `retry` keyword. If `retry` is set to 2, and a job succeeds in a second run (first retry), it won't be retried @@ -2744,7 +2925,7 @@ Possible values for `when` are: Please make sure to update `RETRY_WHEN_IN_DOCUMENTATION` array in `spec/lib/gitlab/ci/config/entry/retry_spec.rb` if you change any of the documented values below. The test there makes sure that all documented - values are really valid as a config option and therefore should always + values are really valid as a configuration option and therefore should always stay in sync with this documentation. --> @@ -2753,16 +2934,18 @@ Possible values for `when` are: - `script_failure`: Retry when the script failed. - `api_failure`: Retry on API failure. - `stuck_or_timeout_failure`: Retry when the job got stuck or timed out. -- `runner_system_failure`: Retry if there was a runner system failure (e.g. setting up the job failed). +- `runner_system_failure`: Retry if there was a runner system failure (for example, job setup failed). - `missing_dependency_failure`: Retry if a dependency was missing. - `runner_unsupported`: Retry if the runner was unsupported. - `stale_schedule`: Retry if a delayed job could not be executed. - `job_execution_timeout`: Retry if the script exceeded the maximum execution time set for the job. -- `archived_failure`: Retry if the job is archived and cannot be run. +- `archived_failure`: Retry if the job is archived and can't be run. - `unmet_prerequisites`: Retry if the job failed to complete prerequisite tasks. - `scheduler_failure`: Retry if the scheduler failed to assign the job to a runner. - `data_integrity_failure`: Retry if there was a structural integrity problem detected. +You can specify the number of [retry attempts for certain stages of job execution](#job-stages-attempts) using variables. + ### `timeout` > [Introduced](https://gitlab.com/gitlab-org/gitlab/issues/14887) in GitLab 12.3. @@ -2780,17 +2963,17 @@ test: ``` The job-level timeout can exceed the -[project-level timeout](../pipelines/settings.md#timeout) but can not +[project-level timeout](../pipelines/settings.md#timeout) but can't exceed the Runner-specific timeout. ### `parallel` -> [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/22631) in GitLab 11.5. +> [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/21480) in GitLab 11.5. `parallel` allows you to configure how many instances of a job to run in parallel. This value has to be greater than or equal to two (2) and less than or equal to 50. -This creates N instances of the same job that run in parallel. They're named +This creates N instances of the same job that run in parallel. They are named sequentially from `job_name 1/N` to `job_name N/N`. For every job, `CI_NODE_INDEX` and `CI_NODE_TOTAL` [environment variables](../variables/README.md#predefined-environment-variables) are set. @@ -2868,7 +3051,7 @@ staging: #### Complex `trigger` syntax for multi-project pipelines -It is possible to configure a branch name that GitLab will use to create +It's possible to configure a branch name that GitLab will use to create a downstream pipeline with: ```yaml @@ -2883,7 +3066,7 @@ staging: branch: stable ``` -It is possible to mirror the status from a triggered pipeline: +It's possible to mirror the status from a triggered pipeline: ```yaml trigger_job: @@ -2892,7 +3075,7 @@ trigger_job: strategy: depend ``` -It is possible to mirror the status from an upstream pipeline: +It's possible to mirror the status from an upstream pipeline: ```yaml upstream_bridge: @@ -2915,7 +3098,7 @@ trigger_job: ``` Similar to [multi-project pipelines](../multi_project_pipelines.md#mirroring-status-from-triggered-pipeline), -it is possible to mirror the status from a triggered pipeline: +it's possible to mirror the status from a triggered pipeline: ```yaml trigger_job: @@ -2971,9 +3154,18 @@ This can help keep your pipeline execution linear. In the example above, jobs fr subsequent stages will wait for the triggered pipeline to successfully complete before starting, at the cost of reduced parallelization. +#### Trigger a pipeline by API call + +Triggers can be used to force a rebuild of a specific branch, tag or commit, +with an API call when a pipeline gets created using a trigger token. + +Not to be confused with the [`trigger`](#trigger) parameter. + +[Read more in the triggers documentation.](../triggers/README.md) + ### `interruptible` -> [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/23464) in GitLab 12.3. +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/32022) in GitLab 12.3. `interruptible` is used to indicate that a job should be canceled if made redundant by a newer pipeline run. Defaults to `false`. This value will only be used if the [automatic cancellation of redundant pipelines feature](../pipelines/settings.md#auto-cancel-pending-pipelines) @@ -2981,8 +3173,8 @@ is enabled. When enabled, a pipeline on the same branch will be canceled when: -- It is made redundant by a newer pipeline run. -- Either all jobs are set as interruptible, or any uninterruptible jobs have not started. +- it's made redundant by a newer pipeline run. +- Either all jobs are set as interruptible, or any uninterruptible jobs haven't started. Pending jobs are always considered interruptible. @@ -3031,7 +3223,7 @@ Sometimes running multiples jobs or pipelines at the same time in an environment can lead to errors during the deployment. To avoid these errors, the `resource_group` attribute can be used to ensure that -the Runner will not run certain jobs simultaneously. +the Runner won't run certain jobs simultaneously. When the `resource_group` key is defined for a job in `.gitlab-ci.yml`, job executions are mutually exclusive across different pipelines for the same project. @@ -3048,7 +3240,7 @@ deploy-to-production: ``` In this case, if a `deploy-to-production` job is running in a pipeline, and a new -`deploy-to-production` job is created in a different pipeline, it will not run until +`deploy-to-production` job is created in a different pipeline, it won't run until the currently running/pending `deploy-to-production` job is finished. As a result, you can ensure that concurrent deployments will never happen to the production environment. @@ -3057,514 +3249,8 @@ is when deploying to physical devices. You may have more than one physical devic one can be deployed to, but there can be only one deployment per device at any given time. NOTE: **Note:** -This key can only contain letters, digits, `-`, `_`, `/`, `$`, `{`, `}`, `.`, and spaces, but it cannot start or end with `/`. - -### `include` - -> - Introduced in [GitLab Premium](https://about.gitlab.com/pricing/) 10.5. -> - Available for Starter, Premium and Ultimate since 10.6. -> - [Moved](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/21603) to GitLab Core in 11.4. - -Using the `include` keyword, you can allow the inclusion of external YAML files. -`include` requires the external YAML file to have the extensions `.yml` or `.yaml`, -otherwise the external file will not be included. - -The files defined in `include` are: - -- Deep merged with those in `.gitlab-ci.yml`. -- Always evaluated first and merged with the content of `.gitlab-ci.yml`, - regardless of the position of the `include` keyword. - -TIP: **Tip:** -Use merging to customize and override included CI/CD configurations with local -definitions. - -NOTE: **Note:** -Using YAML aliases across different YAML files sourced by `include` is not -supported. You must only refer to aliases in the same file. Instead -of using YAML anchors, you can use the [`extends` keyword](#extends). - -`include` supports four include methods: - -- [`local`](#includelocal) -- [`file`](#includefile) -- [`template`](#includetemplate) -- [`remote`](#includeremote) - -See [usage examples](#include-examples). - -NOTE: **Note:** -`.gitlab-ci.yml` configuration included by all methods is evaluated at pipeline creation. -The configuration is a snapshot in time and persisted in the database. Any changes to -referenced `.gitlab-ci.yml` configuration will not be reflected in GitLab until the next pipeline is created. - -#### `include:local` - -`include:local` includes a file from the same repository as `.gitlab-ci.yml`. -It's referenced using full paths relative to the root directory (`/`). - -You can only use files that are currently tracked by Git on the same branch -your configuration file is on. In other words, when using a `include:local`, make -sure that both `.gitlab-ci.yml` and the local file are on the same branch. - -All [nested includes](#nested-includes) will be executed in the scope of the same project, -so it is possible to use local, project, remote, or template includes. - -NOTE: **Note:** -Including local files through Git submodules paths is not supported. - -Example: - -```yaml -include: - - local: '/templates/.gitlab-ci-template.yml' -``` - -#### `include:file` - -> [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/issues/53903) in GitLab 11.7. - -To include files from another private project under the same GitLab instance, -use `include:file`. This file is referenced using full paths relative to the -root directory (`/`). For example: - -```yaml -include: - - project: 'my-group/my-project' - file: '/templates/.gitlab-ci-template.yml' -``` - -You can also specify `ref`, with the default being the `HEAD` of the project: - -```yaml -include: - - project: 'my-group/my-project' - ref: master - file: '/templates/.gitlab-ci-template.yml' - - - project: 'my-group/my-project' - ref: v1.0.0 - file: '/templates/.gitlab-ci-template.yml' - - - project: 'my-group/my-project' - ref: 787123b47f14b552955ca2786bc9542ae66fee5b # Git SHA - file: '/templates/.gitlab-ci-template.yml' -``` - -All [nested includes](#nested-includes) will be executed in the scope of the target project, -so it is possible to use local (relative to target project), project, remote -or template includes. - -#### `include:template` - -> [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/issues/53445) in GitLab 11.7. - -`include:template` can be used to include `.gitlab-ci.yml` templates that are -[shipped with GitLab](https://gitlab.com/gitlab-org/gitlab/tree/master/lib/gitlab/ci/templates). - -For example: - -```yaml -# File sourced from GitLab's template collection -include: - - template: Auto-DevOps.gitlab-ci.yml -``` - -Multiple `include:template` files: - -```yaml -include: - - template: Android-Fastlane.gitlab-ci.yml - - template: Auto-DevOps.gitlab-ci.yml -``` - -All [nested includes](#nested-includes) will be executed only with the permission of the user, -so it is possible to use project, remote or template includes. - -#### `include:remote` - -`include:remote` can be used to include a file from a different location, -using HTTP/HTTPS, referenced by using the full URL. The remote file must be -publicly accessible through a simple GET request as authentication schemas -in the remote URL is not supported. For example: - -```yaml -include: - - remote: 'https://gitlab.com/awesome-project/raw/master/.gitlab-ci-template.yml' -``` - -All nested includes will be executed without context as public user, so only another remote, -or public project, or template is allowed. - -#### Nested includes - -> [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/issues/56836) in GitLab 11.9. - -Nested includes allow you to compose a set of includes. -A total of 100 includes is allowed. -Duplicate includes are considered a configuration error. - -> [Introduced](https://gitlab.com/gitlab-org/gitlab/issues/28212) in GitLab 12.4. - -A hard limit of 30 seconds was set for resolving all files. - -#### `include` examples - -Here are a few more `include` examples. - -##### Single string or array of multiple values - -You can include your extra YAML file(s) either as a single string or -an array of multiple values. The following examples are all valid. - -Single string with the `include:local` method implied: - -```yaml -include: '/templates/.after-script-template.yml' -``` - -Array with `include` method implied: - -```yaml -include: - - 'https://gitlab.com/awesome-project/raw/master/.before-script-template.yml' - - '/templates/.after-script-template.yml' -``` - -Single string with `include` method specified explicitly: - -```yaml -include: - remote: 'https://gitlab.com/awesome-project/raw/master/.before-script-template.yml' -``` - -Array with `include:remote` being the single item: - -```yaml -include: - - remote: 'https://gitlab.com/awesome-project/raw/master/.before-script-template.yml' -``` - -Array with multiple `include` methods specified explicitly: - -```yaml -include: - - remote: 'https://gitlab.com/awesome-project/raw/master/.before-script-template.yml' - - local: '/templates/.after-script-template.yml' - - template: Auto-DevOps.gitlab-ci.yml -``` - -Array mixed syntax: - -```yaml -include: - - 'https://gitlab.com/awesome-project/raw/master/.before-script-template.yml' - - '/templates/.after-script-template.yml' - - template: Auto-DevOps.gitlab-ci.yml - - project: 'my-group/my-project' - ref: master - file: '/templates/.gitlab-ci-template.yml' -``` - -##### Re-using a `before_script` template - -In the following example, the content of `.before-script-template.yml` will be -automatically fetched and evaluated along with the content of `.gitlab-ci.yml`. - -Content of `https://gitlab.com/awesome-project/raw/master/.before-script-template.yml`: - -```yaml -before_script: - - apt-get update -qq && apt-get install -y -qq sqlite3 libsqlite3-dev nodejs - - gem install bundler --no-document - - bundle install --jobs $(nproc) "${FLAGS[@]}" -``` - -Content of `.gitlab-ci.yml`: - -```yaml -include: 'https://gitlab.com/awesome-project/raw/master/.before-script-template.yml' - -rspec: - script: - - bundle exec rspec -``` - -##### Overriding external template values - -The following example shows specific YAML-defined variables and details of the -`production` job from an include file being customized in `.gitlab-ci.yml`. - -Content of `https://company.com/autodevops-template.yml`: - -```yaml -variables: - POSTGRES_USER: user - POSTGRES_PASSWORD: testing_password - POSTGRES_DB: $CI_ENVIRONMENT_SLUG - -production: - stage: production - script: - - install_dependencies - - deploy - environment: - name: production - url: https://$CI_PROJECT_PATH_SLUG.$KUBE_INGRESS_BASE_DOMAIN - only: - - master -``` - -Content of `.gitlab-ci.yml`: - -```yaml -include: 'https://company.com/autodevops-template.yml' - -image: alpine:latest - -variables: - POSTGRES_USER: root - POSTGRES_PASSWORD: secure_password - -stages: - - build - - test - - production - -production: - environment: - url: https://domain.com -``` - -In this case, the variables `POSTGRES_USER` and `POSTGRES_PASSWORD` along -with the environment url of the `production` job defined in -`autodevops-template.yml` have been overridden by new values defined in -`.gitlab-ci.yml`. - -The merging lets you extend and override dictionary mappings, but -you cannot add or modify items to an included array. For example, to add -an additional item to the production job script, you must repeat the -existing script items: - -Content of `https://company.com/autodevops-template.yml`: - -```yaml -production: - stage: production - script: - - install_dependencies - - deploy -``` - -Content of `.gitlab-ci.yml`: - -```yaml -include: 'https://company.com/autodevops-template.yml' - -stages: - - production - -production: - script: - - install_dependencies - - deploy - - notify_owner -``` - -In this case, if `install_dependencies` and `deploy` were not repeated in -`.gitlab-ci.yml`, they would not be part of the script for the `production` -job in the combined CI configuration. - -##### Using nested includes - -The examples below show how includes can be nested from different sources -using a combination of different methods. - -In this example, `.gitlab-ci.yml` includes local the file `/.gitlab-ci/another-config.yml`: - -```yaml -include: - - local: /.gitlab-ci/another-config.yml -``` - -The `/.gitlab-ci/another-config.yml` includes a template and the `/templates/docker-workflow.yml` file -from another project: - -```yaml -include: - - template: Bash.gitlab-ci.yml - - project: group/my-project - file: /templates/docker-workflow.yml -``` - -The `/templates/docker-workflow.yml` present in `group/my-project` includes two local files -of the `group/my-project`: - -```yaml -include: - - local: /templates/docker-build.yml - - local: /templates/docker-testing.yml -``` - -Our `/templates/docker-build.yml` present in `group/my-project` adds a `docker-build` job: - -```yaml -docker-build: - script: docker build -t my-image . -``` - -Our second `/templates/docker-test.yml` present in `group/my-project` adds a `docker-test` job: - -```yaml -docker-test: - script: docker run my-image /run/tests.sh -``` - -### `extends` - -> Introduced in GitLab 11.3. - -`extends` defines entry names that a job that uses `extends` is going to -inherit from. - -It is an alternative to using [YAML anchors](#anchors) and is a little -more flexible and readable: - -```yaml -.tests: - script: rake test - stage: test - only: - refs: - - branches - -rspec: - extends: .tests - script: rake rspec - only: - variables: - - $RSPEC -``` - -In the example above, the `rspec` job inherits from the `.tests` template job. -GitLab will perform a reverse deep merge based on the keys. GitLab will: - -- Merge the `rspec` contents into `.tests` recursively. -- Not merge the values of the keys. - -This results in the following `rspec` job: - -```yaml -rspec: - script: rake rspec - stage: test - only: - refs: - - branches - variables: - - $RSPEC -``` - -NOTE: **Note:** -Note that `script: rake test` has been overwritten by `script: rake rspec`. - -If you do want to include the `rake test`, see [`before_script` and `after_script`](#before_script-and-after_script). - -`.tests` in this example is a [hidden key](#hidden-keys-jobs), but it's -possible to inherit from regular jobs as well. - -`extends` supports multi-level inheritance, however it is not recommended to -use more than three levels. The maximum nesting level that is supported is 10. -The following example has two levels of inheritance: - -```yaml -.tests: - only: - - pushes - -.rspec: - extends: .tests - script: rake rspec - -rspec 1: - variables: - RSPEC_SUITE: '1' - extends: .rspec - -rspec 2: - variables: - RSPEC_SUITE: '2' - extends: .rspec - -spinach: - extends: .tests - script: rake spinach -``` - -In GitLab 12.0 and later, it's also possible to use multiple parents for -`extends`. The algorithm used for merge is "closest scope wins", so -keys from the last member will always shadow anything defined on other -levels. For example: - -```yaml -.only-important: - only: - - master - - stable - tags: - - production - -.in-docker: - tags: - - docker - image: alpine - -rspec: - extends: - - .only-important - - .in-docker - script: - - rake rspec -``` - -This results in the following `rspec` job: - -```yaml -rspec: - only: - - master - - stable - tags: - - docker - image: alpine - script: - - rake rspec -``` - -### Using `extends` and `include` together - -`extends` works across configuration files combined with `include`. - -For example, if you have a local `included.yml` file: - -```yaml -.template: - script: - - echo Hello! -``` - -Then, in `.gitlab-ci.yml` you can use it like this: - -```yaml -include: included.yml - -useTemplate: - image: alpine - extends: .template -``` - -This will run a job called `useTemplate` that runs `echo Hello!` as defined in -the `.template` job, and uses the `alpine` Docker image as defined in the local job. +This key can only contain letters, digits, `-`, `_`, `/`, `$`, `{`, `}`, `.`, and spaces. +It can't start or end with `/`. ### `pages` @@ -3576,7 +3262,7 @@ requirements below must be met: - `artifacts` with a path to the `public/` directory must be defined. The example below simply moves all files from the root of the project to the -`public/` directory. The `.public` workaround is so `cp` doesn't also copy +`public/` directory. The `.public` workaround is so `cp` does not also copy `public/` to itself in an infinite loop: ```yaml @@ -3595,13 +3281,13 @@ pages: Read more on [GitLab Pages user documentation](../../user/project/pages/index.md). -### `variables` +## `variables` > Introduced in GitLab Runner v0.5.0. NOTE: **Note:** Integers (as well as strings) are legal both for variable's name and value. -Floats are not legal and cannot be used. +Floats are not legal and can't be used. GitLab CI/CD allows you to define variables inside `.gitlab-ci.yml` that are then passed in the job environment. They can be set globally and per-job. @@ -3632,7 +3318,7 @@ which can be set in GitLab's UI. Learn more about [variables and their priority](../variables/README.md). -#### Git strategy +### Git strategy > - Introduced in GitLab 8.9 as an experimental feature. > - `GIT_STRATEGY=none` requires GitLab Runner v1.7+. @@ -3655,7 +3341,7 @@ variables: ``` `fetch` is faster as it re-uses the local working copy (falling back to `clone` -if it doesn't exist). `git clean` is used to undo any changes made by the last +if it does not exist). `git clean` is used to undo any changes made by the last job, and `git fetch` is used to retrieve commits made since the last job ran. ```yaml @@ -3664,9 +3350,9 @@ variables: ``` `none` also re-uses the local working copy, but skips all Git operations -(including GitLab Runner's pre-clone script, if present). It is mostly useful -for jobs that operate exclusively on artifacts (e.g., `deploy`). Git repository -data may be present, but it is certain to be out of date, so you should only +(including GitLab Runner's pre-clone script, if present). It's mostly useful +for jobs that operate exclusively on artifacts (for examples `deploy`). Git repository +data may be present, but it's certain to be out of date, so you should only rely on files brought into the local working copy from cache or artifacts. ```yaml @@ -3679,7 +3365,7 @@ NOTE: **Note:** `GIT_STRATEGY` is not supported for but may be in the future. See the [support Git strategy with Kubernetes executor feature proposal](https://gitlab.com/gitlab-org/gitlab-runner/issues/3847) for updates. -#### Git submodule strategy +### Git submodule strategy > Requires GitLab Runner v1.10+. @@ -3689,10 +3375,10 @@ globally or per-job in the [`variables`](#variables) section. There are three possible values: `none`, `normal`, and `recursive`: -- `none` means that submodules will not be included when fetching the project +- `none` means that submodules won't be included when fetching the project code. This is the default, which matches the pre-v1.10 behavior. -- `normal` means that only the top-level submodules will be included. It is +- `normal` means that only the top-level submodules will be included. It's equivalent to: ```shell @@ -3703,7 +3389,7 @@ There are three possible values: `none`, `normal`, and `recursive`: - `recursive` means that all submodules (including submodules of submodules) will be included. This feature needs Git v1.8.1 and later. When using a GitLab Runner with an executor not based on Docker, make sure the Git version - meets that requirement. It is equivalent to: + meets that requirement. It's equivalent to: ```shell git submodule sync --recursive @@ -3717,7 +3403,7 @@ Note that for this feature to work correctly, the submodules must be configured - a relative path to another repository on the same GitLab server. See the [Git submodules](../git_submodules.md) documentation. -#### Git checkout +### Git checkout > Introduced in GitLab Runner 9.3. @@ -3746,7 +3432,7 @@ script: - git merge $CI_COMMIT_SHA ``` -#### Git clean flags +### Git clean flags > Introduced in GitLab Runner 11.10 @@ -3773,7 +3459,7 @@ script: - ls -al cache/ ``` -#### Job stages attempts +### Job stages attempts > Introduced in GitLab, it requires GitLab Runner v1.9+. @@ -3798,7 +3484,7 @@ variables: You can set them globally or per-job in the [`variables`](#variables) section. -#### Shallow cloning +### Shallow cloning > Introduced in GitLab 8.9 as an experimental feature. @@ -3816,7 +3502,7 @@ jobs, jobs may fail. Since Git fetching and cloning is based on a ref, such as a branch name, Runners can't clone a specific commit SHA. If there are multiple jobs in the queue, or -you are retrying an old job, the commit to be tested needs to be within the +you're retrying an old job, the commit to be tested needs to be within the Git history that is cloned. Setting too small a value for `GIT_DEPTH` can make it impossible to run these old commits. You will see `unresolved reference` in job logs. You should then reconsider changing `GIT_DEPTH` to a higher value. @@ -3833,46 +3519,9 @@ variables: You can set it globally or per-job in the [`variables`](#variables) section. -## Deprecated parameters - -The following parameters are deprecated. - -### Globally-defined `types` - -CAUTION: **Deprecated:** -`types` is deprecated, and could be removed in a future release. -Use [`stages`](#stages) instead. - -### Job-defined `type` - -CAUTION: **Deprecated:** -`type` is deprecated, and could be removed in one of the future releases. -Use [`stage`](#stage) instead. - -### Globally-defined `image`, `services`, `cache`, `before_script`, `after_script` - -Defining `image`, `services`, `cache`, `before_script`, and -`after_script` globally is deprecated. Support could be removed -from a future release. - -Use [`default:`](#setting-default-parameters) instead. For example: - -```yaml -default: - image: ruby:2.5 - services: - - docker:dind - cache: - paths: [vendor/] - before_script: - - bundle install --path vendor/ - after_script: - - rm -rf tmp/ -``` - -## Custom build directories +### Custom build directories -> [Introduced](https://gitlab.com/gitlab-org/gitlab-runner/-/merge_requests/1267) in GitLab Runner 11.10 +> [Introduced](https://gitlab.com/gitlab-org/gitlab-runner/-/issues/2211) in GitLab Runner 11.10 NOTE: **Note:** This can only be used when `custom_build_dir` is enabled in the [Runner's @@ -3885,7 +3534,7 @@ specific directory (Go projects, for example). In that case, you can specify the `GIT_CLONE_PATH` variable to tell the Runner in which directory to clone the repository: -```yml +```yaml variables: GIT_CLONE_PATH: $CI_BUILDS_DIR/project-name @@ -3898,12 +3547,12 @@ The `GIT_CLONE_PATH` has to always be within `$CI_BUILDS_DIR`. The directory set is dependent on executor and configuration of [runners.builds_dir](https://docs.gitlab.com/runner/configuration/advanced-configuration.html#the-runners-section) setting. -### Handling concurrency +#### Handling concurrency An executor using a concurrency greater than `1` might lead to failures because multiple jobs might be working on the same directory if the `builds_dir` is shared between jobs. -GitLab Runner does not try to prevent this situation. It is up to the administrator +GitLab Runner does not try to prevent this situation. It's up to the administrator and developers to comply with the requirements of Runner configuration. To avoid this scenario, you can use a unique path within `$CI_BUILDS_DIR`, because Runner @@ -3915,7 +3564,7 @@ exposes two additional variables that provide a unique `ID` of concurrency: The most stable configuration that should work well in any scenario and on any executor is to use `$CI_CONCURRENT_ID` in the `GIT_CLONE_PATH`. For example: -```yml +```yaml variables: GIT_CLONE_PATH: $CI_BUILDS_DIR/$CI_CONCURRENT_ID/project-name @@ -3927,7 +3576,7 @@ test: The `$CI_CONCURRENT_PROJECT_ID` should be used in conjunction with `$CI_PROJECT_PATH` as the `$CI_PROJECT_PATH` provides a path of a repository. That is, `group/subgroup/project`. For example: -```yml +```yaml variables: GIT_CLONE_PATH: $CI_BUILDS_DIR/$CI_CONCURRENT_ID/$CI_PROJECT_PATH @@ -3936,15 +3585,15 @@ test: - pwd ``` -### Nested paths +#### Nested paths The value of `GIT_CLONE_PATH` is expanded once and nesting variables -within it is not supported. +within is not supported. For example, you define both the variables below in your `.gitlab-ci.yml` file: -```yml +```yaml variables: GOPATH: $CI_BUILDS_DIR/go GIT_CLONE_PATH: $GOPATH/src/namespace/project @@ -3962,39 +3611,13 @@ of `.gitlab-ci.yml`. Read more about the various [YAML features](https://learnxinyminutes.com/docs/yaml/). -### Hidden keys (jobs) - -> Introduced in GitLab 8.6 and GitLab Runner v1.1.1. - -If you want to temporarily 'disable' a job, rather than commenting out all the -lines where the job is defined: - -```yaml -#hidden_job: -# script: -# - run test -``` - -you can instead start its name with a dot (`.`) and it will not be processed by -GitLab CI/CD. In the following example, `.hidden_job` will be ignored: - -```yaml -.hidden_job: - script: - - run test -``` - -Use this feature to ignore jobs, or use the -[special YAML features](#special-yaml-features) and transform the hidden keys -into templates. - ### Anchors > Introduced in GitLab 8.6 and GitLab Runner v1.1.1. YAML has a handy feature called 'anchors', which lets you easily duplicate content across your document. Anchors can be used to duplicate/inherit -properties, and is a perfect example to be used with [hidden keys](#hidden-keys-jobs) +properties, and is a perfect example to be used with [hidden jobs](#hide-jobs) to provide templates for your jobs. The following example uses anchors and map merging. It will create two jobs, @@ -4108,7 +3731,7 @@ test:mysql: - ruby ``` -You can see that the hidden keys are conveniently used as templates. +You can see that the hidden jobs are conveniently used as templates. NOTE: **Note:** You can't use YAML anchors across multiple files when leveraging the [`include`](#include) @@ -4183,14 +3806,39 @@ job_no_git_strategy: script: echo $SAMPLE_VARIABLE ``` -## Triggers +### Hide jobs -Triggers can be used to force a rebuild of a specific branch, tag or commit, -with an API call when a pipeline gets created using a trigger token. +> Introduced in GitLab 8.6 and GitLab Runner v1.1.1. -Not to be confused with [`trigger`](#trigger). +If you want to temporarily 'disable' a job, rather than commenting out all the +lines where the job is defined: -[Read more in the triggers documentation.](../triggers/README.md) +```yaml +#hidden_job: +# script: +# - run test +``` + +you can instead start its name with a dot (`.`) and it won't be processed by +GitLab CI/CD. In the following example, `.hidden_job` will be ignored: + +```yaml +.hidden_job: + script: + - run test +``` + +Use this feature to ignore jobs, or use the +[special YAML features](#special-yaml-features) and transform the hidden jobs +into templates. + +## Skip Pipeline + +If your commit message contains `[ci skip]` or `[skip ci]`, using any +capitalization, the commit will be created but the pipeline will be skipped. + +Alternatively, one can pass the `ci.skip` [Git push option](../../user/project/push_options.md#push-options-for-gitlab-cicd) +if using Git 2.10 or newer. ## Processing Git pushes @@ -4201,13 +3849,42 @@ This limitation does not affect any of the updated Merge Request pipelines, all updated Merge Requests will have a pipeline created when using [pipelines for merge requests](../merge_request_pipelines/index.md). -## Skipping jobs +## Deprecated parameters -If your commit message contains `[ci skip]` or `[skip ci]`, using any -capitalization, the commit will be created but the pipeline will be skipped. +The following parameters are deprecated. -Alternatively, one can pass the `ci.skip` [Git push option](../../user/project/push_options.md#push-options-for-gitlab-cicd) -if using Git 2.10 or newer. +### Globally-defined `types` + +CAUTION: **Deprecated:** +`types` is deprecated, and could be removed in a future release. +Use [`stages`](#stages) instead. + +### Job-defined `type` + +CAUTION: **Deprecated:** +`type` is deprecated, and could be removed in one of the future releases. +Use [`stage`](#stage) instead. + +### Globally-defined `image`, `services`, `cache`, `before_script`, `after_script` + +Defining `image`, `services`, `cache`, `before_script`, and +`after_script` globally is deprecated. Support could be removed +from a future release. + +Use [`default:`](#global-defaults) instead. For example: + +```yaml +default: + image: ruby:2.5 + services: + - docker:dind + cache: + paths: [vendor/] + before_script: + - bundle install --path vendor/ + after_script: + - rm -rf tmp/ +``` <!-- ## Troubleshooting diff --git a/doc/ci/yaml/includes.md b/doc/ci/yaml/includes.md new file mode 100644 index 00000000000..a7b626bdd7c --- /dev/null +++ b/doc/ci/yaml/includes.md @@ -0,0 +1,213 @@ +# GitLab CI/CD YAML includes + +In addition to the [`includes` examples](README.md#include) listed in the +[GitLab CI YAML reference](README.md), this page lists more variations of `include` +usage. + +## Single string or array of multiple values + +You can include your extra YAML file(s) either as a single string or +an array of multiple values. The following examples are all valid. + +Single string with the `include:local` method implied: + +```yaml +include: '/templates/.after-script-template.yml' +``` + +Array with `include` method implied: + +```yaml +include: + - 'https://gitlab.com/awesome-project/raw/master/.before-script-template.yml' + - '/templates/.after-script-template.yml' +``` + +Single string with `include` method specified explicitly: + +```yaml +include: + remote: 'https://gitlab.com/awesome-project/raw/master/.before-script-template.yml' +``` + +Array with `include:remote` being the single item: + +```yaml +include: + - remote: 'https://gitlab.com/awesome-project/raw/master/.before-script-template.yml' +``` + +Array with multiple `include` methods specified explicitly: + +```yaml +include: + - remote: 'https://gitlab.com/awesome-project/raw/master/.before-script-template.yml' + - local: '/templates/.after-script-template.yml' + - template: Auto-DevOps.gitlab-ci.yml +``` + +Array mixed syntax: + +```yaml +include: + - 'https://gitlab.com/awesome-project/raw/master/.before-script-template.yml' + - '/templates/.after-script-template.yml' + - template: Auto-DevOps.gitlab-ci.yml + - project: 'my-group/my-project' + ref: master + file: '/templates/.gitlab-ci-template.yml' +``` + +## Re-using a `before_script` template + +In the following example, the content of `.before-script-template.yml` will be +automatically fetched and evaluated along with the content of `.gitlab-ci.yml`. + +Content of `https://gitlab.com/awesome-project/raw/master/.before-script-template.yml`: + +```yaml +before_script: + - apt-get update -qq && apt-get install -y -qq sqlite3 libsqlite3-dev nodejs + - gem install bundler --no-document + - bundle install --jobs $(nproc) "${FLAGS[@]}" +``` + +Content of `.gitlab-ci.yml`: + +```yaml +include: 'https://gitlab.com/awesome-project/raw/master/.before-script-template.yml' + +rspec: + script: + - bundle exec rspec +``` + +## Overriding external template values + +The following example shows specific YAML-defined variables and details of the +`production` job from an include file being customized in `.gitlab-ci.yml`. + +Content of `https://company.com/autodevops-template.yml`: + +```yaml +variables: + POSTGRES_USER: user + POSTGRES_PASSWORD: testing_password + POSTGRES_DB: $CI_ENVIRONMENT_SLUG + +production: + stage: production + script: + - install_dependencies + - deploy + environment: + name: production + url: https://$CI_PROJECT_PATH_SLUG.$KUBE_INGRESS_BASE_DOMAIN + only: + - master +``` + +Content of `.gitlab-ci.yml`: + +```yaml +include: 'https://company.com/autodevops-template.yml' + +image: alpine:latest + +variables: + POSTGRES_USER: root + POSTGRES_PASSWORD: secure_password + +stages: + - build + - test + - production + +production: + environment: + url: https://domain.com +``` + +In this case, the variables `POSTGRES_USER` and `POSTGRES_PASSWORD` along +with the environment URL of the `production` job defined in +`autodevops-template.yml` have been overridden by new values defined in +`.gitlab-ci.yml`. + +The merging lets you extend and override dictionary mappings, but +you cannot add or modify items to an included array. For example, to add +an additional item to the production job script, you must repeat the +existing script items: + +Content of `https://company.com/autodevops-template.yml`: + +```yaml +production: + stage: production + script: + - install_dependencies + - deploy +``` + +Content of `.gitlab-ci.yml`: + +```yaml +include: 'https://company.com/autodevops-template.yml' + +stages: + - production + +production: + script: + - install_dependencies + - deploy + - notify_owner +``` + +In this case, if `install_dependencies` and `deploy` were not repeated in +`.gitlab-ci.yml`, they would not be part of the script for the `production` +job in the combined CI configuration. + +## Using nested includes + +The examples below show how includes can be nested from different sources +using a combination of different methods. + +In this example, `.gitlab-ci.yml` includes local the file `/.gitlab-ci/another-config.yml`: + +```yaml +include: + - local: /.gitlab-ci/another-config.yml +``` + +The `/.gitlab-ci/another-config.yml` includes a template and the `/templates/docker-workflow.yml` file +from another project: + +```yaml +include: + - template: Bash.gitlab-ci.yml + - project: group/my-project + file: /templates/docker-workflow.yml +``` + +The `/templates/docker-workflow.yml` present in `group/my-project` includes two local files +of the `group/my-project`: + +```yaml +include: + - local: /templates/docker-build.yml + - local: /templates/docker-testing.yml +``` + +Our `/templates/docker-build.yml` present in `group/my-project` adds a `docker-build` job: + +```yaml +docker-build: + script: docker build -t my-image . +``` + +Our second `/templates/docker-test.yml` present in `group/my-project` adds a `docker-test` job: + +```yaml +docker-test: + script: docker run my-image /run/tests.sh +``` |
