diff options
| author | GitLab Bot <gitlab-bot@gitlab.com> | 2021-12-20 13:37:47 +0000 |
|---|---|---|
| committer | GitLab Bot <gitlab-bot@gitlab.com> | 2021-12-20 13:37:47 +0000 |
| commit | aee0a117a889461ce8ced6fcf73207fe017f1d99 (patch) | |
| tree | 891d9ef189227a8445d83f35c1b0fc99573f4380 /doc/ci | |
| parent | 8d46af3258650d305f53b819eabf7ab18d22f59e (diff) | |
| download | gitlab-ce-aee0a117a889461ce8ced6fcf73207fe017f1d99.tar.gz | |
Add latest changes from gitlab-org/gitlab@14-6-stable-eev14.6.0-rc42
Diffstat (limited to 'doc/ci')
52 files changed, 3375 insertions, 3095 deletions
diff --git a/doc/ci/caching/index.md b/doc/ci/caching/index.md index 57cbf387115..491454aed28 100644 --- a/doc/ci/caching/index.md +++ b/doc/ci/caching/index.md @@ -27,7 +27,7 @@ can't link to files outside it. ### Cache -- Define cache per job by using the `cache:` keyword. Otherwise it is disabled. +- Define cache per job by using the `cache` keyword. Otherwise it is disabled. - Subsequent pipelines can use the cache. - Subsequent jobs in the same pipeline can use the cache, if the dependencies are identical. - Different projects cannot share the cache. diff --git a/doc/ci/ci_cd_for_external_repos/index.md b/doc/ci/ci_cd_for_external_repos/index.md index fbfcdcbf64f..7bc138d083d 100644 --- a/doc/ci/ci_cd_for_external_repos/index.md +++ b/doc/ci/ci_cd_for_external_repos/index.md @@ -9,33 +9,34 @@ type: index, howto >[Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/4642) in GitLab 10.6. -GitLab CI/CD can be used with: +INFO: +Get external repo access and more by upgrading to GitLab Ultimate. +[Try a free 30-day trial now](https://about.gitlab.com/free-trial/index.html?glm_source=docs.gitlab.com&glm_content=p-ci-cd-external-docs). -- [GitHub](github_integration.md). -- [Bitbucket Cloud](bitbucket_integration.md). -- Any other Git server. +GitLab CI/CD can be used with [GitHub](github_integration.md), [Bitbucket Cloud](bitbucket_integration.md), or any other +Git server. Instead of moving your entire project to GitLab, you can connect your external repository to get the benefits of GitLab CI/CD. Connecting an external repository sets up [repository mirroring](../../user/project/repository/mirror/index.md) -and create a lightweight project with issues, merge requests, wiki, and +and creates a lightweight project with issues, merge requests, wiki, and snippets disabled. These features [can be re-enabled later](../../user/project/settings/index.md#sharing-and-permissions). +## Connect to an external repository + To connect to an external repository: <!-- vale gitlab.Spelling = NO --> -1. On the top menu, select **Projects > Create new project**. +1. On the top bar, select **Menu > Projects > Create new project**. 1. Select **Run CI/CD for external repository**. 1. Select **GitHub** or **Repo by URL**. 1. Complete the fields. <!-- vale gitlab.Spelling = YES --> - - ## Pipelines for external pull requests > [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/65139) in GitLab 12.3. diff --git a/doc/ci/directed_acyclic_graph/index.md b/doc/ci/directed_acyclic_graph/index.md index f26a678962a..abf43390834 100644 --- a/doc/ci/directed_acyclic_graph/index.md +++ b/doc/ci/directed_acyclic_graph/index.md @@ -66,9 +66,9 @@ as quickly as possible. ## Usage -Relationships are defined between jobs using the [`needs:` keyword](../yaml/index.md#needs). +Relationships are defined between jobs using the [`needs` keyword](../yaml/index.md#needs). -Note that `needs:` also works with the [parallel](../yaml/index.md#parallel) keyword, +Note that `needs` also works with the [parallel](../yaml/index.md#parallel) keyword, giving you powerful options for parallelization within your pipeline. ## Limitations @@ -87,7 +87,7 @@ are certain use cases that you may need to work around. For more information, ch The needs visualization makes it easier to visualize the relationships between dependent jobs in a DAG. This graph displays all the jobs in a pipeline that need or are needed by other jobs. Jobs with no relationships are not displayed in this view. -To see the needs visualization, click on the **Needs** tab when viewing a pipeline that uses the `needs:` keyword. +To see the needs visualization, click on the **Needs** tab when viewing a pipeline that uses the `needs` keyword.  diff --git a/doc/ci/docker/using_docker_build.md b/doc/ci/docker/using_docker_build.md index f6a6e892177..3a05e9aa7d9 100644 --- a/doc/ci/docker/using_docker_build.md +++ b/doc/ci/docker/using_docker_build.md @@ -548,7 +548,7 @@ kubectl create configmap docker-daemon --namespace gitlab-runner --from-file /tm ``` NOTE: -Make sure to use the namespace that the GitLab Runner Kubernetes executor uses +Make sure to use the namespace that the Kubernetes executor for GitLab Runner uses to create job pods in. After the ConfigMap is created, you can update the `config.toml` diff --git a/doc/ci/environments/deployment_safety.md b/doc/ci/environments/deployment_safety.md index 78f30b29e06..ca7b01edf39 100644 --- a/doc/ci/environments/deployment_safety.md +++ b/doc/ci/environments/deployment_safety.md @@ -119,17 +119,17 @@ The other pipelines don't get the protected variable. You can also We recommend that you use protected variables on protected environments to make sure that the secrets aren't exposed unintentionally. You can also define production secrets on the [runner side](../runners/configure_runners.md#prevent-runners-from-revealing-sensitive-information). -This prevents other maintainers from reading the secrets and makes sure that the runner only runs on -protected branches. +This prevents other users with the [Maintainer role](../../user/permissions.md) from reading the secrets and makes sure +that the runner only runs on protected branches. For more information, see [pipeline security](../pipelines/index.md#pipeline-security-on-protected-branches). ## Separate project for deployments -All project maintainers have access to production secrets. If you need to limit the number of users +All users with the Maintainer role for the project have access to production secrets. If you need to limit the number of users that can deploy to a production environment, you can create a separate project and configure a new permission model that isolates the CD permissions from the original project and prevents the -original project's maintainers from accessing the production secret and CD configuration. You can +original users with the Maintainer role for the project from accessing the production secret and CD configuration. You can connect the CD project to your development projects by using [multi-project pipelines](../pipelines/multi_project_pipelines.md). ## Protect `gitlab-ci.yml` from change diff --git a/doc/ci/environments/index.md b/doc/ci/environments/index.md index 87d7b9f9e30..561507cab97 100644 --- a/doc/ci/environments/index.md +++ b/doc/ci/environments/index.md @@ -256,7 +256,7 @@ GitLab supports the [dotenv (`.env`)](https://github.com/bkeepers/dotenv) file f and expands the `environment:url` value with variables defined in the `.env` file. To use this feature, specify the -[`artifacts:reports:dotenv`](../yaml/index.md#artifactsreportsdotenv) keyword in `.gitlab-ci.yml`. +[`artifacts:reports:dotenv`](../yaml/artifacts_reports.md#artifactsreportsdotenv) keyword in `.gitlab-ci.yml`. <i class="fa fa-youtube-play youtube" aria-hidden="true"></i> For an overview, see [Set dynamic URLs after a job finished](https://youtu.be/70jDXtOf4Ig). @@ -294,7 +294,7 @@ As soon as the `review` job finishes, GitLab updates the `review/your-branch-nam environment's URL. It parses the `deploy.env` report artifact, registers a list of variables as runtime-created, uses it for expanding `environment:url: $DYNAMIC_ENVIRONMENT_URL` and sets it to the environment URL. -You can also specify a static part of the URL at `environment:url:`, such as +You can also specify a static part of the URL at `environment:url`, such as `https://$DYNAMIC_ENVIRONMENT_URL`. If the value of `DYNAMIC_ENVIRONMENT_URL` is `example.com`, the final result is `https://example.com`. @@ -303,7 +303,7 @@ The assigned URL for the `review/your-branch-name` environment is visible in the Note the following: - `stop_review` doesn't generate a dotenv report artifact, so it doesn't recognize the - `DYNAMIC_ENVIRONMENT_URL` environment variable. Therefore you shouldn't set `environment:url:` in the + `DYNAMIC_ENVIRONMENT_URL` environment variable. Therefore you shouldn't set `environment:url` in the `stop_review` job. - If the environment URL isn't valid (for example, the URL is malformed), the system doesn't update the environment URL. @@ -447,6 +447,63 @@ try to check out the code after the branch is deleted. Read more in the [`.gitlab-ci.yml` reference](../yaml/index.md#environmenton_stop). +#### Stop an environment when another job is finished + +You can set an environment to stop when another job is finished. + +In your `.gitlab-ci.yml` file, specify in the [`on_stop`](../yaml/index.md#environmenton_stop) +keyword the name of the job that stops the environment. + +The following example shows a `review_app` job that calls a `stop_review_app` job after the first +job is finished. The `stop_review_app` is triggered based on what is defined under `when`. In this +case, it is set to `manual`, so it needs a +[manual action](../jobs/job_control.md#create-a-job-that-must-be-run-manually) +from the GitLab UI to run. + +Both jobs must have the same rules or only/except configuration. +In this example, if the configuration is not identical: + +- The `stop_review_app` job might not be included in all pipelines that include the `review_app` job. +- It is not possible to trigger the `action: stop` to stop the environment automatically. + +Also in the example, `GIT_STRATEGY` is set to `none`. If the +`stop_review_app` job is [automatically triggered](../environments/index.md#stop-an-environment), +the runner won't try to check out the code after the branch is deleted. + +The example also overwrites global variables. If your `stop` `environment` job depends +on global variables, use [anchor variables](../yaml/yaml_optimization.md#yaml-anchors-for-variables) when you set the `GIT_STRATEGY` +to change the job without overriding the global variables. + +The `stop_review_app` job **must** have the following keywords defined: + +- `when`, defined at either: + - [The job level](../yaml/index.md#when). + - [In a rules clause](../yaml/index.md#rules). If you use `rules` and `when: manual`, you should + also set [`allow_failure: true`](../yaml/index.md#allow_failure) so the pipeline can complete + even if the job doesn't run. +- `environment:name` +- `environment:action` + +```yaml +review_app: + stage: deploy + script: make deploy-app + environment: + name: review/$CI_COMMIT_REF_SLUG + url: https://$CI_ENVIRONMENT_SLUG.example.com + on_stop: stop_review_app + +stop_review_app: + stage: deploy + variables: + GIT_STRATEGY: none + script: make delete-app + when: manual + environment: + name: review/$CI_COMMIT_REF_SLUG + action: stop +``` + #### Stop an environment after a certain time period > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/20956) in GitLab 12.8. @@ -716,10 +773,11 @@ fetch = +refs/environments/*:refs/remotes/origin/environments/* ### Archive Old Deployments -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/73628) in GitLab 14.5. +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/73628) in GitLab 14.5. +> - [Enabled on GitLab.com and self-managed](https://gitlab.com/gitlab-org/gitlab/-/issues/337507) in GitLab 14.6. FLAG: -On self-managed GitLab, by default this feature is not available. To make it available per project or for your entire instance, ask an administrator to [enable the feature flag](../../administration/feature_flags.md) named `deployments_archive`. On GitLab.com, this feature will be rolled out gradually. +On self-managed GitLab, by default this feature is available. To hide the feature per project or for your entire instance, ask an administrator to [disable the feature flag](../../administration/feature_flags.md) named `deployments_archive`. On GitLab.com, this feature is available. When a new deployment happens in your project, GitLab creates [a special Git-ref to the deployment](#check-out-deployments-locally). @@ -878,11 +936,6 @@ To ensure the `action: stop` can always run when needed, you can: > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/21182) in GitLab 14.4. -FLAG: -On self-managed GitLab, by default this bug fix is available. To hide the bug fix per project, -ask an administrator to [disable the feature flag](../../administration/feature_flags.md) named `surface_environment_creation_failure`. -On GitLab.com, this bug fix is available. - If your project is configured to [create a dynamic environment](#create-a-dynamic-environment), you might encounter this error because the dynamically generated parameter can't be used for creating an environment. diff --git a/doc/ci/environments/protected_environments.md b/doc/ci/environments/protected_environments.md index 55b63dd090d..57fd72863c1 100644 --- a/doc/ci/environments/protected_environments.md +++ b/doc/ci/environments/protected_environments.md @@ -79,7 +79,7 @@ Alternatively, you can use the API to protect an environment: $ curl --request POST --header "PRIVATE-TOKEN: <your_access_token>" \ --data "user_id=3222377&access_level=20" "https://gitlab.com/api/v4/groups/9899826/members" - {"id":3222377,"name":"Sean Carroll","username":"sfcarroll","state":"active","avatar_url":"https://assets.gitlab-static.net/uploads/-/system/user/avatar/3222377/avatar.png","web_url":"https://gitlab.com/sfcarroll","access_level":20,"created_at":"2020-10-26T17:37:50.309Z","expires_at":null} + {"id":3222377,"name":"Sean Carroll","username":"sfcarroll","state":"active","avatar_url":"https://gitlab.com/uploads/-/system/user/avatar/3222377/avatar.png","web_url":"https://gitlab.com/sfcarroll","access_level":20,"created_at":"2020-10-26T17:37:50.309Z","expires_at":null} ``` 1. Use the API to add the group to the project as a reporter: @@ -194,16 +194,15 @@ and are protected at the same time. In an enterprise organization, with thousands of projects under a single group, ensuring that all of the [project-level protected environments](#protecting-environments) are properly configured is not a scalable solution. For example, a developer -might gain privileged access to a higher environment when they are added as a -maintainer to a new project. Group-level protected environments can be a solution -in this situation. +might gain privileged access to a higher environment when they are given the [Maintainer role](../../user/permissions.md) +for a new project. Group-level protected environments can be a solution in this situation. To maximize the effectiveness of group-level protected environments, [group-level memberships](../../user/group/index.md) must be correctly configured: -- Operators should be assigned the [maintainer role](../../user/permissions.md) - (or above) to the top-level group. They can maintain CI/CD configurations for +- Operators should be given at least the [Maintainer role](../../user/permissions.md) + for the top-level group. They can maintain CI/CD configurations for the higher environments (such as production) in the group-level settings page, which includes group-level protected environments, [group-level runners](../runners/runners_scope.md#group-runners), and @@ -211,9 +210,9 @@ configured: configurations are inherited to the child projects as read-only entries. This ensures that only operators can configure the organization-wide deployment ruleset. -- Developers should be assigned the [developer role](../../user/permissions.md) - (or below) at the top-level group, or explicitly assigned to a child project - as maintainers. They do *NOT* have access to the CI/CD configurations in the +- Developers should be given no more than the [Developer role](../../user/permissions.md) + for the top-level group, or explicitly given the [Maintainer role](../../user/permissions.md) for a child project + They do *NOT* have access to the CI/CD configurations in the top-level group, so operators can ensure that the critical configuration won't be accidentally changed by the developers. - For sub-groups and child projects: @@ -225,7 +224,7 @@ configured: environment configurations exist, to run a deployment job, the user must be allowed in **both** rulesets. - In a project or a subgroup of the top-level group, developers can be - safely assigned the Maintainer role to tune their lower environments (such + safely assigned the [Maintainer role](../../user/permissions.md) to tune their lower environments (such as `testing`). Having this configuration in place: diff --git a/doc/ci/examples/authenticating-with-hashicorp-vault/index.md b/doc/ci/examples/authenticating-with-hashicorp-vault/index.md index 238c76b2c3c..1141583df3f 100644 --- a/doc/ci/examples/authenticating-with-hashicorp-vault/index.md +++ b/doc/ci/examples/authenticating-with-hashicorp-vault/index.md @@ -22,8 +22,7 @@ This tutorial assumes you are familiar with GitLab CI/CD and Vault. To follow along, you must have: - An account on GitLab. -- A running Vault server and access to it is required to configure authentication and create roles - and policies. For HashiCorp Vaults, this can be the Open Source or Enterprise version. +- Access to a running Vault server (at least v1.2.0) to configure authentication and to create roles and policies. For HashiCorp Vaults, this can be the Open Source or Enterprise version. NOTE: You must replace the `vault.example.com` URL below with the URL of your Vault server, and `gitlab.example.com` with the URL of your GitLab instance. diff --git a/doc/ci/examples/semantic-release.md b/doc/ci/examples/semantic-release.md index eb758218f17..c74af852a9a 100644 --- a/doc/ci/examples/semantic-release.md +++ b/doc/ci/examples/semantic-release.md @@ -56,7 +56,7 @@ default: { echo "@${CI_PROJECT_ROOT_NAMESPACE}:registry=${CI_API_V4_URL}/projects/${CI_PROJECT_ID}/packages/npm/" echo "${CI_API_V4_URL#https?}/projects/${CI_PROJECT_ID}/packages/npm/:_authToken=\${CI_JOB_TOKEN}" - } | tee --append .npmrc + } | tee -a .npmrc cache: key: ${CI_COMMIT_REF_SLUG} paths: diff --git a/doc/ci/git_submodules.md b/doc/ci/git_submodules.md index dc5faf0188e..2a002b8fb9f 100644 --- a/doc/ci/git_submodules.md +++ b/doc/ci/git_submodules.md @@ -59,7 +59,7 @@ To make submodules work correctly in CI/CD jobs: variables: GIT_SUBMODULE_STRATEGY: recursive ``` - + If you use the [`CI_JOB_TOKEN`](jobs/ci_job_token.md) to clone a submodule in a pipeline job, the user executing the job must be assigned to a role that has [permission](../user/permissions.md#gitlab-cicd-permissions) to trigger a pipeline diff --git a/doc/ci/jobs/ci_job_token.md b/doc/ci/jobs/ci_job_token.md index b6a3011a3d6..532a0dffbce 100644 --- a/doc/ci/jobs/ci_job_token.md +++ b/doc/ci/jobs/ci_job_token.md @@ -4,7 +4,7 @@ group: Pipeline Execution info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- -# GitLab CI/CD job token +# GitLab CI/CD job token **(FREE)** When a pipeline job is about to run, GitLab generates a unique token and injects it as the [`CI_JOB_TOKEN` predefined variable](../variables/predefined_variables.md). @@ -61,11 +61,7 @@ tries to steal tokens from other jobs. > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/328553) in GitLab 14.1. [Deployed behind the `:ci_scoped_job_token` feature flag](../../user/feature_flags.md), disabled by default. > - [Enabled on GitLab.com and self-managed](https://gitlab.com/gitlab-org/gitlab/-/issues/332272) in GitLab 14.4. - -FLAG: -On self-managed GitLab, by default this feature is available. To hide the feature, -ask an administrator to [disable the `ci_scoped_job_token` flag](../../administration/feature_flags.md). -On GitLab.com, this feature is available. +> - [Feature flag removed](https://gitlab.com/gitlab-org/gitlab/-/issues/332272) in GitLab 14.6. You can limit the access scope of a project's CI/CD job token to increase the job token's security. A job token might give extra permissions that aren't necessary @@ -95,7 +91,7 @@ The job token scope is only for controlling access to private projects. 1. On the left sidebar, select **Settings > CI/CD**. 1. Expand **Token Access**. 1. Toggle **Limit CI_JOB_TOKEN access** to enabled. -1. (Optional) Add existing projects to the token's access scope. The user adding a +1. Optional. Add existing projects to the token's access scope. The user adding a project must have the [maintainer role](../../user/permissions.md) in both projects. There is [a proposal](https://gitlab.com/groups/gitlab-org/-/epics/3559) to improve @@ -121,7 +117,7 @@ trigger_pipeline: ``` If you use the `CI_PIPELINE_SOURCE` [predefined CI/CD variable](../variables/predefined_variables.md) -in a pipeline triggered this way, [the value is `pipeline` (not `triggered`)](../triggers/index.md#authentication-tokens). +in a pipeline triggered this way, [the value is `pipeline` (not `triggered`)](../triggers/index.md#configure-cicd-jobs-to-run-in-triggered-pipelines). ## Download an artifact from a different pipeline **(PREMIUM)** diff --git a/doc/ci/jobs/job_control.md b/doc/ci/jobs/job_control.md index 0f92ae5ca49..596df34b5c2 100644 --- a/doc/ci/jobs/job_control.md +++ b/doc/ci/jobs/job_control.md @@ -79,7 +79,7 @@ job: - In **all other cases**, the job is added to the pipeline, with `when: on_success`. WARNING: -If you use a `when:` clause as the final rule (not including `when: never`), two +If you use a `when` clause as the final rule (not including `when: never`), two simultaneous pipelines may start. Both push pipelines and merge request pipelines can be triggered by the same event (a push to the source branch for an open merge request). See how to [prevent duplicate pipelines](#avoid-duplicate-pipelines) @@ -153,7 +153,7 @@ To avoid duplicate pipelines, you can: - Use [`workflow`](../yaml/index.md#workflow) to specify which types of pipelines can run. - Rewrite the rules to run the job only in very specific cases, - and avoid a final `when:` rule: + and avoid a final `when` rule: ```yaml job: @@ -225,7 +225,7 @@ check the value of the `$CI_PIPELINE_SOURCE` variable: | `pipeline` | For [multi-project pipelines](../pipelines/multi_project_pipelines.md) created by [using the API with `CI_JOB_TOKEN`](../pipelines/multi_project_pipelines.md#create-multi-project-pipelines-by-using-the-api), or the [`trigger`](../yaml/index.md#trigger) keyword. | | `push` | For pipelines triggered by a `git push` event, including for branches and tags. | | `schedule` | For [scheduled pipelines](../pipelines/schedules.md). | -| `trigger` | For pipelines created by using a [trigger token](../triggers/index.md#authentication-tokens). | +| `trigger` | For pipelines created by using a [trigger token](../triggers/index.md#configure-cicd-jobs-to-run-in-triggered-pipelines). | | `web` | For pipelines created by using **Run pipeline** button in the GitLab UI, from the project's **CI/CD > Pipelines** section. | | `webide` | For pipelines created by using the [WebIDE](../../user/project/web_ide/index.md). | @@ -335,7 +335,7 @@ to control when to add jobs to pipelines. In the following example, `job` runs only for: - Git tags -- [Triggers](../triggers/index.md#authentication-tokens) +- [Triggers](../triggers/index.md#configure-cicd-jobs-to-run-in-triggered-pipelines) - [Scheduled pipelines](../pipelines/schedules.md) ```yaml @@ -480,8 +480,8 @@ All files are considered to have changed when a scheduled pipeline runs. If you use multiple keywords with `only` or `except`, the keywords are evaluated as a single conjoined expression. That is: -- `only:` includes the job if **all** of the keys have at least one condition that matches. -- `except:` excludes the job if **any** of the keys have at least one condition that matches. +- `only` includes the job if **all** of the keys have at least one condition that matches. +- `except` excludes the job if **any** of the keys have at least one condition that matches. With `only`, individual keys are logically joined by an `AND`. A job is added to the pipeline if the following is true: @@ -634,7 +634,7 @@ timed rollout 10%: start_in: 30 minutes ``` -To stop the active timer of a delayed job, click the **{time-out}** (**Unschedule**) button. +To stop the active timer of a delayed job, select **Unschedule** (**{time-out}**). This job can no longer be scheduled to run automatically. You can, however, execute the job manually. To start a delayed job immediately, select **Play** (**{play}**). diff --git a/doc/ci/large_repositories/index.md b/doc/ci/large_repositories/index.md index 76e34df1f8c..f3044a03e04 100644 --- a/doc/ci/large_repositories/index.md +++ b/doc/ci/large_repositories/index.md @@ -259,5 +259,5 @@ For very active repositories with a large number of references and files, you ca must be configured per-repository. The pack-objects cache also automatically works for forks. On GitLab.com, where the pack-objects cache is enabled on all Gitaly servers, we found that we no longer need a pre-clone step for `gitlab-org/gitlab` development. - Optimize your CI/CD jobs by seeding repository data in a pre-clone step with the - [`pre_clone_script`](https://docs.gitlab.com/runner/configuration/advanced-configuration.html#the-runners-section) of GitLab Runner. See the - [Runner Cloud for Linux](../runners/runner_cloud/linux_runner_cloud.md#pre-clone-script) for more details. + [`pre_clone_script`](https://docs.gitlab.com/runner/configuration/advanced-configuration.html#the-runners-section) of GitLab Runner. See + [SaaS runners on Linux](../runners/saas/linux_saas_runner.md#pre-clone-script) for details. diff --git a/doc/ci/metrics_reports.md b/doc/ci/metrics_reports.md index 5343af16489..eb302b9ed7f 100644 --- a/doc/ci/metrics_reports.md +++ b/doc/ci/metrics_reports.md @@ -37,7 +37,7 @@ For an MR, the values of these metrics from the feature branch are compared to t ## How to set it up -Add a job that creates a [metrics report](yaml/index.md#artifactsreportsmetrics) (default filename: `metrics.txt`). The file should conform to the [OpenMetrics](https://openmetrics.io/) format. +Add a job that creates a [metrics report](yaml/artifacts_reports.md#artifactsreportsmetrics) (default filename: `metrics.txt`). The file should conform to the [OpenMetrics](https://openmetrics.io/) format. For example: diff --git a/doc/ci/migration/jenkins.md b/doc/ci/migration/jenkins.md index c2c06375d7b..ef6f28e36e5 100644 --- a/doc/ci/migration/jenkins.md +++ b/doc/ci/migration/jenkins.md @@ -78,7 +78,7 @@ There are some high level differences between the products worth mentioning: - on [schedule](../pipelines/schedules.md) - from the [GitLab UI](../pipelines/index.md#run-a-pipeline-manually) - by [API call](../triggers/index.md) - - by [webhook](../triggers/index.md#triggering-a-pipeline-from-a-webhook) + - by [webhook](../triggers/index.md#use-a-webhook) - by [ChatOps](../chatops/index.md) - You can control which jobs run in which cases, depending on how they are triggered, @@ -146,15 +146,15 @@ as well. Jenkins Pipelines are based on [Groovy](https://groovy-lang.org/), so the pipeline specification is written as code. GitLab works a bit differently, we use the more highly structured [YAML](https://yaml.org/) format, which -places scripting elements inside of `script:` blocks separate from the pipeline specification itself. +places scripting elements inside of `script` blocks separate from the pipeline specification itself. This is a strength of GitLab, in that it helps keep the learning curve much simpler to get up and running and avoids some of the problem of unconstrained complexity which can make your Jenkinsfile hard to understand and manage. That said, we do of course still value DRY (don't repeat yourself) principles and want to ensure that -behaviors of your jobs can be codified once and applied as needed. You can use the `extends:` syntax to -[reuse configuration in your jobs](../yaml/index.md#extends), and `include:` can +behaviors of your jobs can be codified once and applied as needed. You can use the `extends` syntax to +[reuse configuration in your jobs](../yaml/index.md#extends), and `include` can be used to [reuse pipeline configurations](../yaml/index.md#include) in pipelines in different projects: @@ -174,7 +174,7 @@ rspec: ## Artifact publishing Artifacts may work a bit differently than you've used them with Jenkins. In GitLab, any job can define -a set of artifacts to be saved by using the `artifacts:` keyword. This can be configured to point to a file +a set of artifacts to be saved by using the `artifacts` keyword. This can be configured to point to a file or set of files that can then be persisted from job to job. Read more on our detailed [artifacts documentation](../pipelines/job_artifacts.md): @@ -271,7 +271,7 @@ default: GitLab CI/CD also lets you define stages, but is a little bit more free-form to configure. The GitLab [`stages` keyword](../yaml/index.md#stages) is a top level setting that enumerates the list of stages, but you are not required to nest individual jobs underneath the `stages` section. Any job defined in the `.gitlab-ci.yml` can be made a part of any stage through use of the -[`stage:` keyword](../yaml/index.md#stage). +[`stage` keyword](../yaml/index.md#stage). Note that, unless otherwise specified, every pipeline is instantiated with a `build`, `test`, and `deploy` stage which are run in that order. Jobs that have no `stage` defined are placed by default in the `test` stage. diff --git a/doc/ci/pipelines/job_artifacts.md b/doc/ci/pipelines/job_artifacts.md index 7ecee5508ef..e47b6dddc5f 100644 --- a/doc/ci/pipelines/job_artifacts.md +++ b/doc/ci/pipelines/job_artifacts.md @@ -48,7 +48,171 @@ is used. If you run two types of pipelines (like branch and scheduled) for the same ref, the pipeline that finishes later creates the job artifact. -For more examples, view the [keyword reference for the `.gitlab-ci.yml` file](../yaml/index.md#artifacts). +To disable artifact passing, define the job with empty [dependencies](../yaml/index.md#dependencies): + +```yaml +job: + stage: build + script: make build + dependencies: [] +``` + +You may want to create artifacts only for tagged releases to avoid filling the +build server storage with temporary build artifacts. For example, use [`rules`](../yaml/index.md#rules) +to create artifacts only for tags: + +```yaml +default-job: + script: + - mvn test -U + rules: + - if: $CI_COMMIT_BRANCH + +release-job: + script: + - mvn package -U + artifacts: + paths: + - target/*.war + rules: + - if: $CI_COMMIT_TAG +``` + +You can use wildcards for directories too. For example, if you want to get all the +files inside the directories that end with `xyz`: + +```yaml +job: + artifacts: + paths: + - path/*xyz/* +``` + +### Use CI/CD variables to define the artifacts name + +You can use [CI/CD variables](../variables/index.md) to dynamically define the +artifacts file's name. + +For example, to create an archive with a name of the current job: + +```yaml +job: + artifacts: + name: "$CI_JOB_NAME" + paths: + - binaries/ +``` + +To create an archive with a name of the current branch or tag including only +the binaries directory: + +```yaml +job: + artifacts: + name: "$CI_COMMIT_REF_NAME" + paths: + - binaries/ +``` + +If your branch-name contains forward slashes +(for example `feature/my-feature`) it's advised to use `$CI_COMMIT_REF_SLUG` +instead of `$CI_COMMIT_REF_NAME` for proper naming of the artifact. + +To create an archive with a name of the current job and the current branch or +tag including only the binaries directory: + +```yaml +job: + artifacts: + name: "$CI_JOB_NAME-$CI_COMMIT_REF_NAME" + paths: + - binaries/ +``` + +To create an archive with a name of the current [stage](../yaml/index.md#stages) and branch name: + +```yaml +job: + artifacts: + name: "$CI_JOB_STAGE-$CI_COMMIT_REF_NAME" + paths: + - binaries/ +``` + +If you use **Windows Batch** to run your shell scripts you must replace +`$` with `%`: + +```yaml +job: + artifacts: + name: "%CI_JOB_STAGE%-%CI_COMMIT_REF_NAME%" + paths: + - binaries/ +``` + +If you use **Windows PowerShell** to run your shell scripts you must replace +`$` with `$env:`: + +```yaml +job: + artifacts: + name: "$env:CI_JOB_STAGE-$env:CI_COMMIT_REF_NAME" + paths: + - binaries/ +``` + +### Exclude files from job artifacts + +Use [`artifacts:exclude`](../yaml/index.md#artifactsexclude) to prevent files from +being added to an artifacts archive. + +For example, to store all files in `binaries/`, but not `*.o` files located in +subdirectories of `binaries/`. + +```yaml +artifacts: + paths: + - binaries/ + exclude: + - binaries/**/*.o +``` + +Unlike [`artifacts:paths`](../yaml/index.md#artifactspaths), `exclude` paths are not recursive. +To exclude all of the contents of a directory, match them explicitly rather +than matching the directory itself. + +For example, to store all files in `binaries/` but nothing located in the `temp/` subdirectory: + +```yaml +artifacts: + paths: + - binaries/ + exclude: + - binaries/temp/**/* +``` + +### Add untracked files to artifacts + +Use [`artifacts:untracked`](../yaml/index.md#artifactsuntracked) to add all Git untracked +files as artifacts (along with the paths defined in [`artifacts:paths`](../yaml/index.md#artifactspaths)). + +Save all Git untracked files and files in `binaries`: + +```yaml +artifacts: + untracked: true + paths: + - binaries/ +``` + +Save all untracked files but [exclude](../yaml/index.md#artifactsexclude) `*.txt`: + +```yaml +artifacts: + untracked: true + exclude: + - "*.txt" +``` ## Download job artifacts @@ -103,6 +267,35 @@ To delete a job: 1. On the top right of the job's log, select **Erase job log** (**{remove}**). 1. On the confirmation dialog, select **OK**. +## Expose job artifacts in the merge request UI + +Use the [`artifacts:expose_as`](../yaml/index.md#artifactsexpose_as) keyword to expose +[job artifacts](../pipelines/job_artifacts.md) in the [merge request](../../user/project/merge_requests/index.md) UI. + +For example, to match a single file: + +```yaml +test: + script: ["echo 'test' > file.txt"] + artifacts: + expose_as: 'artifact 1' + paths: ['file.txt'] +``` + +With this configuration, GitLab adds a link **artifact 1** to the relevant merge request +that points to `file.txt`. To access the link, select **View exposed artifact** +below the pipeline graph in the merge request overview. + +An example that matches an entire directory: + +```yaml +test: + script: ["mkdir test && echo 'test' > test/file.txt"] + artifacts: + expose_as: 'artifact 1' + paths: ['test/'] +``` + ## Retrieve job artifacts for other projects To retrieve a job artifact from a different project, you might need to use a @@ -182,6 +375,14 @@ job artifacts are deleted. > - [Feature flag removed](https://gitlab.com/gitlab-org/gitlab/-/issues/229936) in GitLab 13.4. > - [Made optional with a CI/CD setting](https://gitlab.com/gitlab-org/gitlab/-/issues/241026) in GitLab 13.8. +By default artifacts are always kept for the most recent successful pipeline for +each ref. This means that the latest artifacts do not immediately expire according +to the `expire_in` specification. + +If a new pipeline for the same ref completes successfully, the previous pipeline's +artifacts are deleted according to the `expire_in` configuration. The artifacts +of the new pipeline are kept automatically. + Keeping the latest artifacts can use a large amount of storage space in projects with a lot of jobs or large artifacts. If the latest artifacts are not needed in a project, you can disable this behavior to save space: @@ -194,9 +395,6 @@ a project, you can disable this behavior to save space: You can disable this behavior for all projects on a self-managed instance in the [instance's CI/CD settings](../../user/admin_area/settings/continuous_integration.md#keep-the-latest-artifacts-for-all-jobs-in-the-latest-successful-pipelines). -When you disable the feature, the latest artifacts do not immediately expire. -A new pipeline must run before the latest artifacts can expire and be deleted. - ## Troubleshooting job artifacts ### Error message `No files to upload` diff --git a/doc/ci/pipelines/merge_request_pipelines.md b/doc/ci/pipelines/merge_request_pipelines.md index 119633d38e2..85e5b62b0c4 100644 --- a/doc/ci/pipelines/merge_request_pipelines.md +++ b/doc/ci/pipelines/merge_request_pipelines.md @@ -112,11 +112,11 @@ C: - merge_requests ``` -- `A` and `B` always run, because they get the `only:` rule to execute in all cases. +- `A` and `B` always run, because they get the `only` rule to execute in all cases. - `C` only runs for merge requests. It doesn't run for any pipeline except a merge request pipeline. -In this example, you don't have to add the `only:` rule to all of your jobs to make +In this example, you don't have to add the `only` rule to all of your jobs to make them always run. You can use this format to set up a Review App, which helps to save resources. diff --git a/doc/ci/pipelines/merge_trains.md b/doc/ci/pipelines/merge_trains.md index 6074909a887..593cdb68b3f 100644 --- a/doc/ci/pipelines/merge_trains.md +++ b/doc/ci/pipelines/merge_trains.md @@ -11,6 +11,10 @@ last_update: 2019-07-03 > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/9186) in GitLab 12.0. > - [Squash and merge](../../user/project/merge_requests/squash_and_merge.md) support [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/13001) in GitLab 12.6. +INFO: +Get merge trains and more in GitLab Ultimate. +[Try a free 30-day trial now](https://about.gitlab.com/free-trial/index.html?glm_source=docs.gitlab.com&glm_content=p-ci-cd-external-docs). + For more information about why you might want to use merge trains, read [How merge trains keep your master green](https://about.gitlab.com/blog/2020/01/30/all-aboard-merge-trains/). When [pipelines for merged results](pipelines_for_merged_results.md) are @@ -35,7 +39,8 @@ If the pipeline for the merge request at the front of the train completes succes the changes are merged into the target branch, and the other pipelines continue to run. -To add a merge request to a merge train, you need [permissions](../../user/permissions.md) to push to the target branch. +To add a merge request to a merge train, you need [permissions](../../user/permissions.md) to merge or push to the +target branch. Each merge train can run a maximum of **twenty** pipelines in parallel. If more than twenty merge requests are added to the merge train, the merge requests diff --git a/doc/ci/pipelines/multi_project_pipelines.md b/doc/ci/pipelines/multi_project_pipelines.md index 30b3bc2e277..8a83e7e31f4 100644 --- a/doc/ci/pipelines/multi_project_pipelines.md +++ b/doc/ci/pipelines/multi_project_pipelines.md @@ -213,7 +213,7 @@ In the upstream pipeline: ``` 1. Set the `test` job in the downstream pipeline to inherit the variables from the `build_vars` - job in the upstream project with `needs:`. The `test` job inherits the variables in the + job in the upstream project with `needs`. The `test` job inherits the variables in the `dotenv` report and it can access `BUILD_VERSION` in the script: ```yaml diff --git a/doc/ci/pipelines/parent_child_pipelines.md b/doc/ci/pipelines/parent_child_pipelines.md index 64f4160c963..5e4b707a38c 100644 --- a/doc/ci/pipelines/parent_child_pipelines.md +++ b/doc/ci/pipelines/parent_child_pipelines.md @@ -31,10 +31,9 @@ set of concurrently running child pipelines, but within the same project: - Child pipelines still execute each of their jobs according to a stage sequence, but would be free to continue forward through their stages without waiting for unrelated jobs in the parent pipeline to finish. -- The configuration is split up into smaller child pipeline configurations, which are +- The configuration is split up into smaller child pipeline configurations. Each child pipeline contains only relevant steps which are easier to understand. This reduces the cognitive load to understand the overall configuration. - Imports are done at the child pipeline level, reducing the likelihood of collisions. -- Each pipeline has only relevant steps, making it easier to understand what's going on. Child pipelines work well with other GitLab CI/CD features: @@ -43,7 +42,7 @@ Child pipelines work well with other GitLab CI/CD features: - Since the parent pipeline in `.gitlab-ci.yml` and the child pipeline run as normal pipelines, they can have their own behaviors and sequencing in relation to triggers. -See the [`trigger:`](../yaml/index.md#trigger) keyword documentation for full details on how to +See the [`trigger`](../yaml/index.md#trigger) keyword documentation for full details on how to include the child pipeline configuration. <i class="fa fa-youtube-play youtube" aria-hidden="true"></i> @@ -85,7 +84,7 @@ microservice_a: file: '/path/to/child-pipeline.yml' ``` -The maximum number of entries that are accepted for `trigger:include:` is three. +The maximum number of entries that are accepted for `trigger:include` is three. Similar to [multi-project pipelines](multi_project_pipelines.md#mirror-status-of-a-triggered-pipeline-in-the-trigger-job), we can set the parent pipeline to depend on the status of the child pipeline upon completion: diff --git a/doc/ci/pipelines/pipeline_architectures.md b/doc/ci/pipelines/pipeline_architectures.md index 1b23727b142..3ff22a16900 100644 --- a/doc/ci/pipelines/pipeline_architectures.md +++ b/doc/ci/pipelines/pipeline_architectures.md @@ -211,7 +211,7 @@ trigger_b: ``` Example child `a` pipeline configuration, located in `/a/.gitlab-ci.yml`, making -use of the DAG `needs:` keyword: +use of the DAG `needs` keyword: ```yaml stages: @@ -240,7 +240,7 @@ deploy_a: ``` Example child `b` pipeline configuration, located in `/b/.gitlab-ci.yml`, making -use of the DAG `needs:` keyword: +use of the DAG `needs` keyword: ```yaml stages: diff --git a/doc/ci/pipelines/pipelines_for_merged_results.md b/doc/ci/pipelines/pipelines_for_merged_results.md index 2acef9be557..718519faf48 100644 --- a/doc/ci/pipelines/pipelines_for_merged_results.md +++ b/doc/ci/pipelines/pipelines_for_merged_results.md @@ -10,6 +10,10 @@ last_update: 2019-07-03 > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/7380) in GitLab 11.10. +INFO: +Get these pipelines and more in GitLab Ultimate. +[Try a free 30-day trial now](https://about.gitlab.com/free-trial/index.html?glm_source=docs.gitlab.com&glm_content=p-ci-cd-external-docs). + When you submit a merge request, you are requesting to merge changes from a source branch into a target branch. By default, the CI pipeline runs jobs against the source branch. diff --git a/doc/ci/pipelines/settings.md b/doc/ci/pipelines/settings.md index a8ecb5e0d74..cf470836e32 100644 --- a/doc/ci/pipelines/settings.md +++ b/doc/ci/pipelines/settings.md @@ -134,7 +134,7 @@ For example: - `my/path/.my-custom-file.yml@mygroup/another-project` - `my/path/.my-custom-file.yml@mygroup/another-project:refname` -If the configuration file is in a separate project, you can more set more granular permissions. For example: +If the configuration file is in a separate project, you can set more granular permissions. For example: - Create a public project to host the configuration file. - Give write permissions on the project only to users who are allowed to edit the file. @@ -267,7 +267,7 @@ when merging a merge request would cause the project's test coverage to decline. Follow these steps to enable the `Coverage-Check` MR approval rule: -1. Set up a [`coverage:`](../yaml/index.md#coverage) regular expression for all jobs you want to include in the overall coverage value. +1. Set up a [`coverage`](../yaml/index.md#coverage) regular expression for all jobs you want to include in the overall coverage value. 1. Go to your project and select **Settings > General**. 1. Expand **Merge request approvals**. 1. Select **Enable** next to the `Coverage-Check` approval rule. diff --git a/doc/ci/resource_groups/index.md b/doc/ci/resource_groups/index.md index 25dacc9c437..d31fb5561e9 100644 --- a/doc/ci/resource_groups/index.md +++ b/doc/ci/resource_groups/index.md @@ -52,7 +52,7 @@ deploy: ``` With this configuration, the safety on the deployments is assured while you -can still run `build` jobs concurrently for maximizing the pipeline efficency. +can still run `build` jobs concurrently for maximizing the pipeline efficiency. ## Requirements diff --git a/doc/ci/runners/build_cloud/linux_build_cloud.md b/doc/ci/runners/build_cloud/linux_build_cloud.md index e8bad31c821..2892a30cd2e 100644 --- a/doc/ci/runners/build_cloud/linux_build_cloud.md +++ b/doc/ci/runners/build_cloud/linux_build_cloud.md @@ -1,9 +1,9 @@ --- -redirect_to: '../runner_cloud/linux_runner_cloud.md' +redirect_to: '../saas/linux_saas_runner.md' remove_date: '2022-02-05' --- -This document was moved to [another location](../runner_cloud/linux_runner_cloud.md). +This document was moved to [another location](../saas/linux_saas_runner.md). <!-- This redirect file can be deleted after 2022-02-05. --> <!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/#move-or-rename-a-page --> diff --git a/doc/ci/runners/build_cloud/macos/environment.md b/doc/ci/runners/build_cloud/macos/environment.md index aaef0d07098..a534e87cc34 100644 --- a/doc/ci/runners/build_cloud/macos/environment.md +++ b/doc/ci/runners/build_cloud/macos/environment.md @@ -1,9 +1,9 @@ --- -redirect_to: '../../runner_cloud/macos/environment.md' +redirect_to: '../../saas/macos/environment.md' remove_date: '2022-02-05' --- -This document was moved to [another location](../../runner_cloud/macos/environment.md). +This document was moved to [another location](../../saas/macos/environment.md). <!-- This redirect file can be deleted after 2022-02-05. --> <!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/#move-or-rename-a-page --> diff --git a/doc/ci/runners/build_cloud/macos_build_cloud.md b/doc/ci/runners/build_cloud/macos_build_cloud.md index e478f93f34c..50b7e0cfb79 100644 --- a/doc/ci/runners/build_cloud/macos_build_cloud.md +++ b/doc/ci/runners/build_cloud/macos_build_cloud.md @@ -1,9 +1,9 @@ --- -redirect_to: '../runner_cloud/macos_runner_cloud.md' +redirect_to: '../saas/macos_saas_runner.md' remove_date: '2022-02-05' --- -This document was moved to [another location](../runner_cloud/macos_runner_cloud.md). +This document was moved to [another location](../saas/macos_saas_runner.md). <!-- This redirect file can be deleted after 2022-02-05. --> <!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/#move-or-rename-a-page --> diff --git a/doc/ci/runners/build_cloud/windows_build_cloud.md b/doc/ci/runners/build_cloud/windows_build_cloud.md index 8d57ecf27ed..fb64938eb9f 100644 --- a/doc/ci/runners/build_cloud/windows_build_cloud.md +++ b/doc/ci/runners/build_cloud/windows_build_cloud.md @@ -1,9 +1,9 @@ --- -redirect_to: '../runner_cloud/windows_runner_cloud.md' +redirect_to: '../saas/windows_saas_runner.md' remove_date: '2022-02-05' --- -This document was moved to [another location](../runner_cloud/windows_runner_cloud.md). +This document was moved to [another location](../saas/windows_saas_runner.md). <!-- This redirect file can be deleted after 2022-02-05. --> <!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/#move-or-rename-a-page --> diff --git a/doc/ci/runners/configure_runners.md b/doc/ci/runners/configure_runners.md index 9e30f4dbf4d..b2885262e9d 100644 --- a/doc/ci/runners/configure_runners.md +++ b/doc/ci/runners/configure_runners.md @@ -297,6 +297,7 @@ globally or for individual jobs: - [`TRANSFER_METER_FREQUENCY`](#artifact-and-cache-settings) (artifact/cache meter update frequency) - [`ARTIFACT_COMPRESSION_LEVEL`](#artifact-and-cache-settings) (artifact archiver compression level) - [`CACHE_COMPRESSION_LEVEL`](#artifact-and-cache-settings) (cache archiver compression level) +- [`CACHE_REQUEST_TIMEOUT`](#artifact-and-cache-settings) (cache request timeout) You can also use variables to configure how many times a runner [attempts certain stages of job execution](#job-stages-attempts). @@ -637,7 +638,10 @@ For [GitLab Pages](../../user/project/pages/index.md) to serve should use the `ARTIFACT_COMPRESSION_LEVEL: fastest` setting, as only uncompressed zip archives support this feature. -A meter can also be enabled to provide the rate of transfer for uploads and downloads. +A meter can be enabled to provide the rate of transfer for uploads and downloads. + +You can set a maximum time for cache upload and download with the `CACHE_REQUEST_TIMEOUT` setting. +This setting can be useful when slow cache uploads substantially increase the duration of your job. ```yaml variables: @@ -649,6 +653,9 @@ variables: # Use no compression for caches CACHE_COMPRESSION_LEVEL: "fastest" + + # Set maximum duration of cache upload and download + CACHE_REQUEST_TIMEOUT: 5 ``` | Variable | Description | @@ -656,3 +663,4 @@ variables: | `TRANSFER_METER_FREQUENCY` | Specify how often to print the meter's transfer rate. It can be set to a duration (for example, `1s` or `1m30s`). A duration of `0` disables the meter (default). When a value is set, the pipeline shows a progress meter for artifact and cache uploads and downloads. | | `ARTIFACT_COMPRESSION_LEVEL` | To adjust compression ratio, set to `fastest`, `fast`, `default`, `slow`, or `slowest`. This setting works with the Fastzip archiver only, so the GitLab Runner feature flag [`FF_USE_FASTZIP`](https://docs.gitlab.com/runner/configuration/feature-flags.html#available-feature-flags) must also be enabled. | | `CACHE_COMPRESSION_LEVEL` | To adjust compression ratio, set to `fastest`, `fast`, `default`, `slow`, or `slowest`. This setting works with the Fastzip archiver only, so the GitLab Runner feature flag [`FF_USE_FASTZIP`](https://docs.gitlab.com/runner/configuration/feature-flags.html#available-feature-flags) must also be enabled. | +| `CACHE_REQUEST_TIMEOUT` | Configure the maximum duration of cache upload and download operations for a single job in minutes. Default is `10` minutes. | diff --git a/doc/ci/runners/index.md b/doc/ci/runners/index.md index d408bc46609..b4e9fe818cf 100644 --- a/doc/ci/runners/index.md +++ b/doc/ci/runners/index.md @@ -5,12 +5,12 @@ info: To determine the technical writer assigned to the Stage/Group associated w type: reference --- -# GitLab Runner Cloud **(FREE)** +# Runner SaaS **(FREE SAAS)** -If you are using self-managed GitLab or you want to use your own runners on GitLab.com, you can +If you are using self-managed GitLab or you use GitLab.com but want to use your own runners, you can [install and configure your own runners](https://docs.gitlab.com/runner/install/). -If you are using GitLab SaaS (GitLab.com), your CI jobs automatically run on runners in the GitLab Runner Cloud. +If you are using GitLab SaaS (GitLab.com), your CI jobs automatically run on runners provided by GitLab. No configuration is required. Your jobs can run on: - [Linux runners](build_cloud/linux_build_cloud.md). diff --git a/doc/ci/runners/runner_cloud/linux_runner_cloud.md b/doc/ci/runners/runner_cloud/linux_runner_cloud.md index d0fedfcabb2..2892a30cd2e 100644 --- a/doc/ci/runners/runner_cloud/linux_runner_cloud.md +++ b/doc/ci/runners/runner_cloud/linux_runner_cloud.md @@ -1,186 +1,9 @@ --- -stage: Verify -group: Runner -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +redirect_to: '../saas/linux_saas_runner.md' +remove_date: '2022-02-05' --- -# Runner Cloud for Linux **(FREE)** +This document was moved to [another location](../saas/linux_saas_runner.md). -Runner Cloud runners for Linux run in autoscale mode and are powered by Google Cloud Platform. - -Autoscaling means reduced queue times to spin up CI/CD jobs, and isolated VMs for each job, thus maximizing security. These shared runners are available on GitLab.com. - -GitLab offers Ultimate tier capabilities and included CI/CD minutes per group per month for our [Open Source](https://about.gitlab.com/solutions/open-source/join/), [Education](https://about.gitlab.com/solutions/education/), and [Startups](https://about.gitlab.com/solutions/startups/) programs. For private projects, GitLab offers various [plans](https://about.gitlab.com/pricing/), starting with a Free tier. - -All your CI/CD jobs run on [n1-standard-1 instances](https://cloud.google.com/compute/docs/machine-types) with 3.75GB of RAM, Google COS and the latest Docker Engine -installed. Instances provide 1 vCPU and 25GB of HDD disk space. The default -region of the VMs is US East1. -Each instance is used only for one job. This ensures that any sensitive data left on the system can't be accessed by other people's CI/CD jobs. - -NOTE: -The final disk space your jobs can use will be less than 25GB. Some disk space allocated to the instance will be occupied by the operating system, the Docker image, and a copy of your cloned repository. - -The `gitlab-shared-runners-manager-X.gitlab.com` fleet of runners are dedicated for GitLab projects as well as community forks of them. They use a slightly larger machine type (n1-standard-2) and have a bigger SSD disk size. They don't run untagged jobs and unlike the general fleet of shared runners, the instances are re-used up to 40 times. - -Jobs handled by the shared runners on GitLab.com (`shared-runners-manager-X.gitlab.com`), -**time out after 3 hours**, regardless of the timeout configured in a -project. Check the issues [#4010](https://gitlab.com/gitlab-com/infrastructure/-/issues/4010) and [#4070](https://gitlab.com/gitlab-com/infrastructure/-/issues/4070) for the reference. - -Below are the runners' settings. - -| Setting | GitLab.com | Default | -| ----------- | ----------------- | ---------- | -| Executor | `docker+machine` | - | -| Default Docker image | `ruby:2.5` | - | -| `privileged` (run [Docker in Docker](https://hub.docker.com/_/docker/)) | `true` | `false` | - -These runners share a [distributed cache](https://docs.gitlab.com/runner/configuration/autoscale.html#distributed-runners-caching) through use of a Google Cloud Storage (GCS) bucket. Cache contents not updated within the last 14 days are automatically removed through use of an [object lifecycle management policy](https://cloud.google.com/storage/docs/lifecycle). - -## Pre-clone script - -Cloud runners for Linux provide a way to run commands in a CI -job before the runner attempts to run `git init` and `git fetch` to -download a GitLab repository. The -[`pre_clone_script`](https://docs.gitlab.com/runner/configuration/advanced-configuration.html#the-runners-section) -can be used for: - -- Seeding the build directory with repository data -- Sending a request to a server -- Downloading assets from a CDN -- Any other commands that must run before the `git init` - -To use this feature, define a [CI/CD variable](../../../ci/variables/index.md#custom-cicd-variables) called -`CI_PRE_CLONE_SCRIPT` that contains a bash script. - -NOTE: -The `CI_PRE_CLONE_SCRIPT` variable does not work on Windows runners. - -### Pre-clone script example - -This example was used in the `gitlab-org/gitlab` project until November 2021. -The project no longer uses this optimization because the [pack-objects cache](../../../administration/gitaly/configure_gitaly.md#pack-objects-cache) -lets Gitaly serve the full CI/CD fetch traffic. See [Git fetch caching](../../../development/pipelines.md#git-fetch-caching). - -The `CI_PRE_CLONE_SCRIPT` was defined as a project CI/CD variable: - -```shell -( - echo "Downloading archived master..." - wget -O /tmp/gitlab.tar.gz https://storage.googleapis.com/gitlab-ci-git-repo-cache/project-278964/gitlab-master-shallow.tar.gz - - if [ ! -f /tmp/gitlab.tar.gz ]; then - echo "Repository cache not available, cloning a new directory..." - exit - fi - - rm -rf $CI_PROJECT_DIR - echo "Extracting tarball into $CI_PROJECT_DIR..." - mkdir -p $CI_PROJECT_DIR - cd $CI_PROJECT_DIR - tar xzf /tmp/gitlab.tar.gz - rm -f /tmp/gitlab.tar.gz - chmod a+w $CI_PROJECT_DIR -) -``` - -The first step of the script downloads `gitlab-master.tar.gz` from Google Cloud Storage. -There was a [GitLab CI/CD job named `cache-repo`](https://gitlab.com/gitlab-org/gitlab/-/blob/5fb40526c8c8aaafc5f92eab36d5bbddaca3893d/.gitlab/ci/cache-repo.gitlab-ci.yml) -that was responsible for keeping that archive up-to-date. Every two hours on a scheduled pipeline, -it did the following: - -1. Create a fresh clone of the `gitlab-org/gitlab` repository on GitLab.com. -1. Save the data as a `.tar.gz`. -1. Upload it into the Google Cloud Storage bucket. - -When a job ran with this configuration, the output looked similar to: - -```shell -$ eval "$CI_PRE_CLONE_SCRIPT" -Downloading archived master... -Extracting tarball into /builds/gitlab-org/gitlab... -Fetching changes... -Reinitialized existing Git repository in /builds/gitlab-org/gitlab/.git/ -``` - -The `Reinitialized existing Git repository` message shows that -the pre-clone step worked. The runner runs `git init`, which -overwrites the Git configuration with the appropriate settings to fetch -from the GitLab repository. - -`CI_REPO_CACHE_CREDENTIALS` must contain the Google Cloud service account -JSON for uploading to the `gitlab-ci-git-repo-cache` bucket. - -Note that this bucket should be located in the same continent as the -runner, or [you can incur network egress charges](https://cloud.google.com/storage/pricing). - -## `config.toml` - -The full contents of our `config.toml` are: - -NOTE: -Settings that are not public are shown as `X`. - -**Google Cloud Platform** - -```toml -concurrent = X -check_interval = 1 -metrics_server = "X" -sentry_dsn = "X" - -[[runners]] - name = "docker-auto-scale" - request_concurrency = X - url = "https://gitlab.com/" - token = "SHARED_RUNNER_TOKEN" - pre_clone_script = "eval \"$CI_PRE_CLONE_SCRIPT\"" - executor = "docker+machine" - environment = [ - "DOCKER_DRIVER=overlay2", - "DOCKER_TLS_CERTDIR=" - ] - limit = X - [runners.docker] - image = "ruby:2.5" - privileged = true - volumes = [ - "/certs/client", - "/dummy-sys-class-dmi-id:/sys/class/dmi/id:ro" # Make kaniko builds work on GCP. - ] - [runners.machine] - IdleCount = 50 - IdleTime = 3600 - MaxBuilds = 1 # For security reasons we delete the VM after job has finished so it's not reused. - MachineName = "srm-%s" - MachineDriver = "google" - MachineOptions = [ - "google-project=PROJECT", - "google-disk-size=25", - "google-machine-type=n1-standard-1", - "google-username=core", - "google-tags=gitlab-com,srm", - "google-use-internal-ip", - "google-zone=us-east1-d", - "engine-opt=mtu=1460", # Set MTU for container interface, for more information check https://gitlab.com/gitlab-org/gitlab-runner/-/issues/3214#note_82892928 - "google-machine-image=PROJECT/global/images/IMAGE", - "engine-opt=ipv6", # This will create IPv6 interfaces in the containers. - "engine-opt=fixed-cidr-v6=fc00::/7", - "google-operation-backoff-initial-interval=2" # Custom flag from forked docker-machine, for more information check https://github.com/docker/machine/pull/4600 - ] - [[runners.machine.autoscaling]] - Periods = ["* * * * * sat,sun *"] - Timezone = "UTC" - IdleCount = 70 - IdleTime = 3600 - [[runners.machine.autoscaling]] - Periods = ["* 30-59 3 * * * *", "* 0-30 4 * * * *"] - Timezone = "UTC" - IdleCount = 700 - IdleTime = 3600 - [runners.cache] - Type = "gcs" - Shared = true - [runners.cache.gcs] - CredentialsFile = "/path/to/file" - BucketName = "bucket-name" -``` +<!-- This redirect file can be deleted after 2022-02-05. --> +<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/#move-or-rename-a-page --> diff --git a/doc/ci/runners/runner_cloud/macos/environment.md b/doc/ci/runners/runner_cloud/macos/environment.md index ddefad775c1..37ad21c28fc 100644 --- a/doc/ci/runners/runner_cloud/macos/environment.md +++ b/doc/ci/runners/runner_cloud/macos/environment.md @@ -1,43 +1,9 @@ --- -stage: Verify -group: Runner -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +redirect_to: '../../saas/macos/environment.md' +remove_date: '2022-02-05' --- -# VM instances and images for Runner Cloud for macOS **(FREE)** +This document was moved to [another location](../../saas/macos/environment.md). -When you use the Runner Cloud for macOS: - -- Each of your jobs runs in a newly provisioned VM, which is dedicated to the specific job. -- The VM is active only for the duration of the job and immediately deleted. - -## VM types - -The virtual machine where your job runs has `sudo` access with no password. -For the Beta, there is only one available machine type, `gbc-macos-large`. - -| Instance type | vCPUS | Memory (GB) | -| --------- | --- | ------- | -| `gbc-macos-large` | 4 | 10 | - -## VM images - -You can execute your build on one of the following images. -You specify this image in your `.gitlab-ci.yml` file. - -Each image is running a specific version of macOS and Xcode. - -| VM image | Included software | -|---------------------------|--------------------| -| macos-10.13-xcode-7 | <https://gitlab.com/gitlab-org/ci-cd/shared-runners/images/macstadium/orka/-/blob/main/toolchain/high-sierra.yml> | -| macos-10.13-xcode-8 | <https://gitlab.com/gitlab-org/ci-cd/shared-runners/images/macstadium/orka/-/blob/main/toolchain/high-sierra.yml> | -| macos-10.13-xcode-9 | <https://gitlab.com/gitlab-org/ci-cd/shared-runners/images/macstadium/orka/-/blob/main/toolchain/high-sierra.yml> | -| macos-10.14-xcode-10 | <https://gitlab.com/gitlab-org/ci-cd/shared-runners/images/macstadium/orka/-/blob/main/toolchain/mojave.yml> | -| macos-10.15-xcode-11 | <https://gitlab.com/gitlab-org/ci-cd/shared-runners/images/macstadium/orka/-/blob/main/toolchain/catalina.yml> | -| macos-11-xcode-12 | <https://gitlab.com/gitlab-org/ci-cd/shared-runners/images/macstadium/orka/-/blob/main/toolchain/big-sur.yml> | - -### Image update policy - -- Support for new macOS versions is planned. -- Additional details on the support policy and image update release process are documented - [in this project](https://gitlab.com/gitlab-org/ci-cd/shared-runners/images/macstadium/orka/-/blob/55bf59c8fa88712960afff2bf6ecc5daa879a8f5/docs/overview.md#os-images). +<!-- This redirect file can be deleted after 2022-02-05. --> +<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/#move-or-rename-a-page -->
\ No newline at end of file diff --git a/doc/ci/runners/runner_cloud/macos_runner_cloud.md b/doc/ci/runners/runner_cloud/macos_runner_cloud.md index 332284fa8c1..50b7e0cfb79 100644 --- a/doc/ci/runners/runner_cloud/macos_runner_cloud.md +++ b/doc/ci/runners/runner_cloud/macos_runner_cloud.md @@ -1,62 +1,9 @@ --- -stage: Verify -group: Runner -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +redirect_to: '../saas/macos_saas_runner.md' +remove_date: '2022-02-05' --- -# Runner Cloud for macOS (Beta) **(FREE SAAS)** +This document was moved to [another location](../saas/macos_saas_runner.md). -The Runner Cloud for macOS Beta provides on-demand runners integrated with GitLab SaaS [CI/CD](../../../ci/index.md). -Use these runners to build, test, and deploy apps for the Apple ecosystem (macOS, iOS, tvOS). You can take advantage -of all the capabilities of the GitLab single DevOps platform and not have to manage or operate a -build environment. - -Cloud runners for macOS are in [Beta](https://about.gitlab.com/handbook/product/gitlab-the-product/#beta) -and shouldn't be relied upon for mission-critical production jobs. - -## Quickstart - -To start using Runner Cloud for macOS Beta, you must submit an access request [issue](https://gitlab.com/gitlab-com/macos-buildcloud-runners-beta/-/issues/new?issuable_template=beta_access_request). After your -access has been granted and your build environment configured, you must configure your -`.gitlab-ci.yml` pipeline file: - -1. Add a `.gitlab-ci.yml` file to your project repository. -1. Specify the [image](macos/environment.md#vm-images) you want to use. -1. Commit a change to your repository. - -The runners automatically run your build. - -## Example `.gitlab-ci.yml` file - -The following sample `.gitlab-ci.yml` file shows how to start using the runners for macOS: - -```yaml -.macos_buildcloud_runners: - tags: - - shared-macos-amd64 - image: macos-11-xcode-12 - -stages: - - build - - test - -before_script: - - echo "started by ${GITLAB_USER_NAME}" - -build: - extends: - - .macos_buildcloud_runners - stage: build - script: - - echo "running scripts in the build job" - -test: - extends: - - .macos_buildcloud_runners - stage: test - script: - - echo "running scripts in the test job" -``` - -NOTE: -During the Beta period, the architecture of this solution will change. Rather than the jobs running on a specific VM instance, they will run on an ephemeral VM instance that is created by an autoscaling instance, known as the Runner Manager. We will notify all Beta participants of any downtime required to do this work. +<!-- This redirect file can be deleted after 2022-02-05. --> +<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/#move-or-rename-a-page --> diff --git a/doc/ci/runners/runner_cloud/windows_runner_cloud.md b/doc/ci/runners/runner_cloud/windows_runner_cloud.md index ef4d4076c91..fb64938eb9f 100644 --- a/doc/ci/runners/runner_cloud/windows_runner_cloud.md +++ b/doc/ci/runners/runner_cloud/windows_runner_cloud.md @@ -1,155 +1,9 @@ --- -stage: Verify -group: Runner -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +redirect_to: '../saas/windows_saas_runner.md' +remove_date: '2022-02-05' --- -# Runner Cloud for Windows (beta) **(FREE)** +This document was moved to [another location](../saas/windows_saas_runner.md). -Runner Cloud runners for Windows are in [beta](https://about.gitlab.com/handbook/product/gitlab-the-product/#beta) -and shouldn't be used for production workloads. - -During this beta period, the [shared runner pipeline quota](../../../user/admin_area/settings/continuous_integration.md#shared-runners-pipeline-minutes-quota) -applies for groups and projects in the same manner as Linux runners. This may -change when the beta period ends, as discussed in this [related issue](https://gitlab.com/gitlab-org/gitlab/-/issues/30834). - -Windows runners on GitLab.com autoscale by launching virtual machines on -the Google Cloud Platform. This solution uses an -[autoscaling driver](https://gitlab.com/gitlab-org/ci-cd/custom-executor-drivers/autoscaler/tree/master/docs/readme.md) -developed by GitLab for the [custom executor](https://docs.gitlab.com/runner/executors/custom.html). -Windows runners execute your CI/CD jobs on `n1-standard-2` instances with -2 vCPUs and 7.5 GB RAM. You can find a full list of available Windows packages in -the [package documentation](https://gitlab.com/gitlab-org/ci-cd/shared-runners/images/gcp/windows-containers/blob/main/cookbooks/preinstalled-software/README.md). - -We want to keep iterating to get Windows runners in a stable state and -[generally available](https://about.gitlab.com/handbook/product/gitlab-the-product/#generally-available-ga). -You can follow our work towards this goal in the -[related epic](https://gitlab.com/groups/gitlab-org/-/epics/2162). - -## Configuration - -The full contents of our `config.toml` are: - -NOTE: -Settings that aren't public are shown as `X`. - -```toml -concurrent = X -check_interval = 3 - -[[runners]] - name = "windows-runner" - url = "https://gitlab.com/" - token = "TOKEN" - executor = "custom" - builds_dir = "C:\\GitLab-Runner\\builds" - cache_dir = "C:\\GitLab-Runner\\cache" - shell = "powershell" - [runners.custom] - config_exec = "C:\\GitLab-Runner\\autoscaler\\autoscaler.exe" - config_args = ["--config", "C:\\GitLab-Runner\\autoscaler\\config.toml", "custom", "config"] - prepare_exec = "C:\\GitLab-Runner\\autoscaler\\autoscaler.exe" - prepare_args = ["--config", "C:\\GitLab-Runner\\autoscaler\\config.toml", "custom", "prepare"] - run_exec = "C:\\GitLab-Runner\\autoscaler\\autoscaler.exe" - run_args = ["--config", "C:\\GitLab-Runner\\autoscaler\\config.toml", "custom", "run"] - cleanup_exec = "C:\\GitLab-Runner\\autoscaler\\autoscaler.exe" - cleanup_args = ["--config", "C:\\GitLab-Runner\\autoscaler\\config.toml", "custom", "cleanup"] -``` - -The full contents of our `autoscaler/config.toml` are: - -```toml -Provider = "gcp" -Executor = "winrm" -OS = "windows" -LogLevel = "info" -LogFormat = "text" -LogFile = "C:\\GitLab-Runner\\autoscaler\\autoscaler.log" -VMTag = "windows" - -[GCP] - ServiceAccountFile = "PATH" - Project = "some-project-df9323" - Zone = "us-east1-c" - MachineType = "n1-standard-2" - Image = "IMAGE" - DiskSize = 50 - DiskType = "pd-standard" - Subnetwork = "default" - Network = "default" - Tags = ["TAGS"] - Username = "gitlab_runner" - -[WinRM] - MaximumTimeout = 3600 - ExecutionMaxRetries = 0 - -[ProviderCache] - Enabled = true - Directory = "C:\\GitLab-Runner\\autoscaler\\machines" -``` - -## Example `.gitlab-ci.yml` file - -Below is a sample `.gitlab-ci.yml` file that shows how to start using the runners for Windows: - -```yaml -.shared_windows_runners: - tags: - - shared-windows - - windows - - windows-1809 - -stages: - - build - - test - -before_script: - - Set-Variable -Name "time" -Value (date -Format "%H:%m") - - echo ${time} - - echo "started by ${GITLAB_USER_NAME}" - -build: - extends: - - .shared_windows_runners - stage: build - script: - - echo "running scripts in the build job" - -test: - extends: - - .shared_windows_runners - stage: test - script: - - echo "running scripts in the test job" -``` - -## Limitations and known issues - -- All the limitations mentioned in our [beta - definition](https://about.gitlab.com/handbook/product/#beta). -- The average provisioning time for a new Windows VM is 5 minutes. - This means that you may notice slower build start times - on the Windows runner fleet during the beta. In a future - release we intend to update the autoscaler to enable - the pre-provisioning of virtual machines. This is intended to significantly reduce - the time it takes to provision a VM on the Windows fleet. You can - follow along in the [related issue](https://gitlab.com/gitlab-org/ci-cd/custom-executor-drivers/autoscaler/-/issues/32). -- The Windows runner fleet may be unavailable occasionally - for maintenance or updates. -- The Windows runner virtual machine instances do not use the - GitLab Docker executor. This means that you can't specify - [`image`](../../../ci/yaml/index.md#image) or [`services`](../../../ci/yaml/index.md#services) in - your pipeline configuration. -- For the beta release, we have included a set of software packages in - the base VM image. If your CI job requires additional software that's - not included in this list, then you must add installation - commands to [`before_script`](../../../ci/yaml/index.md#before_script) or [`script`](../../../ci/yaml/index.md#script) to install the required - software. Note that each job runs on a new VM instance, so the - installation of additional software packages needs to be repeated for - each job in your pipeline. -- The job may stay in a pending state for longer than the - Linux runners. -- There is the possibility that we introduce breaking changes which will - require updates to pipelines that are using the Windows runner - fleet. +<!-- This redirect file can be deleted after 2022-02-05. --> +<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/#move-or-rename-a-page --> diff --git a/doc/ci/runners/saas/linux_saas_runner.md b/doc/ci/runners/saas/linux_saas_runner.md new file mode 100644 index 00000000000..4d1e628b8e7 --- /dev/null +++ b/doc/ci/runners/saas/linux_saas_runner.md @@ -0,0 +1,188 @@ +--- +stage: Verify +group: Runner +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +--- + +# SaaS runners on Linux **(FREE SAAS)** + +SaaS runners on Linux are autoscaled ephemeral Google Cloud Platform virtual machines. + +Autoscaling means reduced queue times to spin up CI/CD jobs, and isolated VMs for each job, thus maximizing security. These shared runners are available on GitLab.com. + +GitLab offers Ultimate tier capabilities and included CI/CD minutes per group per month for our [Open Source](https://about.gitlab.com/solutions/open-source/join/), [Education](https://about.gitlab.com/solutions/education/), and [Startups](https://about.gitlab.com/solutions/startups/) programs. For private projects, GitLab offers various [plans](https://about.gitlab.com/pricing/), starting with a Free tier. + +All your CI/CD jobs run on [n1-standard-1 instances](https://cloud.google.com/compute/docs/machine-types) with 3.75GB of RAM, Google COS and the latest Docker Engine +installed. Instances provide 1 vCPU and 25GB of HDD disk space. The default +region of the VMs is US East1. +Each instance is used only for one job. This ensures that any sensitive data left on the system can't be accessed by other people's CI/CD jobs. + +NOTE: +The final disk space your jobs can use will be less than 25GB. Some disk space allocated to the instance will be occupied by the operating system, the Docker image, and a copy of your cloned repository. + +The `gitlab-shared-runners-manager-X.gitlab.com` fleet of runners are dedicated for GitLab projects as well as community forks of them. They use a slightly larger machine type (n1-standard-2) and have a bigger SSD disk size. They don't run untagged jobs and unlike the general fleet of shared runners, the instances are re-used up to 40 times. + +Jobs handled by shared runners on GitLab.com (`shared-runners-manager-X.gitlab.com`) +**time out after 3 hours**, regardless of the timeout configured in a +project. Check issue [#4010](https://gitlab.com/gitlab-com/infrastructure/-/issues/4010) and [#4070](https://gitlab.com/gitlab-com/infrastructure/-/issues/4070) for the reference. + +Jobs handled by shared runners on Windows and macOS on GitLab.com **time out after 1 hour** while this service is in the Beta stage. + +Below are the runners' settings. + +| Setting | GitLab.com | Default | +| ----------- | ----------------- | ---------- | +| Executor | `docker+machine` | - | +| Default Docker image | `ruby:2.5` | - | +| `privileged` (run [Docker in Docker](https://hub.docker.com/_/docker/)) | `true` | `false` | + +These runners share a [distributed cache](https://docs.gitlab.com/runner/configuration/autoscale.html#distributed-runners-caching) through use of a Google Cloud Storage (GCS) bucket. Cache contents not updated within the last 14 days are automatically removed through use of an [object lifecycle management policy](https://cloud.google.com/storage/docs/lifecycle). + +## Pre-clone script + +With SaaS runners on Linux, you can run commands in a CI +job before the runner attempts to run `git init` and `git fetch` to +download a GitLab repository. The +[`pre_clone_script`](https://docs.gitlab.com/runner/configuration/advanced-configuration.html#the-runners-section) +can be used for: + +- Seeding the build directory with repository data +- Sending a request to a server +- Downloading assets from a CDN +- Any other commands that must run before the `git init` + +To use this feature, define a [CI/CD variable](../../../ci/variables/index.md#custom-cicd-variables) called +`CI_PRE_CLONE_SCRIPT` that contains a bash script. + +NOTE: +The `CI_PRE_CLONE_SCRIPT` variable does not work on Windows runners. + +### Pre-clone script example + +This example was used in the `gitlab-org/gitlab` project until November 2021. +The project no longer uses this optimization because the [pack-objects cache](../../../administration/gitaly/configure_gitaly.md#pack-objects-cache) +lets Gitaly serve the full CI/CD fetch traffic. See [Git fetch caching](../../../development/pipelines.md#git-fetch-caching). + +The `CI_PRE_CLONE_SCRIPT` was defined as a project CI/CD variable: + +```shell +( + echo "Downloading archived master..." + wget -O /tmp/gitlab.tar.gz https://storage.googleapis.com/gitlab-ci-git-repo-cache/project-278964/gitlab-master-shallow.tar.gz + + if [ ! -f /tmp/gitlab.tar.gz ]; then + echo "Repository cache not available, cloning a new directory..." + exit + fi + + rm -rf $CI_PROJECT_DIR + echo "Extracting tarball into $CI_PROJECT_DIR..." + mkdir -p $CI_PROJECT_DIR + cd $CI_PROJECT_DIR + tar xzf /tmp/gitlab.tar.gz + rm -f /tmp/gitlab.tar.gz + chmod a+w $CI_PROJECT_DIR +) +``` + +The first step of the script downloads `gitlab-master.tar.gz` from Google Cloud Storage. +There was a [GitLab CI/CD job named `cache-repo`](https://gitlab.com/gitlab-org/gitlab/-/blob/5fb40526c8c8aaafc5f92eab36d5bbddaca3893d/.gitlab/ci/cache-repo.gitlab-ci.yml) +that was responsible for keeping that archive up-to-date. Every two hours on a scheduled pipeline, +it did the following: + +1. Create a fresh clone of the `gitlab-org/gitlab` repository on GitLab.com. +1. Save the data as a `.tar.gz`. +1. Upload it into the Google Cloud Storage bucket. + +When a job ran with this configuration, the output looked similar to: + +```shell +$ eval "$CI_PRE_CLONE_SCRIPT" +Downloading archived master... +Extracting tarball into /builds/gitlab-org/gitlab... +Fetching changes... +Reinitialized existing Git repository in /builds/gitlab-org/gitlab/.git/ +``` + +The `Reinitialized existing Git repository` message shows that +the pre-clone step worked. The runner runs `git init`, which +overwrites the Git configuration with the appropriate settings to fetch +from the GitLab repository. + +`CI_REPO_CACHE_CREDENTIALS` must contain the Google Cloud service account +JSON for uploading to the `gitlab-ci-git-repo-cache` bucket. + +Note that this bucket should be located in the same continent as the +runner, or [you can incur network egress charges](https://cloud.google.com/storage/pricing). + +## `config.toml` + +The full contents of our `config.toml` are: + +NOTE: +Settings that are not public are shown as `X`. + +**Google Cloud Platform** + +```toml +concurrent = X +check_interval = 1 +metrics_server = "X" +sentry_dsn = "X" + +[[runners]] + name = "docker-auto-scale" + request_concurrency = X + url = "https://gitlab.com/" + token = "SHARED_RUNNER_TOKEN" + pre_clone_script = "eval \"$CI_PRE_CLONE_SCRIPT\"" + executor = "docker+machine" + environment = [ + "DOCKER_DRIVER=overlay2", + "DOCKER_TLS_CERTDIR=" + ] + limit = X + [runners.docker] + image = "ruby:2.5" + privileged = true + volumes = [ + "/certs/client", + "/dummy-sys-class-dmi-id:/sys/class/dmi/id:ro" # Make kaniko builds work on GCP. + ] + [runners.machine] + IdleCount = 50 + IdleTime = 3600 + MaxBuilds = 1 # For security reasons we delete the VM after job has finished so it's not reused. + MachineName = "srm-%s" + MachineDriver = "google" + MachineOptions = [ + "google-project=PROJECT", + "google-disk-size=25", + "google-machine-type=n1-standard-1", + "google-username=core", + "google-tags=gitlab-com,srm", + "google-use-internal-ip", + "google-zone=us-east1-d", + "engine-opt=mtu=1460", # Set MTU for container interface, for more information check https://gitlab.com/gitlab-org/gitlab-runner/-/issues/3214#note_82892928 + "google-machine-image=PROJECT/global/images/IMAGE", + "engine-opt=ipv6", # This will create IPv6 interfaces in the containers. + "engine-opt=fixed-cidr-v6=fc00::/7", + "google-operation-backoff-initial-interval=2" # Custom flag from forked docker-machine, for more information check https://github.com/docker/machine/pull/4600 + ] + [[runners.machine.autoscaling]] + Periods = ["* * * * * sat,sun *"] + Timezone = "UTC" + IdleCount = 70 + IdleTime = 3600 + [[runners.machine.autoscaling]] + Periods = ["* 30-59 3 * * * *", "* 0-30 4 * * * *"] + Timezone = "UTC" + IdleCount = 700 + IdleTime = 3600 + [runners.cache] + Type = "gcs" + Shared = true + [runners.cache.gcs] + CredentialsFile = "/path/to/file" + BucketName = "bucket-name" +``` diff --git a/doc/ci/runners/saas/macos/environment.md b/doc/ci/runners/saas/macos/environment.md new file mode 100644 index 00000000000..3332eab9b44 --- /dev/null +++ b/doc/ci/runners/saas/macos/environment.md @@ -0,0 +1,43 @@ +--- +stage: Verify +group: Runner +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +--- + +# VM instances and images for SaaS runners on macOS **(FREE SAAS)** + +When you use SaaS runners on macOS: + +- Each of your jobs runs in a newly provisioned VM, which is dedicated to the specific job. +- The VM is active only for the duration of the job and immediately deleted. + +## VM types + +The virtual machine where your job runs has `sudo` access with no password. +For the Beta, there is only one available machine type, `gbc-macos-large`. + +| Instance type | vCPUS | Memory (GB) | +| --------- | --- | ------- | +| `gbc-macos-large` | 4 | 10 | + +## VM images + +You can execute your build on one of the following images. +You specify this image in your `.gitlab-ci.yml` file. + +Each image is running a specific version of macOS and Xcode. + +| VM image | Included software | +|---------------------------|--------------------| +| macos-10.13-xcode-7 | <https://gitlab.com/gitlab-org/ci-cd/shared-runners/images/macstadium/orka/-/blob/main/toolchain/high-sierra.yml> | +| macos-10.13-xcode-8 | <https://gitlab.com/gitlab-org/ci-cd/shared-runners/images/macstadium/orka/-/blob/main/toolchain/high-sierra.yml> | +| macos-10.13-xcode-9 | <https://gitlab.com/gitlab-org/ci-cd/shared-runners/images/macstadium/orka/-/blob/main/toolchain/high-sierra.yml> | +| macos-10.14-xcode-10 | <https://gitlab.com/gitlab-org/ci-cd/shared-runners/images/macstadium/orka/-/blob/main/toolchain/mojave.yml> | +| macos-10.15-xcode-11 | <https://gitlab.com/gitlab-org/ci-cd/shared-runners/images/macstadium/orka/-/blob/main/toolchain/catalina.yml> | +| macos-11-xcode-12 | <https://gitlab.com/gitlab-org/ci-cd/shared-runners/images/macstadium/orka/-/blob/main/toolchain/big-sur.yml> | + +### Image update policy + +- Support for new macOS versions is planned. +- Additional details on the support policy and image update release process are documented + [in this project](https://gitlab.com/gitlab-org/ci-cd/shared-runners/images/macstadium/orka/-/blob/55bf59c8fa88712960afff2bf6ecc5daa879a8f5/docs/overview.md#os-images). diff --git a/doc/ci/runners/saas/macos_saas_runner.md b/doc/ci/runners/saas/macos_saas_runner.md new file mode 100644 index 00000000000..40c4deb51aa --- /dev/null +++ b/doc/ci/runners/saas/macos_saas_runner.md @@ -0,0 +1,63 @@ +--- +stage: Verify +group: Runner +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +--- + +# SaaS runners on macOS (Beta) **(FREE SAAS)** + +SaaS runners on macOS provide an on-demand macOS build environment integrated with +GitLab SaaS [CI/CD](../../../ci/index.md). +Use these runners to build, test, and deploy apps for the Apple ecosystem (macOS, iOS, tvOS). You can take advantage +of all the capabilities of the GitLab single DevOps platform and not have to manage or operate a +build environment. + +SaaS runners on macOS are in [Beta](https://about.gitlab.com/handbook/product/gitlab-the-product/#beta) +and shouldn't be relied upon for mission-critical production jobs. + +## Quickstart + +To start using SaaS runners on macOS, you must submit an access request [issue](https://gitlab.com/gitlab-com/macos-buildcloud-runners-beta/-/issues/new?issuable_template=beta_access_request). After your +access has been granted and your build environment configured, you must configure your +`.gitlab-ci.yml` pipeline file: + +1. Add a `.gitlab-ci.yml` file to your project repository. +1. Specify the [image](macos/environment.md#vm-images) you want to use. +1. Commit a change to your repository. + +The runners automatically run your build. + +## Example `.gitlab-ci.yml` file + +The following sample `.gitlab-ci.yml` file shows how to start using the SaaS runners on macOS: + +```yaml +.macos_saas_runners: + tags: + - shared-macos-amd64 + image: macos-11-xcode-12 + +stages: + - build + - test + +before_script: + - echo "started by ${GITLAB_USER_NAME}" + +build: + extends: + - .macos_saas_runners + stage: build + script: + - echo "running scripts in the build job" + +test: + extends: + - .macos_saas_runners + stage: test + script: + - echo "running scripts in the test job" +``` + +NOTE: +During the Beta period, the architecture of this solution will change. Rather than the jobs running on a specific VM instance, they will run on an ephemeral VM instance that is created by an autoscaling instance, known as the Runner Manager. We will notify all Beta participants of any downtime required to do this work. diff --git a/doc/ci/runners/saas/windows_saas_runner.md b/doc/ci/runners/saas/windows_saas_runner.md new file mode 100644 index 00000000000..87ee542fb14 --- /dev/null +++ b/doc/ci/runners/saas/windows_saas_runner.md @@ -0,0 +1,155 @@ +--- +stage: Verify +group: Runner +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +--- + +# SaaS runners on Windows (beta) **(FREE SAAS)** + +SaaS runners on Windows are in [beta](https://about.gitlab.com/handbook/product/gitlab-the-product/#beta) +and shouldn't be used for production workloads. + +During this beta period, the [shared runner pipeline quota](../../../user/admin_area/settings/continuous_integration.md#shared-runners-pipeline-minutes-quota) +applies for groups and projects in the same manner as Linux runners. This may +change when the beta period ends, as discussed in this [related issue](https://gitlab.com/gitlab-org/gitlab/-/issues/30834). + +Windows runners on GitLab.com autoscale by launching virtual machines on +the Google Cloud Platform. This solution uses an +[autoscaling driver](https://gitlab.com/gitlab-org/ci-cd/custom-executor-drivers/autoscaler/tree/master/docs/readme.md) +developed by GitLab for the [custom executor](https://docs.gitlab.com/runner/executors/custom.html). +Windows runners execute your CI/CD jobs on `n1-standard-2` instances with +2 vCPUs and 7.5 GB RAM. You can find a full list of available Windows packages in +the [package documentation](https://gitlab.com/gitlab-org/ci-cd/shared-runners/images/gcp/windows-containers/blob/main/cookbooks/preinstalled-software/README.md). + +We want to keep iterating to get Windows runners in a stable state and +[generally available](https://about.gitlab.com/handbook/product/gitlab-the-product/#generally-available-ga). +You can follow our work towards this goal in the +[related epic](https://gitlab.com/groups/gitlab-org/-/epics/2162). + +## Configuration + +The full contents of our `config.toml` are: + +NOTE: +Settings that aren't public are shown as `X`. + +```toml +concurrent = X +check_interval = 3 + +[[runners]] + name = "windows-runner" + url = "https://gitlab.com/" + token = "TOKEN" + executor = "custom" + builds_dir = "C:\\GitLab-Runner\\builds" + cache_dir = "C:\\GitLab-Runner\\cache" + shell = "powershell" + [runners.custom] + config_exec = "C:\\GitLab-Runner\\autoscaler\\autoscaler.exe" + config_args = ["--config", "C:\\GitLab-Runner\\autoscaler\\config.toml", "custom", "config"] + prepare_exec = "C:\\GitLab-Runner\\autoscaler\\autoscaler.exe" + prepare_args = ["--config", "C:\\GitLab-Runner\\autoscaler\\config.toml", "custom", "prepare"] + run_exec = "C:\\GitLab-Runner\\autoscaler\\autoscaler.exe" + run_args = ["--config", "C:\\GitLab-Runner\\autoscaler\\config.toml", "custom", "run"] + cleanup_exec = "C:\\GitLab-Runner\\autoscaler\\autoscaler.exe" + cleanup_args = ["--config", "C:\\GitLab-Runner\\autoscaler\\config.toml", "custom", "cleanup"] +``` + +The full contents of our `autoscaler/config.toml` are: + +```toml +Provider = "gcp" +Executor = "winrm" +OS = "windows" +LogLevel = "info" +LogFormat = "text" +LogFile = "C:\\GitLab-Runner\\autoscaler\\autoscaler.log" +VMTag = "windows" + +[GCP] + ServiceAccountFile = "PATH" + Project = "some-project-df9323" + Zone = "us-east1-c" + MachineType = "n1-standard-2" + Image = "IMAGE" + DiskSize = 50 + DiskType = "pd-standard" + Subnetwork = "default" + Network = "default" + Tags = ["TAGS"] + Username = "gitlab_runner" + +[WinRM] + MaximumTimeout = 3600 + ExecutionMaxRetries = 0 + +[ProviderCache] + Enabled = true + Directory = "C:\\GitLab-Runner\\autoscaler\\machines" +``` + +## Example `.gitlab-ci.yml` file + +Below is a sample `.gitlab-ci.yml` file that shows how to start using the runners for Windows: + +```yaml +.shared_windows_runners: + tags: + - shared-windows + - windows + - windows-1809 + +stages: + - build + - test + +before_script: + - Set-Variable -Name "time" -Value (date -Format "%H:%m") + - echo ${time} + - echo "started by ${GITLAB_USER_NAME}" + +build: + extends: + - .shared_windows_runners + stage: build + script: + - echo "running scripts in the build job" + +test: + extends: + - .shared_windows_runners + stage: test + script: + - echo "running scripts in the test job" +``` + +## Limitations and known issues + +- All the limitations mentioned in our [beta + definition](https://about.gitlab.com/handbook/product/#beta). +- The average provisioning time for a new Windows VM is 5 minutes. + This means that you may notice slower build start times + on the Windows runner fleet during the beta. In a future + release we intend to update the autoscaler to enable + the pre-provisioning of virtual machines. This is intended to significantly reduce + the time it takes to provision a VM on the Windows fleet. You can + follow along in the [related issue](https://gitlab.com/gitlab-org/ci-cd/custom-executor-drivers/autoscaler/-/issues/32). +- The Windows runner fleet may be unavailable occasionally + for maintenance or updates. +- The Windows runner virtual machine instances do not use the + GitLab Docker executor. This means that you can't specify + [`image`](../../../ci/yaml/index.md#image) or [`services`](../../../ci/yaml/index.md#services) in + your pipeline configuration. +- For the beta release, we have included a set of software packages in + the base VM image. If your CI job requires additional software that's + not included in this list, then you must add installation + commands to [`before_script`](../../../ci/yaml/index.md#before_script) or [`script`](../../../ci/yaml/index.md#script) to install the required + software. Note that each job runs on a new VM instance, so the + installation of additional software packages needs to be repeated for + each job in your pipeline. +- The job may stay in a pending state for longer than the + Linux runners. +- There is the possibility that we introduce breaking changes which will + require updates to pipelines that are using the Windows runner + fleet. diff --git a/doc/ci/secrets/index.md b/doc/ci/secrets/index.md index 4d42bc69df8..c0a763c80f0 100644 --- a/doc/ci/secrets/index.md +++ b/doc/ci/secrets/index.md @@ -53,6 +53,7 @@ and supports multiple secrets engines. To configure your Vault server: +1. Ensure your Vault server is running on version 1.2.0 or higher. 1. Enable the authentication method by running these commands. They provide your Vault server the [JSON Web Key Set](https://tools.ietf.org/html/rfc7517) (JWKS) endpoint for your GitLab instance, so Vault can fetch the public signing key and verify the JSON Web Token (JWT) when authenticating: @@ -85,10 +86,10 @@ To configure your Vault server: to provide details about your Vault server: - `VAULT_SERVER_URL` - The URL of your Vault server, such as `https://vault.example.com:8200`. Required. - - `VAULT_AUTH_ROLE` - (Optional) The role to use when attempting to authenticate. + - `VAULT_AUTH_ROLE` - Optional. The role to use when attempting to authenticate. If no role is specified, Vault uses the [default role](https://www.vaultproject.io/api/auth/jwt#default_role) specified when the authentication method was configured. - - `VAULT_AUTH_PATH` - (Optional) The path where the authentication method is mounted, default is `jwt`. + - `VAULT_AUTH_PATH` - Optional. The path where the authentication method is mounted, default is `jwt`. NOTE: Support for providing these values in the user interface [is tracked in this issue](https://gitlab.com/gitlab-org/gitlab/-/issues/218677). diff --git a/doc/ci/services/gitlab.md b/doc/ci/services/gitlab.md index 5ac66846ab7..689ce884ae4 100644 --- a/doc/ci/services/gitlab.md +++ b/doc/ci/services/gitlab.md @@ -28,7 +28,7 @@ NOTE: Variables set in the GitLab UI are not passed down to the service containers. [Learn more](../variables/index.md#). -Then, commands in `script:` sections in your `.gitlab-ci.yml` file can access the API at `http://gitlab/api/v4`. +Then, commands in `script` sections in your `.gitlab-ci.yml` file can access the API at `http://gitlab/api/v4`. For more information about why `gitlab` is used for the `Host`, see [How services are linked to the job](../docker/using_docker_images.md#extended-docker-configuration-options). diff --git a/doc/ci/test_cases/index.md b/doc/ci/test_cases/index.md index 384bfc10779..4c840125d24 100644 --- a/doc/ci/test_cases/index.md +++ b/doc/ci/test_cases/index.md @@ -11,6 +11,10 @@ type: reference > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/233479) in GitLab 13.6. > - [Feature flag removed](https://gitlab.com/gitlab-org/gitlab/-/issues/241983) in GitLab 13.7. +INFO: +Create test cases in GitLab Ultimate. +[Try it free for 30 days](https://about.gitlab.com/free-trial/index.html?glm_source=docs.gitlab.com&glm_content=u-test-cases-docs). + Test cases in GitLab can help your teams create testing scenarios in their existing development platform. Now your Implementation and Testing teams can collaborate better, as they no longer have to diff --git a/doc/ci/triggers/img/triggers_page.png b/doc/ci/triggers/img/triggers_page.png Binary files differdeleted file mode 100644 index 7dc8f91cf7e..00000000000 --- a/doc/ci/triggers/img/triggers_page.png +++ /dev/null diff --git a/doc/ci/triggers/index.md b/doc/ci/triggers/index.md index afcf8ae629a..d3ac1de7c3b 100644 --- a/doc/ci/triggers/index.md +++ b/doc/ci/triggers/index.md @@ -5,131 +5,121 @@ info: To determine the technical writer assigned to the Stage/Group associated w type: tutorial --- -# Triggering pipelines through the API **(FREE)** +# Trigger pipelines by using the API **(FREE)** -Triggers can be used to force a pipeline rerun of a specific `ref` (branch or -tag) with an API call. +To trigger a pipeline for a specific branch or tag, you can use an API call +to the [pipeline triggers API endpoint](../../api/pipeline_triggers.md). -## Authentication tokens +When authenticating with the API, you can use: -The following methods of authentication are supported: +- A [trigger token](#create-a-trigger-token) to trigger a branch or tag pipeline. +- A [CI/CD job token](../jobs/ci_job_token.md) to trigger a [multi-project pipeline](../pipelines/multi_project_pipelines.md#create-multi-project-pipelines-by-using-the-api). -- Trigger tokens: A unique trigger token can be obtained when [adding a new trigger](#adding-a-new-trigger). -- [CI job tokens](../jobs/ci_job_token.md). +## Create a trigger token -If using the `$CI_PIPELINE_SOURCE` [predefined CI/CD variable](../variables/predefined_variables.md) -to limit which jobs run in a pipeline, the value could be either `pipeline` or `trigger`, -depending on which trigger method is used. +You can trigger a pipeline for a branch or tag by generating a trigger token and using it +to authenticate an API call. The token impersonates a user's project access and permissions. -| `$CI_PIPELINE_SOURCE` value | Trigger method | -|-----------------------------|----------------| -| `pipeline` | Using the `trigger:` keyword in the CI/CD configuration file, or using the trigger API with `$CI_JOB_TOKEN`. | -| `trigger` | Using the trigger API using a generated trigger token | +Prerequisite: -This also applies when using the `pipelines` or `triggers` keywords with the legacy [`only/except` basic syntax](../yaml/index.md#only--except). +- You must have at least the [Maintainer role](../../user/permissions.md) for the project. -## Adding a new trigger +To create a trigger token: -Go to your -**Settings > CI/CD** under **Triggers** to add a new trigger. The **Add trigger** button creates -a new token which you can then use to trigger a rerun of this -particular project's pipeline. - -Every new trigger you create, gets assigned a different token which you can -then use inside your scripts or `.gitlab-ci.yml`. You also have a nice -overview of the time the triggers were last used. - - +1. On the top bar, select **Menu > Projects** and find your project. +1. On the left sidebar, select **Settings > CI/CD**. +1. Expand **Pipeline triggers**. +1. Enter a description and select **Add trigger**. + - You can view and copy the full token for all triggers you have created. + - You can only see the first 4 characters for tokens created by other project members. WARNING: -Passing plain text tokens in public projects is a security issue. Potential -attackers can impersonate the user that exposed their trigger token publicly in -their `.gitlab-ci.yml` file. Use [CI/CD variables](../variables/index.md) -to protect trigger tokens. +It is a security risk to save tokens in plain text in public projects. Potential +attackers could use a trigger token exposed in the `.gitlab-ci.yml` file to impersonate +the user that created the token. Use [masked CI/CD variables](../variables/index.md#mask-a-cicd-variable) +to improve the security of trigger tokens. -## Revoking a trigger +## Trigger a pipeline -You can revoke a trigger any time by going at your project's -**Settings > CI/CD** under **Triggers** and hitting the **Revoke** button. -The action is irreversible. +After you [create a trigger token](#create-a-trigger-token), you can use it to trigger +pipelines with a tool that can access the API, or a webhook. -## Triggering a pipeline +### Use cURL -To trigger a pipeline you need to send a `POST` request to the GitLab API endpoint: +You can use cURL to trigger pipelines with the [pipeline triggers API endpoint](../../api/pipeline_triggers.md). +For example: -```plaintext -POST /projects/:id/trigger/pipeline -``` +- Use a multiline cURL command: -The required parameters are the [trigger's `token`](#authentication-tokens) -and the Git `ref` on which the trigger is performed. Valid refs are -branches or tags. The `:id` of a project can be found by -[querying the API](../../api/projects.md) or by visiting the **CI/CD** -settings page which provides self-explanatory examples. + ```shell + curl --request POST \ + --form token=<token> \ + --formref=<ref_name> \ + "https://gitlab.example.com/api/v4/projects/<project_id>/trigger/pipeline" + ``` -When a rerun of a pipeline is triggered, jobs are labeled as `triggered` in -**CI/CD > Jobs**. +- Use cURL and pass the `<token>` and `<ref_name>` in the query string: -You can see which trigger caused a job to run by visiting the single job page. -A part of the trigger's token is exposed in the UI as you can see from the image -below. + ```shell + curl --request POST \ + "https://gitlab.example.com/api/v4/projects/<project_id>/trigger/pipeline?token=<token>&ref=<ref_name>" + ``` - - -By using cURL you can trigger a pipeline rerun with minimal effort, for example: - -```shell -curl --request POST \ - --form token=TOKEN \ - --form ref=main \ - "https://gitlab.example.com/api/v4/projects/9/trigger/pipeline" -``` +In each example, replace: -In this case, the pipeline for the project with ID `9` runs on the `main` branch. +- The URL with `https://gitlab.com` or the URL of your instance. +- `<token>` with your trigger token. +- `<ref_name>` with a branch or tag name, like `main`. +- `<project_id>` with your project ID, like `123456`. The project ID is displayed + at the top of every project's landing page. -Alternatively, you can pass the `token` and `ref` arguments in the query string: +### Use a CI/CD job -```shell -curl --request POST \ - "https://gitlab.example.com/api/v4/projects/9/trigger/pipeline?token=TOKEN&ref=main" -``` +You can use a CI/CD job with a triggers token to trigger pipelines when another pipeline +runs. -You can also benefit by using triggers in your `.gitlab-ci.yml`. Let's say that -you have two projects, A and B, and you want to trigger a pipeline on the `main` -branch of project B whenever a tag on project A is created. This is the job you -need to add in project A's `.gitlab-ci.yml`: +For example, to trigger a pipeline on the `main` branch of `project-B` when a tag +is created in `project-A`, add the following job to project A's `.gitlab-ci.yml` file: ```yaml trigger_pipeline: stage: deploy script: - - 'curl --request POST --form token=TOKEN --form ref=main "https://gitlab.example.com/api/v4/projects/9/trigger/pipeline"' + - 'curl --fail --request POST --form token=$MY_TRIGGER_TOKEN --form ref=main "https://gitlab.example.com/api/v4/projects/123456/trigger/pipeline"' rules: - if: $CI_COMMIT_TAG ``` -This means that whenever a new tag is pushed on project A, the job runs and the -`trigger_pipeline` job is executed, triggering the pipeline for project B. The -`stage: deploy` ensures that this job runs only after all jobs with -`stage: test` complete successfully. +In this example: -NOTE: -You [cannot use the API to start `when:manual` trigger jobs](https://gitlab.com/gitlab-org/gitlab/-/issues/284086). +- `1234` is the project ID for `project-B`. The project ID is displayed at the top + of every project's landing page. +- The [`rules`](../yaml/index.md#rules) cause the job to run every time a tag is added to `project-A`. +- `MY_TRIGGER_TOKEN` is a [masked CI/CD variables](../variables/index.md#mask-a-cicd-variable) + that contains the trigger token. -## Triggering a pipeline from a webhook +### Use a webhook -To trigger a job from a webhook of another project you need to add the following -webhook URL for Push and Tag events (change the project ID, ref and token): +To trigger a pipeline from another project's webhook, use a webhook URL like the following +for push and tag events: ```plaintext https://gitlab.example.com/api/v4/projects/9/ref/main/trigger/pipeline?token=TOKEN ``` -You should pass `ref` as part of the URL, to take precedence over `ref` from -the webhook body that designates the branch ref that fired the trigger in the -source repository. Be sure to URL-encode `ref` if it contains slashes. +Replace: + +- The URL with `https://gitlab.com` or the URL of your instance. +- `<token>` with your trigger token. +- `<ref_name>` with a branch or tag name, like `main`. +- `<project_id>` with your project ID, like `123456`. The project ID is displayed + at the top of the project's landing page. + +The `ref` in the URL takes precedence over the `ref` in the webhook payload. The +payload `ref` is the branch that fired the trigger in the source repository. +You must URL-encode `ref` if it contains slashes. -### Using webhook payload in the triggered pipeline +#### Use a webhook payload > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/31197) in GitLab 13.9. > - [Feature flag removed](https://gitlab.com/gitlab-org/gitlab/-/issues/321027) in GitLab 13.11. @@ -139,94 +129,68 @@ the `TRIGGER_PAYLOAD` [predefined CI/CD variable](../variables/predefined_variab The payload is exposed as a [file-type variable](../variables/index.md#cicd-variable-types), so you can access the data with `cat $TRIGGER_PAYLOAD` or a similar command. -## Making use of trigger variables +### Pass CI/CD variables in the API call -You can pass any number of arbitrary variables in the trigger API call and they -are available in GitLab CI/CD so that they can be used in your `.gitlab-ci.yml` -file. The parameter is of the form: +You can pass any number of [CI/CD variables](../variables/index.md) in the trigger API call. +These variables have the [highest precedence](../variables/index.md#cicd-variable-precedence), +and override all variables with the same name. -```plaintext -variables[key]=value +The parameter is of the form `variables[key]=value`, for example: + +```shell +curl --request POST \ + --form token=TOKEN \ + --form ref=main \ + --form "variables[UPLOAD_TO_S3]=true" \ + "https://gitlab.example.com/api/v4/projects/123456/trigger/pipeline" ``` -This information is also exposed in the UI. _Values_ are only viewable by users with the Owner and Maintainer role. +CI/CD variables in triggered pipelines display on each job's page, but only +users with the Owner and Maintainer role can view the values.  -Using trigger variables can be proven useful for a variety of reasons: - -- Identifiable jobs. Since the variable is exposed in the UI you can know - why the pipeline was triggered if you pass a variable that explains the - purpose. -- Conditional job processing. You can have conditional jobs that run whenever - a certain variable is present. +## Revoke a trigger token -Consider the following `.gitlab-ci.yml` where we set three -[stages](../yaml/index.md#stages) and the `upload_package` job is run only -when all jobs from the test and build stages pass. When the `UPLOAD_TO_S3` -variable is non-zero, `make upload` is run. +To revoke a trigger token: -```yaml -stages: - - test - - build - - package - -run_tests: - stage: test - script: - - make test - -build_package: - stage: build - script: - - make build - -upload_package: - stage: package - script: - - if [ -n "${UPLOAD_TO_S3}" ]; then make upload; fi -``` +1. On the top bar, select **Menu > Projects** and find your project. +1. On the left sidebar, select **Settings > CI/CD**. +1. Expand **Pipeline triggers**. +1. To the left of the trigger token you want to revoke, select **Revoke** (**{remove}**). -You can then trigger a pipeline while you pass the `UPLOAD_TO_S3` variable -and the script of the `upload_package` job is run: +A revoked trigger token cannot be added back. -```shell -curl --request POST \ - --form token=TOKEN \ - --form ref=main \ - --form "variables[UPLOAD_TO_S3]=true" \ - "https://gitlab.example.com/api/v4/projects/9/trigger/pipeline" -``` +## Configure CI/CD jobs to run in triggered pipelines -Trigger variables have the [highest priority](../variables/index.md#cicd-variable-precedence) -of all types of variables. +To [configure when to run jobs](../jobs/job_control.md) in triggered pipelines: -## Using cron to trigger nightly pipelines +- Use [`rules`](../yaml/index.md#rules) with the `$CI_PIPELINE_SOURCE` [predefined CI/CD variable](../variables/predefined_variables.md). +- Use [`only`/`except`](../yaml/index.md#onlyrefs--exceptrefs) keywords. -Whether you craft a script or just run cURL directly, you can trigger jobs -in conjunction with cron. The example below triggers a job on the `main` branch -of project with ID `9` every night at `00:30`: +| `$CI_PIPELINE_SOURCE` value | `only`/`except` keywords | Trigger method | +|-----------------------------|--------------------------|---------------------| +| `trigger` | `triggers` | In pipelines triggered with the [pipeline triggers API](../../api/pipeline_triggers.md) by using a [trigger token](#create-a-trigger-token). | +| `pipeline` | `pipelines` | In [multi-project pipelines](../pipelines/multi_project_pipelines.md#create-multi-project-pipelines-by-using-the-api) triggered with the [pipeline triggers API](../../api/pipeline_triggers.md) by using the [`$CI_JOB_TOKEN`](../jobs/ci_job_token.md), or by using the [`trigger`](../yaml/index.md#trigger) keyword in the CI/CD configuration file. | -```shell -30 0 * * * curl --request POST --form token=TOKEN --form ref=main "https://gitlab.example.com/api/v4/projects/9/trigger/pipeline" -``` +Additionally, the `$CI_PIPELINE_TRIGGERED` predefined CI/CD variable is set to `true` +in pipelines triggered with a trigger token. -This behavior can also be achieved through the GitLab UI with -[pipeline schedules](../pipelines/schedules.md). +## See which trigger token was used -## Legacy triggers +You can see which trigger caused a job to run by visiting the single job page. +A part of the trigger's token displays on the right of the page, under the job details: -Old triggers, created before GitLab 9.0 are marked as legacy. + -Triggers with the legacy label do not have an associated user and only have -access to the current project. They are considered deprecated and might be -removed with one of the future versions of GitLab. +In pipelines triggered with a trigger token, jobs are labeled as `triggered` in +**CI/CD > Jobs**. ## Troubleshooting -### '404 not found' when triggering a pipeline +### `404 not found` when triggering a pipeline A response of `{"message":"404 Not Found"}` when triggering a pipeline might be caused -by using a Personal Access Token instead of a trigger token. [Add a new trigger](#adding-a-new-trigger) -and use that token to authenticate when triggering a pipeline. +by using a [personal access token](../../user/profile/personal_access_tokens.md) +instead of a trigger token. [Create a new trigger token](#create-a-trigger-token) +and use it instead of the personal access token. diff --git a/doc/ci/troubleshooting.md b/doc/ci/troubleshooting.md index 037e8d3497d..4d550f6da13 100644 --- a/doc/ci/troubleshooting.md +++ b/doc/ci/troubleshooting.md @@ -291,7 +291,7 @@ Pipeline configuration warnings are shown when you: ### "Job may allow multiple pipelines to run for a single action" warning -When you use [`rules`](yaml/index.md#rules) with a `when:` clause without an `if:` +When you use [`rules`](yaml/index.md#rules) with a `when` clause without an `if` clause, multiple pipelines may run. Usually this occurs when you push a commit to a branch that has an open merge request associated with it. diff --git a/doc/ci/unit_test_reports.md b/doc/ci/unit_test_reports.md index e758fbc91dd..55fd8c1eb49 100644 --- a/doc/ci/unit_test_reports.md +++ b/doc/ci/unit_test_reports.md @@ -41,7 +41,7 @@ Consider the following workflow: ## How it works First, GitLab Runner uploads all [JUnit report format XML files](https://www.ibm.com/docs/en/adfz/developer-for-zos/14.1.0?topic=formats-junit-xml-format) -as [artifacts](yaml/index.md#artifactsreportsjunit) to GitLab. Then, when you visit a merge request, GitLab starts +as [artifacts](yaml/artifacts_reports.md#artifactsreportsjunit) to GitLab. Then, when you visit a merge request, GitLab starts comparing the head and base branch's JUnit report format XML files, where: - The base branch is the target branch (usually the default branch). @@ -77,7 +77,7 @@ If a test failed in the project's default branch in the last 14 days, a message ## How to set it up To enable the Unit test reports in merge requests, you need to add -[`artifacts:reports:junit`](yaml/index.md#artifactsreportsjunit) +[`artifacts:reports:junit`](yaml/artifacts_reports.md#artifactsreportsjunit) in `.gitlab-ci.yml`, and specify the path(s) of the generated test reports. The reports must be `.xml` files, otherwise [GitLab returns an Error 500](https://gitlab.com/gitlab-org/gitlab/-/issues/216575). @@ -377,7 +377,7 @@ GitLab does not parse very [large nodes](https://nokogiri.org/tutorials/parsing_ > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/202114) in GitLab 13.0 behind the `:junit_pipeline_screenshots_view` feature flag, disabled by default. > - [Feature flag removed](https://gitlab.com/gitlab-org/gitlab/-/issues/216979) in GitLab 13.12. -Upload your screenshots as [artifacts](yaml/index.md#artifactsreportsjunit) to GitLab. If JUnit +Upload your screenshots as [artifacts](yaml/artifacts_reports.md#artifactsreportsjunit) to GitLab. If JUnit report format XML files contain an `attachment` tag, GitLab parses the attachment. Note that: - The `attachment` tag **must** contain the relative path to `$CI_PROJECT_DIR` of the screenshots you uploaded. For diff --git a/doc/ci/variables/index.md b/doc/ci/variables/index.md index a0c8dbd4e4a..acc3489143a 100644 --- a/doc/ci/variables/index.md +++ b/doc/ci/variables/index.md @@ -166,10 +166,10 @@ To add or update variables in the project settings: - **Key**: Must be one line, with no spaces, using only letters, numbers, or `_`. - **Value**: No limitations. - **Type**: [`File` or `Variable`](#cicd-variable-types). - - **Environment scope**: (Optional) `All`, or specific [environments](../environments/index.md). - - **Protect variable** (Optional): If selected, the variable is only available + - **Environment scope**: Optional. `All`, or specific [environments](../environments/index.md). + - **Protect variable** Optional. If selected, the variable is only available in pipelines that run on protected branches or tags. - - **Mask variable** (Optional): If selected, the variable's **Value** is masked + - **Mask variable** Optional. If selected, the variable's **Value** is masked in job logs. The variable fails to save if the value does not meet the [masking requirements](#mask-a-cicd-variable). @@ -208,10 +208,10 @@ To add a group variable: - **Key**: Must be one line, with no spaces, using only letters, numbers, or `_`. - **Value**: No limitations. - **Type**: [`File` or `Variable`](#cicd-variable-types). - - **Environment scope** (Optional): `All`, or specific [environments](#limit-the-environment-scope-of-a-cicd-variable). **(PREMIUM)** - - **Protect variable** (Optional): If selected, the variable is only available + - **Environment scope** Optional. `All`, or specific [environments](#limit-the-environment-scope-of-a-cicd-variable). **(PREMIUM)** + - **Protect variable** Optional. If selected, the variable is only available in pipelines that run on protected branches or tags. - - **Mask variable** (Optional): If selected, the variable's **Value** is masked + - **Mask variable** Optional. If selected, the variable's **Value** is masked in job logs. The variable fails to save if the value does not meet the [masking requirements](#mask-a-cicd-variable). @@ -248,9 +248,9 @@ To add an instance variable: 10,000 characters is allowed. This is also bounded by the limits of the selected runner operating system. In GitLab 13.0 to 13.2, 700 characters is allowed. - **Type**: [`File` or `Variable`](#cicd-variable-types). - - **Protect variable** (Optional): If selected, the variable is only available + - **Protect variable** Optional. If selected, the variable is only available in pipelines that run on protected branches or tags. - - **Mask variable** (Optional): If selected, the variable's **Value** is not shown + - **Mask variable** Optional. If selected, the variable's **Value** is not shown in job logs. The variable is not saved if the value does not meet the [masking requirements](#mask-a-cicd-variable). ### CI/CD variable types @@ -293,6 +293,11 @@ Use the variables in a job script like this: kubectl config set-cluster e2e --server="$KUBE_URL" --certificate-authority="$KUBE_CA_PEM" ``` +WARNING: +Be careful when assigning the value of a file variable to another variable. The other +variable takes the content of the file as its value, **not** the path to the file. +See [issue 29407](https://gitlab.com/gitlab-org/gitlab/-/issues/29407) for more details. + An alternative to `File` type variables is to: - Read the value of a CI/CD variable (`variable` type). @@ -554,7 +559,7 @@ These variables cannot be used as CI/CD variables to configure a pipeline, but they can be used in job scripts. 1. In the job script, save the variable as a `.env` file. -1. Save the `.env` file as an [`artifacts:reports:dotenv`](../yaml/index.md#artifactsreportsdotenv) +1. Save the `.env` file as an [`artifacts:reports:dotenv`](../yaml/artifacts_reports.md#artifactsreportsdotenv) artifact. 1. Set a job in a later stage to receive the artifact by using the [`dependencies`](../yaml/index.md#dependencies) or the [`needs`](../yaml/index.md#needs) keywords. @@ -607,7 +612,7 @@ which variables take precedence. The order of precedence for variables is (from highest to lowest): -1. [Trigger variables](../triggers/index.md#making-use-of-trigger-variables), +1. [Trigger variables](../triggers/index.md#pass-cicd-variables-in-the-api-call), [scheduled pipeline variables](../pipelines/schedules.md#using-variables), and [manual pipeline run variables](#override-a-variable-when-running-a-pipeline-manually). 1. Project [variables](#custom-cicd-variables). @@ -641,7 +646,7 @@ You can override the value of a variable when you: 1. Create a pipeline by using [the API](../../api/pipelines.md#create-a-new-pipeline). 1. Run a job manually in the UI. 1. Use [push options](../../user/project/push_options.md#push-options-for-gitlab-cicd). -1. Trigger a pipeline by using [the API](../triggers/index.md#making-use-of-trigger-variables). +1. Trigger a pipeline by using [the API](../triggers/index.md#pass-cicd-variables-in-the-api-call). 1. Pass variables to a downstream pipeline [by using the `variable` keyword](../pipelines/multi_project_pipelines.md#pass-cicd-variables-to-a-downstream-pipeline-by-using-the-variables-keyword) or [by using variable inheritance](../pipelines/multi_project_pipelines.md#pass-cicd-variables-to-a-downstream-pipeline-by-using-variable-inheritance). diff --git a/doc/ci/variables/predefined_variables.md b/doc/ci/variables/predefined_variables.md index 45fa1994342..c06ca6878c4 100644 --- a/doc/ci/variables/predefined_variables.md +++ b/doc/ci/variables/predefined_variables.md @@ -14,7 +14,9 @@ Some variables are only available with more recent versions of [GitLab Runner](h You can [output the values of all variables available for a job](index.md#list-all-environment-variables) with a `script` command. -There are also [Kubernetes-specific deployment variables](../../user/project/clusters/deploy_to_cluster.md#deployment-variables). +There are also [Kubernetes-specific deployment variables (deprecated)](../../user/project/clusters/deploy_to_cluster.md#deployment-variables). + +There are also a number of [variables you can use to configure runner behavior](../runners/configure_runners.md#configure-runner-behavior-with-variables) globally or for individual jobs. | Variable | GitLab | Runner | Description | |------------------------------------------|--------|--------|-------------| @@ -75,7 +77,7 @@ There are also [Kubernetes-specific deployment variables](../../user/project/clu | `CI_PAGES_URL` | 11.8 | all | The URL for a GitLab Pages site. Always a subdomain of `CI_PAGES_DOMAIN`. | | `CI_PIPELINE_ID` | 8.10 | all | The instance-level ID of the current pipeline. This ID is unique across all projects on the GitLab instance. | | `CI_PIPELINE_IID` | 11.0 | all | The project-level IID (internal ID) of the current pipeline. This ID is unique only within the current project. | -| `CI_PIPELINE_SOURCE` | 10.0 | all | How the pipeline was triggered. Can be `push`, `web`, `schedule`, `api`, `external`, `chat`, `webide`, `merge_request_event`, `external_pull_request_event`, `parent_pipeline`, [`trigger`, or `pipeline`](../triggers/index.md#authentication-tokens). | +| `CI_PIPELINE_SOURCE` | 10.0 | all | How the pipeline was triggered. Can be `push`, `web`, `schedule`, `api`, `external`, `chat`, `webide`, `merge_request_event`, `external_pull_request_event`, `parent_pipeline`, [`trigger`, or `pipeline`](../triggers/index.md#configure-cicd-jobs-to-run-in-triggered-pipelines). | | `CI_PIPELINE_TRIGGERED` | all | all | `true` if the job was [triggered](../triggers/index.md). | | `CI_PIPELINE_URL` | 11.1 | 0.5 | The URL for the pipeline details. | | `CI_PIPELINE_CREATED_AT` | 13.10 | all | The UTC datetime when the pipeline was created, in [ISO 8601](https://tools.ietf.org/html/rfc3339#appendix-A) format. | @@ -122,7 +124,7 @@ There are also [Kubernetes-specific deployment variables](../../user/project/clu | `GITLAB_USER_ID` | 8.12 | all | The ID of the user who started the job. | | `GITLAB_USER_LOGIN` | 10.0 | all | The username of the user who started the job. | | `GITLAB_USER_NAME` | 10.0 | all | The name of the user who started the job. | -| `TRIGGER_PAYLOAD` | 13.9 | all | The webhook payload. Only available when a pipeline is [triggered with a webhook](../triggers/index.md#using-webhook-payload-in-the-triggered-pipeline). | +| `TRIGGER_PAYLOAD` | 13.9 | all | The webhook payload. Only available when a pipeline is [triggered with a webhook](../triggers/index.md#use-a-webhook-payload). | ## Predefined variables for merge request pipelines diff --git a/doc/ci/yaml/artifacts_reports.md b/doc/ci/yaml/artifacts_reports.md new file mode 100644 index 00000000000..cd38cf58c71 --- /dev/null +++ b/doc/ci/yaml/artifacts_reports.md @@ -0,0 +1,304 @@ +--- +stage: Verify +group: Testing +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +--- + +# GitLab CI/CD artifacts reports types **(FREE)** + +Use [`artifacts:reports`](index.md#artifactsreports) to: + +- Collect test reports, code quality reports, security reports, and other artifacts generated by included templates in + jobs. +- Some of these reports are used to display information in: + - Merge requests. + - Pipeline views. + - [Security dashboards](../../user/application_security/security_dashboard/index.md). + +The test reports are collected regardless of the job results (success or failure). +You can use [`artifacts:expire_in`](index.md#artifactsexpire_in) to set up an expiration +date for their artifacts. + +Some `artifacts:reports` types can be generated by multiple jobs in the same pipeline, and used by merge request or +pipeline features from each job. + +To be able to browse the report output files, include the [`artifacts:paths`](index.md#artifactspaths) keyword. + +NOTE: +Combined reports in parent pipelines using [artifacts from child pipelines](index.md#needspipelinejob) is +not supported. Track progress on adding support in [this issue](https://gitlab.com/gitlab-org/gitlab/-/issues/215725). + +## `artifacts:reports:accessibility` + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/39425) in GitLab 12.8. + +The `accessibility` report uses [pa11y](https://pa11y.org/) to report on the accessibility impact +of changes introduced in merge requests. + +GitLab can display the results of one or more reports in the merge request +[accessibility widget](../../user/project/merge_requests/accessibility_testing.md#accessibility-merge-request-widget). + +For more information, see [Accessibility testing](../../user/project/merge_requests/accessibility_testing.md). + +## `artifacts:reports:api_fuzzing` **(ULTIMATE)** + +> - Introduced in GitLab 13.4. +> - Requires GitLab Runner 13.4 or later. + +The `api_fuzzing` report collects [API Fuzzing bugs](../../user/application_security/api_fuzzing/index.md) +as artifacts. + +GitLab can display the results of one or more reports in: + +- The merge request [security widget](../../user/application_security/api_fuzzing/index.md#view-details-of-an-api-fuzzing-vulnerability). +- The [Project Vulnerability report](../../user/application_security/vulnerability_report/index.md). +- The pipeline [**Security** tab](../../user/application_security/security_dashboard/index.md#pipeline-security). +- The [security dashboard](../../user/application_security/api_fuzzing/index.md#security-dashboard). + +## `artifacts:reports:browser_performance` **(PREMIUM)** + +> [Name changed](https://gitlab.com/gitlab-org/gitlab/-/issues/225914) from `artifacts:reports:performance` in GitLab 14.0. + +The `browser_performance` report collects [Browser Performance Testing metrics](../../user/project/merge_requests/browser_performance_testing.md) +as artifacts. + +GitLab can display the results of one report in the merge request +[browser performance testing widget](../../user/project/merge_requests/browser_performance_testing.md#how-browser-performance-testing-works). + +GitLab cannot display the combined results of multiple `browser_performance` reports. + +## `artifacts:reports:cluster_image_scanning` **(ULTIMATE)** + +> - Introduced in GitLab 14.1. +> - Requires GitLab Runner 14.1 and above. + +The `cluster_image_scanning` report collects `CLUSTER_IMAGE_SCANNING` vulnerabilities. The collected +`CLUSTER_IMAGE_SCANNING` report uploads to GitLab as an artifact. + +GitLab can display the results of one or more reports in: + +- The [security dashboard](../../user/application_security/security_dashboard/index.md). +- The [Project Vulnerability report](../../user/application_security/vulnerability_report/index.md). + +## `artifacts:reports:cobertura` + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/3708) in GitLab 12.9. + +The `cobertura` report collects [Cobertura coverage XML files](../../user/project/merge_requests/test_coverage_visualization.md). +The collected Cobertura coverage reports upload to GitLab as an artifact. + +GitLab can display the results of one or more reports in the merge request +[diff annotations](../../user/project/merge_requests/test_coverage_visualization.md). + +Cobertura was originally developed for Java, but there are many third-party ports for other languages such as +JavaScript, Python, and Ruby. + +## `artifacts:reports:codequality` + +> [Moved](https://gitlab.com/gitlab-org/gitlab/-/issues/212499) to GitLab Free in 13.2. + +The `codequality` report collects [code quality issues](../../user/project/merge_requests/code_quality.md). The +collected code quality report uploads to GitLab as an artifact. + +GitLab can display the results of: + +- One or more reports in the merge request [code quality widget](../../user/project/merge_requests/code_quality.md#code-quality-widget). +- Only one report in: + - The merge request [diff annotations](../../user/project/merge_requests/code_quality.md#code-quality-in-diff-view). + Track progress on adding support for multiple reports in [this issue](https://gitlab.com/gitlab-org/gitlab/-/issues/328257). + - The [full report](../metrics_reports.md). Track progress on adding support for multiple reports in + [this issue](https://gitlab.com/gitlab-org/gitlab/-/issues/9014). + +## `artifacts:reports:container_scanning` **(ULTIMATE)** + +The `container_scanning` report collects [Container Scanning vulnerabilities](../../user/application_security/container_scanning/index.md). +The collected Container Scanning report uploads to GitLab as an artifact. + +GitLab can display the results of one or more reports in: + +- The merge request [container scanning widget](../../user/application_security/container_scanning/index.md). +- The pipeline [**Security** tab](../../user/application_security/security_dashboard/index.md#pipeline-security). +- The [security dashboard](../../user/application_security/security_dashboard/index.md). +- The [Project Vulnerability report](../../user/application_security/vulnerability_report/index.md). + +## `artifacts:reports:coverage_fuzzing` **(ULTIMATE)** + +> - Introduced in GitLab 13.4. +> - Requires GitLab Runner 13.4 or later. + +The `coverage_fuzzing` report collects [coverage fuzzing bugs](../../user/application_security/coverage_fuzzing/index.md). +The collected coverage fuzzing report uploads to GitLab as an artifact. +GitLab can display the results of one or more reports in: + +- The merge request [coverage fuzzing widget](../../user/application_security/coverage_fuzzing/index.md#interacting-with-the-vulnerabilities). +- The pipeline [**Security** tab](../../user/application_security/security_dashboard/index.md#pipeline-security). +- The [Project Vulnerability report](../../user/application_security/vulnerability_report/index.md). +- The [security dashboard](../../user/application_security/security_dashboard/index.md). + +## `artifacts:reports:dast` **(ULTIMATE)** + +The `dast` report collects [DAST vulnerabilities](../../user/application_security/dast/index.md). The collected DAST +report uploads to GitLab as an artifact. + +GitLab can display the results of one or more reports in: + +- The merge request [security widget](../../user/application_security/dast/index.md#view-details-of-a-vulnerability-detected-by-dast). +- The pipeline [**Security** tab](../../user/application_security/security_dashboard/index.md#pipeline-security). +- The [Project Vulnerability report](../../user/application_security/vulnerability_report/index.md). +- The [security dashboard](../../user/application_security/security_dashboard/index.md). + +## `artifacts:reports:dependency_scanning` **(ULTIMATE)** + +The `dependency_scanning` report collects [Dependency Scanning vulnerabilities](../../user/application_security/dependency_scanning/index.md). +The collected Dependency Scanning report uploads to GitLab as an artifact. + +GitLab can display the results of one or more reports in: + +- The merge request [dependency scanning widget](../../user/application_security/dependency_scanning/index.md#overview). +- The pipeline [**Security** tab](../../user/application_security/security_dashboard/index.md#pipeline-security). +- The [security dashboard](../../user/application_security/security_dashboard/index.md). +- The [Project Vulnerability report](../../user/application_security/vulnerability_report/index.md). +- The [dependency list](../../user/application_security/dependency_list/). + +## `artifacts:reports:dotenv` + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/17066) in GitLab 12.9. + +The `dotenv` report collects a set of environment variables as artifacts. + +The collected variables are registered as runtime-created variables of the job, +which you can use to [set dynamic environment URLs after a job finishes](../environments/index.md#set-dynamic-environment-urls-after-a-job-finishes). + +If duplicate environment variables are present in a `dotenv` report: + +- In GitLab 14.6 and later, the last one specified is used. +- In GitLab 14.5 and earlier, an error occurs. + +The exceptions to the [original dotenv rules](https://github.com/motdotla/dotenv#rules) are: + +- The variable key can contain only letters, digits, and underscores (`_`). +- The maximum size of the `.env` file is 5 KB. + This limit [can be changed on self-managed instances](../../administration/instance_limits.md#limit-dotenv-file-size). +- On GitLab.com, [the maximum number of inherited variables](../../user/gitlab_com/index.md#gitlab-cicd) + is 50 for Free, 100 for Premium and 150 for Ultimate. The default for + self-managed instances is 150, and can be changed by changing the + `dotenv_variables` [application limit](../../administration/instance_limits.md#limit-dotenv-variables). +- Variable substitution in the `.env` file is not supported. +- The `.env` file can't have empty lines or comments (starting with `#`). +- Key values in the `env` file cannot have spaces or newline characters (`\n`), including when using single or double quotes. +- Quote escaping during parsing (`key = 'value'` -> `{key: "value"}`) is not supported. + +## `artifacts:reports:junit` + +The `junit` report collects [JUnit report format XML files](https://www.ibm.com/docs/en/adfz/developer-for-zos/14.1.0?topic=formats-junit-xml-format). +The collected Unit test reports upload to GitLab as an artifact. Although JUnit was originally developed in Java, there +are many third-party ports for other languages such as JavaScript, Python, and Ruby. + +See [Unit test reports](../unit_test_reports.md) for more details and examples. +Below is an example of collecting a JUnit report format XML file from Ruby's RSpec test tool: + +```yaml +rspec: + stage: test + script: + - bundle install + - rspec --format RspecJunitFormatter --out rspec.xml + artifacts: + reports: + junit: rspec.xml +``` + +GitLab can display the results of one or more reports in: + +- The merge request [code quality widget](../../ci/unit_test_reports.md#how-it-works). +- The [full report](../../ci/unit_test_reports.md#viewing-unit-test-reports-on-gitlab). + +Some JUnit tools export to multiple XML files. You can specify multiple test report paths in a single job to +concatenate them into a single file. Use either: + +- A filename pattern (`junit: rspec-*.xml`). +- an array of filenames (`junit: [rspec-1.xml, rspec-2.xml, rspec-3.xml]`). +- A Combination of both (`junit: [rspec.xml, test-results/TEST-*.xml]`). + +## `artifacts:reports:license_scanning` **(ULTIMATE)** + +> Introduced in GitLab 12.8. + +The License Compliance report collects [Licenses](../../user/compliance/license_compliance/index.md). The License +Compliance report uploads to GitLab as an artifact. + +GitLab can display the results of one or more reports in: + +- The merge request [license compliance widget](../../user/compliance/license_compliance/index.md). +- The [license list](../../user/compliance/license_compliance/index.md#license-list). + +## `artifacts:reports:load_performance` **(PREMIUM)** + +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/35260) in GitLab 13.2. +> - Requires GitLab Runner 11.5 and above. + +The `load_performance` report collects [Load Performance Testing metrics](../../user/project/merge_requests/load_performance_testing.md). +The report is uploaded to GitLab as an artifact. + +GitLab can display the results of only one report in the merge request +[load testing widget](../../user/project/merge_requests/load_performance_testing.md#how-load-performance-testing-works). + +GitLab cannot display the combined results of multiple `load_performance` reports. + +## `artifacts:reports:metrics` **(PREMIUM)** + +The `metrics` report collects [Metrics](../metrics_reports.md). The collected Metrics report uploads to GitLab as an +artifact. + +GitLab can display the results of one or more reports in the merge request +[metrics reports widget](../../ci/metrics_reports.md#metrics-reports). + +## `artifacts:reports:requirements` **(ULTIMATE)** + +> - [Introduced](https://gitlab.com/groups/gitlab-org/-/epics/2859) in GitLab 13.1. + +The `requirements` report collects `requirements.json` files. The collected Requirements report uploads to GitLab as an +artifact and existing [requirements](../../user/project/requirements/index.md) are marked as Satisfied. + +GitLab can display the results of one or more reports in the +[project requirements](../../user/project/requirements/index.md#view-a-requirement). + +## `artifacts:reports:sast` + +> - [Moved](https://gitlab.com/groups/gitlab-org/-/epics/2098) from GitLab Ultimate to GitLab Free in 13.3. + +The `sast` report collects [SAST vulnerabilities](../../user/application_security/sast/index.md). The collected SAST +report uploads to GitLab as an artifact. + +GitLab can display the results of one or more reports in: + +- The merge request [SAST widget](../../user/application_security/sast/index.md#static-application-security-testing-sast). +- The [security dashboard](../../user/application_security/security_dashboard/index.md). + +## `artifacts:reports:secret_detection` + +> - Introduced in GitLab 13.1. +> - [Moved](https://gitlab.com/gitlab-org/gitlab/-/issues/222788) to GitLab Free in 13.3. +> - Requires GitLab Runner 11.5 and above. + +The `secret-detection` report collects [detected secrets](../../user/application_security/secret_detection/index.md). +The collected Secret Detection report is uploaded to GitLab. + +GitLab can display the results of one or more reports in: + +- The merge request [secret scanning widget](../../user/application_security/secret_detection/index.md). +- The [pipeline **Security** tab](../../user/application_security/index.md#view-security-scan-information-in-the-pipeline-security-tab). +- The [security dashboard](../../user/application_security/security_dashboard/index.md). + +## `artifacts:reports:terraform` + +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/207528) in GitLab 13.0. +> - Requires [GitLab Runner](https://docs.gitlab.com/runner/) 11.5 and above. + +The `terraform` report obtains a Terraform `tfplan.json` file. [JQ processing required to remove credentials](../../user/infrastructure/iac/mr_integration.md#configure-terraform-report-artifacts). +The collected Terraform plan report uploads to GitLab as an artifact. + +GitLab can display the results of one or more reports in the merge request +[terraform widget](../../user/infrastructure/iac/mr_integration.md#output-terraform-plan-information-into-a-merge-request). + +For more information, see [Output `terraform plan` information into a merge request](../../user/infrastructure/iac/mr_integration.md). diff --git a/doc/ci/yaml/includes.md b/doc/ci/yaml/includes.md index 5e2eb53a0ea..3c94ddb3c14 100644 --- a/doc/ci/yaml/includes.md +++ b/doc/ci/yaml/includes.md @@ -238,7 +238,7 @@ In `include` sections in your `.gitlab-ci.yml` file, you can use: In GitLab 14.5 and later, you can also use: -- [Trigger variables](../triggers/index.md#making-use-of-trigger-variables). +- [Trigger variables](../triggers/index.md#pass-cicd-variables-in-the-api-call). - [Scheduled pipeline variables](../pipelines/schedules.md#using-variables). - [Manual pipeline run variables](../variables/index.md#override-a-variable-when-running-a-pipeline-manually). - Pipeline [predefined variables](../variables/predefined_variables.md). diff --git a/doc/ci/yaml/index.md b/doc/ci/yaml/index.md index 5702c7a7dfd..ed05ef08d02 100644 --- a/doc/ci/yaml/index.md +++ b/doc/ci/yaml/index.md @@ -16,6 +16,8 @@ This document lists the configuration options for your GitLab `.gitlab-ci.yml` f When you are editing your `.gitlab-ci.yml` file, you can validate it with the [CI Lint](../lint.md) tool. +If you are editing this page, make sure you follow the [CI/CD YAML reference style guide](../../development/cicd/cicd_reference_documentation_guide.md). + ## Keywords A GitLab CI/CD pipeline configuration includes: @@ -25,9 +27,9 @@ A GitLab CI/CD pipeline configuration includes: | Keyword | Description | |-------------------------|:------------| | [`default`](#default) | Custom default values for job keywords. | + | [`include`](#include) | Import configuration from other YAML files. | | [`stages`](#stages) | The names and order of the pipeline stages. | | [`workflow`](#workflow) | Control what types of pipeline run. | - | [`include`](#include) | Import configuration from other YAML files. | - [Jobs](../jobs/index.md) configured with [job keywords](#job-keywords): @@ -73,7 +75,7 @@ or import additional pipeline configuration. ### `default` You can set global defaults for some keywords. Jobs that do not define one or more -of the listed keywords use the value defined in the `default:` section. +of the listed keywords use the value defined in the `default` section. **Keyword type**: Global keyword. @@ -90,7 +92,7 @@ of the listed keywords use the value defined in the `default:` section. - [`tags`](#tags) - [`timeout`](#timeout) -**Example of `default`:** +**Example of `default`**: ```yaml default: @@ -106,7 +108,7 @@ rspec 2.7: In this example, `ruby:3.0` is the default `image` value for all jobs in the pipeline. The `rspec 2.7` job does not use the default, because it overrides the default with -a job-specific `image:` section: +a job-specific `image` section: **Additional details**: @@ -116,179 +118,6 @@ a job-specific `image:` section: takes precedence and is not replaced by the default. - Control inheritance of default keywords in jobs with [`inherit:default`](#inheritdefault). -### `stages` - -Use `stages` to define stages that contain groups of jobs. Use [`stage`](#stage) -in a job to configure the job to run in a specific stage. - -If `stages` is not defined in the `.gitlab-ci.yml` file, the default pipeline stages are: - -- [`.pre`](#stage-pre) -- `build` -- `test` -- `deploy` -- [`.post`](#stage-post) - -The order of the items in `stages` defines the execution order for jobs: - -- Jobs in the same stage run in parallel. -- Jobs in the next stage run after the jobs from the previous stage complete successfully. - -**Keyword type**: Global keyword. - -**Example of `stages`:** - -```yaml -stages: - - build - - test - - deploy -``` - -In this example: - -1. All jobs in `build` execute in parallel. -1. If all jobs in `build` succeed, the `test` jobs execute in parallel. -1. If all jobs in `test` succeed, the `deploy` jobs execute in parallel. -1. If all jobs in `deploy` succeed, the pipeline is marked as `passed`. - -If any job fails, the pipeline is marked as `failed` and jobs in later stages do not -start. Jobs in the current stage are not stopped and continue to run. - -**Additional details**: - -- If a job does not specify a [`stage`](#stage), the job is assigned the `test` stage. -- If a stage is defined but no jobs use it, the stage is not visible in the pipeline, - which can help [compliance pipeline configurations](../../user/project/settings/index.md#compliance-pipeline-configuration): - - Stages can be defined in the compliance configuration but remain hidden if not used. - - The defined stages become visible when developers use them in job definitions. - -**Related topics**: - -- To make a job start earlier and ignore the stage order, use the [`needs`](#needs) keyword. - -### `workflow` - -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/29654) in GitLab 12.5 - -Use [`workflow`](workflow.md) to control pipeline behavior. - -**Related topics**: - -- [`workflow: rules` examples](workflow.md#workflow-rules-examples) -- [Switch between branch pipelines and merge request pipelines](workflow.md#switch-between-branch-pipelines-and-merge-request-pipelines) - -#### `workflow:rules` - -The `rules` keyword in `workflow` is similar to [`rules:` defined in jobs](#rules), -but controls whether or not a whole pipeline is created. - -When no rules evaluate to true, the pipeline does not run. - -**Possible inputs**: You can use some of the same keywords as job-level [`rules`](#rules): - -- [`rules: if`](#rulesif). -- [`rules: changes`](#ruleschanges). -- [`rules: exists`](#rulesexists). -- [`when`](#when), can only be `always` or `never` when used with `workflow`. -- [`variables`](#workflowrulesvariables). - -**Example of `workflow:rules`:** - -```yaml -workflow: - rules: - - if: $CI_COMMIT_MESSAGE =~ /-draft$/ - when: never - - if: $CI_PIPELINE_SOURCE == "merge_request_event" - - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH -``` - -In this example, pipelines run if the commit message does not have `-drafts` in it -and the pipeline is for either: - -- A merge request -- The default branch. - -**Additional details**: - -- If your rules match both branch pipelines (other than the default branch) and merge request pipelines, - [duplicate pipelines](../jobs/job_control.md#avoid-duplicate-pipelines) can occur. - -**Related topics**: - -- You can use the [`workflow:rules` templates](workflow.md#workflowrules-templates) to import - a preconfigured `workflow: rules` entry. -- [Common `if` clauses for `workflow:rules`](workflow.md#common-if-clauses-for-workflowrules). - -#### `workflow:rules:variables` - -> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/294232) in GitLab 13.11. -> - [Feature flag removed](https://gitlab.com/gitlab-org/gitlab/-/issues/300997) in GitLab 14.1. - -You can use [`variables`](#variables) in `workflow:rules:` to define variables for -specific pipeline conditions. - -When the condition matches, the variable is created and can be used by all jobs -in the pipeline. If the variable is already defined at the global level, the `workflow` -variable takes precedence and overrides the global variable. - -**Keyword type**: Global keyword. - -**Possible inputs**: Variable name and value pairs: - -- The name can use only numbers, letters, and underscores (`_`). -- The value must be a string. - -**Example of `workflow:rules:variables`:** - -```yaml -variables: - DEPLOY_VARIABLE: "default-deploy" - -workflow: - rules: - - if: $CI_COMMIT_REF_NAME == $CI_DEFAULT_BRANCH - variables: - DEPLOY_VARIABLE: "deploy-production" # Override globally-defined DEPLOY_VARIABLE - - if: $CI_COMMIT_REF_NAME =~ /feature/ - variables: - IS_A_FEATURE: "true" # Define a new variable. - - when: always # Run the pipeline in other cases - -job1: - variables: - DEPLOY_VARIABLE: "job1-default-deploy" - rules: - - if: $CI_COMMIT_REF_NAME == $CI_DEFAULT_BRANCH - variables: # Override DEPLOY_VARIABLE defined - DEPLOY_VARIABLE: "job1-deploy-production" # at the job level. - - when: on_success # Run the job in other cases - script: - - echo "Run script with $DEPLOY_VARIABLE as an argument" - - echo "Run another script if $IS_A_FEATURE exists" - -job2: - script: - - echo "Run script with $DEPLOY_VARIABLE as an argument" - - echo "Run another script if $IS_A_FEATURE exists" -``` - -When the branch is the default branch: - -- job1's `DEPLOY_VARIABLE` is `job1-deploy-production`. -- job2's `DEPLOY_VARIABLE` is `deploy-production`. - -When the branch is `feature`: - -- job1's `DEPLOY_VARIABLE` is `job1-default-deploy`, and `IS_A_FEATURE` is `true`. -- job2's `DEPLOY_VARIABLE` is `default-deploy`, and `IS_A_FEATURE` is `true`. - -When the branch is something else: - -- job1's `DEPLOY_VARIABLE` is `job1-default-deploy`. -- job2's `DEPLOY_VARIABLE` is `default-deploy`. - ### `include` > [Moved](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/42861) to GitLab Free in 11.4. @@ -372,8 +201,10 @@ use `include:file`. You can use `include:file` in combination with `include:proj **Keyword type**: Global keyword. -**Possible inputs**: A full path, relative to the root directory (`/`). -The YAML file must have the extension `.yml` or `.yaml`. +**Possible inputs**: + +- A full path, relative to the root directory (`/`). The YAML file must have the + extension `.yml` or `.yaml`. **Example of `include:file`**: @@ -428,10 +259,10 @@ Use `include:remote` with a full URL to include a file from a different location **Keyword type**: Global keyword. -**Possible inputs**: A public URL accessible by an HTTP/HTTPS `GET` request. -Authentication with the remote URL is not supported. +**Possible inputs**: -The YAML file must have the extension `.yml` or `.yaml`. +- A public URL accessible by an HTTP/HTTPS `GET` request. Authentication with the + remote URL is not supported. The YAML file must have the extension `.yml` or `.yaml`. **Example of `include:remote`**: @@ -454,7 +285,9 @@ Use `include:template` to include [`.gitlab-ci.yml` templates](https://gitlab.co **Keyword type**: Global keyword. -**Possible inputs**: [`.gitlab-ci.yml` templates](https://gitlab.com/gitlab-org/gitlab/-/tree/master/lib/gitlab/ci/templates). +**Possible inputs**: + +- [`.gitlab-ci.yml` templates](https://gitlab.com/gitlab-org/gitlab/-/tree/master/lib/gitlab/ci/templates). **Example of `include:template`**: @@ -477,224 +310,189 @@ include: - All [nested includes](includes.md#use-nested-includes) are executed only with the permission of the user, so it's possible to use `project`, `remote`, or `template` includes. -## Job keywords +### `stages` -The following topics explain how to use keywords to configure CI/CD pipelines. +Use `stages` to define stages that contain groups of jobs. Use [`stage`](#stage) +in a job to configure the job to run in a specific stage. -### `image` +If `stages` is not defined in the `.gitlab-ci.yml` file, the default pipeline stages are: -Use `image` to specify a Docker image that the job runs in. +- [`.pre`](#stage-pre) +- `build` +- `test` +- `deploy` +- [`.post`](#stage-post) -**Keyword type**: Job keyword. You can use it only as part of a job or in the -[`default:` section](#default). +The order of the items in `stages` defines the execution order for jobs: -**Possible inputs**: The name of the image, including the registry path if needed, in one of these formats: +- Jobs in the same stage run in parallel. +- Jobs in the next stage run after the jobs from the previous stage complete successfully. -- `<image-name>` (Same as using `<image-name>` with the `latest` tag) -- `<image-name>:<tag>` -- `<image-name>@<digest>` +**Keyword type**: Global keyword. -**Example of `image`**: +**Example of `stages`**: ```yaml -default: - image: ruby:3.0 - -rspec: - script: bundle exec rspec - -rspec 2.7: - image: registry.example.com/my-group/my-project/ruby:2.7 - script: bundle exec rspec +stages: + - build + - test + - deploy ``` -In this example, the `ruby:3.0` image is the default for all jobs in the pipeline. -The `rspec 2.7` job does not use the default, because it overrides the default with -a job-specific `image:` section. +In this example: -**Related topics**: +1. All jobs in `build` execute in parallel. +1. If all jobs in `build` succeed, the `test` jobs execute in parallel. +1. If all jobs in `test` succeed, the `deploy` jobs execute in parallel. +1. If all jobs in `deploy` succeed, the pipeline is marked as `passed`. -- [Run your CI/CD jobs in Docker containers](../docker/using_docker_images.md). +If any job fails, the pipeline is marked as `failed` and jobs in later stages do not +start. Jobs in the current stage are not stopped and continue to run. -#### `image:name` +**Additional details**: -The name of the Docker image that the job runs in. Similar to [`image:`](#image) used by itself. +- If a job does not specify a [`stage`](#stage), the job is assigned the `test` stage. +- If a stage is defined but no jobs use it, the stage is not visible in the pipeline, + which can help [compliance pipeline configurations](../../user/project/settings/index.md#compliance-pipeline-configuration): + - Stages can be defined in the compliance configuration but remain hidden if not used. + - The defined stages become visible when developers use them in job definitions. -**Keyword type**: Job keyword. You can use it only as part of a job or in the -[`default:` section](#default). +**Related topics**: -**Possible inputs**: The name of the image, including the registry path if needed, in one of these formats: +- To make a job start earlier and ignore the stage order, use the [`needs`](#needs) keyword. -- `<image-name>` (Same as using `<image-name>` with the `latest` tag) -- `<image-name>:<tag>` -- `<image-name>@<digest>` +### `workflow` -**Example of `image:name`**: +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/29654) in GitLab 12.5 -```yaml -image: - name: "registry.example.com/my/image:latest" -``` +Use [`workflow`](workflow.md) to control pipeline behavior. **Related topics**: -- [Run your CI/CD jobs in Docker containers](../docker/using_docker_images.md). +- [`workflow: rules` examples](workflow.md#workflow-rules-examples) +- [Switch between branch pipelines and merge request pipelines](workflow.md#switch-between-branch-pipelines-and-merge-request-pipelines) -#### `image:entrypoint` +#### `workflow:rules` -Command or script to execute as the container's entry point. +The `rules` keyword in `workflow` is similar to [`rules` defined in jobs](#rules), +but controls whether or not a whole pipeline is created. -When the Docker container is created, the `entrypoint` is translated to the Docker `--entrypoint` option. -The syntax is similar to the [Dockerfile `ENTRYPOINT` directive](https://docs.docker.com/engine/reference/builder/#entrypoint), -where each shell token is a separate string in the array. +When no rules evaluate to true, the pipeline does not run. -**Keyword type**: Job keyword. You can use it only as part of a job or in the -[`default:` section](#default). +**Possible inputs**: You can use some of the same keywords as job-level [`rules`](#rules): -**Possible inputs**: A string. +- [`rules: if`](#rulesif). +- [`rules: changes`](#ruleschanges). +- [`rules: exists`](#rulesexists). +- [`when`](#when), can only be `always` or `never` when used with `workflow`. +- [`variables`](#workflowrulesvariables). -**Example of `image:entrypoint`**: +**Example of `workflow:rules`**: ```yaml -image: - name: super/sql:experimental - entrypoint: [""] +workflow: + rules: + - if: $CI_COMMIT_MESSAGE =~ /-draft$/ + when: never + - if: $CI_PIPELINE_SOURCE == "merge_request_event" + - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH ``` -**Related topics**: - -- [Override the entrypoint of an image](../docker/using_docker_images.md#override-the-entrypoint-of-an-image). - -#### `services` - -Use `services` to specify an additional Docker image to run scripts in. The [`services` image](../services/index.md) is linked -to the image specified in the [`image`](#image) keyword. - -**Keyword type**: Job keyword. You can use it only as part of a job or in the -[`default:` section](#default). - -**Possible inputs**: The name of the services image, including the registry path if needed, in one of these formats: - -- `<image-name>` (Same as using `<image-name>` with the `latest` tag) -- `<image-name>:<tag>` -- `<image-name>@<digest>` +In this example, pipelines run if the commit message does not have `-drafts` in it +and the pipeline is for either: -**Example of `services`**: +- A merge request +- The default branch. -```yaml -default: - image: - name: ruby:2.6 - entrypoint: ["/bin/bash"] +**Additional details**: - services: - - name: my-postgres:11.7 - alias: db-postgres - entrypoint: ["/usr/local/bin/db-postgres"] - command: ["start"] +- If your rules match both branch pipelines (other than the default branch) and merge request pipelines, + [duplicate pipelines](../jobs/job_control.md#avoid-duplicate-pipelines) can occur. - before_script: - - bundle install +**Related topics**: -test: - script: - - bundle exec rake spec -``` +- You can use the [`workflow:rules` templates](workflow.md#workflowrules-templates) to import + a preconfigured `workflow: rules` entry. +- [Common `if` clauses for `workflow:rules`](workflow.md#common-if-clauses-for-workflowrules). -In this example, the job launches a Ruby container. Then, from that container, the job launches -another container that's running PostgreSQL. Then the job then runs scripts -in that container. +#### `workflow:rules:variables` -**Related topics**: +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/294232) in GitLab 13.11. +> - [Feature flag removed](https://gitlab.com/gitlab-org/gitlab/-/issues/300997) in GitLab 14.1. -- [Available settings for `services`](../services/index.md#available-settings-for-services). -- [Define `services` in the `.gitlab-ci.yml` file](../services/index.md#define-services-in-the-gitlab-ciyml-file). -- [Run your CI/CD jobs in Docker containers](../docker/using_docker_images.md). -- [Use Docker to build Docker images](../docker/using_docker_build.md). +You can use [`variables`](#variables) in `workflow:rules` to define variables for +specific pipeline conditions. -### `script` +When the condition matches, the variable is created and can be used by all jobs +in the pipeline. If the variable is already defined at the global level, the `workflow` +variable takes precedence and overrides the global variable. -Use `script` to specify commands for the runner to execute. +**Keyword type**: Global keyword. -All jobs except [trigger jobs](#trigger) require a `script` keyword. +**Possible inputs**: Variable name and value pairs: -**Keyword type**: Job keyword. You can use it only as part of a job. +- The name can use only numbers, letters, and underscores (`_`). +- The value must be a string. -**Possible inputs**: An array including: +**Example of `workflow:rules:variables`**: -- Single line commands. -- Long commands [split over multiple lines](script.md#split-long-commands). -- [YAML anchors](yaml_optimization.md#yaml-anchors-for-scripts). +```yaml +variables: + DEPLOY_VARIABLE: "default-deploy" -**Example of `script`:** +workflow: + rules: + - if: $CI_COMMIT_REF_NAME == $CI_DEFAULT_BRANCH + variables: + DEPLOY_VARIABLE: "deploy-production" # Override globally-defined DEPLOY_VARIABLE + - if: $CI_COMMIT_REF_NAME =~ /feature/ + variables: + IS_A_FEATURE: "true" # Define a new variable. + - when: always # Run the pipeline in other cases -```yaml job1: - script: "bundle exec rspec" + variables: + DEPLOY_VARIABLE: "job1-default-deploy" + rules: + - if: $CI_COMMIT_REF_NAME == $CI_DEFAULT_BRANCH + variables: # Override DEPLOY_VARIABLE defined + DEPLOY_VARIABLE: "job1-deploy-production" # at the job level. + - when: on_success # Run the job in other cases + script: + - echo "Run script with $DEPLOY_VARIABLE as an argument" + - echo "Run another script if $IS_A_FEATURE exists" job2: script: - - uname -a - - bundle exec rspec + - echo "Run script with $DEPLOY_VARIABLE as an argument" + - echo "Run another script if $IS_A_FEATURE exists" ``` -**Additional details**: - -- When you use [these special characters in `script`](script.md#use-special-characters-with-script), you must use single quotes (`'`) or double quotes (`"`) . - -**Related topics**: - -- You can [ignore non-zero exit codes](script.md#ignore-non-zero-exit-codes). -- [Use color codes with `script`](script.md#add-color-codes-to-script-output) - to make job logs easier to review. -- [Create custom collapsible sections](../jobs/index.md#custom-collapsible-sections) - to simplify job log output. - -#### `before_script` - -Use `before_script` to define an array of commands that should run before each job's -`script` commands, but after [artifacts](#artifacts) are restored. - -**Keyword type**: Job keyword. You can use it only as part of a job or in the -[`default:` section](#default). - -**Possible inputs**: An array including: +When the branch is the default branch: -- Single line commands. -- Long commands [split over multiple lines](script.md#split-long-commands). -- [YAML anchors](yaml_optimization.md#yaml-anchors-for-scripts). +- job1's `DEPLOY_VARIABLE` is `job1-deploy-production`. +- job2's `DEPLOY_VARIABLE` is `deploy-production`. -**Example of `before_script`:** +When the branch is `feature`: -```yaml -job: - before_script: - - echo "Execute this command before any 'script:' commands." - script: - - echo "This command executes after the job's 'before_script' commands." -``` +- job1's `DEPLOY_VARIABLE` is `job1-default-deploy`, and `IS_A_FEATURE` is `true`. +- job2's `DEPLOY_VARIABLE` is `default-deploy`, and `IS_A_FEATURE` is `true`. -**Additional details**: +When the branch is something else: -- Scripts you specify in `before_script` are concatenated with any scripts you specify - in the main [`script`](#script). The combined scripts execute together in a single shell. +- job1's `DEPLOY_VARIABLE` is `job1-default-deploy`. +- job2's `DEPLOY_VARIABLE` is `default-deploy`. -**Related topics**: +## Job keywords -- [Use `before_script` with `default`](script.md#set-a-default-before_script-or-after_script-for-all-jobs) - to define a default array of commands that should run before the `script` commands in all jobs. -- You can [ignore non-zero exit codes](script.md#ignore-non-zero-exit-codes). -- [Use color codes with `before_script`](script.md#add-color-codes-to-script-output) - to make job logs easier to review. -- [Create custom collapsible sections](../jobs/index.md#custom-collapsible-sections) - to simplify job log output. +The following topics explain how to use keywords to configure CI/CD pipelines. -#### `after_script` +### `after_script` Use `after_script` to define an array of commands that run after each job, including failed jobs. **Keyword type**: Job keyword. You can use it only as part of a job or in the -[`default:` section](#default). +[`default` section](#default). **Possible inputs**: An array including: @@ -702,7 +500,7 @@ Use `after_script` to define an array of commands that run after each job, inclu - Long commands [split over multiple lines](script.md#split-long-commands). - [YAML anchors](yaml_optimization.md#yaml-anchors-for-scripts). -**Example of `after_script`:** +**Example of `after_script`**: ```yaml job: @@ -740,1168 +538,922 @@ If a job times out or is cancelled, the `after_script` commands do not execute. - [Create custom collapsible sections](../jobs/index.md#custom-collapsible-sections) to simplify job log output. -### `stage` +### `allow_failure` -Use `stage` to define which [stage](#stages) a job runs in. Jobs in the same -`stage` can execute in parallel (see **Additional details**). +Use `allow_failure` to determine whether a pipeline should continue running when a job fails. -If `stage` is not defined, the job uses the `test` stage by default. +- To let the pipeline continue running subsequent jobs, use `allow_failure: true`. +- To stop the pipeline from running subsequent jobs, use `allow_failure: false`. + +When jobs are allowed to fail (`allow_failure: true`) an orange warning (**{status_warning}**) +indicates that a job failed. However, the pipeline is successful and the associated commit +is marked as passed with no warnings. + +This same warning is displayed when: + +- All other jobs in the stage are successful. +- All other jobs in the pipeline are successful. + +The default value for `allow_failure` is: + +- `true` for [manual jobs](../jobs/job_control.md#create-a-job-that-must-be-run-manually). +- `false` for manual jobs that also use [`rules`](#rules). +- `false` in all other cases. **Keyword type**: Job keyword. You can use it only as part of a job. -**Possible inputs**: An array including any number of stage names. Stage names can be: +**Possible inputs**: -- The [default stages](#stages). -- User-defined stages. +- `true` or `false`. -**Example of `stage`**: +**Example of `allow_failure`**: ```yaml -stages: - - build - - test - - deploy - job1: - stage: build + stage: test script: - - echo "This job compiles code." + - execute_script_1 job2: stage: test script: - - echo "This job tests the compiled code. It runs when the build stage completes." + - execute_script_2 + allow_failure: true job3: - script: - - echo "This job also runs in the test stage". - -job4: stage: deploy script: - - echo "This job deploys the code. It runs when the test stage completes." + - deploy_to_staging ``` -**Additional details**: +In this example, `job1` and `job2` run in parallel: -- Jobs can run in parallel if they run on different runners. -- If you have only one runner, jobs can run in parallel if the runner's - [`concurrent` setting](https://docs.gitlab.com/runner/configuration/advanced-configuration.html#the-global-section) - is greater than `1`. +- If `job1` fails, jobs in the `deploy` stage do not start. +- If `job2` fails, jobs in the `deploy` stage can still start. -#### `stage: .pre` +**Additional details**: -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/31441) in GitLab 12.4. +- You can use `allow_failure` as a subkey of [`rules`](#rulesallow_failure). +- You can use `allow_failure: false` with a manual job to create a [blocking manual job](../jobs/job_control.md#types-of-manual-jobs). + A blocked pipeline does not run any jobs in later stages until the manual job + is started and completes successfully. -Use the `.pre` stage to make a job run at the start of a pipeline. `.pre` is -always the first stage in a pipeline. User-defined stages execute after `.pre`. -You do not have to define `.pre` in [`stages`](#stages). +#### `allow_failure:exit_codes` -You must have a job in at least one stage other than `.pre` or `.post`. +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/273157) in GitLab 13.8. +> - [Feature flag removed](https://gitlab.com/gitlab-org/gitlab/-/issues/292024) in GitLab 13.9. -**Keyword type**: You can only use it with a job's `stage` keyword. +Use `allow_failure:exit_codes` to control when a job should be +allowed to fail. The job is `allow_failure: true` for any of the listed exit codes, +and `allow_failure` false for any other exit code. -**Example of `stage: .pre`**: +**Keyword type**: Job keyword. You can use it only as part of a job. -```yaml -stages: - - build - - test +**Possible inputs**: -job1: - stage: build - script: - - echo "This job runs in the build stage." +- A single exit code. +- An array of exit codes. -first-job: - stage: .pre +**Example of `allow_failure`**: + +```yaml +test_job_1: script: - - echo "This job runs in the .pre stage, before all other stages." + - echo "Run a script that results in exit code 1. This job fails." + - exit 1 + allow_failure: + exit_codes: 137 -job2: - stage: test +test_job_2: script: - - echo "This job runs in the test stage." + - echo "Run a script that results in exit code 137. This job is allowed to fail." + - exit 137 + allow_failure: + exit_codes: + - 137 + - 255 ``` -#### `stage: .post` - -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/31441) in GitLab 12.4. - -Use the `.post` stage to make a job run at the end of a pipeline. `.post` -is always the last stage in a pipeline. User-defined stages execute before `.post`. -You do not have to define `.post` in [`stages`](#stages). - -You must have a job in at least one stage other than `.pre` or `.post`. - -**Keyword type**: You can only use it with a job's `stage` keyword. - -**Example of `stage: .post`**: - -```yaml -stages: - - build - - test - -job1: - stage: build - script: - - echo "This job runs in the build stage." +### `artifacts` -last-job: - stage: .post - script: - - echo "This job runs in the .post stage, after all other stages." +Use `artifacts` to specify which files to save as [job artifacts](../pipelines/job_artifacts.md). +Job artifacts are a list of files and directories that are +attached to the job when it [succeeds, fails, or always](#artifactswhen). -job2: - stage: test - script: - - echo "This job runs in the test stage." -``` +The artifacts are sent to GitLab after the job finishes. They are +available for download in the GitLab UI if the size is smaller than the +the [maximum artifact size](../../user/gitlab_com/index.md#gitlab-cicd). -### `extends` +By default, jobs in later stages automatically download all the artifacts created +by jobs in earlier stages. You can control artifact download behavior in jobs with +[`dependencies`](#dependencies). -Use `extends` to reuse configuration sections. It's an alternative to [YAML anchors](yaml_optimization.md#anchors) -and is a little more flexible and readable. +When using the [`needs`](#needs) keyword, jobs can only download +artifacts from the jobs defined in the `needs` configuration. -**Keyword type**: Job keyword. You can use it only as part of a job. +Job artifacts are only collected for successful jobs by default, and +artifacts are restored after [caches](#cache). -**Possible inputs:** +[Read more about artifacts](../pipelines/job_artifacts.md). -- The name of another job in the pipeline. -- A list (array) of names of other jobs in the pipeline. +#### `artifacts:exclude` -**Example of `extends`:** +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/15122) in GitLab 13.1 +> - Requires GitLab Runner 13.1 -```yaml -.tests: - script: rake test - stage: test - only: - refs: - - branches +Use `artifacts:exclude` to prevent files from being added to an artifacts archive. -rspec: - extends: .tests - script: rake rspec - only: - variables: - - $RSPEC -``` +**Keyword type**: Job keyword. You can use it only as part of a job or in the +[`default` section](#default). -In this example, the `rspec` job uses the configuration from the `.tests` template job. -When creating the pipeline, GitLab: +**Possible inputs**: -- Performs a reverse deep merge based on the keys. -- Merges the `.tests` content with the `rspec` job. -- Doesn't merge the values of the keys. +- An array of file paths, relative to the project directory. +- You can use Wildcards that use [glob](https://en.wikipedia.org/wiki/Glob_(programming)) or + [`doublestar.PathMatch`](https://pkg.go.dev/github.com/bmatcuk/doublestar@v1.2.2?tab=doc#PathMatch) patterns. -The result is this `rspec` job: +**Example of `artifacts:exclude`**: ```yaml -rspec: - script: rake rspec - stage: test - only: - refs: - - branches - variables: - - $RSPEC +artifacts: + paths: + - binaries/ + exclude: + - binaries/**/*.o ``` -**Additional details:** +This example stores all files in `binaries/`, but not `*.o` files located in +subdirectories of `binaries/`. -- In GitLab 12.0 and later, you can use multiple parents for `extends`. -- The `extends` keyword supports up to eleven levels of inheritance, but you should - avoid using more than three levels. -- In the example above, `.tests` is a [hidden job](../jobs/index.md#hide-jobs), - but you can extend configuration from regular jobs as well. +**Additional details**: -**Related topics:** +- `artifacts:exclude` paths are not searched recursively. +- Files matched by [`artifacts:untracked`](#artifactsuntracked) can be excluded using + `artifacts:exclude` too. -- [Reuse configuration sections by using `extends`](yaml_optimization.md#use-extends-to-reuse-configuration-sections). -- Use `extends` to reuse configuration from [included configuration files](yaml_optimization.md#use-extends-and-include-together). - -### `rules` +**Related topics**: -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/27863) in GitLab 12.3. +- [Exclude files from job artifacts](../pipelines/job_artifacts.md#exclude-files-from-job-artifacts). -Use `rules` to include or exclude jobs in pipelines. +#### `artifacts:expire_in` -Rules are evaluated when the pipeline is created, and evaluated *in order* -until the first match. When a match is found, the job is either included or excluded from the pipeline, -depending on the configuration. +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/16267) in GitLab 13.0 behind a disabled feature flag, the latest job artifacts are kept regardless of expiry time. +> - [Made default behavior](https://gitlab.com/gitlab-org/gitlab/-/issues/229936) in GitLab 13.4. +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/241026) in GitLab 13.8, keeping latest job artifacts can be disabled at the project level. +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/276583) in GitLab 13.9, keeping latest job artifacts can be disabled instance-wide. +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/321323) in GitLab 13.12, the latest pipeline artifacts are kept regardless of expiry time. -You cannot use dotenv variables created in job scripts in rules, because rules are evaluated before any jobs run. +Use `expire_in` to specify how long [job artifacts](../pipelines/job_artifacts.md) are stored before +they expire and are deleted. The `expire_in` setting does not affect: -`rules` replaces [`only/except`](#only--except) and they can't be used together -in the same job. If you configure one job to use both keywords, the GitLab returns -a `key may not be used with rules` error. +- Artifacts from the latest job, unless keeping the latest job artifacts is: + - [Disabled at the project level](../pipelines/job_artifacts.md#keep-artifacts-from-most-recent-successful-jobs). + - [Disabled instance-wide](../../user/admin_area/settings/continuous_integration.md#keep-the-latest-artifacts-for-all-jobs-in-the-latest-successful-pipelines). +- [Pipeline artifacts](../pipelines/pipeline_artifacts.md). You can't specify an expiration date for + pipeline artifacts. See [When pipeline artifacts are deleted](../pipelines/pipeline_artifacts.md#when-pipeline-artifacts-are-deleted) + for more information. -`rules` accepts an array of rules defined with: +After their expiry, artifacts are deleted hourly by default (using a cron job), and are not +accessible anymore. -- `if` -- `changes` -- `exists` -- `allow_failure` -- `variables` -- `when` +**Keyword type**: Job keyword. You can use it only as part of a job or in the +[`default` section](#default). -You can combine multiple keywords together for [complex rules](../jobs/job_control.md#complex-rules). +**Possible inputs**: The expiry time. If no unit is provided, the time is in seconds. +Valid values include: -The job is added to the pipeline: +- `'42'` +- `42 seconds` +- `3 mins 4 sec` +- `2 hrs 20 min` +- `2h20min` +- `6 mos 1 day` +- `47 yrs 6 mos and 4d` +- `3 weeks and 2 days` +- `never` -- If an `if`, `changes`, or `exists` rule matches and also has `when: on_success` (default), - `when: delayed`, or `when: always`. -- If a rule is reached that is only `when: on_success`, `when: delayed`, or `when: always`. +**Example of `artifacts:expire_in`**: -The job is not added to the pipeline: +```yaml +job: + artifacts: + expire_in: 1 week +``` -- If no rules match. -- If a rule matches and has `when: never`. +**Additional details**: -You can use [`!reference` tags](yaml_optimization.md#reference-tags) to [reuse `rules` configuration](../jobs/job_control.md#reuse-rules-in-different-jobs) -in different jobs. +- The expiration time period begins when the artifact is uploaded and stored on GitLab. + If the expiry time is not defined, it defaults to the [instance wide setting](../../user/admin_area/settings/continuous_integration.md#default-artifacts-expiration). +- To override the expiration date and protect artifacts from being automatically deleted: + - Select **Keep** on the job page. + - [In GitLab 13.3 and later](https://gitlab.com/gitlab-org/gitlab/-/issues/22761), set the value of + `expire_in` to `never`. -#### `rules:if` +#### `artifacts:expose_as` -Use `rules:if` clauses to specify when to add a job to a pipeline: +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/15018) in GitLab 12.5. -- If an `if` statement is true, add the job to the pipeline. -- If an `if` statement is true, but it's combined with `when: never`, do not add the job to the pipeline. -- If no `if` statements are true, do not add the job to the pipeline. +Use the `artifacts:expose_as` keyword to +[expose job artifacts in the merge request UI](../pipelines/job_artifacts.md#expose-job-artifacts-in-the-merge-request-ui). -`if:` clauses are evaluated based on the values of [predefined CI/CD variables](../variables/predefined_variables.md) -or [custom CI/CD variables](../variables/index.md#custom-cicd-variables). +**Keyword type**: Job keyword. You can use it only as part of a job or in the +[`default` section](#default). -**Keyword type**: Job-specific and pipeline-specific. You can use it as part of a job -to configure the job behavior, or with [`workflow`](#workflow) to configure the pipeline behavior. +**Possible inputs**: -**Possible inputs**: A [CI/CD variable expression](../jobs/job_control.md#cicd-variable-expressions). +- The name to display in the merge request UI for the artifacts download link. + Must be combined with [`artifacts:paths`](#artifactspaths). -**Example of `rules:if`**: +**Example of `artifacts:expose_as`**: ```yaml -job: - script: echo "Hello, Rules!" - rules: - - if: '$CI_MERGE_REQUEST_SOURCE_BRANCH_NAME =~ /^feature/ && $CI_MERGE_REQUEST_TARGET_BRANCH_NAME != $CI_DEFAULT_BRANCH' - when: never - - if: '$CI_MERGE_REQUEST_SOURCE_BRANCH_NAME =~ /^feature/' - when: manual - allow_failure: true - - if: '$CI_MERGE_REQUEST_SOURCE_BRANCH_NAME' +test: + script: ["echo 'test' > file.txt"] + artifacts: + expose_as: 'artifact 1' + paths: ['file.txt'] ``` **Additional details**: -- If a rule matches and has no `when` defined, the rule uses the `when` - defined for the job, which defaults to `on_success` if not defined. -- You can define `when` once per rule, or once at the job-level, which applies to - all rules. You can't mix `when` at the job-level with `when` in rules. -- Unlike variables in [`script`](../variables/index.md#use-cicd-variables-in-job-scripts) - sections, variables in rules expressions are always formatted as `$VARIABLE`. - - You can use `rules:if` with `include` to [conditionally include other configuration files](includes.md#use-rules-with-include). +- If `artifacts:paths` uses [CI/CD variables](../variables/index.md), the artifacts do not display in the UI. +- A maximum of 10 job artifacts per merge request can be exposed. +- Glob patterns are unsupported. +- If a directory is specified and there is more than one file in the directory, + the link is to the job [artifacts browser](../pipelines/job_artifacts.md#download-job-artifacts). +- If [GitLab Pages](../../administration/pages/index.md) is enabled, GitLab automatically + renders the artifacts when the artifacts is a single file with one of these extensions: + - `.html` or `.htm` + - `.txt` + - `.json` + - `.xml` + - `.log` **Related topics**: -- [Common `if` expressions for `rules`](../jobs/job_control.md#common-if-clauses-for-rules). -- [Avoid duplicate pipelines](../jobs/job_control.md#avoid-duplicate-pipelines). - -#### `rules:changes` - -Use `rules:changes` to specify when to add a job to a pipeline by checking for changes -to specific files. - -WARNING: -You should use `rules: changes` only with **branch pipelines** or **merge request pipelines**. -You can use `rules: changes` with other pipeline types, but `rules: changes` always -evaluates to true when there is no Git `push` event. Tag pipelines, scheduled pipelines, -and so on do **not** have a Git `push` event associated with them. A `rules: changes` job -is **always** added to those pipelines if there is no `if:` that limits the job to -branch or merge request pipelines. - -**Keyword type**: Job keyword. You can use it only as part of a job. - -**Possible inputs**: An array of file paths. In GitLab 13.6 and later, -[file paths can include variables](../jobs/job_control.md#variables-in-ruleschanges). - -**Example of `rules:changes`**: - -```yaml -docker build: - script: docker build -t my-image:$CI_COMMIT_REF_SLUG . - rules: - - if: '$CI_PIPELINE_SOURCE == "merge_request_event"' - changes: - - Dockerfile - when: manual - allow_failure: true -``` +- [Expose job artifacts in the merge request UI](../pipelines/job_artifacts.md#expose-job-artifacts-in-the-merge-request-ui). -- If the pipeline is a merge request pipeline, check `Dockerfile` for changes. -- If `Dockerfile` has changed, add the job to the pipeline as a manual job, and the pipeline - continues running even if the job is not triggered (`allow_failure: true`). -- If `Dockerfile` has not changed, do not add job to any pipeline (same as `when: never`). +#### `artifacts:name` -**Additional details**: +Use the `artifacts:name` keyword to define the name of the created artifacts +archive. You can specify a unique name for every archive. -- `rules: changes` works the same way as [`only: changes` and `except: changes`](#onlychanges--exceptchanges). -- You can use `when: never` to implement a rule similar to [`except:changes`](#onlychanges--exceptchanges). -- `changes` resolves to `true` if any of the matching files are changed (an `OR` operation). - -#### `rules:exists` +If not defined, the default name is `artifacts`, which becomes `artifacts.zip` when downloaded. -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/24021) in GitLab 12.4. +**Keyword type**: Job keyword. You can use it only as part of a job or in the +[`default` section](#default). -Use `exists` to run a job when certain files exist in the repository. +**Possible inputs**: -**Keyword type**: Job keyword. You can use it only as part of a job. +- The name of the artifacts archive. Can use [CI/CD variables](../variables/index.md). Must be combined with + [`artifacts:paths`](#artifactspaths). -**Possible inputs**: An array of file paths. Paths are relative to the project directory (`$CI_PROJECT_DIR`) -and can't directly link outside it. File paths can use glob patterns. +**Example of `artifacts:name`**: -**Example of `rules:exists`**: +To create an archive with a name of the current job: ```yaml job: - script: docker build -t my-image:$CI_COMMIT_REF_SLUG . - rules: - - exists: - - Dockerfile + artifacts: + name: "job1-artifacts-file" + paths: + - binaries/ ``` -`job` runs if a `Dockerfile` exists anywhere in the repository. - -**Additional details**: - -- Glob patterns are interpreted with Ruby [`File.fnmatch`](https://docs.ruby-lang.org/en/2.7.0/File.html#method-c-fnmatch) - with the flags `File::FNM_PATHNAME | File::FNM_DOTMATCH | File::FNM_EXTGLOB`. -- For performance reasons, GitLab matches a maximum of 10,000 `exists` patterns or - file paths. After the 10,000th check, rules with patterned globs always match. - In other words, the `exists` rule always assumes a match in projects with more - than 10,000 files. -- `exists` resolves to `true` if any of the listed files are found (an `OR` operation). +**Related topics**: -#### `rules:allow_failure` +- [Use CI/CD variables to define the artifacts name](../pipelines/job_artifacts.md#use-cicd-variables-to-define-the-artifacts-name). -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/30235) in GitLab 12.8. +#### `artifacts:paths` -Use [`allow_failure: true`](#allow_failure) in `rules:` to allow a job to fail -without stopping the pipeline. +Paths are relative to the project directory (`$CI_PROJECT_DIR`) and can't directly +link outside it. -You can also use `allow_failure: true` with a manual job. The pipeline continues -running without waiting for the result of the manual job. `allow_failure: false` -combined with `when: manual` in rules causes the pipeline to wait for the manual -job to run before continuing. +**Keyword type**: Job keyword. You can use it only as part of a job or in the +[`default` section](#default). -**Keyword type**: Job keyword. You can use it only as part of a job. +**Possible inputs**: -**Possible inputs**: `true` or `false`. Defaults to `false` if not defined. +- An array of file paths, relative to the project directory. +- You can use Wildcards that use [glob](https://en.wikipedia.org/wiki/Glob_(programming)) + patterns and: + - In [GitLab Runner 13.0 and later](https://gitlab.com/gitlab-org/gitlab-runner/-/issues/2620), + [`doublestar.Glob`](https://pkg.go.dev/github.com/bmatcuk/doublestar@v1.2.2?tab=doc#Match). + - In GitLab Runner 12.10 and earlier, [`filepath.Match`](https://pkg.go.dev/path/filepath#Match). -**Example of `rules:allow_failure`**: +**Example of `artifacts:paths`**: ```yaml job: - script: echo "Hello, Rules!" - rules: - - if: '$CI_MERGE_REQUEST_TARGET_BRANCH_NAME == $CI_DEFAULT_BRANCH' - when: manual - allow_failure: true + artifacts: + paths: + - binaries/ + - .config ``` -If the rule matches, then the job is a manual job with `allow_failure: true`. +This example creates an artifact with `.config` and all the files in the `binaries` directory. **Additional details**: -- The rule-level `rules:allow_failure` overrides the job-level [`allow_failure`](#allow_failure), - and only applies when the specific rule triggers the job. - -#### `rules:variables` +- If not used with [`artifacts:name`](#artifactsname) defined, the artifacts file + is named `artifacts`, which becomes `artifacts.zip` when downloaded. -> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/209864) in GitLab 13.7. -> - [Feature flag removed](https://gitlab.com/gitlab-org/gitlab/-/issues/289803) in GitLab 13.10. +**Related topics**: -Use [`variables`](#variables) in `rules:` to define variables for specific conditions. +- To restrict which jobs a specific job fetches artifacts from, see [`dependencies`](#dependencies). +- [Create job artifacts](../pipelines/job_artifacts.md#create-job-artifacts). -**Keyword type**: Job-specific. You can use it only as part of a job. - -**Possible inputs**: A hash of variables in the format `VARIABLE-NAME: value`. +#### `artifacts:public` -**Example of `rules:variables`**: +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/49775) in GitLab 13.8 +> - It's [deployed behind a feature flag](../../user/feature_flags.md), disabled by default. +> - It's disabled on GitLab.com. +> - It's recommended for production use. -```yaml -job: - variables: - DEPLOY_VARIABLE: "default-deploy" - rules: - - if: $CI_COMMIT_REF_NAME == $CI_DEFAULT_BRANCH - variables: # Override DEPLOY_VARIABLE defined - DEPLOY_VARIABLE: "deploy-production" # at the job level. - - if: $CI_COMMIT_REF_NAME =~ /feature/ - variables: - IS_A_FEATURE: "true" # Define a new variable. - script: - - echo "Run script with $DEPLOY_VARIABLE as an argument" - - echo "Run another script if $IS_A_FEATURE exists" -``` +FLAG: +On self-managed GitLab, by default this feature is not available. To make it available, +ask an administrator to [enable the feature flag](../../administration/feature_flags.md) named `non_public_artifacts`. On +GitLab.com, this feature is not available. -### `only` / `except` +Use `artifacts:public` to determine whether the job artifacts should be +publicly available. -NOTE: -`only` and `except` are not being actively developed. [`rules`](#rules) is the preferred -keyword to control when to add jobs to pipelines. +When `artifacts:public` is `true` (default), the artifacts in +public pipelines are available for download by anonymous and guest users. -You can use `only` and `except` to control when to add jobs to pipelines. +To deny read access for anonymous and guest users to artifacts in public +pipelines, set `artifacts:public` to `false`: -- Use `only` to define when a job runs. -- Use `except` to define when a job **does not** run. +**Keyword type**: Job keyword. You can use it only as part of a job or in the +[`default` section](#default). -Four keywords can be used with `only` and `except`: +**Possible inputs**: -- [`refs`](#onlyrefs--exceptrefs) -- [`variables`](#onlyvariables--exceptvariables) -- [`changes`](#onlychanges--exceptchanges) -- [`kubernetes`](#onlykubernetes--exceptkubernetes) +- `true` (default if not defined) or `false`. -See [specify when jobs run with `only` and `except`](../jobs/job_control.md#specify-when-jobs-run-with-only-and-except) -for more details and examples. +**Example of `artifacts:paths`**: -#### `only:refs` / `except:refs` +```yaml +job: + artifacts: + public: false +``` -Use the `only:refs` and `except:refs` keywords to control when to add jobs to a -pipeline based on branch names or pipeline types. +#### `artifacts:reports` -**Keyword type**: Job keyword. You can use it only as part of a job. +Use [`artifacts:reports`](artifacts_reports.md) to collect artifacts generated by +included templates in jobs. -**Possible inputs**: An array including any number of: +**Keyword type**: Job keyword. You can use it only as part of a job or in the +[`default` section](#default). -- Branch names, for example `main` or `my-feature-branch`. -- [Regular expressions](../jobs/job_control.md#only--except-regex-syntax) - that match against branch names, for example `/^feature-.*/`. -- The following keywords: +**Possible inputs**: - | **Value** | **Description** | - | -------------------------|-----------------| - | `api` | For pipelines triggered by the [pipelines API](../../api/pipelines.md#create-a-new-pipeline). | - | `branches` | When the Git reference for a pipeline is a branch. | - | `chat` | For pipelines created by using a [GitLab ChatOps](../chatops/index.md) command. | - | `external` | When you use CI services other than GitLab. | - | `external_pull_requests` | When an external pull request on GitHub is created or updated (See [Pipelines for external pull requests](../ci_cd_for_external_repos/index.md#pipelines-for-external-pull-requests)). | - | `merge_requests` | For pipelines created when a merge request is created or updated. Enables [merge request pipelines](../pipelines/merge_request_pipelines.md), [merged results pipelines](../pipelines/pipelines_for_merged_results.md), and [merge trains](../pipelines/merge_trains.md). | - | `pipelines` | For [multi-project pipelines](../pipelines/multi_project_pipelines.md) created by [using the API with `CI_JOB_TOKEN`](../pipelines/multi_project_pipelines.md#create-multi-project-pipelines-by-using-the-api), or the [`trigger`](#trigger) keyword. | - | `pushes` | For pipelines triggered by a `git push` event, including for branches and tags. | - | `schedules` | For [scheduled pipelines](../pipelines/schedules.md). | - | `tags` | When the Git reference for a pipeline is a tag. | - | `triggers` | For pipelines created by using a [trigger token](../triggers/index.md#authentication-tokens). | - | `web` | For pipelines created by selecting **Run pipeline** in the GitLab UI, from the project's **CI/CD > Pipelines** section. | +- See list of available [artifacts reports types](artifacts_reports.md). -**Example of `only:refs` and `except:refs`**: +**Example of `artifacts:reports`**: ```yaml -job1: - script: echo - only: - - main - - /^issue-.*$/ - - merge_requests - -job2: - script: echo - except: - - main - - /^stable-branch.*$/ - - schedules +rspec: + stage: test + script: + - bundle install + - rspec --format RspecJunitFormatter --out rspec.xml + artifacts: + reports: + junit: rspec.xml ``` -**Additional details:** - -- Scheduled pipelines run on specific branches, so jobs configured with `only: branches` - run on scheduled pipelines too. Add `except: schedules` to prevent jobs with `only: branches` - from running on scheduled pipelines. -- `only` or `except` used without any other keywords are equivalent to `only: refs` - or `except: refs`. For example, the following two jobs configurations have the same - behavior: - - ```yaml - job1: - script: echo - only: - - branches - - job2: - script: echo - only: - refs: - - branches - ``` - -- If a job does not use `only`, `except`, or [`rules`](#rules), then `only` is set to `branches` - and `tags` by default. +**Additional details**: - For example, `job1` and `job2` are equivalent: +- Combining reports in parent pipelines using [artifacts from child pipelines](#needspipelinejob) is + not supported. Track progress on adding support in [this issue](https://gitlab.com/gitlab-org/gitlab/-/issues/215725). +- To be able to browse the report output files, include the [`artifacts:paths`](#artifactspaths) keyword. Please note that this will upload and store the artifact twice. +- The test reports are collected regardless of the job results (success or failure). + You can use [`artifacts:expire_in`](#artifactsexpire_in) to set up an expiration + date for artifacts reports. - ```yaml - job1: - script: echo 'test' +#### `artifacts:untracked` - job2: - script: echo 'test' - only: - - branches - - tags - ``` +Use `artifacts:untracked` to add all Git untracked files as artifacts (along +with the paths defined in `artifacts:paths`). `artifacts:untracked` ignores configuration +in the repository's `.gitignore` file. -#### `only:variables` / `except:variables` +**Keyword type**: Job keyword. You can use it only as part of a job or in the +[`default` section](#default). -Use the `only:variables` or `except:variables` keywords to control when to add jobs -to a pipeline, based on the status of [CI/CD variables](../variables/index.md). +**Possible inputs**: -**Keyword type**: Job keyword. You can use it only as part of a job. +- `true` or `false` (default if not defined). -**Possible inputs**: An array of [CI/CD variable expressions](../jobs/job_control.md#cicd-variable-expressions). +**Example of `artifacts:untracked`**: -**Example of `only:variables`**: +Save all Git untracked files: ```yaml -deploy: - script: cap staging deploy - only: - variables: - - $RELEASE == "staging" - - $STAGING +job: + artifacts: + untracked: true ``` **Related topics**: -- [`only:variables` and `except:variables` examples](../jobs/job_control.md#only-variables--except-variables-examples). - -#### `only:changes` / `except:changes` +- [Add untracked files to artifacts](../pipelines/job_artifacts.md#add-untracked-files-to-artifacts). -Use the `changes` keyword with `only` to run a job, or with `except` to skip a job, -when a Git push event modifies a file. - -Use `changes` in pipelines with the following refs: +#### `artifacts:when` -- `branches` -- `external_pull_requests` -- `merge_requests` (see additional details about [using `only:changes` with pipelines for merge requests](../jobs/job_control.md#use-onlychanges-with-pipelines-for-merge-requests)) +Use `artifacts:when` to upload artifacts on job failure or despite the +failure. -**Keyword type**: Job keyword. You can use it only as part of a job. +**Keyword type**: Job keyword. You can use it only as part of a job or in the +[`default` section](#default). -**Possible inputs**: An array including any number of: +**Possible inputs**: -- Paths to files. -- Wildcard paths for single directories, for example `path/to/directory/*`, or a directory - and all its subdirectories, for example `path/to/directory/**/*`. -- Wildcard ([glob](https://en.wikipedia.org/wiki/Glob_(programming))) paths for all - files with the same extension or multiple extensions, for example `*.md` or `path/to/directory/*.{rb,py,sh}`. -- Wildcard paths to files in the root directory, or all directories, wrapped in double quotes. - For example `"*.json"` or `"**/*.json"`. +- `on_success` (default): Upload artifacts only when the job succeeds. +- `on_failure`: Upload artifacts only when the job fails. +- `always`: Always upload artifacts. For example, when + [uploading artifacts](../unit_test_reports.md#viewing-junit-screenshots-on-gitlab) + required to troubleshoot failing tests. -**Example of `only:changes`**: +**Example of `artifacts:when`**: ```yaml -docker build: - script: docker build -t my-image:$CI_COMMIT_REF_SLUG . - only: - refs: - - branches - changes: - - Dockerfile - - docker/scripts/* - - dockerfiles/**/* - - more_scripts/*.{rb,py,sh} - - "**/*.json" +job: + artifacts: + when: on_failure ``` -**Additional details**: - -- `changes` resolves to `true` if any of the matching files are changed (an `OR` operation). -- If you use refs other than `branches`, `external_pull_requests`, or `merge_requests`, - `changes` can't determine if a given file is new or old and always returns `true`. -- If you use `only: changes` with other refs, jobs ignore the changes and always run. -- If you use `except: changes` with other refs, jobs ignore the changes and never run. - -**Related topics**: +### `before_script` -- [`only: changes` and `except: changes` examples](../jobs/job_control.md#onlychanges--exceptchanges-examples). -- If you use `changes` with [only allow merge requests to be merged if the pipeline succeeds](../../user/project/merge_requests/merge_when_pipeline_succeeds.md#only-allow-merge-requests-to-be-merged-if-the-pipeline-succeeds), - you should [also use `only:merge_requests`](../jobs/job_control.md#use-onlychanges-with-pipelines-for-merge-requests). -- Use `changes` with [new branches or tags *without* pipelines for merge requests](../jobs/job_control.md#use-onlychanges-without-pipelines-for-merge-requests). -- Use `changes` with [scheduled pipelines](../jobs/job_control.md#use-onlychanges-with-scheduled-pipelines). - -#### `only:kubernetes` / `except:kubernetes` +Use `before_script` to define an array of commands that should run before each job's +`script` commands, but after [artifacts](#artifacts) are restored. -Use `only:kubernetes` or `except:kubernetes` to control if jobs are added to the pipeline -when the Kubernetes service is active in the project. +**Keyword type**: Job keyword. You can use it only as part of a job or in the +[`default` section](#default). -**Keyword type**: Job-specific. You can use it only as part of a job. +**Possible inputs**: An array including: -**Possible inputs**: The `kubernetes` strategy accepts only the `active` keyword. +- Single line commands. +- Long commands [split over multiple lines](script.md#split-long-commands). +- [YAML anchors](yaml_optimization.md#yaml-anchors-for-scripts). -**Example of `only:kubernetes`**: +**Example of `before_script`**: ```yaml -deploy: - only: - kubernetes: active +job: + before_script: + - echo "Execute this command before any 'script:' commands." + script: + - echo "This command executes after the job's 'before_script' commands." ``` -In this example, the `deploy` job runs only when the Kubernetes service is active -in the project. - -### `needs` - -> - [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/47063) in GitLab 12.2. -> - In GitLab 12.3, maximum number of jobs in `needs` array raised from five to 50. -> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/30631) in GitLab 12.8, `needs: []` lets jobs start immediately. -> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/30632) in GitLab 14.2, you can refer to jobs in the same stage as the job you are configuring. +**Additional details**: -Use `needs:` to execute jobs out-of-order. Relationships between jobs -that use `needs` can be visualized as a [directed acyclic graph](../directed_acyclic_graph/index.md). +- Scripts you specify in `before_script` are concatenated with any scripts you specify + in the main [`script`](#script). The combined scripts execute together in a single shell. -You can ignore stage ordering and run some jobs without waiting for others to complete. -Jobs in multiple stages can run concurrently. +**Related topics**: -**Keyword type**: Job keyword. You can use it only as part of a job. +- [Use `before_script` with `default`](script.md#set-a-default-before_script-or-after_script-for-all-jobs) + to define a default array of commands that should run before the `script` commands in all jobs. +- You can [ignore non-zero exit codes](script.md#ignore-non-zero-exit-codes). +- [Use color codes with `before_script`](script.md#add-color-codes-to-script-output) + to make job logs easier to review. +- [Create custom collapsible sections](../jobs/index.md#custom-collapsible-sections) + to simplify job log output. -**Possible inputs**: +### `cache` -- An array of jobs. -- An empty array (`[]`), to set the job to start as soon as the pipeline is created. +Use `cache` to specify a list of files and directories to +cache between jobs. You can only use paths that are in the local working copy. -**Example of `needs`**: +Caching is shared between pipelines and jobs. Caches are restored before [artifacts](#artifacts). -```yaml -linux:build: - stage: build - script: echo "Building linux..." +Learn more about caches in [Caching in GitLab CI/CD](../caching/index.md). -mac:build: - stage: build - script: echo "Building mac..." +#### `cache:paths` -lint: - stage: test - needs: [] - script: echo "Linting..." +Use the `cache:paths` keyword to choose which files or directories to cache. -linux:rspec: - stage: test - needs: ["linux:build"] - script: echo "Running rspec on linux..." +**Keyword type**: Job keyword. You can use it only as part of a job or in the +[`default` section](#default). -mac:rspec: - stage: test - needs: ["mac:build"] - script: echo "Running rspec on mac..." +**Possible inputs**: -production: - stage: deploy - script: echo "Running production..." -``` +- An array of paths relative to the project directory (`$CI_PROJECT_DIR`). + You can use wildcards that use [glob](https://en.wikipedia.org/wiki/Glob_(programming)) + patterns: + - In [GitLab Runner 13.0 and later](https://gitlab.com/gitlab-org/gitlab-runner/-/issues/2620), + [`doublestar.Glob`](https://pkg.go.dev/github.com/bmatcuk/doublestar@v1.2.2?tab=doc#Match). + - In GitLab Runner 12.10 and earlier, + [`filepath.Match`](https://pkg.go.dev/path/filepath#Match). -This example creates four paths of execution: +**Example of `cache:paths`**: -- Linter: The `lint` job runs immediately without waiting for the `build` stage - to complete because it has no needs (`needs: []`). -- Linux path: The `linux:rspec` job runs as soon as the `linux:build` - job finishes, without waiting for `mac:build` to finish. -- macOS path: The `mac:rspec` jobs runs as soon as the `mac:build` - job finishes, without waiting for `linux:build` to finish. -- The `production` job runs as soon as all previous jobs finish: - `linux:build`, `linux:rspec`, `mac:build`, `mac:rspec`. +Cache all files in `binaries` that end in `.apk` and the `.config` file: -**Additional details**: +```yaml +rspec: + script: + - echo "This job uses a cache." + cache: + key: binaries-cache + paths: + - binaries/*.apk + - .config +``` -- The maximum number of jobs that a single job can have in the `needs:` array is limited: - - For GitLab.com, the limit is 50. For more information, see our - [infrastructure issue](https://gitlab.com/gitlab-com/gl-infra/infrastructure/-/issues/7541). - - For self-managed instances, the default limit is 50. This limit [can be changed](../../administration/cicd.md#set-the-needs-job-limit). -- If `needs:` refers to a job that uses the [`parallel`](#parallel) keyword, - it depends on all jobs created in parallel, not just one job. It also downloads - artifacts from all the parallel jobs by default. If the artifacts have the same - name, they overwrite each other and only the last one downloaded is saved. -- In [GitLab 14.1 and later](https://gitlab.com/gitlab-org/gitlab/-/issues/30632) you - can refer to jobs in the same stage as the job you are configuring. This feature is - enabled on GitLab.com and ready for production use. On self-managed [GitLab 14.2 and later](https://gitlab.com/gitlab-org/gitlab/-/issues/30632) - this feature is available by default. -- In GitLab 14.0 and older, you can only refer to jobs in earlier stages. Stages must be - explicitly defined for all jobs that use the `needs:` keyword, or are referenced - in a job's `needs:` section. -- In GitLab 13.9 and older, if `needs:` refers to a job that might not be added to - a pipeline because of `only`, `except`, or `rules`, the pipeline might fail to create. +**Related topics**: -#### `needs:artifacts` +- See the [common `cache` use cases](../caching/index.md#common-use-cases-for-caches) for more + `cache:paths` examples. -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/14311) in GitLab 12.6. +#### `cache:key` -When a job uses `needs`, it no longer downloads all artifacts from previous stages -by default, because jobs with `needs` can start before earlier stages complete. With -`needs` you can only download artifacts from the jobs listed in the `needs:` configuration. +Use the `cache:key` keyword to give each cache a unique identifying key. All jobs +that use the same cache key use the same cache, including in different pipelines. -Use `artifacts: true` (default) or `artifacts: false` to control when artifacts are -downloaded in jobs that use `needs`. +If not set, the default key is `default`. All jobs with the `cache` keyword but +no `cache:key` share the `default` cache. -**Keyword type**: Job keyword. You can use it only as part of a job. Must be used with `needs:job`. +**Keyword type**: Job keyword. You can use it only as part of a job or in the +[`default` section](#default). **Possible inputs**: -- `true` (default) or `false`. +- A string. +- A [predefined variables](../variables/index.md). +- A combination of both. -**Example of `needs:artifacts`**: +**Example of `cache:key`**: ```yaml -test-job1: - stage: test - needs: - - job: build_job1 - artifacts: true - -test-job2: - stage: test - needs: - - job: build_job2 - artifacts: false - -test-job3: - needs: - - job: build_job1 - artifacts: true - - job: build_job2 - - build_job3 +cache-job: + script: + - echo "This job uses a cache." + cache: + key: binaries-cache-$CI_COMMIT_REF_SLUG + paths: + - binaries/ ``` -In this example: +**Additional details**: -- The `test-job1` job downloads the `build_job1` artifacts -- The `test-job2` job does not download the `build_job2` artifacts. -- The `test-job3` job downloads the artifacts from all three `build_jobs`, because - `artifacts:` is `true`, or defaults to `true`, for all three needed jobs. +- If you use **Windows Batch** to run your shell scripts you must replace + `$` with `%`. For example: `key: %CI_COMMIT_REF_SLUG%` +- The `cache:key` value can't contain: -**Additional details**: + - The `/` character, or the equivalent URI-encoded `%2F`. + - Only the `.` character (any number), or the equivalent URI-encoded `%2E`. -- In GitLab 12.6 and later, you can't combine the [`dependencies`](#dependencies) keyword - with `needs`. +- The cache is shared between jobs, so if you're using different + paths for different jobs, you should also set a different `cache:key`. + Otherwise cache content can be overwritten. -#### `needs:project` **(PREMIUM)** +**Related topics**: -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/14311) in GitLab 12.7. +- You can specify a [fallback cache key](../caching/index.md#use-a-fallback-cache-key) + to use if the specified `cache:key` is not found. +- You can [use multiple cache keys](../caching/index.md#use-multiple-caches) in a single job. +- See the [common `cache` use cases](../caching/index.md#common-use-cases-for-caches) for more + `cache:key` examples. -Use `needs:project` to download artifacts from up to five jobs in other pipelines. -The artifacts are downloaded from the latest successful pipeline for the specified ref. +##### `cache:key:files` -If there is a pipeline running for the specified ref, a job with `needs:project` -does not wait for the pipeline to complete. Instead, the job downloads the artifact -from the latest pipeline that completed successfully. +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/18986) in GitLab 12.5. -`needs:project` must be used with `job:`, `ref:`, and `artifacts:`. +Use the `cache:key:files` keyword to generate a new key when one or two specific files +change. `cache:key:files` lets you reuse some caches, and rebuild them less often, +which speeds up subsequent pipeline runs. -**Keyword type**: Job keyword. You can use it only as part of a job. +**Keyword type**: Job keyword. You can use it only as part of a job or in the +[`default` section](#default). **Possible inputs**: -- `needs:project`: A full project path, including namespace and group. If the - project is in the same group or namespace, you can omit them from the `project:` - keyword. For example: `project: group/project-name` or `project: project-name`. -- `job`: The job to download artifacts from. -- `ref`: The ref to download artifacts from. -- `artifacts`: Must be `true` to download artifacts. +- An array of one or two file paths. -**Examples of `needs:project`**: +**Example of `cache:key:files`**: ```yaml -build_job: - stage: build +cache-job: script: - - ls -lhR - needs: - - project: namespace/group/project-name - job: build-1 - ref: main - artifacts: true + - echo "This job uses a cache." + cache: + key: + files: + - Gemfile.lock + - package.json + paths: + - vendor/ruby + - node_modules ``` -In this example, `build_job` downloads the artifacts from the latest successful `build-1` job -on the `main` branch in the `group/project-name` project. - -In GitLab 13.3 and later, you can use [CI/CD variables](../variables/index.md) in `needs:project`, -for example: - -```yaml -build_job: - stage: build - script: - - ls -lhR - needs: - - project: $CI_PROJECT_PATH - job: $DEPENDENCY_JOB_NAME - ref: $ARTIFACTS_DOWNLOAD_REF - artifacts: true -``` +This example creates a cache for Ruby and Node.js dependencies. The cache +is tied to the current versions of the `Gemfile.lock` and `package.json` files. When one of +these files changes, a new cache key is computed and a new cache is created. Any future +job runs that use the same `Gemfile.lock` and `package.json` with `cache:key:files` +use the new cache, instead of rebuilding the dependencies. **Additional details**: -- To download artifacts from a different pipeline in the current project, set `project:` - to be the same as the current project, but use a different ref than the current pipeline. - Concurrent pipelines running on the same ref could override the artifacts. -- The user running the pipeline must have at least the Reporter role for the group or project, - or the group/project must have public visibility. -- You can't use `needs:project` in the same job as [`trigger`](#trigger). -- When using `needs:project` to download artifacts from another pipeline, the job does not wait for - the needed job to complete. [Directed acyclic graph](../directed_acyclic_graph/index.md) - behavior is limited to jobs in the same pipeline. Make sure that the needed job in the other - pipeline completes before the job that needs it tries to download the artifacts. -- You can't download artifacts from jobs that run in [`parallel:`](#parallel). -- Support for [CI/CD variables](../variables/index.md) in `project`, `job`, and `ref` was - [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/202093) in GitLab 13.3. - [Feature flag removed](https://gitlab.com/gitlab-org/gitlab/-/issues/235761) in GitLab 13.4. - -**Related topics**: - -- To download artifacts between [parent-child pipelines](../pipelines/parent_child_pipelines.md), - use [`needs:pipeline:job`](#needspipelinejob). +- The cache `key` is a SHA computed from the most recent commits +that changed each listed file. + If neither file is changed in any commits, the fallback key is `default`. -#### `needs:pipeline:job` +##### `cache:key:prefix` -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/255983) in GitLab 13.7. +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/18986) in GitLab 12.5. -A [child pipeline](../pipelines/parent_child_pipelines.md) can download artifacts from a job in -its parent pipeline or another child pipeline in the same parent-child pipeline hierarchy. +Use `cache:key:prefix` to combine a prefix with the SHA computed for [`cache:key:files`](#cachekeyfiles). -**Keyword type**: Job keyword. You can use it only as part of a job. +**Keyword type**: Job keyword. You can use it only as part of a job or in the +[`default` section](#default). **Possible inputs**: -- `needs:pipeline`: A pipeline ID. Must be a pipeline present in the same parent-child pipeline hierarchy. -- `job:`: The job to download artifacts from. - -**Example of `needs:pipeline:job`**: - -- Parent pipeline (`.gitlab-ci.yml`): - - ```yaml - create-artifact: - stage: build - script: echo 'sample artifact' > artifact.txt - artifacts: - paths: [artifact.txt] - - child-pipeline: - stage: test - trigger: - include: child.yml - strategy: depend - variables: - PARENT_PIPELINE_ID: $CI_PIPELINE_ID - ``` +- A string +- A [predefined variables](../variables/index.md) +- A combination of both. -- Child pipeline (`child.yml`): +**Example of `cache:key:prefix`**: - ```yaml - use-artifact: - script: cat artifact.txt - needs: - - pipeline: $PARENT_PIPELINE_ID - job: create-artifact - ``` +```yaml +rspec: + script: + - echo "This rspec job uses a cache." + cache: + key: + files: + - Gemfile.lock + prefix: $CI_JOB_NAME + paths: + - vendor/ruby +``` -In this example, the `create-artifact` job in the parent pipeline creates some artifacts. -The `child-pipeline` job triggers a child pipeline, and passes the `CI_PIPELINE_ID` -variable to the child pipeline as a new `PARENT_PIPELINE_ID` variable. The child pipeline -can use that variable in `needs:pipeline` to download artifacts from the parent pipeline. +For example, adding a `prefix` of `$CI_JOB_NAME` causes the key to look like `rspec-feef9576d21ee9b6a32e30c5c79d0a0ceb68d1e5`. +If a branch changes `Gemfile.lock`, that branch has a new SHA checksum for `cache:key:files`. +A new cache key is generated, and a new cache is created for that key. If `Gemfile.lock` +is not found, the prefix is added to `default`, so the key in the example would be `rspec-default`. **Additional details**: -- The `pipeline` attribute does not accept the current pipeline ID (`$CI_PIPELINE_ID`). - To download artifacts from a job in the current pipeline, use [`needs`](#needsartifacts). - -#### `needs:optional` - -> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/30680) in GitLab 13.10. -> - [Feature flag removed](https://gitlab.com/gitlab-org/gitlab/-/issues/323891) in GitLab 14.0. - -To need a job that sometimes does not exist in the pipeline, add `optional: true` -to the `needs` configuration. If not defined, `optional: false` is the default. +- If no file in `cache:key:files` is changed in any commits, the prefix is added to the `default` key. -Jobs that use [`rules`](#rules), [`only`, or `except`](#only--except), might -not always exist in a pipeline. When the pipeline is created, GitLab checks the `needs` -relationships before starting it. Without `optional: true`, needs relationships that -point to a job that does not exist stops the pipeline from starting and causes a pipeline -error similar to: +#### `cache:untracked` -- `'job1' job needs 'job2' job, but it was not added to the pipeline` +Use `untracked: true` to cache all files that are untracked in your Git repository: -**Keyword type**: Job keyword. You can use it only as part of a job. +**Keyword type**: Job keyword. You can use it only as part of a job or in the +[`default` section](#default). **Possible inputs**: -- `job:`: The job to make optional. - `true` or `false` (default). -**Example of `needs:optional`**: +**Example of `cache:untracked`**: ```yaml -build: - stage: build - rules: - - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH - rspec: - stage: test - needs: - - job: build - optional: true + script: test + cache: + untracked: true ``` -In this example: +**Additional details**: -- When the branch is the default branch, the `build` job exists in the pipeline, and the `rspec` - job waits for it to complete before starting. -- When the branch is not the default branch, the `build` job does not exist in the pipeline. - The `rspec` job runs immediately (similar to `needs: []`) because its `needs` - relationship to the `build` job is optional. +- You can combine `cache:untracked` with `cache:paths` to cache all untracked files + as well as files in the configured paths. For example: -#### `needs:pipeline` + ```yaml + rspec: + script: test + cache: + untracked: true + paths: + - binaries/ + ``` -You can mirror the pipeline status from an upstream pipeline to a bridge job by -using the `needs:pipeline` keyword. The latest pipeline status from the default branch is -replicated to the bridge job. +#### `cache:when` -**Keyword type**: Job keyword. You can use it only as part of a job. +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/18969) in GitLab 13.5 and GitLab Runner v13.5.0. + +Use `cache:when` to define when to save the cache, based on the status of the job. + +**Keyword type**: Job keyword. You can use it only as part of a job or in the +[`default` section](#default). **Possible inputs**: -- A full project path, including namespace and group. If the - project is in the same group or namespace, you can omit them from the `project:` - keyword. For example: `project: group/project-name` or `project: project-name`. +- `on_success` (default): Save the cache only when the job succeeds. +- `on_failure`: Save the cache only when the job fails. +- `always`: Always save the cache. -**Example of `needs:pipeline`**: +**Example of `cache:when`**: ```yaml -upstream_bridge: - stage: test - needs: - pipeline: other/project +rspec: + script: rspec + cache: + paths: + - rspec/ + when: 'always' ``` -**Additional details**: +This example stores the cache whether or not the job fails or succeeds. -- If you add the `job` keyword to `needs:pipeline`, the job no longer mirrors the - pipeline status. The behavior changes to [`needs:pipeline:job`](#needspipelinejob). +#### `cache:policy` -### `tags` +To change the upload and download behavior of a cache, use the `cache:policy` keyword. +By default, the job downloads the cache when the job starts, and uploads changes +to the cache when the job ends. This caching style is the `pull-push` policy (default). -> - A limit of 50 tags per job [enabled on GitLab.com](https://gitlab.com/gitlab-org/gitlab/-/issues/338929) in GitLab 14.3. -> - A limit of 50 tags per job [enabled on self-managed](https://gitlab.com/gitlab-org/gitlab/-/issues/339855) in GitLab 14.3. +To set a job to only download the cache when the job starts, but never upload changes +when the job finishes, use `cache:policy:pull`. -Use `tags` to select a specific runner from the list of all runners that are -available for the project. +To set a job to only upload a cache when the job finishes, but never download the +cache when the job starts, use `cache:policy:push`. -When you register a runner, you can specify the runner's tags, for -example `ruby`, `postgres`, or `development`. To pick up and run a job, a runner must -be assigned every tag listed in the job. +Use the `pull` policy when you have many jobs executing in parallel that use the same cache. +This policy speeds up job execution and reduces load on the cache server. You can +use a job with the `push` policy to build the cache. **Keyword type**: Job keyword. You can use it only as part of a job or in the -[`default:` section](#default). +[`default` section](#default). **Possible inputs**: -- An array of tag names. -- [CI/CD variables](../runners/configure_runners.md#use-cicd-variables-in-tags) in GitLab 14.1 and later. +- `pull` +- `push` +- `pull-push` (default) -**Example of `tags`**: +**Example of `cache:policy`**: ```yaml -job: - tags: - - ruby - - postgres +prepare-dependencies-job: + stage: build + cache: + key: gems + paths: + - vendor/bundle + policy: push + script: + - echo "This job only downloads dependencies and builds the cache." + - echo "Downloading dependencies..." + +faster-test-job: + stage: test + cache: + key: gems + paths: + - vendor/bundle + policy: pull + script: + - echo "This job script uses the cache, but does not update it." + - echo "Running tests..." ``` -In this example, only runners with *both* the `ruby` and `postgres` tags can run the job. +### `coverage` -**Additional details**: +Use `coverage` with a custom regular expression to configure how code coverage +is extracted from the job output. The coverage is shown in the UI if at least one +line in the job output matches the regular expression. -- In [GitLab 14.3](https://gitlab.com/gitlab-org/gitlab/-/issues/338479) and later, - the number of tags must be less than `50`. +To extract the code coverage value in the matching line, GitLab uses this +regular expression: `\d+(\.\d+)?`. -**Related topics**: +**Possible inputs**: -- [Use tags to control which jobs a runner can run](../runners/configure_runners.md#use-tags-to-control-which-jobs-a-runner-can-run). +- A regular expression. Must start and end with `/`. -### `allow_failure` +**Example of `coverage`**: -Use `allow_failure` to determine whether a pipeline should continue running when a job fails. +```yaml +job1: + script: rspec + coverage: '/Code coverage: \d+\.\d+/' +``` -- To let the pipeline continue running subsequent jobs, use `allow_failure: true`. -- To stop the pipeline from running subsequent jobs, use `allow_failure: false`. +In this example: -When jobs are allowed to fail (`allow_failure: true`) an orange warning (**{status_warning}**) -indicates that a job failed. However, the pipeline is successful and the associated commit -is marked as passed with no warnings. +1. GitLab checks the job log for a line that matches the regular expression. A line + like `Code coverage: 67.89` would match. +1. GitLab then checks the line to find a match to `\d+(\.\d+)?`. The sample matching + line above gives a code coverage of `67.89`. -This same warning is displayed when: +**Additional details**: -- All other jobs in the stage are successful. -- All other jobs in the pipeline are successful. +- If there is more than one matched line in the job output, the last line is used. +- Leading zeros are removed. +- Coverage output from [child pipelines](../pipelines/parent_child_pipelines.md) + is not recorded or displayed. Check [the related issue](https://gitlab.com/gitlab-org/gitlab/-/issues/280818) + for more details. -The default value for `allow_failure` is: +### `dast_configuration` **(ULTIMATE)** -- `true` for [manual jobs](../jobs/job_control.md#create-a-job-that-must-be-run-manually). -- `false` for manual jobs that also use [`rules`](#rules). -- `false` in all other cases. +> [Introduced](https://gitlab.com/groups/gitlab-org/-/epics/5981) in GitLab 14.1. -**Keyword type**: Job keyword. You can use it only as part of a job. +Use the `dast_configuration` keyword to specify a site profile and scanner profile to be used in a +CI/CD configuration. Both profiles must first have been created in the project. The job's stage must +be `dast`. -**Possible inputs**: `true` or `false`. +**Keyword type**: Job keyword. You can use only as part of a job. -**Example of `allow_failure`**: +**Possible inputs**: One each of `site_profile` and `scanner_profile`. + +- Use `site_profile` to specify the site profile to be used in the job. +- Use `scanner_profile` to specify the scanner profile to be used in the job. + +**Example of `dast_configuration`**: ```yaml -job1: - stage: test - script: - - execute_script_1 +stages: + - build + - dast -job2: - stage: test - script: - - execute_script_2 - allow_failure: true +include: + - template: DAST.gitlab-ci.yml -job3: - stage: deploy - script: - - deploy_to_staging +dast: + dast_configuration: + site_profile: "Example Co" + scanner_profile: "Quick Passive Test" ``` -In this example, `job1` and `job2` run in parallel: - -- If `job1` fails, jobs in the `deploy` stage do not start. -- If `job2` fails, jobs in the `deploy` stage can still start. +In this example, the `dast` job extends the `dast` configuration added with the `include` keyword +to select a specific site profile and scanner profile. **Additional details**: -- You can use `allow_failure` as a subkey of [`rules:`](#rulesallow_failure). -- You can use `allow_failure: false` with a manual job to create a [blocking manual job](../jobs/job_control.md#types-of-manual-jobs). - A blocked pipeline does not run any jobs in later stages until the manual job - is started and completes successfully. - -#### `allow_failure:exit_codes` - -> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/273157) in GitLab 13.8. -> - [Feature flag removed](https://gitlab.com/gitlab-org/gitlab/-/issues/292024) in GitLab 13.9. - -Use `allow_failure:exit_codes` to control when a job should be -allowed to fail. The job is `allow_failure: true` for any of the listed exit codes, -and `allow_failure` false for any other exit code. - -**Keyword type**: Job keyword. You can use it only as part of a job. - -**Possible inputs**: - -- A single exit code. -- An array of exit codes. +- Settings contained in either a site profile or scanner profile take precedence over those + contained in the DAST template. -**Example of `allow_failure`**: +**Related topics**: -```yaml -test_job_1: - script: - - echo "Run a script that results in exit code 1. This job fails." - - exit 1 - allow_failure: - exit_codes: 137 +- [Site profile](../../user/application_security/dast/index.md#site-profile). +- [Scanner profile](../../user/application_security/dast/index.md#scanner-profile). -test_job_2: - script: - - echo "Run a script that results in exit code 137. This job is allowed to fail." - - exit 137 - allow_failure: - exit_codes: - - 137 - - 255 -``` +### `dependencies` -### `when` +Use the `dependencies` keyword to define a list of jobs to fetch [artifacts](#artifacts) from. +You can also set a job to download no artifacts at all. -Use `when` to configure the conditions for when jobs run. If not defined in a job, -the default value is `when: on_success`. +If you do not use `dependencies`, all artifacts from previous stages are passed to each job. **Keyword type**: Job keyword. You can use it only as part of a job. **Possible inputs**: -- `on_success` (default): Run the job only when all jobs in earlier stages succeed - or have `allow_failure: true`. -- `manual`: Run the job only when [triggered manually](../jobs/job_control.md#create-a-job-that-must-be-run-manually). -- `always`: Run the job regardless of the status of jobs in earlier stages. -- `on_failure`: Run the job only when at least one job in an earlier stage fails. -- `delayed`: [Delay the execution of a job](../jobs/job_control.md#run-a-job-after-a-delay) - for a specified duration. -- `never`: Don't run the job. +- The names of jobs to fetch artifacts from. +- An empty array (`[]`), to configure the job to not download any artifacts. -**Example of `when`**: +**Example of `dependencies`**: ```yaml -stages: - - build - - cleanup_build - - test - - deploy - - cleanup +build osx: + stage: build + script: make build:osx + artifacts: + paths: + - binaries/ -build_job: +build linux: stage: build - script: - - make build + script: make build:linux + artifacts: + paths: + - binaries/ -cleanup_build_job: - stage: cleanup_build - script: - - cleanup build when failed - when: on_failure +test osx: + stage: test + script: make test:osx + dependencies: + - build:osx -test_job: +test linux: stage: test - script: - - make test + script: make test:linux + dependencies: + - build:linux -deploy_job: +deploy: stage: deploy - script: - - make deploy - when: manual - -cleanup_job: - stage: cleanup - script: - - cleanup after jobs - when: always + script: make deploy ``` -In this example, the script: +In this example, two jobs have artifacts: `build osx` and `build linux`. When `test osx` is executed, +the artifacts from `build osx` are downloaded and extracted in the context of the build. +The same thing happens for `test linux` and artifacts from `build linux`. -1. Executes `cleanup_build_job` only when `build_job` fails. -1. Always executes `cleanup_job` as the last step in pipeline regardless of - success or failure. -1. Executes `deploy_job` when you run it manually in the GitLab UI. +The `deploy` job downloads artifacts from all previous jobs because of +the [stage](#stages) precedence. **Additional details**: -- In [GitLab 13.5 and later](https://gitlab.com/gitlab-org/gitlab/-/issues/201938), you - can use `when:manual` in the same job as [`trigger`](#trigger). In GitLab 13.4 and - earlier, using them together causes the error `jobs:#{job-name} when should be on_success, on_failure or always`. -- The default behavior of `allow_failure` changes to `true` with `when: manual`. - However, if you use `when: manual` with [`rules`](#rules), `allow_failure` defaults - to `false`. - -**Related topics**: - -- `when` can be used with [`rules`](#rules) for more dynamic job control. -- `when` can be used with [`workflow`](#workflow) to control when a pipeline can start. +- The job status does not matter. If a job fails or it's a manual job that isn't triggered, no error occurs. +- If the artifacts of a dependent job are [expired](#artifactsexpire_in) or + [deleted](../pipelines/job_artifacts.md#delete-job-artifacts), then the job fails. ### `environment` @@ -1999,23 +1551,19 @@ environment. Use the `action` keyword to specify jobs that prepare, start, or stop environments. -| **Value** | **Description** | -|-----------|---------------------------------------------------------------------------------------------------------------------------------------------------------------| -| `start` | Default value. Indicates that job starts the environment. The deployment is created after the job starts. | -| `prepare` | Indicates that the job is only preparing the environment. It does not trigger deployments. [Read more about preparing environments](../environments/index.md#prepare-an-environment-without-creating-a-deployment). | -| `stop` | Indicates that job stops deployment. See the example below. | +**Keyword type**: Job keyword. You can use it only as part of a job. -Take for instance: +**Possible inputs**: One of the following keywords: -```yaml -review_app: - stage: deploy - script: make deploy-app - environment: - name: review/$CI_COMMIT_REF_SLUG - url: https://$CI_ENVIRONMENT_SLUG.example.com - on_stop: stop_review_app +| **Value** | **Description** | +|:----------|:----------------| +| `start` | Default value. Indicates that the job starts the environment. The deployment is created after the job starts. | +| `prepare` | Indicates that the job is only preparing the environment. It does not trigger deployments. [Read more about preparing environments](../environments/index.md#prepare-an-environment-without-creating-a-deployment). | +| `stop` | Indicates that the job stops a deployment. For more detail, read [Stop an environment](../environments/index.md#stop-an-environment). | + +**Example of `environment:action`**: +```yaml stop_review_app: stage: deploy variables: @@ -2027,39 +1575,6 @@ stop_review_app: action: stop ``` -In the above example, the `review_app` job deploys to the `review` -environment. A new `stop_review_app` job is listed under `on_stop`. -After the `review_app` job is finished, it triggers the -`stop_review_app` job based on what is defined under `when`. In this case, -it is set to `manual`, so it needs a [manual action](../jobs/job_control.md#create-a-job-that-must-be-run-manually) from -the GitLab UI to run. - -Also in the example, `GIT_STRATEGY` is set to `none`. If the -`stop_review_app` job is [automatically triggered](../environments/index.md#stop-an-environment), -the runner won't try to check out the code after the branch is deleted. - -The example also overwrites global variables. If your `stop` `environment` job depends -on global variables, use [anchor variables](yaml_optimization.md#yaml-anchors-for-variables) when you set the `GIT_STRATEGY` -to change the job without overriding the global variables. - -The `stop_review_app` job is **required** to have the following keywords defined: - -- `when`, defined at either: - - [The job level](#when). - - [In a rules clause](#rules). If you use `rules:` and `when: manual`, you should - also set [`allow_failure: true`](#allow_failure) so the pipeline can complete - even if the job doesn't run. -- `environment:name` -- `environment:action` - -Additionally, both jobs should have matching [`rules`](#only--except) -or [`only/except`](#only--except) configuration. - -In the examples above, if the configuration is not identical: - -- The `stop_review_app` job might not be included in all pipelines that include the `review_app` job. -- It is not possible to trigger the `action: stop` to stop the environment automatically. - #### `environment:auto_stop_in` > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/20956) in GitLab 12.8. @@ -2185,1211 +1700,869 @@ The common use case is to create dynamic environments for branches and use them as Review Apps. You can see an example that uses Review Apps at <https://gitlab.com/gitlab-examples/review-apps-nginx/>. -### `cache` - -Use `cache` to specify a list of files and directories to -cache between jobs. You can only use paths that are in the local working copy. - -Caching is shared between pipelines and jobs. Caches are restored before [artifacts](#artifacts). - -Learn more about caches in [Caching in GitLab CI/CD](../caching/index.md). - -#### `cache:paths` - -Use the `cache:paths` keyword to choose which files or directories to cache. +### `extends` -**Keyword type**: Job keyword. You can use it only as part of a job or in the -[`default:` section](#default). +Use `extends` to reuse configuration sections. It's an alternative to [YAML anchors](yaml_optimization.md#anchors) +and is a little more flexible and readable. -**Possible inputs**: An array of paths relative to the project directory (`$CI_PROJECT_DIR`). -You can use wildcards that use [glob](https://en.wikipedia.org/wiki/Glob_(programming)) -patterns: +**Keyword type**: Job keyword. You can use it only as part of a job. -- In [GitLab Runner 13.0 and later](https://gitlab.com/gitlab-org/gitlab-runner/-/issues/2620), -[`doublestar.Glob`](https://pkg.go.dev/github.com/bmatcuk/doublestar@v1.2.2?tab=doc#Match). -- In GitLab Runner 12.10 and earlier, -[`filepath.Match`](https://pkg.go.dev/path/filepath#Match). +**Possible inputs**: -**Example of `cache:paths`**: +- The name of another job in the pipeline. +- A list (array) of names of other jobs in the pipeline. -Cache all files in `binaries` that end in `.apk` and the `.config` file: +**Example of `extends`**: ```yaml +.tests: + script: rake test + stage: test + only: + refs: + - branches + rspec: - script: - - echo "This job uses a cache." - cache: - key: binaries-cache - paths: - - binaries/*.apk - - .config + extends: .tests + script: rake rspec + only: + variables: + - $RSPEC ``` -**Related topics**: - -- See the [common `cache` use cases](../caching/index.md#common-use-cases-for-caches) for more - `cache:paths` examples. - -#### `cache:key` - -Use the `cache:key` keyword to give each cache a unique identifying key. All jobs -that use the same cache key use the same cache, including in different pipelines. - -If not set, the default key is `default`. All jobs with the `cache:` keyword but -no `cache:key` share the `default` cache. - -**Keyword type**: Job keyword. You can use it only as part of a job or in the -[`default:` section](#default). - -**Possible inputs**: +In this example, the `rspec` job uses the configuration from the `.tests` template job. +When creating the pipeline, GitLab: -- A string. -- A [predefined variables](../variables/index.md). -- A combination of both. +- Performs a reverse deep merge based on the keys. +- Merges the `.tests` content with the `rspec` job. +- Doesn't merge the values of the keys. -**Example of `cache:key`**: +The result is this `rspec` job: ```yaml -cache-job: - script: - - echo "This job uses a cache." - cache: - key: binaries-cache-$CI_COMMIT_REF_SLUG - paths: - - binaries/ +rspec: + script: rake rspec + stage: test + only: + refs: + - branches + variables: + - $RSPEC ``` **Additional details**: -- If you use **Windows Batch** to run your shell scripts you must replace - `$` with `%`. For example: `key: %CI_COMMIT_REF_SLUG%` -- The `cache:key` value can't contain: - - - The `/` character, or the equivalent URI-encoded `%2F`. - - Only the `.` character (any number), or the equivalent URI-encoded `%2E`. - -- The cache is shared between jobs, so if you're using different - paths for different jobs, you should also set a different `cache:key`. - Otherwise cache content can be overwritten. +- In GitLab 12.0 and later, you can use multiple parents for `extends`. +- The `extends` keyword supports up to eleven levels of inheritance, but you should + avoid using more than three levels. +- In the example above, `.tests` is a [hidden job](../jobs/index.md#hide-jobs), + but you can extend configuration from regular jobs as well. **Related topics**: -- You can specify a [fallback cache key](../caching/index.md#use-a-fallback-cache-key) - to use if the specified `cache:key` is not found. -- You can [use multiple cache keys](../caching/index.md#use-multiple-caches) in a single job. -- See the [common `cache` use cases](../caching/index.md#common-use-cases-for-caches) for more - `cache:key` examples. - -##### `cache:key:files` - -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/18986) in GitLab 12.5. - -Use the `cache:key:files` keyword to generate a new key when one or two specific files -change. `cache:key:files` lets you reuse some caches, and rebuild them less often, -which speeds up subsequent pipeline runs. - -**Keyword type**: Job keyword. You can use it only as part of a job or in the -[`default:` section](#default). - -**Possible inputs**: An array of one or two file paths. - -**Example of `cache:key:files`**: - -```yaml -cache-job: - script: - - echo "This job uses a cache." - cache: - key: - files: - - Gemfile.lock - - package.json - paths: - - vendor/ruby - - node_modules -``` - -This example creates a cache for Ruby and Node.js dependencies. The cache -is tied to the current versions of the `Gemfile.lock` and `package.json` files. When one of -these files changes, a new cache key is computed and a new cache is created. Any future -job runs that use the same `Gemfile.lock` and `package.json` with `cache:key:files` -use the new cache, instead of rebuilding the dependencies. - -**Additional details**: - -- The cache `key` is a SHA computed from the most recent commits -that changed each listed file. - If neither file is changed in any commits, the fallback key is `default`. - -##### `cache:key:prefix` +- [Reuse configuration sections by using `extends`](yaml_optimization.md#use-extends-to-reuse-configuration-sections). +- Use `extends` to reuse configuration from [included configuration files](yaml_optimization.md#use-extends-and-include-together). -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/18986) in GitLab 12.5. +### `image` -Use `cache:key:prefix` to combine a prefix with the SHA computed for [`cache:key:files`](#cachekeyfiles). +Use `image` to specify a Docker image that the job runs in. **Keyword type**: Job keyword. You can use it only as part of a job or in the -[`default:` section](#default). +[`default` section](#default). -**Possible inputs**: +**Possible inputs**: The name of the image, including the registry path if needed, in one of these formats: -- A string -- A [predefined variables](../variables/index.md) -- A combination of both. +- `<image-name>` (Same as using `<image-name>` with the `latest` tag) +- `<image-name>:<tag>` +- `<image-name>@<digest>` -**Example of `cache:key:prefix`**: +**Example of `image`**: ```yaml +default: + image: ruby:3.0 + rspec: - script: - - echo "This rspec job uses a cache." - cache: - key: - files: - - Gemfile.lock - prefix: $CI_JOB_NAME - paths: - - vendor/ruby + script: bundle exec rspec + +rspec 2.7: + image: registry.example.com/my-group/my-project/ruby:2.7 + script: bundle exec rspec ``` -For example, adding a `prefix` of `$CI_JOB_NAME` causes the key to look like `rspec-feef9576d21ee9b6a32e30c5c79d0a0ceb68d1e5`. -If a branch changes `Gemfile.lock`, that branch has a new SHA checksum for `cache:key:files`. -A new cache key is generated, and a new cache is created for that key. If `Gemfile.lock` -is not found, the prefix is added to `default`, so the key in the example would be `rspec-default`. +In this example, the `ruby:3.0` image is the default for all jobs in the pipeline. +The `rspec 2.7` job does not use the default, because it overrides the default with +a job-specific `image` section. -**Additional details**: +**Related topics**: -- If no file in `cache:key:files` is changed in any commits, the prefix is added to the `default` key. +- [Run your CI/CD jobs in Docker containers](../docker/using_docker_images.md). -#### `cache:untracked` +#### `image:name` -Use `untracked: true` to cache all files that are untracked in your Git repository: +The name of the Docker image that the job runs in. Similar to [`image`](#image) used by itself. **Keyword type**: Job keyword. You can use it only as part of a job or in the -[`default:` section](#default). +[`default` section](#default). -**Possible inputs**: `true` or `false` (default). +**Possible inputs**: The name of the image, including the registry path if needed, in one of these formats: -**Example of `cache:untracked`**: +- `<image-name>` (Same as using `<image-name>` with the `latest` tag) +- `<image-name>:<tag>` +- `<image-name>@<digest>` + +**Example of `image:name`**: ```yaml -rspec: - script: test - cache: - untracked: true +image: + name: "registry.example.com/my/image:latest" ``` -**Additional details**: - -- You can combine `cache:untracked` with `cache:paths` to cache all untracked files - as well as files in the configured paths. For example: +**Related topics**: - ```yaml - rspec: - script: test - cache: - untracked: true - paths: - - binaries/ - ``` +- [Run your CI/CD jobs in Docker containers](../docker/using_docker_images.md). -#### `cache:when` +#### `image:entrypoint` -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/18969) in GitLab 13.5 and GitLab Runner v13.5.0. +Command or script to execute as the container's entry point. -Use `cache:when` to define when to save the cache, based on the status of the job. +When the Docker container is created, the `entrypoint` is translated to the Docker `--entrypoint` option. +The syntax is similar to the [Dockerfile `ENTRYPOINT` directive](https://docs.docker.com/engine/reference/builder/#entrypoint), +where each shell token is a separate string in the array. **Keyword type**: Job keyword. You can use it only as part of a job or in the -[`default:` section](#default). +[`default` section](#default). **Possible inputs**: -- `on_success` (default): Save the cache only when the job succeeds. -- `on_failure`: Save the cache only when the job fails. -- `always`: Always save the cache. +- A string. -**Example of `cache:when`**: +**Example of `image:entrypoint`**: ```yaml -rspec: - script: rspec - cache: - paths: - - rspec/ - when: 'always' +image: + name: super/sql:experimental + entrypoint: [""] ``` -This example stores the cache whether or not the job fails or succeeds. +**Related topics**: -#### `cache:policy` +- [Override the entrypoint of an image](../docker/using_docker_images.md#override-the-entrypoint-of-an-image). -To change the upload and download behavior of a cache, use the `cache:policy` keyword. -By default, the job downloads the cache when the job starts, and uploads changes -to the cache when the job ends. This caching style is the `pull-push` policy (default). +### `inherit` -To set a job to only download the cache when the job starts, but never upload changes -when the job finishes, use `cache:policy:pull`. +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/207484) in GitLab 12.9. -To set a job to only upload a cache when the job finishes, but never download the -cache when the job starts, use `cache:policy:push`. +Use `inherit` to [control inheritance of globally-defined defaults and variables](../jobs/index.md#control-the-inheritance-of-default-keywords-and-global-variables). -Use the `pull` policy when you have many jobs executing in parallel that use the same cache. -This policy speeds up job execution and reduces load on the cache server. You can -use a job with the `push` policy to build the cache. +#### `inherit:default` -**Keyword type**: Job keyword. You can use it only as part of a job or in the -[`default:` section](#default). +Use `inherit:default` to control the inheritance of [default keywords](#default). + +**Keyword type**: Job keyword. You can use it only as part of a job. **Possible inputs**: -- `pull` -- `push` -- `pull-push` (default) +- `true` (default) or `false` to enable or disable the inheritance of all default keywords. +- A list of specific default keywords to inherit. -**Example of `cache:policy`**: +**Example of `inherit:default`**: ```yaml -prepare-dependencies-job: - stage: build - cache: - key: gems - paths: - - vendor/bundle - policy: push - script: - - echo "This job only downloads dependencies and builds the cache." - - echo "Downloading dependencies..." +default: + retry: 2 + image: ruby:3.0 + interruptible: true -faster-test-job: - stage: test - cache: - key: gems - paths: - - vendor/bundle - policy: pull - script: - - echo "This job script uses the cache, but does not update it." - - echo "Running tests..." +job1: + script: echo "This job does not inherit any default keywords." + inherit: + default: false + +job2: + script: echo "This job inherits only the two listed default keywords. It does not inherit 'interruptible'." + inherit: + default: + - retry + - image ``` -### `dependencies` +**Additional details**: -Use the `dependencies` keyword to define a list of jobs to fetch [artifacts](#artifacts) from. -You can also set a job to download no artifacts at all. +- You can also list default keywords to inherit on one line: `default: [keyword1, keyword2]` -If you do not use `dependencies`, all artifacts from previous stages are passed to each job. +#### `inherit:variables` + +Use `inherit:variables` to control the inheritance of [global variables](#variables) keywords. **Keyword type**: Job keyword. You can use it only as part of a job. **Possible inputs**: -- The names of jobs to fetch artifacts from. -- An empty array (`[]`), to configure the job to not download any artifacts. +- `true` (default) or `false` to enable or disable the inheritance of all global variables. +- A list of specific variables to inherit. -**Example of `dependencies`**: +**Example of `inherit:variables`**: ```yaml -build osx: - stage: build - script: make build:osx - artifacts: - paths: - - binaries/ - -build linux: - stage: build - script: make build:linux - artifacts: - paths: - - binaries/ - -test osx: - stage: test - script: make test:osx - dependencies: - - build:osx +variables: + VARIABLE1: "This is variable 1" + VARIABLE2: "This is variable 2" + VARIABLE3: "This is variable 3" -test linux: - stage: test - script: make test:linux - dependencies: - - build:linux +job1: + script: echo "This job does not inherit any global variables." + inherit: + variables: false -deploy: - stage: deploy - script: make deploy +job2: + script: echo "This job inherits only the two listed global variables. It does not inherit 'VARIABLE3'." + inherit: + variables: + - VARIABLE1 + - VARIABLE2 ``` -In this example, two jobs have artifacts: `build osx` and `build linux`. When `test osx` is executed, -the artifacts from `build osx` are downloaded and extracted in the context of the build. -The same thing happens for `test linux` and artifacts from `build linux`. - -The `deploy` job downloads artifacts from all previous jobs because of -the [stage](#stages) precedence. - **Additional details**: -- The job status does not matter. If a job fails or it's a manual job that isn't triggered, no error occurs. -- If the artifacts of a dependent job are [expired](#artifactsexpire_in) or - [deleted](../pipelines/job_artifacts.md#delete-job-artifacts), then the job fails. - -### `artifacts` - -Use `artifacts` to specify a list of files and directories that are -attached to the job when it [succeeds, fails, or always](#artifactswhen). - -The artifacts are sent to GitLab after the job finishes. They are -available for download in the GitLab UI if the size is not -larger than the [maximum artifact size](../../user/gitlab_com/index.md#gitlab-cicd). +- You can also list global variables to inherit on one line: `variables: [VARIABLE1, VARIABLE2]` -By default, jobs in later stages automatically download all the artifacts created -by jobs in earlier stages. You can control artifact download behavior in jobs with -[`dependencies`](#dependencies). +### `interruptible` -When using the [`needs`](#needs) keyword, jobs can only download -artifacts from the jobs defined in the `needs` configuration. +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/32022) in GitLab 12.3. -Job artifacts are only collected for successful jobs by default, and -artifacts are restored after [caches](#cache). +Use `interruptible` if a job should be canceled when a newer pipeline starts before the job completes. -[Read more about artifacts](../pipelines/job_artifacts.md). +This keyword is used with the [automatic cancellation of redundant pipelines](../pipelines/settings.md#auto-cancel-redundant-pipelines) +feature. When enabled, a running job with `interruptible: true` can be cancelled when +a new pipeline starts on the same branch. -#### `artifacts:exclude` +You can't cancel subsequent jobs after a job with `interruptible: false` starts. -> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/15122) in GitLab 13.1 -> - Requires GitLab Runner 13.1 +**Keyword type**: Job keyword. You can use it only as part of a job or in the +[`default` section](#default). -`exclude` makes it possible to prevent files from being added to an artifacts -archive. +**Possible inputs**: -Similar to [`artifacts:paths`](#artifactspaths), `exclude` paths are relative -to the project directory. You can use Wildcards that use -[glob](https://en.wikipedia.org/wiki/Glob_(programming)) or -[`doublestar.PathMatch`](https://pkg.go.dev/github.com/bmatcuk/doublestar@v1.2.2?tab=doc#PathMatch) patterns. +- `true` or `false` (default). -For example, to store all files in `binaries/`, but not `*.o` files located in -subdirectories of `binaries/`: +**Example of `interruptible`**: ```yaml -artifacts: - paths: - - binaries/ - exclude: - - binaries/**/*.o -``` +stages: + - stage1 + - stage2 + - stage3 -Unlike [`artifacts:paths`](#artifactspaths), `exclude` paths are not recursive. To exclude all of the contents of a directory, you can match them explicitly rather than matching the directory itself. +step-1: + stage: stage1 + script: + - echo "Can be canceled." + interruptible: true -For example, to store all files in `binaries/` but nothing located in the `temp/` subdirectory: +step-2: + stage: stage2 + script: + - echo "Can not be canceled." -```yaml -artifacts: - paths: - - binaries/ - exclude: - - binaries/temp/**/* +step-3: + stage: stage3 + script: + - echo "Because step-2 can not be canceled, this step can never be canceled, even though it's set as interruptible." + interruptible: true ``` -Files matched by [`artifacts:untracked`](#artifactsuntracked) can be excluded using -`artifacts:exclude` too. +In this example, a new pipeline causes a running pipeline to be: -#### `artifacts:expire_in` +- Canceled, if only `step-1` is running or pending. +- Not canceled, after `step-2` starts. -> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/16267) in GitLab 13.0 behind a disabled feature flag, the latest job artifacts are kept regardless of expiry time. -> - [Made default behavior](https://gitlab.com/gitlab-org/gitlab/-/issues/229936) in GitLab 13.4. -> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/241026) in GitLab 13.8, keeping latest job artifacts can be disabled at the project level. -> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/276583) in GitLab 13.9, keeping latest job artifacts can be disabled instance-wide. -> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/321323) in GitLab 13.12, the latest pipeline artifacts are kept regardless of expiry time. +**Additional details**: -Use `expire_in` to specify how long [job artifacts](../pipelines/job_artifacts.md) are stored before -they expire and are deleted. The `expire_in` setting does not affect: +- Only set `interruptible: true` if the job can be safely canceled after it has started, + like a build job. Deployment jobs usually shouldn't be cancelled, to prevent partial deployments. +- To completely cancel a running pipeline, all jobs must have `interruptible: true`, + or `interruptible: false` jobs must not have started. -- Artifacts from the latest job, unless keeping the latest job artifacts is: - - [Disabled at the project level](../pipelines/job_artifacts.md#keep-artifacts-from-most-recent-successful-jobs). - - [Disabled instance-wide](../../user/admin_area/settings/continuous_integration.md#keep-the-latest-artifacts-for-all-jobs-in-the-latest-successful-pipelines). -- [Pipeline artifacts](../pipelines/pipeline_artifacts.md). You can't specify an expiration date for - pipeline artifacts. See [When pipeline artifacts are deleted](../pipelines/pipeline_artifacts.md#when-pipeline-artifacts-are-deleted) - for more information. +### `needs` -The value of `expire_in` is an elapsed time in seconds, unless a unit is provided. Valid values -include: +> - [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/47063) in GitLab 12.2. +> - In GitLab 12.3, maximum number of jobs in `needs` array raised from five to 50. +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/30631) in GitLab 12.8, `needs: []` lets jobs start immediately. +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/30632) in GitLab 14.2, you can refer to jobs in the same stage as the job you are configuring. -- `'42'` -- `42 seconds` -- `3 mins 4 sec` -- `2 hrs 20 min` -- `2h20min` -- `6 mos 1 day` -- `47 yrs 6 mos and 4d` -- `3 weeks and 2 days` -- `never` +Use `needs` to execute jobs out-of-order. Relationships between jobs +that use `needs` can be visualized as a [directed acyclic graph](../directed_acyclic_graph/index.md). -To expire artifacts one week after being uploaded: +You can ignore stage ordering and run some jobs without waiting for others to complete. +Jobs in multiple stages can run concurrently. -```yaml -job: - artifacts: - expire_in: 1 week -``` +**Keyword type**: Job keyword. You can use it only as part of a job. -The expiration time period begins when the artifact is uploaded and stored on GitLab. If the expiry -time is not defined, it defaults to the -[instance wide setting](../../user/admin_area/settings/continuous_integration.md#default-artifacts-expiration) -(30 days by default). +**Possible inputs**: -To override the expiration date and protect artifacts from being automatically deleted: +- An array of jobs. +- An empty array (`[]`), to set the job to start as soon as the pipeline is created. -- Select **Keep** on the job page. -- [In GitLab 13.3 and later](https://gitlab.com/gitlab-org/gitlab/-/issues/22761), set the value of - `expire_in` to `never`. +**Example of `needs`**: -After their expiry, artifacts are deleted hourly by default (using a cron job), and are not -accessible anymore. +```yaml +linux:build: + stage: build + script: echo "Building linux..." -#### `artifacts:expose_as` +mac:build: + stage: build + script: echo "Building mac..." -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/15018) in GitLab 12.5. +lint: + stage: test + needs: [] + script: echo "Linting..." -Use the `expose_as` keyword to expose [job artifacts](../pipelines/job_artifacts.md) -in the [merge request](../../user/project/merge_requests/index.md) UI. +linux:rspec: + stage: test + needs: ["linux:build"] + script: echo "Running rspec on linux..." -For example, to match a single file: +mac:rspec: + stage: test + needs: ["mac:build"] + script: echo "Running rspec on mac..." -```yaml -test: - script: ["echo 'test' > file.txt"] - artifacts: - expose_as: 'artifact 1' - paths: ['file.txt'] +production: + stage: deploy + script: echo "Running production..." ``` -With this configuration, GitLab adds a link **artifact 1** to the relevant merge request -that points to `file1.txt`. To access the link, select **View exposed artifact** -below the pipeline graph in the merge request overview. - -An example that matches an entire directory: +This example creates four paths of execution: -```yaml -test: - script: ["mkdir test && echo 'test' > test/file.txt"] - artifacts: - expose_as: 'artifact 1' - paths: ['test/'] -``` +- Linter: The `lint` job runs immediately without waiting for the `build` stage + to complete because it has no needs (`needs: []`). +- Linux path: The `linux:rspec` job runs as soon as the `linux:build` + job finishes, without waiting for `mac:build` to finish. +- macOS path: The `mac:rspec` jobs runs as soon as the `mac:build` + job finishes, without waiting for `linux:build` to finish. +- The `production` job runs as soon as all previous jobs finish: + `linux:build`, `linux:rspec`, `mac:build`, `mac:rspec`. -Note the following: +**Additional details**: -- Artifacts do not display in the merge request UI when using variables to define the `artifacts:paths`. -- A maximum of 10 job artifacts per merge request can be exposed. -- Glob patterns are unsupported. -- If a directory is specified, the link is to the job [artifacts browser](../pipelines/job_artifacts.md#download-job-artifacts) if there is more than - one file in the directory. -- For exposed single file artifacts with `.html`, `.htm`, `.txt`, `.json`, `.xml`, - and `.log` extensions, if [GitLab Pages](../../administration/pages/index.md) is: - - Enabled, GitLab automatically renders the artifact. - - Not enabled, the file is displayed in the artifacts browser. +- The maximum number of jobs that a single job can have in the `needs` array is limited: + - For GitLab.com, the limit is 50. For more information, see our + [infrastructure issue](https://gitlab.com/gitlab-com/gl-infra/infrastructure/-/issues/7541). + - For self-managed instances, the default limit is 50. This limit [can be changed](../../administration/cicd.md#set-the-needs-job-limit). +- If `needs` refers to a job that uses the [`parallel`](#parallel) keyword, + it depends on all jobs created in parallel, not just one job. It also downloads + artifacts from all the parallel jobs by default. If the artifacts have the same + name, they overwrite each other and only the last one downloaded is saved. +- In [GitLab 14.1 and later](https://gitlab.com/gitlab-org/gitlab/-/issues/30632) you + can refer to jobs in the same stage as the job you are configuring. This feature is + enabled on GitLab.com and ready for production use. On self-managed [GitLab 14.2 and later](https://gitlab.com/gitlab-org/gitlab/-/issues/30632) + this feature is available by default. +- In GitLab 14.0 and older, you can only refer to jobs in earlier stages. Stages must be + explicitly defined for all jobs that use the `needs` keyword, or are referenced + in a job's `needs` section. +- In GitLab 13.9 and older, if `needs` refers to a job that might not be added to + a pipeline because of `only`, `except`, or `rules`, the pipeline might fail to create. -#### `artifacts:name` +#### `needs:artifacts` -Use the `name` directive to define the name of the created artifacts -archive. You can specify a unique name for every archive. The `artifacts:name` -variable can make use of any of the [predefined variables](../variables/index.md). -The default name is `artifacts`, which becomes `artifacts.zip` when you download it. +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/14311) in GitLab 12.6. -To create an archive with a name of the current job: +When a job uses `needs`, it no longer downloads all artifacts from previous stages +by default, because jobs with `needs` can start before earlier stages complete. With +`needs` you can only download artifacts from the jobs listed in the `needs` configuration. -```yaml -job: - artifacts: - name: "$CI_JOB_NAME" - paths: - - binaries/ -``` +Use `artifacts: true` (default) or `artifacts: false` to control when artifacts are +downloaded in jobs that use `needs`. -To create an archive with a name of the current branch or tag including only -the binaries directory: +**Keyword type**: Job keyword. You can use it only as part of a job. Must be used with `needs:job`. -```yaml -job: - artifacts: - name: "$CI_COMMIT_REF_NAME" - paths: - - binaries/ -``` +**Possible inputs**: -If your branch-name contains forward slashes -(for example `feature/my-feature`) it's advised to use `$CI_COMMIT_REF_SLUG` -instead of `$CI_COMMIT_REF_NAME` for proper naming of the artifact. +- `true` (default) or `false`. -To create an archive with a name of the current job and the current branch or -tag including only the binaries directory: +**Example of `needs:artifacts`**: ```yaml -job: - artifacts: - name: "$CI_JOB_NAME-$CI_COMMIT_REF_NAME" - paths: - - binaries/ -``` +test-job1: + stage: test + needs: + - job: build_job1 + artifacts: true -To create an archive with a name of the current [stage](#stages) and branch name: +test-job2: + stage: test + needs: + - job: build_job2 + artifacts: false -```yaml -job: - artifacts: - name: "$CI_JOB_STAGE-$CI_COMMIT_REF_NAME" - paths: - - binaries/ +test-job3: + needs: + - job: build_job1 + artifacts: true + - job: build_job2 + - build_job3 ``` ---- - -If you use **Windows Batch** to run your shell scripts you must replace -`$` with `%`: - -```yaml -job: - artifacts: - name: "%CI_JOB_STAGE%-%CI_COMMIT_REF_NAME%" - paths: - - binaries/ -``` +In this example: -If you use **Windows PowerShell** to run your shell scripts you must replace -`$` with `$env:`: +- The `test-job1` job downloads the `build_job1` artifacts +- The `test-job2` job does not download the `build_job2` artifacts. +- The `test-job3` job downloads the artifacts from all three `build_jobs`, because + `artifacts` is `true`, or defaults to `true`, for all three needed jobs. -```yaml -job: - artifacts: - name: "$env:CI_JOB_STAGE-$env:CI_COMMIT_REF_NAME" - paths: - - binaries/ -``` +**Additional details**: -#### `artifacts:paths` +- In GitLab 12.6 and later, you can't combine the [`dependencies`](#dependencies) keyword + with `needs`. -Paths are relative to the project directory (`$CI_PROJECT_DIR`) and can't directly -link outside it. You can use Wildcards that use [glob](https://en.wikipedia.org/wiki/Glob_(programming)) -patterns and: +#### `needs:project` **(PREMIUM)** -- In [GitLab Runner 13.0 and later](https://gitlab.com/gitlab-org/gitlab-runner/-/issues/2620), - [`doublestar.Glob`](https://pkg.go.dev/github.com/bmatcuk/doublestar@v1.2.2?tab=doc#Match). -- In GitLab Runner 12.10 and earlier, [`filepath.Match`](https://pkg.go.dev/path/filepath#Match). +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/14311) in GitLab 12.7. -To restrict which jobs a specific job fetches artifacts from, see [dependencies](#dependencies). +Use `needs:project` to download artifacts from up to five jobs in other pipelines. +The artifacts are downloaded from the latest successful pipeline for the specified ref. -Send all files in `binaries` and `.config`: +If there is a pipeline running for the specified ref, a job with `needs:project` +does not wait for the pipeline to complete. Instead, the job downloads the artifact +from the latest pipeline that completed successfully. -```yaml -artifacts: - paths: - - binaries/ - - .config -``` +`needs:project` must be used with `job`, `ref`, and `artifacts`. -To disable artifact passing, define the job with empty [dependencies](#dependencies): +**Keyword type**: Job keyword. You can use it only as part of a job. -```yaml -job: - stage: build - script: make build - dependencies: [] -``` +**Possible inputs**: -You may want to create artifacts only for tagged releases to avoid filling the -build server storage with temporary build artifacts. +- `needs:project`: A full project path, including namespace and group. +- `job`: The job to download artifacts from. +- `ref`: The ref to download artifacts from. +- `artifacts`: Must be `true` to download artifacts. -Create artifacts only for tags (`default-job` doesn't create artifacts): +**Examples of `needs:project`**: ```yaml -default-job: - script: - - mvn test -U - rules: - - if: $CI_COMMIT_BRANCH - -release-job: +build_job: + stage: build script: - - mvn package -U - artifacts: - paths: - - target/*.war - rules: - - if: $CI_COMMIT_TAG -``` - -You can use wildcards for directories too. For example, if you want to get all the files inside the directories that end with `xyz`: - -```yaml -job: - artifacts: - paths: - - path/*xyz/* + - ls -lhR + needs: + - project: namespace/group/project-name + job: build-1 + ref: main + artifacts: true ``` -#### `artifacts:public` - -> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/49775) in GitLab 13.8 -> - It's [deployed behind a feature flag](../../user/feature_flags.md), disabled by default. -> - It's disabled on GitLab.com. -> - It's recommended for production use. - -Use `artifacts:public` to determine whether the job artifacts should be -publicly available. - -The default for `artifacts:public` is `true` which means that the artifacts in -public pipelines are available for download by anonymous and guest users: - -```yaml -artifacts: - public: true -``` +In this example, `build_job` downloads the artifacts from the latest successful `build-1` job +on the `main` branch in the `group/project-name` project. -To deny read access for anonymous and guest users to artifacts in public -pipelines, set `artifacts:public` to `false`: +In GitLab 13.3 and later, you can use [CI/CD variables](../variables/index.md) in `needs:project`, +for example: ```yaml -artifacts: - public: false +build_job: + stage: build + script: + - ls -lhR + needs: + - project: $CI_PROJECT_PATH + job: $DEPENDENCY_JOB_NAME + ref: $ARTIFACTS_DOWNLOAD_REF + artifacts: true ``` -#### `artifacts:reports` - -Use [`artifacts:reports`](#artifactsreports) to: - -- Collect test reports, code quality reports, security reports, and other artifacts generated by included templates in - jobs. -- Some of these reports are used to display information in: - - Merge requests. - - Pipeline views. - - [Security dashboards](../../user/application_security/security_dashboard/index.md). - -The test reports are collected regardless of the job results (success or failure). -You can use [`artifacts:expire_in`](#artifactsexpire_in) to set up an expiration -date for their artifacts. - -Some `artifacts:reports` types can be generated by multiple jobs in the same pipeline, and used by merge request or -pipeline features from each job. - -To be able to browse the report output files, include the [`artifacts:paths`](#artifactspaths) keyword. - -NOTE: -Combined reports in parent pipelines using [artifacts from child pipelines](#needspipelinejob) is -not supported. Track progress on adding support in [this issue](https://gitlab.com/gitlab-org/gitlab/-/issues/215725). - -##### `artifacts:reports:accessibility` **(FREE)** - -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/39425) in GitLab 12.8. - -The `accessibility` report uses [pa11y](https://pa11y.org/) to report on the accessibility impact -of changes introduced in merge requests. - -GitLab can display the results of one or more reports in the merge request -[accessibility widget](../../user/project/merge_requests/accessibility_testing.md#accessibility-merge-request-widget). - -For more information, see [Accessibility testing](../../user/project/merge_requests/accessibility_testing.md). - -##### `artifacts:reports:api_fuzzing` **(ULTIMATE)** - -> - Introduced in GitLab 13.4. -> - Requires GitLab Runner 13.4 or later. - -The `api_fuzzing` report collects [API Fuzzing bugs](../../user/application_security/api_fuzzing/index.md) -as artifacts. - -GitLab can display the results of one or more reports in: - -- The merge request [security widget](../../user/application_security/api_fuzzing/index.md#view-details-of-an-api-fuzzing-vulnerability). -- The [Project Vulnerability report](../../user/application_security/vulnerability_report/index.md). -- The pipeline [**Security** tab](../../user/application_security/security_dashboard/index.md#pipeline-security). -- The [security dashboard](../../user/application_security/api_fuzzing/index.md#security-dashboard). - -##### `artifacts:reports:browser_performance` **(PREMIUM)** - -> [Name changed](https://gitlab.com/gitlab-org/gitlab/-/issues/225914) from `artifacts:reports:performance` in GitLab 14.0. - -The `browser_performance` report collects [Browser Performance Testing metrics](../../user/project/merge_requests/browser_performance_testing.md) -as artifacts. - -GitLab can display the results of one report in the merge request -[browser performance testing widget](../../user/project/merge_requests/browser_performance_testing.md#how-browser-performance-testing-works). - -GitLab cannot display the combined results of multiple `browser_performance` reports. - -##### `artifacts:reports:cluster_image_scanning` **(ULTIMATE)** - -> - Introduced in GitLab 14.1. -> - Requires GitLab Runner 14.1 and above. - -The `cluster_image_scanning` report collects `CLUSTER_IMAGE_SCANNING` vulnerabilities. The collected -`CLUSTER_IMAGE_SCANNING` report uploads to GitLab as an artifact. - -GitLab can display the results of one or more reports in: - -- The [security dashboard](../../user/application_security/security_dashboard/index.md). -- The [Project Vulnerability report](../../user/application_security/vulnerability_report/index.md). - -##### `artifacts:reports:cobertura` - -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/3708) in GitLab 12.9. - -The `cobertura` report collects [Cobertura coverage XML files](../../user/project/merge_requests/test_coverage_visualization.md). -The collected Cobertura coverage reports upload to GitLab as an artifact. - -GitLab can display the results of one or more reports in the merge request -[diff annotations](../../user/project/merge_requests/test_coverage_visualization.md). - -Cobertura was originally developed for Java, but there are many third-party ports for other languages such as -JavaScript, Python, and Ruby. - -##### `artifacts:reports:codequality` - -> [Moved](https://gitlab.com/gitlab-org/gitlab/-/issues/212499) to GitLab Free in 13.2. - -The `codequality` report collects [code quality issues](../../user/project/merge_requests/code_quality.md). The -collected code quality report uploads to GitLab as an artifact. +**Additional details**: -GitLab can display the results of: +- To download artifacts from a different pipeline in the current project, set `project` + to be the same as the current project, but use a different ref than the current pipeline. + Concurrent pipelines running on the same ref could override the artifacts. +- The user running the pipeline must have at least the Reporter role for the group or project, + or the group/project must have public visibility. +- You can't use `needs:project` in the same job as [`trigger`](#trigger). +- When using `needs:project` to download artifacts from another pipeline, the job does not wait for + the needed job to complete. [Directed acyclic graph](../directed_acyclic_graph/index.md) + behavior is limited to jobs in the same pipeline. Make sure that the needed job in the other + pipeline completes before the job that needs it tries to download the artifacts. +- You can't download artifacts from jobs that run in [`parallel`](#parallel). +- Support for [CI/CD variables](../variables/index.md) in `project`, `job`, and `ref` was + [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/202093) in GitLab 13.3. + [Feature flag removed](https://gitlab.com/gitlab-org/gitlab/-/issues/235761) in GitLab 13.4. -- One or more reports in the merge request [code quality widget](../../user/project/merge_requests/code_quality.md#code-quality-widget). -- Only one report in: - - The merge request [diff annotations](../../user/project/merge_requests/code_quality.md#code-quality-in-diff-view). - Track progress on adding support for multiple reports in [this issue](https://gitlab.com/gitlab-org/gitlab/-/issues/328257). - - The [full report](../metrics_reports.md). Track progress on adding support for multiple reports in - [this issue](https://gitlab.com/gitlab-org/gitlab/-/issues/9014). +**Related topics**: -##### `artifacts:reports:container_scanning` **(ULTIMATE)** +- To download artifacts between [parent-child pipelines](../pipelines/parent_child_pipelines.md), + use [`needs:pipeline:job`](#needspipelinejob). -The `container_scanning` report collects [Container Scanning vulnerabilities](../../user/application_security/container_scanning/index.md). -The collected Container Scanning report uploads to GitLab as an artifact. +#### `needs:pipeline:job` -GitLab can display the results of one or more reports in: +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/255983) in GitLab 13.7. -- The merge request [container scanning widget](../../user/application_security/container_scanning/index.md). -- The pipeline [**Security** tab](../../user/application_security/security_dashboard/index.md#pipeline-security). -- The [security dashboard](../../user/application_security/security_dashboard/index.md). -- The [Project Vulnerability report](../../user/application_security/vulnerability_report/index.md). +A [child pipeline](../pipelines/parent_child_pipelines.md) can download artifacts from a job in +its parent pipeline or another child pipeline in the same parent-child pipeline hierarchy. -##### `artifacts:reports:coverage_fuzzing` **(ULTIMATE)** +**Keyword type**: Job keyword. You can use it only as part of a job. -> - Introduced in GitLab 13.4. -> - Requires GitLab Runner 13.4 or later. +**Possible inputs**: -The `coverage_fuzzing` report collects [coverage fuzzing bugs](../../user/application_security/coverage_fuzzing/index.md). -The collected coverage fuzzing report uploads to GitLab as an artifact. -GitLab can display the results of one or more reports in: +- `needs:pipeline`: A pipeline ID. Must be a pipeline present in the same parent-child pipeline hierarchy. +- `job`: The job to download artifacts from. -- The merge request [coverage fuzzing widget](../../user/application_security/coverage_fuzzing/index.md#interacting-with-the-vulnerabilities). -- The pipeline [**Security** tab](../../user/application_security/security_dashboard/index.md#pipeline-security). -- The [Project Vulnerability report](../../user/application_security/vulnerability_report/index.md). -- The [security dashboard](../../user/application_security/security_dashboard/index.md). +**Example of `needs:pipeline:job`**: -##### `artifacts:reports:dast` **(ULTIMATE)** +- Parent pipeline (`.gitlab-ci.yml`): -The `dast` report collects [DAST vulnerabilities](../../user/application_security/dast/index.md). The collected DAST -report uploads to GitLab as an artifact. + ```yaml + create-artifact: + stage: build + script: echo "sample artifact" > artifact.txt + artifacts: + paths: [artifact.txt] -GitLab can display the results of one or more reports in: + child-pipeline: + stage: test + trigger: + include: child.yml + strategy: depend + variables: + PARENT_PIPELINE_ID: $CI_PIPELINE_ID + ``` -- The merge request [security widget](../../user/application_security/dast/index.md#view-details-of-a-vulnerability-detected-by-dast). -- The pipeline [**Security** tab](../../user/application_security/security_dashboard/index.md#pipeline-security). -- The [Project Vulnerability report](../../user/application_security/vulnerability_report/index.md). -- The [security dashboard](../../user/application_security/security_dashboard/index.md). +- Child pipeline (`child.yml`): -##### `artifacts:reports:dependency_scanning` **(ULTIMATE)** + ```yaml + use-artifact: + script: cat artifact.txt + needs: + - pipeline: $PARENT_PIPELINE_ID + job: create-artifact + ``` -The `dependency_scanning` report collects [Dependency Scanning vulnerabilities](../../user/application_security/dependency_scanning/index.md). -The collected Dependency Scanning report uploads to GitLab as an artifact. +In this example, the `create-artifact` job in the parent pipeline creates some artifacts. +The `child-pipeline` job triggers a child pipeline, and passes the `CI_PIPELINE_ID` +variable to the child pipeline as a new `PARENT_PIPELINE_ID` variable. The child pipeline +can use that variable in `needs:pipeline` to download artifacts from the parent pipeline. -GitLab can display the results of one or more reports in: +**Additional details**: -- The merge request [dependency scanning widget](../../user/application_security/dependency_scanning/index.md#overview). -- The pipeline [**Security** tab](../../user/application_security/security_dashboard/index.md#pipeline-security). -- The [security dashboard](../../user/application_security/security_dashboard/index.md). -- The [Project Vulnerability report](../../user/application_security/vulnerability_report/index.md). -- The [dependency list](../../user/application_security/dependency_list/). +- The `pipeline` attribute does not accept the current pipeline ID (`$CI_PIPELINE_ID`). + To download artifacts from a job in the current pipeline, use [`needs`](#needsartifacts). -##### `artifacts:reports:dotenv` +#### `needs:optional` -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/17066) in GitLab 12.9. +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/30680) in GitLab 13.10. +> - [Feature flag removed](https://gitlab.com/gitlab-org/gitlab/-/issues/323891) in GitLab 14.0. -The `dotenv` report collects a set of environment variables as artifacts. +To need a job that sometimes does not exist in the pipeline, add `optional: true` +to the `needs` configuration. If not defined, `optional: false` is the default. -The collected variables are registered as runtime-created variables of the job, -which is useful to [set dynamic environment URLs after a job finishes](../environments/index.md#set-dynamic-environment-urls-after-a-job-finishes). +Jobs that use [`rules`](#rules), [`only`, or `except`](#only--except), might +not always exist in a pipeline. When the pipeline is created, GitLab checks the `needs` +relationships before starting it. Without `optional: true`, needs relationships that +point to a job that does not exist stops the pipeline from starting and causes a pipeline +error similar to: -The exceptions to the [original dotenv rules](https://github.com/motdotla/dotenv#rules) are: +- `'job1' job needs 'job2' job, but it was not added to the pipeline` -- The variable key can contain only letters, digits, and underscores (`_`). -- The maximum size of the `.env` file is 5 KB. -- In GitLab 13.5 and older, the maximum number of inherited variables is 10. -- In [GitLab 13.6 and later](https://gitlab.com/gitlab-org/gitlab/-/issues/247913), - the maximum number of inherited variables is 20. -- Variable substitution in the `.env` file is not supported. -- The `.env` file can't have empty lines or comments (starting with `#`). -- Key values in the `env` file cannot have spaces or newline characters (`\n`), including when using single or double quotes. -- Quote escaping during parsing (`key = 'value'` -> `{key: "value"}`) is not supported. +**Keyword type**: Job keyword. You can use it only as part of a job. -##### `artifacts:reports:junit` +**Possible inputs**: -The `junit` report collects [JUnit report format XML files](https://www.ibm.com/docs/en/adfz/developer-for-zos/14.1.0?topic=formats-junit-xml-format). -The collected Unit test reports upload to GitLab as an artifact. Although JUnit was originally developed in Java, there -are many third-party ports for other languages such as JavaScript, Python, and Ruby. +- `job`: The job to make optional. +- `true` or `false` (default). -See [Unit test reports](../unit_test_reports.md) for more details and examples. -Below is an example of collecting a JUnit report format XML file from Ruby's RSpec test tool: +**Example of `needs:optional`**: ```yaml +build: + stage: build + rules: + - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH + rspec: stage: test - script: - - bundle install - - rspec --format RspecJunitFormatter --out rspec.xml - artifacts: - reports: - junit: rspec.xml + needs: + - job: build + optional: true ``` -GitLab can display the results of one or more reports in: - -- The merge request [code quality widget](../../ci/unit_test_reports.md#how-it-works). -- The [full report](../../ci/unit_test_reports.md#viewing-unit-test-reports-on-gitlab). - -Some JUnit tools export to multiple XML files. You can specify multiple test report paths in a single job to -concatenate them into a single file. Use either: - -- A filename pattern (`junit: rspec-*.xml`). -- an array of filenames (`junit: [rspec-1.xml, rspec-2.xml, rspec-3.xml]`). -- A Combination of both (`junit: [rspec.xml, test-results/TEST-*.xml]`). - -##### `artifacts:reports:license_scanning` **(ULTIMATE)** - -> Introduced in GitLab 12.8. - -The License Compliance report collects [Licenses](../../user/compliance/license_compliance/index.md). The License -Compliance report uploads to GitLab as an artifact. - -GitLab can display the results of one or more reports in: - -- The merge request [license compliance widget](../../user/compliance/license_compliance/index.md). -- The [license list](../../user/compliance/license_compliance/index.md#license-list). - -##### `artifacts:reports:load_performance` **(PREMIUM)** - -> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/35260) in GitLab 13.2. -> - Requires GitLab Runner 11.5 and above. - -The `load_performance` report collects [Load Performance Testing metrics](../../user/project/merge_requests/load_performance_testing.md). -The report is uploaded to GitLab as an artifact. - -GitLab can display the results of only one report in the merge request -[load testing widget](../../user/project/merge_requests/load_performance_testing.md#how-load-performance-testing-works). - -GitLab cannot display the combined results of multiple `load_performance` reports. - -##### `artifacts:reports:metrics` **(PREMIUM)** +In this example: -The `metrics` report collects [Metrics](../metrics_reports.md). The collected Metrics report uploads to GitLab as an -artifact. +- When the branch is the default branch, the `build` job exists in the pipeline, and the `rspec` + job waits for it to complete before starting. +- When the branch is not the default branch, the `build` job does not exist in the pipeline. + The `rspec` job runs immediately (similar to `needs: []`) because its `needs` + relationship to the `build` job is optional. -GitLab can display the results of one or more reports in the merge request -[metrics reports widget](../../ci/metrics_reports.md#metrics-reports). +#### `needs:pipeline` -##### `artifacts:reports:requirements` **(ULTIMATE)** +You can mirror the pipeline status from an upstream pipeline to a bridge job by +using the `needs:pipeline` keyword. The latest pipeline status from the default branch is +replicated to the bridge job. -> - [Introduced](https://gitlab.com/groups/gitlab-org/-/epics/2859) in GitLab 13.1. +**Keyword type**: Job keyword. You can use it only as part of a job. -The `requirements` report collects `requirements.json` files. The collected Requirements report uploads to GitLab as an -artifact and existing [requirements](../../user/project/requirements/index.md) are marked as Satisfied. +**Possible inputs**: -GitLab can display the results of one or more reports in the -[project requirements](../../user/project/requirements/index.md#view-a-requirement). +- A full project path, including namespace and group. If the + project is in the same group or namespace, you can omit them from the `project` + keyword. For example: `project: group/project-name` or `project: project-name`. -##### `artifacts:reports:sast` +**Example of `needs:pipeline`**: -> - [Moved](https://gitlab.com/groups/gitlab-org/-/epics/2098) from GitLab Ultimate to GitLab Free in 13.3. +```yaml +upstream_bridge: + stage: test + needs: + pipeline: other/project +``` -The `sast` report collects [SAST vulnerabilities](../../user/application_security/sast/index.md). The collected SAST -report uploads to GitLab as an artifact. +**Additional details**: -GitLab can display the results of one or more reports in: +- If you add the `job` keyword to `needs:pipeline`, the job no longer mirrors the + pipeline status. The behavior changes to [`needs:pipeline:job`](#needspipelinejob). -- The merge request [SAST widget](../../user/application_security/sast/index.md#static-application-security-testing-sast). -- The [security dashboard](../../user/application_security/security_dashboard/index.md). +### `only` / `except` -##### `artifacts:reports:secret_detection` +NOTE: +`only` and `except` are not being actively developed. [`rules`](#rules) is the preferred +keyword to control when to add jobs to pipelines. -> - Introduced in GitLab 13.1. -> - [Moved](https://gitlab.com/gitlab-org/gitlab/-/issues/222788) to GitLab Free in 13.3. -> - Requires GitLab Runner 11.5 and above. +You can use `only` and `except` to control when to add jobs to pipelines. -The `secret-detection` report collects [detected secrets](../../user/application_security/secret_detection/index.md). -The collected Secret Detection report is uploaded to GitLab. +- Use `only` to define when a job runs. +- Use `except` to define when a job **does not** run. -GitLab can display the results of one or more reports in: +Four keywords can be used with `only` and `except`: -- The merge request [secret scanning widget](../../user/application_security/secret_detection/index.md). -- The [pipeline **Security** tab](../../user/application_security/index.md#view-security-scan-information-in-the-pipeline-security-tab). -- The [security dashboard](../../user/application_security/security_dashboard/index.md). +- [`refs`](#onlyrefs--exceptrefs) +- [`variables`](#onlyvariables--exceptvariables) +- [`changes`](#onlychanges--exceptchanges) +- [`kubernetes`](#onlykubernetes--exceptkubernetes) -##### `artifacts:reports:terraform` +See [specify when jobs run with `only` and `except`](../jobs/job_control.md#specify-when-jobs-run-with-only-and-except) +for more details and examples. -> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/207528) in GitLab 13.0. -> - Requires [GitLab Runner](https://docs.gitlab.com/runner/) 11.5 and above. +#### `only:refs` / `except:refs` -The `terraform` report obtains a Terraform `tfplan.json` file. [JQ processing required to remove credentials](../../user/infrastructure/iac/mr_integration.md#configure-terraform-report-artifacts). -The collected Terraform plan report uploads to GitLab as an artifact. +Use the `only:refs` and `except:refs` keywords to control when to add jobs to a +pipeline based on branch names or pipeline types. -GitLab can display the results of one or more reports in the merge request -[terraform widget](../../user/infrastructure/iac/mr_integration.md#output-terraform-plan-information-into-a-merge-request). +**Keyword type**: Job keyword. You can use it only as part of a job. -For more information, see [Output `terraform plan` information into a merge request](../../user/infrastructure/iac/mr_integration.md). +**Possible inputs**: An array including any number of: -#### `artifacts:untracked` +- Branch names, for example `main` or `my-feature-branch`. +- [Regular expressions](../jobs/job_control.md#only--except-regex-syntax) + that match against branch names, for example `/^feature-.*/`. +- The following keywords: -Use `artifacts:untracked` to add all Git untracked files as artifacts (along -with the paths defined in `artifacts:paths`). `artifacts:untracked` ignores configuration -in the repository's `.gitignore` file. + | **Value** | **Description** | + | -------------------------|-----------------| + | `api` | For pipelines triggered by the [pipelines API](../../api/pipelines.md#create-a-new-pipeline). | + | `branches` | When the Git reference for a pipeline is a branch. | + | `chat` | For pipelines created by using a [GitLab ChatOps](../chatops/index.md) command. | + | `external` | When you use CI services other than GitLab. | + | `external_pull_requests` | When an external pull request on GitHub is created or updated (See [Pipelines for external pull requests](../ci_cd_for_external_repos/index.md#pipelines-for-external-pull-requests)). | + | `merge_requests` | For pipelines created when a merge request is created or updated. Enables [merge request pipelines](../pipelines/merge_request_pipelines.md), [merged results pipelines](../pipelines/pipelines_for_merged_results.md), and [merge trains](../pipelines/merge_trains.md). | + | `pipelines` | For [multi-project pipelines](../pipelines/multi_project_pipelines.md) created by [using the API with `CI_JOB_TOKEN`](../pipelines/multi_project_pipelines.md#create-multi-project-pipelines-by-using-the-api), or the [`trigger`](#trigger) keyword. | + | `pushes` | For pipelines triggered by a `git push` event, including for branches and tags. | + | `schedules` | For [scheduled pipelines](../pipelines/schedules.md). | + | `tags` | When the Git reference for a pipeline is a tag. | + | `triggers` | For pipelines created by using a [trigger token](../triggers/index.md#configure-cicd-jobs-to-run-in-triggered-pipelines). | + | `web` | For pipelines created by selecting **Run pipeline** in the GitLab UI, from the project's **CI/CD > Pipelines** section. | -Send all Git untracked files: +**Example of `only:refs` and `except:refs`**: ```yaml -artifacts: - untracked: true -``` - -Send all Git untracked files and files in `binaries`: +job1: + script: echo + only: + - main + - /^issue-.*$/ + - merge_requests -```yaml -artifacts: - untracked: true - paths: - - binaries/ +job2: + script: echo + except: + - main + - /^stable-branch.*$/ + - schedules ``` -Send all untracked files but [exclude](#artifactsexclude) `*.txt`: +**Additional details**: -```yaml -artifacts: - untracked: true - exclude: - - "*.txt" -``` +- Scheduled pipelines run on specific branches, so jobs configured with `only: branches` + run on scheduled pipelines too. Add `except: schedules` to prevent jobs with `only: branches` + from running on scheduled pipelines. +- `only` or `except` used without any other keywords are equivalent to `only: refs` + or `except: refs`. For example, the following two jobs configurations have the same + behavior: -#### `artifacts:when` + ```yaml + job1: + script: echo + only: + - branches -Use `artifacts:when` to upload artifacts on job failure or despite the -failure. + job2: + script: echo + only: + refs: + - branches + ``` -`artifacts:when` can be set to one of the following values: +- If a job does not use `only`, `except`, or [`rules`](#rules), then `only` is set to `branches` + and `tags` by default. -1. `on_success` (default): Upload artifacts only when the job succeeds. -1. `on_failure`: Upload artifacts only when the job fails. -1. `always`: Always upload artifacts. For example, when - [uploading artifacts](../unit_test_reports.md#viewing-junit-screenshots-on-gitlab) required to - troubleshoot failing tests. + For example, `job1` and `job2` are equivalent: -For example, to upload artifacts only when a job fails: + ```yaml + job1: + script: echo "test" -```yaml -job: - artifacts: - when: on_failure -``` + job2: + script: echo "test" + only: + - branches + - tags + ``` -### `coverage` +#### `only:variables` / `except:variables` -Use `coverage` with a custom regular expression to configure how code coverage -is extracted from the job output. The coverage is shown in the UI if at least one -line in the job output matches the regular expression. +Use the `only:variables` or `except:variables` keywords to control when to add jobs +to a pipeline, based on the status of [CI/CD variables](../variables/index.md). -To extract the code coverage value in the matching line, GitLab uses this -regular expression: `\d+(\.\d+)?`. +**Keyword type**: Job keyword. You can use it only as part of a job. -**Possible inputs**: A regular expression. Must start and end with `/`. +**Possible inputs**: -**Example of `coverage`**: +- An array of [CI/CD variable expressions](../jobs/job_control.md#cicd-variable-expressions). + +**Example of `only:variables`**: ```yaml -job1: - script: rspec - coverage: '/Code coverage: \d+\.\d+/' +deploy: + script: cap staging deploy + only: + variables: + - $RELEASE == "staging" + - $STAGING ``` -In this example: - -1. GitLab checks the job log for a line that matches the regular expression. A line - like `Code coverage: 67.89` would match. -1. GitLab then checks the line to find a match to `\d+(\.\d+)?`. The sample matching - line above gives a code coverage of `67.89`. +**Related topics**: -**Additional details**: +- [`only:variables` and `except:variables` examples](../jobs/job_control.md#only-variables--except-variables-examples). -- If there is more than one matched line in the job output, the last line is used. -- Leading zeros are removed. -- Coverage output from [child pipelines](../pipelines/parent_child_pipelines.md) - is not recorded or displayed. Check [the related issue](https://gitlab.com/gitlab-org/gitlab/-/issues/280818) - for more details. +#### `only:changes` / `except:changes` -### `dast_configuration` **(ULTIMATE)** +Use the `changes` keyword with `only` to run a job, or with `except` to skip a job, +when a Git push event modifies a file. -> [Introduced](https://gitlab.com/groups/gitlab-org/-/epics/5981) in GitLab 14.1. +Use `changes` in pipelines with the following refs: -Use the `dast_configuration` keyword to specify a site profile and scanner profile to be used in a -CI/CD configuration. Both profiles must first have been created in the project. The job's stage must -be `dast`. +- `branches` +- `external_pull_requests` +- `merge_requests` (see additional details about [using `only:changes` with pipelines for merge requests](../jobs/job_control.md#use-onlychanges-with-pipelines-for-merge-requests)) -**Keyword type**: Job keyword. You can use only as part of a job. +**Keyword type**: Job keyword. You can use it only as part of a job. -**Possible inputs**: One each of `site_profile` and `scanner_profile`. +**Possible inputs**: An array including any number of: -- Use `site_profile` to specify the site profile to be used in the job. -- Use `scanner_profile` to specify the scanner profile to be used in the job. +- Paths to files. +- Wildcard paths for single directories, for example `path/to/directory/*`, or a directory + and all its subdirectories, for example `path/to/directory/**/*`. +- Wildcard ([glob](https://en.wikipedia.org/wiki/Glob_(programming))) paths for all + files with the same extension or multiple extensions, for example `*.md` or `path/to/directory/*.{rb,py,sh}`. +- Wildcard paths to files in the root directory, or all directories, wrapped in double quotes. + For example `"*.json"` or `"**/*.json"`. -**Example of `dast_configuration`**: +**Example of `only:changes`**: ```yaml -stages: - - build - - dast - -include: - - template: DAST.gitlab-ci.yml - -dast: - dast_configuration: - site_profile: "Example Co" - scanner_profile: "Quick Passive Test" +docker build: + script: docker build -t my-image:$CI_COMMIT_REF_SLUG . + only: + refs: + - branches + changes: + - Dockerfile + - docker/scripts/* + - dockerfiles/**/* + - more_scripts/*.{rb,py,sh} + - "**/*.json" ``` -In this example, the `dast` job extends the `dast` configuration added with the `include:` keyword -to select a specific site profile and scanner profile. - **Additional details**: -- Settings contained in either a site profile or scanner profile take precedence over those - contained in the DAST template. +- `changes` resolves to `true` if any of the matching files are changed (an `OR` operation). +- If you use refs other than `branches`, `external_pull_requests`, or `merge_requests`, + `changes` can't determine if a given file is new or old and always returns `true`. +- If you use `only: changes` with other refs, jobs ignore the changes and always run. +- If you use `except: changes` with other refs, jobs ignore the changes and never run. **Related topics**: -- [Site profile](../../user/application_security/dast/index.md#site-profile). -- [Scanner profile](../../user/application_security/dast/index.md#scanner-profile). - -### `retry` +- [`only: changes` and `except: changes` examples](../jobs/job_control.md#onlychanges--exceptchanges-examples). +- If you use `changes` with [only allow merge requests to be merged if the pipeline succeeds](../../user/project/merge_requests/merge_when_pipeline_succeeds.md#only-allow-merge-requests-to-be-merged-if-the-pipeline-succeeds), + you should [also use `only:merge_requests`](../jobs/job_control.md#use-onlychanges-with-pipelines-for-merge-requests). +- Use `changes` with [new branches or tags *without* pipelines for merge requests](../jobs/job_control.md#use-onlychanges-without-pipelines-for-merge-requests). +- Use `changes` with [scheduled pipelines](../jobs/job_control.md#use-onlychanges-with-scheduled-pipelines). -Use `retry` to configure how many times a job is retried if it fails. -If not defined, defaults to `0` and jobs do not retry. +#### `only:kubernetes` / `except:kubernetes` -When a job fails, the job is processed up to two more times, until it succeeds or -reaches the maximum number of retries. +Use `only:kubernetes` or `except:kubernetes` to control if jobs are added to the pipeline +when the Kubernetes service is active in the project. -By default, all failure types cause the job to be retried. Use [`retry:when`](#retrywhen) -to select which failures to retry on. +**Keyword type**: Job-specific. You can use it only as part of a job. -**Keyword type**: Job keyword. You can use it only as part of a job or in the -[`default:` section](#default). +**Possible inputs**: -**Possible inputs**: `0` (default), `1`, or `2`. +- The `kubernetes` strategy accepts only the `active` keyword. -**Example of `retry`**: +**Example of `only:kubernetes`**: ```yaml -test: - script: rspec - retry: 2 +deploy: + only: + kubernetes: active ``` -#### `retry:when` - -Use `retry:when` with `retry:max` to retry jobs for only specific failure cases. -`retry:max` is the maximum number of retries, like [`retry`](#retry), and can be -`0`, `1`, or `2`. - -**Keyword type**: Job keyword. You can use it only as part of a job or in the -[`default:` section](#default). - -**Possible inputs**: A single failure type, or an array of one or more failure types: - -<!-- - If you change any of the values below, make sure to update the `RETRY_WHEN_IN_DOCUMENTATION` - array in `spec/lib/gitlab/ci/config/entry/retry_spec.rb`. - The test there makes sure that all documented - values are valid as a configuration option and therefore should always - stay in sync with this documentation. ---> - -- `always`: Retry on any failure (default). -- `unknown_failure`: Retry when the failure reason is unknown. -- `script_failure`: Retry when the script failed. -- `api_failure`: Retry on API failure. -- `stuck_or_timeout_failure`: Retry when the job got stuck or timed out. -- `runner_system_failure`: Retry if there is a runner system failure (for example, job setup failed). -- `runner_unsupported`: Retry if the runner is unsupported. -- `stale_schedule`: Retry if a delayed job could not be executed. -- `job_execution_timeout`: Retry if the script exceeded the maximum execution time set for the job. -- `archived_failure`: Retry if the job is archived and can't be run. -- `unmet_prerequisites`: Retry if the job failed to complete prerequisite tasks. -- `scheduler_failure`: Retry if the scheduler failed to assign the job to a runner. -- `data_integrity_failure`: Retry if there is a structural integrity problem detected. +In this example, the `deploy` job runs only when the Kubernetes service is active +in the project. -**Example of `retry:when`** (single failure type): +### `pages` -```yaml -test: - script: rspec - retry: - max: 2 - when: runner_system_failure -``` +Use `pages` to define a [GitLab Pages](../../user/project/pages/index.md) job that +uploads static content to GitLab. The content is then published as a website. -If there is a failure other than a runner system failure, the job is not retried. +**Keyword type**: Job name. -**Example of `retry:when`** (array of failure types): +**Example of `pages`**: ```yaml -test: - script: rspec - retry: - max: 2 - when: - - runner_system_failure - - stuck_or_timeout_failure +pages: + stage: deploy + script: + - mkdir .public + - cp -r * .public + - mv .public public + artifacts: + paths: + - public + rules: + - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH ``` -**Related topics**: - -You can specify the number of [retry attempts for certain stages of job execution](../runners/configure_runners.md#job-stages-attempts) -using variables. - -### `timeout` - -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/14887) in GitLab 12.3. - -Use `timeout` to configure a timeout for a specific job. If the job runs for longer -than the timeout, the job fails. - -The job-level timeout can be longer than the [project-level timeout](../pipelines/settings.md#set-a-limit-for-how-long-jobs-can-run). -but can't be longer than the [runner's timeout](../runners/configure_runners.md#set-maximum-job-timeout-for-a-runner). - -**Keyword type**: Job keyword. You can use it only as part of a job or in the -[`default:` section](#default). - -**Possible inputs**: A period of time written in natural language. For example, these are all equivalent: - -- `3600 seconds` -- `60 minutes` -- `one hour` +This example moves all files from the root of the project to the `public/` directory. +The `.public` workaround is so `cp` does not also copy `public/` to itself in an infinite loop. -**Example of `timeout`**: +**Additional details**: -```yaml -build: - script: build.sh - timeout: 3 hours 30 minutes +You must: -test: - script: rspec - timeout: 3h 30m -``` +- Place any static content in a `public/` directory. +- Define [`artifacts`](#artifacts) with a path to the `public/` directory. ### `parallel` @@ -3401,7 +2574,9 @@ Parallel jobs are named sequentially from `job_name 1/N` to `job_name N/N`. **Keyword type**: Job keyword. You can use it only as part of a job. -**Possible inputs**: A numeric value from `2` to `50`. +**Possible inputs**: + +- A numeric value from `2` to `50`. **Example of `parallel`**: @@ -3434,7 +2609,9 @@ Multiple runners must exist, or a single runner must be configured to run multip **Keyword type**: Job keyword. You can use it only as part of a job. -**Possible inputs**: A numeric value from `2` to `50`. +**Possible inputs**: + +- A numeric value from `2` to `50`. **Example of `parallel:matrix`**: @@ -3477,175 +2654,6 @@ deploystacks: [vultr, processing] - [Run a one-dimensional matrix of parallel jobs](../jobs/job_control.md#run-a-one-dimensional-matrix-of-parallel-jobs). - [Run a matrix of triggered parallel jobs](../jobs/job_control.md#run-a-matrix-of-parallel-trigger-jobs). -### `trigger` - -> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/8997) in GitLab Premium 11.8. -> - [Moved](https://gitlab.com/gitlab-org/gitlab/-/issues/199224) to GitLab Free in 12.8. - -Use `trigger` to start a downstream pipeline that is either: - -- [A multi-project pipeline](../pipelines/multi_project_pipelines.md). -- [A child pipeline](../pipelines/parent_child_pipelines.md). - -**Keyword type**: Job keyword. You can use it only as part of a job. - -**Possible inputs**: - -- For multi-project pipelines, path to the downstream project. -- For child pipelines, path to the child pipeline CI/CD configuration file. - -**Example of `trigger` for multi-project pipeline**: - -```yaml -rspec: - stage: test - script: bundle exec rspec - -staging: - stage: deploy - trigger: my/deployment -``` - -**Example of `trigger` for child pipelines**: - -```yaml -trigger_job: - trigger: - include: path/to/child-pipeline.yml -``` - -**Additional details**: - -- Jobs with `trigger` can only use a [limited set of keywords](../pipelines/multi_project_pipelines.md#define-multi-project-pipelines-in-your-gitlab-ciyml-file). - For example, you can't run commands with [`script`](#script), [`before_script`](#before_script), - or [`after_script`](#after_script). -- In [GitLab 13.5 and later](https://gitlab.com/gitlab-org/gitlab/-/issues/201938), you - can use [`when:manual`](#when) in the same job as `trigger`. In GitLab 13.4 and - earlier, using them together causes the error `jobs:#{job-name} when should be on_success, on_failure or always`. -- In [GitLab 13.2 and later](https://gitlab.com/gitlab-org/gitlab/-/issues/197140/), you can - view which job triggered a downstream pipeline in the [pipeline graph](../pipelines/index.md#visualize-pipelines). - -**Related topics**: - -- [Multi-project pipeline configuration examples](../pipelines/multi_project_pipelines.md#define-multi-project-pipelines-in-your-gitlab-ciyml-file). -- [Child pipeline configuration examples](../pipelines/parent_child_pipelines.md#examples). -- To force a rebuild of a specific branch, tag, or commit, you can - [use an API call with a trigger token](../triggers/index.md). - The trigger token is different than the `trigger` keyword. - -#### `trigger:strategy` - -Use `trigger:strategy` to force the `trigger` job to wait for the downstream pipeline to complete -before it is marked as **success**. - -This behavior is different than the default, which is for the `trigger` job to be marked as -**success** as soon as the downstream pipeline is created. - -This setting makes your pipeline execution linear rather than parallel. - -**Example of `trigger:strategy`**: - -```yaml -trigger_job: - trigger: - include: path/to/child-pipeline.yml - strategy: depend -``` - -In this example, jobs from subsequent stages wait for the triggered pipeline to -successfully complete before starting. - -### `interruptible` - -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/32022) in GitLab 12.3. - -Use `interruptible` if a job should be canceled when a newer pipeline starts before the job completes. - -This keyword is used with the [automatic cancellation of redundant pipelines](../pipelines/settings.md#auto-cancel-redundant-pipelines) -feature. When enabled, a running job with `interruptible: true` can be cancelled when -a new pipeline starts on the same branch. - -You can't cancel subsequent jobs after a job with `interruptible: false` starts. - -**Keyword type**: Job keyword. You can use it only as part of a job or in the -[`default:` section](#default). - -**Possible inputs**: `true` or `false` (default). - -**Example of `interruptible`**: - -```yaml -stages: - - stage1 - - stage2 - - stage3 - -step-1: - stage: stage1 - script: - - echo "Can be canceled." - interruptible: true - -step-2: - stage: stage2 - script: - - echo "Can not be canceled." - -step-3: - stage: stage3 - script: - - echo "Because step-2 can not be canceled, this step can never be canceled, even though it's set as interruptible." - interruptible: true -``` - -In this example, a new pipeline causes a running pipeline to be: - -- Canceled, if only `step-1` is running or pending. -- Not canceled, after `step-2` starts. - -**Additional details**: - -- Only set `interruptible: true` if the job can be safely canceled after it has started, - like a build job. Deployment jobs usually shouldn't be cancelled, to prevent partial deployments. -- To completely cancel a running pipeline, all jobs must have `interruptible: true`, - or `interruptible: false` jobs must not have started. - -### `resource_group` - -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/15536) in GitLab 12.7. - -Use `resource_group` to create a [resource group](../resource_groups/index.md) that -ensures a job is mutually exclusive across different pipelines for the same project. - -For example, if multiple jobs that belong to the same resource group are queued simultaneously, -only one of the jobs starts. The other jobs wait until the `resource_group` is free. - -Resource groups behave similar to semaphores in other programming languages. - -You can define multiple resource groups per environment. For example, -when deploying to physical devices, you might have multiple physical devices. Each device -can be deployed to, but only one deployment can occur per device at any given time. - -**Keyword type**: Job keyword. You can use it only as part of a job. - -**Possible inputs**: Only letters, digits, `-`, `_`, `/`, `$`, `{`, `}`, `.`, and spaces. -It can't start or end with `/`. - -**Example of `resource_group`**: - -```yaml -deploy-to-production: - script: deploy - resource_group: production -``` - -In this example, two `deploy-to-production` jobs in two separate pipelines can never run at the same time. As a result, -you can ensure that concurrent deployments never happen to the production environment. - -**Related topics**: - -- [Pipeline-level concurrency control with cross-project/parent-child pipelines](../resource_groups/index.md#pipeline-level-concurrency-control-with-cross-projectparent-child-pipelines). - ### `release` > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/19298) in GitLab 13.2. @@ -3660,7 +2668,7 @@ you can use this image from the GitLab Container Registry: `registry.gitlab.com/ **Keyword type**: Job keyword. You can use it only as part of a job. -**Possible inputs**: The `release:` subkeys: +**Possible inputs**: The `release` subkeys: - [`tag_name`](#releasetag_name) - [`name`](#releasename) (optional) @@ -3679,7 +2687,7 @@ you can use this image from the GitLab Container Registry: `registry.gitlab.com/ rules: - if: $CI_COMMIT_TAG # Run this job when a tag is created manually script: - - echo 'Running the release job.' + - echo "Running the release job." release: name: 'Release $CI_COMMIT_TAG' description: 'Release created using the release-cli.' @@ -3697,7 +2705,7 @@ This example creates a release: ```yaml script: - - echo 'release job' + - echo "release job" ``` An [issue](https://gitlab.com/gitlab-org/gitlab/-/issues/223856) exists to remove this requirement. @@ -3723,7 +2731,9 @@ New tags use the SHA associated with the pipeline. **Keyword type**: Job keyword. You can use it only as part of a job. -**Possible inputs**: A tag name. Can use [CI/CD variables](../variables/index.md). +**Possible inputs**: + +- A tag name. Can use [CI/CD variables](../variables/index.md). **Example of `release:tag_name`**: @@ -3735,7 +2745,7 @@ To create a release when a new tag is added to the project: ```yaml job: - script: echo 'Running the release job for the new tag.' + script: echo "Running the release job for the new tag." release: tag_name: $CI_COMMIT_TAG description: 'Release description' @@ -3748,7 +2758,7 @@ should **not** configure the job to run only for new tags. A semantic versioning ```yaml job: - script: echo 'Running the release job and creating a new tag.' + script: echo "Running the release job and creating a new tag." release: tag_name: ${MAJOR}_${MINOR}_${REVISION} description: 'Release description' @@ -3762,7 +2772,9 @@ The release name. If omitted, it is populated with the value of `release: tag_na **Keyword type**: Job keyword. You can use it only as part of a job. -**Possible inputs**: A text string. +**Possible inputs**: + +- A text string. **Example of `release:name`**: @@ -3849,6 +2861,405 @@ assets: link_type: 'other' # optional ``` +### `resource_group` + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/15536) in GitLab 12.7. + +Use `resource_group` to create a [resource group](../resource_groups/index.md) that +ensures a job is mutually exclusive across different pipelines for the same project. + +For example, if multiple jobs that belong to the same resource group are queued simultaneously, +only one of the jobs starts. The other jobs wait until the `resource_group` is free. + +Resource groups behave similar to semaphores in other programming languages. + +You can define multiple resource groups per environment. For example, +when deploying to physical devices, you might have multiple physical devices. Each device +can be deployed to, but only one deployment can occur per device at any given time. + +**Keyword type**: Job keyword. You can use it only as part of a job. + +**Possible inputs**: + +- Only letters, digits, `-`, `_`, `/`, `$`, `{`, `}`, `.`, and spaces. + It can't start or end with `/`. + +**Example of `resource_group`**: + +```yaml +deploy-to-production: + script: deploy + resource_group: production +``` + +In this example, two `deploy-to-production` jobs in two separate pipelines can never run at the same time. As a result, +you can ensure that concurrent deployments never happen to the production environment. + +**Related topics**: + +- [Pipeline-level concurrency control with cross-project/parent-child pipelines](../resource_groups/index.md#pipeline-level-concurrency-control-with-cross-projectparent-child-pipelines). + +### `retry` + +Use `retry` to configure how many times a job is retried if it fails. +If not defined, defaults to `0` and jobs do not retry. + +When a job fails, the job is processed up to two more times, until it succeeds or +reaches the maximum number of retries. + +By default, all failure types cause the job to be retried. Use [`retry:when`](#retrywhen) +to select which failures to retry on. + +**Keyword type**: Job keyword. You can use it only as part of a job or in the +[`default` section](#default). + +**Possible inputs**: + +- `0` (default), `1`, or `2`. + +**Example of `retry`**: + +```yaml +test: + script: rspec + retry: 2 +``` + +#### `retry:when` + +Use `retry:when` with `retry:max` to retry jobs for only specific failure cases. +`retry:max` is the maximum number of retries, like [`retry`](#retry), and can be +`0`, `1`, or `2`. + +**Keyword type**: Job keyword. You can use it only as part of a job or in the +[`default` section](#default). + +**Possible inputs**: + +- A single failure type, or an array of one or more failure types: + +<!-- + If you change any of the values below, make sure to update the `RETRY_WHEN_IN_DOCUMENTATION` + array in `spec/lib/gitlab/ci/config/entry/retry_spec.rb`. + The test there makes sure that all documented + values are valid as a configuration option and therefore should always + stay in sync with this documentation. +--> + +- `always`: Retry on any failure (default). +- `unknown_failure`: Retry when the failure reason is unknown. +- `script_failure`: Retry when the script failed. +- `api_failure`: Retry on API failure. +- `stuck_or_timeout_failure`: Retry when the job got stuck or timed out. +- `runner_system_failure`: Retry if there is a runner system failure (for example, job setup failed). +- `runner_unsupported`: Retry if the runner is unsupported. +- `stale_schedule`: Retry if a delayed job could not be executed. +- `job_execution_timeout`: Retry if the script exceeded the maximum execution time set for the job. +- `archived_failure`: Retry if the job is archived and can't be run. +- `unmet_prerequisites`: Retry if the job failed to complete prerequisite tasks. +- `scheduler_failure`: Retry if the scheduler failed to assign the job to a runner. +- `data_integrity_failure`: Retry if there is a structural integrity problem detected. + +**Example of `retry:when`** (single failure type): + +```yaml +test: + script: rspec + retry: + max: 2 + when: runner_system_failure +``` + +If there is a failure other than a runner system failure, the job is not retried. + +**Example of `retry:when`** (array of failure types): + +```yaml +test: + script: rspec + retry: + max: 2 + when: + - runner_system_failure + - stuck_or_timeout_failure +``` + +**Related topics**: + +You can specify the number of [retry attempts for certain stages of job execution](../runners/configure_runners.md#job-stages-attempts) +using variables. + +### `rules` + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/27863) in GitLab 12.3. + +Use `rules` to include or exclude jobs in pipelines. + +Rules are evaluated when the pipeline is created, and evaluated *in order* +until the first match. When a match is found, the job is either included or excluded from the pipeline, +depending on the configuration. + +You cannot use dotenv variables created in job scripts in rules, because rules are evaluated before any jobs run. + +`rules` replaces [`only/except`](#only--except) and they can't be used together +in the same job. If you configure one job to use both keywords, the GitLab returns +a `key may not be used with rules` error. + +`rules` accepts an array of rules defined with: + +- `if` +- `changes` +- `exists` +- `allow_failure` +- `variables` +- `when` + +You can combine multiple keywords together for [complex rules](../jobs/job_control.md#complex-rules). + +The job is added to the pipeline: + +- If an `if`, `changes`, or `exists` rule matches and also has `when: on_success` (default), + `when: delayed`, or `when: always`. +- If a rule is reached that is only `when: on_success`, `when: delayed`, or `when: always`. + +The job is not added to the pipeline: + +- If no rules match. +- If a rule matches and has `when: never`. + +You can use [`!reference` tags](yaml_optimization.md#reference-tags) to [reuse `rules` configuration](../jobs/job_control.md#reuse-rules-in-different-jobs) +in different jobs. + +#### `rules:if` + +Use `rules:if` clauses to specify when to add a job to a pipeline: + +- If an `if` statement is true, add the job to the pipeline. +- If an `if` statement is true, but it's combined with `when: never`, do not add the job to the pipeline. +- If no `if` statements are true, do not add the job to the pipeline. + +`if` clauses are evaluated based on the values of [predefined CI/CD variables](../variables/predefined_variables.md) +or [custom CI/CD variables](../variables/index.md#custom-cicd-variables). + +**Keyword type**: Job-specific and pipeline-specific. You can use it as part of a job +to configure the job behavior, or with [`workflow`](#workflow) to configure the pipeline behavior. + +**Possible inputs**: + +- A [CI/CD variable expression](../jobs/job_control.md#cicd-variable-expressions). + +**Example of `rules:if`**: + +```yaml +job: + script: echo "Hello, Rules!" + rules: + - if: '$CI_MERGE_REQUEST_SOURCE_BRANCH_NAME =~ /^feature/ && $CI_MERGE_REQUEST_TARGET_BRANCH_NAME != $CI_DEFAULT_BRANCH' + when: never + - if: '$CI_MERGE_REQUEST_SOURCE_BRANCH_NAME =~ /^feature/' + when: manual + allow_failure: true + - if: '$CI_MERGE_REQUEST_SOURCE_BRANCH_NAME' +``` + +**Additional details**: + +- If a rule matches and has no `when` defined, the rule uses the `when` + defined for the job, which defaults to `on_success` if not defined. +- You can define `when` once per rule, or once at the job-level, which applies to + all rules. You can't mix `when` at the job-level with `when` in rules. +- Unlike variables in [`script`](../variables/index.md#use-cicd-variables-in-job-scripts) + sections, variables in rules expressions are always formatted as `$VARIABLE`. + - You can use `rules:if` with `include` to [conditionally include other configuration files](includes.md#use-rules-with-include). + +**Related topics**: + +- [Common `if` expressions for `rules`](../jobs/job_control.md#common-if-clauses-for-rules). +- [Avoid duplicate pipelines](../jobs/job_control.md#avoid-duplicate-pipelines). + +#### `rules:changes` + +Use `rules:changes` to specify when to add a job to a pipeline by checking for changes +to specific files. + +WARNING: +You should use `rules: changes` only with **branch pipelines** or **merge request pipelines**. +You can use `rules: changes` with other pipeline types, but `rules: changes` always +evaluates to true when there is no Git `push` event. Tag pipelines, scheduled pipelines, +and so on do **not** have a Git `push` event associated with them. A `rules: changes` job +is **always** added to those pipelines if there is no `if` that limits the job to +branch or merge request pipelines. + +**Keyword type**: Job keyword. You can use it only as part of a job. + +**Possible inputs**: + +- An array of file paths. In GitLab 13.6 and later, [file paths can include variables](../jobs/job_control.md#variables-in-ruleschanges). + +**Example of `rules:changes`**: + +```yaml +docker build: + script: docker build -t my-image:$CI_COMMIT_REF_SLUG . + rules: + - if: '$CI_PIPELINE_SOURCE == "merge_request_event"' + changes: + - Dockerfile + when: manual + allow_failure: true +``` + +- If the pipeline is a merge request pipeline, check `Dockerfile` for changes. +- If `Dockerfile` has changed, add the job to the pipeline as a manual job, and the pipeline + continues running even if the job is not triggered (`allow_failure: true`). +- If `Dockerfile` has not changed, do not add job to any pipeline (same as `when: never`). + +**Additional details**: + +- `rules: changes` works the same way as [`only: changes` and `except: changes`](#onlychanges--exceptchanges). +- You can use `when: never` to implement a rule similar to [`except:changes`](#onlychanges--exceptchanges). +- `changes` resolves to `true` if any of the matching files are changed (an `OR` operation). + +#### `rules:exists` + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/24021) in GitLab 12.4. + +Use `exists` to run a job when certain files exist in the repository. + +**Keyword type**: Job keyword. You can use it only as part of a job. + +**Possible inputs**: + +- An array of file paths. Paths are relative to the project directory (`$CI_PROJECT_DIR`) + and can't directly link outside it. File paths can use glob patterns. + +**Example of `rules:exists`**: + +```yaml +job: + script: docker build -t my-image:$CI_COMMIT_REF_SLUG . + rules: + - exists: + - Dockerfile +``` + +`job` runs if a `Dockerfile` exists anywhere in the repository. + +**Additional details**: + +- Glob patterns are interpreted with Ruby [`File.fnmatch`](https://docs.ruby-lang.org/en/2.7.0/File.html#method-c-fnmatch) + with the flags `File::FNM_PATHNAME | File::FNM_DOTMATCH | File::FNM_EXTGLOB`. +- For performance reasons, GitLab matches a maximum of 10,000 `exists` patterns or + file paths. After the 10,000th check, rules with patterned globs always match. + In other words, the `exists` rule always assumes a match in projects with more + than 10,000 files. +- `exists` resolves to `true` if any of the listed files are found (an `OR` operation). + +#### `rules:allow_failure` + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/30235) in GitLab 12.8. + +Use [`allow_failure: true`](#allow_failure) in `rules` to allow a job to fail +without stopping the pipeline. + +You can also use `allow_failure: true` with a manual job. The pipeline continues +running without waiting for the result of the manual job. `allow_failure: false` +combined with `when: manual` in rules causes the pipeline to wait for the manual +job to run before continuing. + +**Keyword type**: Job keyword. You can use it only as part of a job. + +**Possible inputs**: + +- `true` or `false`. Defaults to `false` if not defined. + +**Example of `rules:allow_failure`**: + +```yaml +job: + script: echo "Hello, Rules!" + rules: + - if: '$CI_MERGE_REQUEST_TARGET_BRANCH_NAME == $CI_DEFAULT_BRANCH' + when: manual + allow_failure: true +``` + +If the rule matches, then the job is a manual job with `allow_failure: true`. + +**Additional details**: + +- The rule-level `rules:allow_failure` overrides the job-level [`allow_failure`](#allow_failure), + and only applies when the specific rule triggers the job. + +#### `rules:variables` + +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/209864) in GitLab 13.7. +> - [Feature flag removed](https://gitlab.com/gitlab-org/gitlab/-/issues/289803) in GitLab 13.10. + +Use [`variables`](#variables) in `rules` to define variables for specific conditions. + +**Keyword type**: Job-specific. You can use it only as part of a job. + +**Possible inputs**: + +- A hash of variables in the format `VARIABLE-NAME: value`. + +**Example of `rules:variables`**: + +```yaml +job: + variables: + DEPLOY_VARIABLE: "default-deploy" + rules: + - if: $CI_COMMIT_REF_NAME == $CI_DEFAULT_BRANCH + variables: # Override DEPLOY_VARIABLE defined + DEPLOY_VARIABLE: "deploy-production" # at the job level. + - if: $CI_COMMIT_REF_NAME =~ /feature/ + variables: + IS_A_FEATURE: "true" # Define a new variable. + script: + - echo "Run script with $DEPLOY_VARIABLE as an argument" + - echo "Run another script if $IS_A_FEATURE exists" +``` + +### `script` + +Use `script` to specify commands for the runner to execute. + +All jobs except [trigger jobs](#trigger) require a `script` keyword. + +**Keyword type**: Job keyword. You can use it only as part of a job. + +**Possible inputs**: An array including: + +- Single line commands. +- Long commands [split over multiple lines](script.md#split-long-commands). +- [YAML anchors](yaml_optimization.md#yaml-anchors-for-scripts). + +**Example of `script`**: + +```yaml +job1: + script: "bundle exec rspec" + +job2: + script: + - uname -a + - bundle exec rspec +``` + +**Additional details**: + +- When you use [these special characters in `script`](script.md#use-special-characters-with-script), you must use single quotes (`'`) or double quotes (`"`) . + +**Related topics**: + +- You can [ignore non-zero exit codes](script.md#ignore-non-zero-exit-codes). +- [Use color codes with `script`](script.md#add-color-codes-to-script-output) + to make job logs easier to review. +- [Create custom collapsible sections](../jobs/index.md#custom-collapsible-sections) + to simplify job log output. + ### `secrets` **(PREMIUM)** > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/33014) in GitLab 13.4. @@ -3926,7 +3337,9 @@ the secret value directly in the variable. **Keyword type**: Job keyword. You can use it only as part of a job. -**Possible inputs**: `true` (default) or `false`. +**Possible inputs**: + +- `true` (default) or `false`. **Example of `secrets:file`**: @@ -3943,118 +3356,323 @@ job: - The `file` keyword is a setting for the CI/CD variable and must be nested under the CI/CD variable name, not in the `vault` section. -### `pages` +### `services` -Use `pages` to define a [GitLab Pages](../../user/project/pages/index.md) job that -uploads static content to GitLab. The content is then published as a website. +Use `services` to specify an additional Docker image to run scripts in. The [`services` image](../services/index.md) is linked +to the image specified in the [`image`](#image) keyword. -**Keyword type**: Job name. +**Keyword type**: Job keyword. You can use it only as part of a job or in the +[`default` section](#default). -**Example of `pages`**: +**Possible inputs**: The name of the services image, including the registry path if needed, in one of these formats: + +- `<image-name>` (Same as using `<image-name>` with the `latest` tag) +- `<image-name>:<tag>` +- `<image-name>@<digest>` + +**Example of `services`**: ```yaml -pages: - stage: deploy +default: + image: + name: ruby:2.6 + entrypoint: ["/bin/bash"] + + services: + - name: my-postgres:11.7 + alias: db-postgres + entrypoint: ["/usr/local/bin/db-postgres"] + command: ["start"] + + before_script: + - bundle install + +test: script: - - mkdir .public - - cp -r * .public - - mv .public public - artifacts: - paths: - - public - rules: - - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH + - bundle exec rake spec ``` -This example moves all files from the root of the project to the `public/` directory. -The `.public` workaround is so `cp` does not also copy `public/` to itself in an infinite loop. +In this example, the job launches a Ruby container. Then, from that container, the job launches +another container that's running PostgreSQL. Then the job then runs scripts +in that container. + +**Related topics**: + +- [Available settings for `services`](../services/index.md#available-settings-for-services). +- [Define `services` in the `.gitlab-ci.yml` file](../services/index.md#define-services-in-the-gitlab-ciyml-file). +- [Run your CI/CD jobs in Docker containers](../docker/using_docker_images.md). +- [Use Docker to build Docker images](../docker/using_docker_build.md). + +### `stage` + +Use `stage` to define which [stage](#stages) a job runs in. Jobs in the same +`stage` can execute in parallel (see **Additional details**). + +If `stage` is not defined, the job uses the `test` stage by default. + +**Keyword type**: Job keyword. You can use it only as part of a job. + +**Possible inputs**: An array including any number of stage names. Stage names can be: + +- The [default stages](#stages). +- User-defined stages. + +**Example of `stage`**: + +```yaml +stages: + - build + - test + - deploy + +job1: + stage: build + script: + - echo "This job compiles code." + +job2: + stage: test + script: + - echo "This job tests the compiled code. It runs when the build stage completes." + +job3: + script: + - echo "This job also runs in the test stage". + +job4: + stage: deploy + script: + - echo "This job deploys the code. It runs when the test stage completes." +``` **Additional details**: -You must: +- Jobs can run in parallel if they run on different runners. +- If you have only one runner, jobs can run in parallel if the runner's + [`concurrent` setting](https://docs.gitlab.com/runner/configuration/advanced-configuration.html#the-global-section) + is greater than `1`. -- Place any static content in a `public/` directory. -- Define [`artifacts`](#artifacts) with a path to the `public/` directory. +#### `stage: .pre` -### `inherit` +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/31441) in GitLab 12.4. -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/207484) in GitLab 12.9. +Use the `.pre` stage to make a job run at the start of a pipeline. `.pre` is +always the first stage in a pipeline. User-defined stages execute after `.pre`. +You do not have to define `.pre` in [`stages`](#stages). -Use `inherit:` to [control inheritance of globally-defined defaults and variables](../jobs/index.md#control-the-inheritance-of-default-keywords-and-global-variables). +You must have a job in at least one stage other than `.pre` or `.post`. -#### `inherit:default` +**Keyword type**: You can only use it with a job's `stage` keyword. -Use `inherit:default` to control the inheritance of [default keywords](#default). +**Example of `stage: .pre`**: -**Keyword type**: Job keyword. You can use it only as part of a job. +```yaml +stages: + - build + - test -**Possible inputs**: +job1: + stage: build + script: + - echo "This job runs in the build stage." -- `true` (default) or `false` to enable or disable the inheritance of all default keywords. -- A list of specific default keywords to inherit. +first-job: + stage: .pre + script: + - echo "This job runs in the .pre stage, before all other stages." + +job2: + stage: test + script: + - echo "This job runs in the test stage." +``` + +#### `stage: .post` + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/31441) in GitLab 12.4. -**Example of `inherit:default`:** +Use the `.post` stage to make a job run at the end of a pipeline. `.post` +is always the last stage in a pipeline. User-defined stages execute before `.post`. +You do not have to define `.post` in [`stages`](#stages). + +You must have a job in at least one stage other than `.pre` or `.post`. + +**Keyword type**: You can only use it with a job's `stage` keyword. + +**Example of `stage: .post`**: ```yaml -default: - retry: 2 - image: ruby:3.0 - interruptible: true +stages: + - build + - test job1: - script: echo "This job does not inherit any default keywords." - inherit: - default: false + stage: build + script: + - echo "This job runs in the build stage." + +last-job: + stage: .post + script: + - echo "This job runs in the .post stage, after all other stages." job2: - script: echo "This job inherits only the two listed default keywords. It does not inherit 'interruptible'." - inherit: - default: - - retry - - image + stage: test + script: + - echo "This job runs in the test stage." ``` -**Additional details:** +### `tags` -- You can also list default keywords to inherit on one line: `default: [keyword1, keyword2]` +> - A limit of 50 tags per job [enabled on GitLab.com](https://gitlab.com/gitlab-org/gitlab/-/issues/338929) in GitLab 14.3. +> - A limit of 50 tags per job [enabled on self-managed](https://gitlab.com/gitlab-org/gitlab/-/issues/339855) in GitLab 14.3. -#### `inherit:variables` +Use `tags` to select a specific runner from the list of all runners that are +available for the project. -Use `inherit:variables` to control the inheritance of [global variables](#variables) keywords. +When you register a runner, you can specify the runner's tags, for +example `ruby`, `postgres`, or `development`. To pick up and run a job, a runner must +be assigned every tag listed in the job. + +**Keyword type**: Job keyword. You can use it only as part of a job or in the +[`default` section](#default). + +**Possible inputs**: + +- An array of tag names. +- [CI/CD variables](../runners/configure_runners.md#use-cicd-variables-in-tags) in GitLab 14.1 and later. + +**Example of `tags`**: + +```yaml +job: + tags: + - ruby + - postgres +``` + +In this example, only runners with *both* the `ruby` and `postgres` tags can run the job. + +**Additional details**: + +- In [GitLab 14.3](https://gitlab.com/gitlab-org/gitlab/-/issues/338479) and later, + the number of tags must be less than `50`. + +**Related topics**: + +- [Use tags to control which jobs a runner can run](../runners/configure_runners.md#use-tags-to-control-which-jobs-a-runner-can-run). + +### `timeout` + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/14887) in GitLab 12.3. + +Use `timeout` to configure a timeout for a specific job. If the job runs for longer +than the timeout, the job fails. + +The job-level timeout can be longer than the [project-level timeout](../pipelines/settings.md#set-a-limit-for-how-long-jobs-can-run). +but can't be longer than the [runner's timeout](../runners/configure_runners.md#set-maximum-job-timeout-for-a-runner). + +**Keyword type**: Job keyword. You can use it only as part of a job or in the +[`default` section](#default). + +**Possible inputs**: A period of time written in natural language. For example, these are all equivalent: + +- `3600 seconds` +- `60 minutes` +- `one hour` + +**Example of `timeout`**: + +```yaml +build: + script: build.sh + timeout: 3 hours 30 minutes + +test: + script: rspec + timeout: 3h 30m +``` + +### `trigger` + +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/8997) in GitLab Premium 11.8. +> - [Moved](https://gitlab.com/gitlab-org/gitlab/-/issues/199224) to GitLab Free in 12.8. + +Use `trigger` to start a downstream pipeline that is either: + +- [A multi-project pipeline](../pipelines/multi_project_pipelines.md). +- [A child pipeline](../pipelines/parent_child_pipelines.md). **Keyword type**: Job keyword. You can use it only as part of a job. **Possible inputs**: -- `true` (default) or `false` to enable or disable the inheritance of all global variables. -- A list of specific variables to inherit. +- For multi-project pipelines, path to the downstream project. +- For child pipelines, path to the child pipeline CI/CD configuration file. -**Example of `inherit:variables`:** +**Example of `trigger` for multi-project pipeline**: ```yaml -variables: - VARIABLE1: "This is variable 1" - VARIABLE2: "This is variable 2" - VARIABLE3: "This is variable 3" +rspec: + stage: test + script: bundle exec rspec -job1: - script: echo "This job does not inherit any global variables." - inherit: - variables: false +staging: + stage: deploy + trigger: my/deployment +``` -job2: - script: echo "This job inherits only the two listed global variables. It does not inherit 'VARIABLE3'." - inherit: - variables: - - VARIABLE1 - - VARIABLE2 +**Example of `trigger` for child pipelines**: + +```yaml +trigger_job: + trigger: + include: path/to/child-pipeline.yml ``` -**Additional details:** +**Additional details**: -- You can also list global variables to inherit on one line: `variables: [VARIABLE1, VARIABLE2]` +- Jobs with `trigger` can only use a [limited set of keywords](../pipelines/multi_project_pipelines.md#define-multi-project-pipelines-in-your-gitlab-ciyml-file). + For example, you can't run commands with [`script`](#script), [`before_script`](#before_script), + or [`after_script`](#after_script). +- You [cannot use the API to start `when:manual` trigger jobs](https://gitlab.com/gitlab-org/gitlab/-/issues/284086). +- In [GitLab 13.5 and later](https://gitlab.com/gitlab-org/gitlab/-/issues/201938), you + can use [`when:manual`](#when) in the same job as `trigger`. In GitLab 13.4 and + earlier, using them together causes the error `jobs:#{job-name} when should be on_success, on_failure or always`. +- In [GitLab 13.2 and later](https://gitlab.com/gitlab-org/gitlab/-/issues/197140/), you can + view which job triggered a downstream pipeline in the [pipeline graph](../pipelines/index.md#visualize-pipelines). + +**Related topics**: + +- [Multi-project pipeline configuration examples](../pipelines/multi_project_pipelines.md#define-multi-project-pipelines-in-your-gitlab-ciyml-file). +- [Child pipeline configuration examples](../pipelines/parent_child_pipelines.md#examples). +- To run a pipeline for a specific branch, tag, or commit, you can use a [trigger token](../triggers/index.md) + to authenticate with the [pipeline triggers API](../../api/pipeline_triggers.md). + The trigger token is different than the `trigger` keyword. + +#### `trigger:strategy` + +Use `trigger:strategy` to force the `trigger` job to wait for the downstream pipeline to complete +before it is marked as **success**. + +This behavior is different than the default, which is for the `trigger` job to be marked as +**success** as soon as the downstream pipeline is created. + +This setting makes your pipeline execution linear rather than parallel. + +**Example of `trigger:strategy`**: + +```yaml +trigger_job: + trigger: + include: path/to/child-pipeline.yml + strategy: depend +``` + +In this example, jobs from subsequent stages wait for the triggered pipeline to +successfully complete before starting. -## `variables` +### `variables` [CI/CD variables](../variables/index.md) are configurable values that are passed to jobs. Use `variables` to create [custom variables](../variables/index.md#custom-cicd-variables). @@ -4071,10 +3689,11 @@ variable defined, the [job-level variable takes precedence](../variables/index.m **Possible inputs**: Variable name and value pairs: -- The name can use only numbers, letters, and underscores (`_`). +- The name can use only numbers, letters, and underscores (`_`). In some shells, + the first character must be a letter. - The value must be a string. -**Examples of `variables`:** +**Examples of `variables`**: ```yaml variables: @@ -4106,7 +3725,7 @@ deploy_review_job: automatically creates and makes available in the job. - You can [configure runner behavior with variables](../runners/configure_runners.md#configure-runner-behavior-with-variables). -### `variables:description` +#### `variables:description` > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/30101) in GitLab 13.7. @@ -4117,7 +3736,9 @@ Must be used with `value`, for the variable value. **Keyword type**: Global keyword. You cannot set job-level variables to be pre-filled when you run a pipeline manually. -**Possible inputs**: A string. +**Possible inputs**: + +- A string. **Example of `variables:description`**: @@ -4128,6 +3749,84 @@ variables: description: "The deployment target. Change this variable to 'canary' or 'production' if needed." ``` +### `when` + +Use `when` to configure the conditions for when jobs run. If not defined in a job, +the default value is `when: on_success`. + +**Keyword type**: Job keyword. You can use it only as part of a job. + +**Possible inputs**: + +- `on_success` (default): Run the job only when all jobs in earlier stages succeed + or have `allow_failure: true`. +- `manual`: Run the job only when [triggered manually](../jobs/job_control.md#create-a-job-that-must-be-run-manually). +- `always`: Run the job regardless of the status of jobs in earlier stages. +- `on_failure`: Run the job only when at least one job in an earlier stage fails. +- `delayed`: [Delay the execution of a job](../jobs/job_control.md#run-a-job-after-a-delay) + for a specified duration. +- `never`: Don't run the job. + +**Example of `when`**: + +```yaml +stages: + - build + - cleanup_build + - test + - deploy + - cleanup + +build_job: + stage: build + script: + - make build + +cleanup_build_job: + stage: cleanup_build + script: + - cleanup build when failed + when: on_failure + +test_job: + stage: test + script: + - make test + +deploy_job: + stage: deploy + script: + - make deploy + when: manual + +cleanup_job: + stage: cleanup + script: + - cleanup after jobs + when: always +``` + +In this example, the script: + +1. Executes `cleanup_build_job` only when `build_job` fails. +1. Always executes `cleanup_job` as the last step in pipeline regardless of + success or failure. +1. Executes `deploy_job` when you run it manually in the GitLab UI. + +**Additional details**: + +- In [GitLab 13.5 and later](https://gitlab.com/gitlab-org/gitlab/-/issues/201938), you + can use `when:manual` in the same job as [`trigger`](#trigger). In GitLab 13.4 and + earlier, using them together causes the error `jobs:#{job-name} when should be on_success, on_failure or always`. +- The default behavior of `allow_failure` changes to `true` with `when: manual`. + However, if you use `when: manual` with [`rules`](#rules), `allow_failure` defaults + to `false`. + +**Related topics**: + +- `when` can be used with [`rules`](#rules) for more dynamic job control. +- `when` can be used with [`workflow`](#workflow) to control when a pipeline can start. + ## Deprecated keywords The following keywords are deprecated. @@ -4150,7 +3849,7 @@ Defining `image`, `services`, `cache`, `before_script`, and `after_script` globally is deprecated. Support could be removed from a future release. -Use [`default:`](#default) instead. For example: +Use [`default`](#default) instead. For example: ```yaml default: diff --git a/doc/ci/yaml/script.md b/doc/ci/yaml/script.md index c1b283ff10f..fdec0947df5 100644 --- a/doc/ci/yaml/script.md +++ b/doc/ci/yaml/script.md @@ -13,7 +13,7 @@ You can use special syntax in [`script`](index.md#script) sections to: - [Create custom collapsible sections](../jobs/index.md#custom-collapsible-sections) to simplify job log output. -## Use special characters with `script:` +## Use special characters with `script` Sometimes, `script` commands must be wrapped in single or double quotes. For example, commands that contain a colon (`:`) must be wrapped in single quotes (`'`). @@ -101,7 +101,7 @@ WARNING: If multiple commands are combined into one command string, only the last command's failure or success is reported. [Failures from earlier commands are ignored due to a bug](https://gitlab.com/gitlab-org/gitlab-runner/-/issues/25394). -To work around this, run each command as a separate `script:` item, or add an `exit 1` +To work around this, run each command as a separate `script` item, or add an `exit 1` command to each command string. You can use the `|` (literal) YAML multiline block scalar indicator to write diff --git a/doc/ci/yaml/workflow.md b/doc/ci/yaml/workflow.md index 67ca1150553..332214638d8 100644 --- a/doc/ci/yaml/workflow.md +++ b/doc/ci/yaml/workflow.md @@ -4,7 +4,7 @@ group: Pipeline Authoring info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- -# GitLab CI/CD `workflow` keyword +# GitLab CI/CD `workflow` keyword **(FREE)** Use the [`workflow`](index.md#workflow) keyword to control when pipelines are created. |
