diff options
| author | GitLab Bot <gitlab-bot@gitlab.com> | 2022-08-18 08:17:02 +0000 |
|---|---|---|
| committer | GitLab Bot <gitlab-bot@gitlab.com> | 2022-08-18 08:17:02 +0000 |
| commit | b39512ed755239198a9c294b6a45e65c05900235 (patch) | |
| tree | d234a3efade1de67c46b9e5a38ce813627726aa7 /doc/ci | |
| parent | d31474cf3b17ece37939d20082b07f6657cc79a9 (diff) | |
| download | gitlab-ce-b39512ed755239198a9c294b6a45e65c05900235.tar.gz | |
Add latest changes from gitlab-org/gitlab@15-3-stable-eev15.3.0-rc42
Diffstat (limited to 'doc/ci')
37 files changed, 618 insertions, 158 deletions
diff --git a/doc/ci/cloud_services/aws/index.md b/doc/ci/cloud_services/aws/index.md index dc8dfd95899..2e1abf83a11 100644 --- a/doc/ci/cloud_services/aws/index.md +++ b/doc/ci/cloud_services/aws/index.md @@ -62,16 +62,14 @@ After you configure the OIDC and role, the GitLab CI/CD job can retrieve a tempo assume role: script: - > - STS=($(aws sts assume-role-with-web-identity + export $(printf "AWS_ACCESS_KEY_ID=%s AWS_SECRET_ACCESS_KEY=%s AWS_SESSION_TOKEN=%s" + $(aws sts assume-role-with-web-identity --role-arn ${ROLE_ARN} --role-session-name "GitLabRunner-${CI_PROJECT_ID}-${CI_PIPELINE_ID}" --web-identity-token $CI_JOB_JWT_V2 --duration-seconds 3600 --query 'Credentials.[AccessKeyId,SecretAccessKey,SessionToken]' --output text)) - - export AWS_ACCESS_KEY_ID="${STS[0]}" - - export AWS_SECRET_ACCESS_KEY="${STS[1]}" - - export AWS_SESSION_TOKEN="${STS[2]}" - aws sts get-caller-identity ``` diff --git a/doc/ci/cloud_services/google_cloud/index.md b/doc/ci/cloud_services/google_cloud/index.md index f4e4a2046ba..54265816868 100644 --- a/doc/ci/cloud_services/google_cloud/index.md +++ b/doc/ci/cloud_services/google_cloud/index.md @@ -67,7 +67,7 @@ inside the Workload Identity Pool created in the previous step, using the follow | `google.subject` | `assertion.sub` | | `attribute.X` | `assertion.X` | - You can also [build complex attributes](https://cloud.google.com/iam/help/workload-identity/attribute-mapping) + You can also [build complex attributes](https://cloud.google.com/iam/docs/workload-identity-federation#mapping) using Common Expression Language (CEL). You must map every attribute that you want to use for permission granting. For example, if you want to map permissions in the next step based on the user's email address, you must map `attribute.user_email` to `assertion.user_email`. diff --git a/doc/ci/docker/using_docker_build.md b/doc/ci/docker/using_docker_build.md index df0c7b69d46..ea4ad25637b 100644 --- a/doc/ci/docker/using_docker_build.md +++ b/doc/ci/docker/using_docker_build.md @@ -14,6 +14,9 @@ test it, and publish it to a container registry. To run Docker commands in your CI/CD jobs, you must configure GitLab Runner to support `docker` commands. +If you want to build Docker images without enabling privileged mode on the runner, +you can use a [Docker alternative](#docker-alternatives). + ## Enable Docker commands in your CI/CD jobs To enable Docker commands for your CI/CD jobs, you can use: @@ -22,9 +25,6 @@ To enable Docker commands for your CI/CD jobs, you can use: - [Docker-in-Docker](#use-docker-in-docker) - [Docker socket binding](#use-docker-socket-binding) -If you don't want to execute a runner in privileged mode, -but want to use `docker build`, you can also use [`kaniko`](using_kaniko.md) or [`buildah`](https://github.com/containers/buildah). - If you are using shared runners on GitLab.com, [learn more about how these runners are configured](../runners/index.md). @@ -133,7 +133,7 @@ To use Docker-in-Docker with TLS enabled: - This command registers a new runner to use the `docker:20.10.16` image. To start the build and service containers, it uses the `privileged` mode. - If you want to use [Docker-in-Docker](https://www.docker.com/blog/docker-can-now-run-within-docker/), + If you want to use Docker-in-Docker, you must always use `privileged = true` in your Docker containers. - This command mounts `/certs/client` for the service and build container, which is needed for the Docker client to use the @@ -345,8 +345,8 @@ not without its own challenges: root file system, you can use the job's working directory as a mount point for child containers. For example, if you have files you want to share with a child container, you might create a subdirectory under `/builds/$CI_PROJECT_PATH` - and use it as your mount point. For a more detailed explanation, view [issue - #41227](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/41227). + and use it as your mount point. For a more detailed explanation, view + [issue #41227](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/41227). ```yaml variables: @@ -406,8 +406,8 @@ sudo gitlab-runner register -n \ ##### Enable registry mirror for `docker:dind` service When the Docker daemon starts inside of the service container, it uses -the default configuration. You may want to configure a [registry -mirror](https://docs.docker.com/registry/recipes/mirror/) for +the default configuration. You may want to configure a +[registry mirror](https://docs.docker.com/registry/recipes/mirror/) for performance improvements and to ensure you don't reach Docker Hub rate limits. ###### The service in the `.gitlab-ci.yml` file @@ -831,6 +831,44 @@ If you're running multiple runners, you have to modify all configuration files. Read more about the [runner configuration](https://docs.gitlab.com/runner/configuration/) and [using the OverlayFS storage driver](https://docs.docker.com/engine/userguide/storagedriver/overlayfs-driver/). +## Docker alternatives + +To build Docker images without enabling privileged mode on the runner, you can +use one of these alternatives: + +- [`kaniko`](using_kaniko.md). +- [`buildah`](https://github.com/containers/buildah). + +For example, with `buildah`: + +```yaml +# Some details from https://major.io/2019/05/24/build-containers-in-gitlab-ci-with-buildah/ + +build: + stage: build + image: quay.io/buildah/stable + variables: + # Use vfs with buildah. Docker offers overlayfs as a default, but buildah + # cannot stack overlayfs on top of another overlayfs filesystem. + STORAGE_DRIVER: vfs + # Write all image metadata in the docker format, not the standard OCI format. + # Newer versions of docker can handle the OCI format, but older versions, like + # the one shipped with Fedora 30, cannot handle the format. + BUILDAH_FORMAT: docker + # You may need this workaround for some errors: https://stackoverflow.com/a/70438141/1233435 + BUILDAH_ISOLATION: chroot + FQ_IMAGE_NAME: "${CI_REGISTRY_IMAGE}/test" + before_script: + # Log in to the GitLab container registry + - export REGISTRY_AUTH_FILE=${HOME}/auth.json + - echo "$CI_REGISTRY_PASSWORD" | buildah login -u "$CI_REGISTRY_USER" --password-stdin $CI_REGISTRY + script: + - buildah images + - buildah build -t $FQ_IMAGE_NAME + - buildah images + - buildah push $FQ_IMAGE_NAME +``` + ## Use the GitLab Container Registry After you've built a Docker image, you can push it up to the built-in diff --git a/doc/ci/docker/using_kaniko.md b/doc/ci/docker/using_kaniko.md index c2b18dc6467..712fd7b45d6 100644 --- a/doc/ci/docker/using_kaniko.md +++ b/doc/ci/docker/using_kaniko.md @@ -64,7 +64,7 @@ build: entrypoint: [""] script: - mkdir -p /kaniko/.docker - - echo "{\"auths\":{\"${CI_REGISTRY}\":{\"auth\":\"$(printf "%s:%s" "${CI_REGISTRY_USER}" "${CI_REGISTRY_PASSWORD}" | base64 | tr -d '\n')\"}}}" > /kaniko/.docker/config.json + - echo "{\"auths\":{\"${CI_REGISTRY}\":{\"auth\":\"$(printf "%s:%s" "${CI_REGISTRY_USER}" "${CI_REGISTRY_PASSWORD}" | base64 -w 0)\"}}}" > /kaniko/.docker/config.json - >- /kaniko/executor --context "${CI_PROJECT_DIR}" @@ -119,6 +119,13 @@ build: - if: $CI_COMMIT_TAG ``` +## Build a multi-arch image + +You can build [multi-arch images](https://www.docker.com/blog/multi-arch-build-and-images-the-simple-way/) +inside a container by using [`manifest-tool`](https://github.com/estesp/manifest-tool). + +For a detailed guide on how to build a multi-arch image, read [Building a multi-arch container image in unprivileged containers](https://ingenuity.siemens.com/2022/07/building-a-multi-arch-container-image-in-unprivileged-containers/). + ## Using a registry with a custom certificate When trying to push to a Docker registry that uses a certificate that is signed diff --git a/doc/ci/environments/index.md b/doc/ci/environments/index.md index 6ffa68e4873..7747f5e9b78 100644 --- a/doc/ci/environments/index.md +++ b/doc/ci/environments/index.md @@ -21,7 +21,6 @@ GitLab: If you have a deployment service like [Kubernetes](../../user/infrastructure/clusters/index.md) associated with your project, you can use it to assist with your deployments. -You can even access a [web terminal](#web-terminals-deprecated) for your environment from within GitLab. ## View environments and deployments @@ -374,6 +373,9 @@ To retry or rollback a deployment: ### Environment URL +> - [Fixed](https://gitlab.com/gitlab-org/gitlab/-/issues/337417) to persist arbitrary URLs in GitLab 15.2 [with a flag](../../administration/feature_flags.md) named `soft_validation_on_external_url`. Disabled by default. +> - [Generally available](https://gitlab.com/gitlab-org/gitlab/-/issues/337417) in GitLab 15.3. [Feature flag `soft_validation_on_external_url`](https://gitlab.com/gitlab-org/gitlab/-/issues/367206) removed. + The [environment URL](../yaml/index.md#environmenturl) is displayed in a few places in GitLab: diff --git a/doc/ci/environments/protected_environments.md b/doc/ci/environments/protected_environments.md index 35ee4f9dd33..17eccc38747 100644 --- a/doc/ci/environments/protected_environments.md +++ b/doc/ci/environments/protected_environments.md @@ -70,7 +70,7 @@ Alternatively, you can use the API to protect an environment: name: ${CI_JOB_NAME} ``` -1. Use the UI to [create a new group](../../user/group/index.md#create-a-group). +1. Use the UI to [create a new group](../../user/group/manage.md#create-a-group). For example, this group is called `protected-access-group` and has the group ID `9899826`. Note that the rest of the examples in these steps use this group. @@ -189,11 +189,14 @@ and are protected at the same time. ### Configure group-level memberships +> - Operators are required to have Owner+ role from the original Maintainer+ role and this role change is introduced from GitLab 15.3 [with a flag](https://gitlab.com/gitlab-org/gitlab/-/issues/369873) named `group_level_protected_environment_settings_permission`. Enabled by default. +> - Original behavior where Operators are required to have Maintainer+ role can be achieved by enabling [flag](https://gitlab.com/gitlab-org/gitlab/-/issues/369875) named `override_group_level_protected_environment_settings_permission`. Disabled by default. + To maximize the effectiveness of group-level protected environments, [group-level memberships](../../user/group/index.md) must be correctly configured: -- Operators should be given at least the Maintainer role +- Operators should be given at least the Owner role for the top-level group. They can maintain CI/CD configurations for the higher environments (such as production) in the group-level settings page, which includes group-level protected environments, @@ -203,7 +206,7 @@ configured: This ensures that only operators can configure the organization-wide deployment ruleset. - Developers should be given no more than the Developer role - for the top-level group, or explicitly given the Maintainer role for a child project + for the top-level group, or explicitly given the Owner role for a child project They do *not* have access to the CI/CD configurations in the top-level group, so operators can ensure that the critical configuration won't be accidentally changed by the developers. diff --git a/doc/ci/examples/authenticating-with-hashicorp-vault/index.md b/doc/ci/examples/authenticating-with-hashicorp-vault/index.md index 90cbcb9e240..00025a66936 100644 --- a/doc/ci/examples/authenticating-with-hashicorp-vault/index.md +++ b/doc/ci/examples/authenticating-with-hashicorp-vault/index.md @@ -180,13 +180,13 @@ $ vault write auth/jwt/role/myproject-production - <<EOF EOF ``` -This example uses [bound_claims](https://www.vaultproject.io/api/auth/jwt#bound_claims) to specify that only a JWT with matching values for the specified claims is allowed to authenticate. +This example uses [bound_claims](https://www.vaultproject.io/api-docs/auth/jwt#bound_claims) to specify that only a JWT with matching values for the specified claims is allowed to authenticate. Combined with [protected branches](../../../user/project/protected_branches.md), you can restrict who is able to authenticate and read the secrets. -[`token_explicit_max_ttl`](https://www.vaultproject.io/api/auth/jwt#token_explicit_max_ttl) specifies that the token issued by Vault, upon successful authentication, has a hard lifetime limit of 60 seconds. +[`token_explicit_max_ttl`](https://www.vaultproject.io/api-docs/auth/jwt#token_explicit_max_ttl) specifies that the token issued by Vault, upon successful authentication, has a hard lifetime limit of 60 seconds. -[`user_claim`](https://www.vaultproject.io/api/auth/jwt#user_claim) specifies the name for the Identity alias created by Vault upon a successful login. +[`user_claim`](https://www.vaultproject.io/api-docs/auth/jwt#user_claim) specifies the name for the Identity alias created by Vault upon a successful login. [`bound_claims_type`](https://www.vaultproject.io/api-docs/auth/jwt#bound_claims_type) configures the interpretation of the `bound_claims` values. If set to `glob`, the values are interpreted as globs, with `*` matching any number of characters. @@ -212,7 +212,7 @@ Role example to support the templated policy above, mapping the claim field `pro } ``` -For the full list of options, see Vault's [Create Role documentation](https://www.vaultproject.io/api/auth/jwt#create-role). +For the full list of options, see Vault's [Create Role documentation](https://www.vaultproject.io/api-docs/auth/jwt#create-role). WARNING: Always restrict your roles to project or namespace by using one of the provided claims (for example, `project_id` or `namespace_id`). Otherwise any JWT generated by this instance may be allowed to authenticate using this role. @@ -225,14 +225,15 @@ $ vault write auth/jwt/config \ bound_issuer="gitlab.example.com" ``` -[bound_issuer](https://www.vaultproject.io/api/auth/jwt#inlinecode-bound_issuer) specifies that only a JWT with the issuer (that is, the `iss` claim) set to `gitlab.example.com` can use this method to authenticate, and that the JWKS endpoint (`https://gitlab.example.com/-/jwks`) should be used to validate the token. +[bound_issuer](https://www.vaultproject.io/api-docs/auth/jwt#bound_issuer) specifies that only a JWT with the issuer (that is, the `iss` claim) set to `gitlab.example.com` can use this method to authenticate, and that the JWKS endpoint (`https://gitlab.example.com/-/jwks`) should be used to validate the token. -For the full list of available configuration options, see Vault's [API documentation](https://www.vaultproject.io/api/auth/jwt#configure). +For the full list of available configuration options, see Vault's [API documentation](https://www.vaultproject.io/api-docs/auth/jwt#configure). The following job, when run for the default branch, is able to read secrets under `secret/myproject/staging/`, but not the secrets under `secret/myproject/production/`: ```yaml read_secrets: + image: vault:latest script: # Check job's ref name - echo $CI_COMMIT_REF_NAME @@ -241,7 +242,7 @@ read_secrets: # Vault's address can be provided here or as CI/CD variable - export VAULT_ADDR=http://vault.example.com:8200 # Authenticate and get token. Token expiry time and other properties can be configured - # when configuring JWT Auth - https://www.vaultproject.io/api/auth/jwt#parameters-1 + # when configuring JWT Auth - https://www.vaultproject.io/api-docs/auth/jwt#parameters-1 - export VAULT_TOKEN="$(vault write -field=token auth/jwt/login role=myproject-staging jwt=$CI_JOB_JWT)" # Now use the VAULT_TOKEN to read the secret and store it in an environment variable - export PASSWORD="$(vault kv get -field=password secret/myproject/staging/db)" @@ -261,6 +262,7 @@ The following job is able to authenticate using the `myproject-production` role ```yaml read_secrets: + image: vault:latest script: # Check job's ref name - echo $CI_COMMIT_REF_NAME @@ -269,7 +271,7 @@ read_secrets: # Vault's address can be provided here or as CI/CD variable - export VAULT_ADDR=http://vault.example.com:8200 # Authenticate and get token. Token expiry time and other properties can be configured - # when configuring JWT Auth - https://www.vaultproject.io/api/auth/jwt#parameters-1 + # when configuring JWT Auth - https://www.vaultproject.io/api-docs/auth/jwt#parameters-1 - export VAULT_TOKEN="$(vault write -field=token auth/jwt/login role=myproject-production jwt=$CI_JOB_JWT)" # Now use the VAULT_TOKEN to read the secret and store it in environment variable - export PASSWORD="$(vault kv get -field=password secret/myproject/production/db)" @@ -288,9 +290,9 @@ and GitLab features. For example, restrict the token by: for specific groups using `group_claim`. - Hard coding values for Vault bound claims based on the `user_login` and `user_email` of specific users. -- Setting Vault time limits for TTL of the token as specified in [`token_explicit_max_ttl`](https://www.vaultproject.io/api/auth/jwt#token_explicit_max_ttl), +- Setting Vault time limits for TTL of the token as specified in [`token_explicit_max_ttl`](https://www.vaultproject.io/api-docs/auth/jwt#token_explicit_max_ttl), where the token expires after authentication. -- Scoping the JWT to [GitLab projected branches](../../../user/project/protected_branches.md) +- Scoping the JWT to [GitLab protected branches](../../../user/project/protected_branches.md) that are restricted to a subset of project users. -- Scoping the JWT to [GitLab projected tags](../../../user/project/protected_tags.md), +- Scoping the JWT to [GitLab protected tags](../../../user/project/protected_tags.md), that are restricted to a subset of project users. diff --git a/doc/ci/git_submodules.md b/doc/ci/git_submodules.md index 5f22fb894e5..fbecb1cd964 100644 --- a/doc/ci/git_submodules.md +++ b/doc/ci/git_submodules.md @@ -84,3 +84,9 @@ hidden files. If there is no `.gitmodules` file, it's possible the submodule settings are in a [`git config`](https://www.atlassian.com/git/tutorials/setting-up-a-repository/git-config) file. + +### `fatal: run_command returned non-zero status` error + +This error can happen in a job when working with submodules and the `GIT_STRATEGY` is set to `fetch`. + +Setting the `GIT_STRATEGY` to `clone` should resolve the issue. diff --git a/doc/ci/interactive_web_terminal/index.md b/doc/ci/interactive_web_terminal/index.md index f49fdd6c39f..e6a9f1fa646 100644 --- a/doc/ci/interactive_web_terminal/index.md +++ b/doc/ci/interactive_web_terminal/index.md @@ -18,8 +18,8 @@ taken to protect the users. NOTE: [Shared runners on GitLab.com](../runners/index.md) do not -provide an interactive web terminal. Follow [this -issue](https://gitlab.com/gitlab-org/gitlab/-/issues/24674) for progress on +provide an interactive web terminal. Follow +[this issue](https://gitlab.com/gitlab-org/gitlab/-/issues/24674) for progress on adding support. For groups and projects hosted on GitLab.com, interactive web terminals are available when using your own group or project runner. @@ -27,8 +27,8 @@ terminals are available when using your own group or project runner. Two things need to be configured for the interactive web terminal to work: -- The runner needs to have [`[session_server]` configured - properly](https://docs.gitlab.com/runner/configuration/advanced-configuration.html#the-session_server-section) +- The runner needs to have + [`[session_server]` configured properly](https://docs.gitlab.com/runner/configuration/advanced-configuration.html#the-session_server-section) - If you are using a reverse proxy with your GitLab instance, web terminals need to be [enabled](../../administration/integration/terminal.md#enabling-and-disabling-terminal-support) @@ -54,8 +54,8 @@ Not all executors are NOTE: The `docker` executor does not keep running after the build script is finished. At that point, the terminal automatically -disconnects and does not wait for the user to finish. Please follow [this -issue](https://gitlab.com/gitlab-org/gitlab-runner/-/issues/3605) for updates on +disconnects and does not wait for the user to finish. Please follow +[this issue](https://gitlab.com/gitlab-org/gitlab-runner/-/issues/3605) for updates on improving this behavior. Sometimes, when a job is running, things don't go as you would expect, and it @@ -63,8 +63,7 @@ would be helpful if one can have a shell to aid debugging. When a job is running, on the right panel you can see a button `debug` that opens the terminal for the current job. - + When clicked, a new tab opens to the terminal page where you can access the terminal and type commands like a normal shell. diff --git a/doc/ci/jobs/ci_job_token.md b/doc/ci/jobs/ci_job_token.md index 9567bc9cd65..93f22da648a 100644 --- a/doc/ci/jobs/ci_job_token.md +++ b/doc/ci/jobs/ci_job_token.md @@ -13,6 +13,7 @@ You can use a GitLab CI/CD job token to authenticate with specific API endpoints - Packages: - [Package Registry](../../user/packages/package_registry/index.md#use-gitlab-cicd-to-build-packages). + - [Packages API](../../api/packages.md) (project-level). - [Container Registry](../../user/packages/container_registry/index.md#build-and-push-by-using-gitlab-cicd) (the `$CI_REGISTRY_PASSWORD` is `$CI_JOB_TOKEN`). - [Container Registry API](../../api/container_registry.md) diff --git a/doc/ci/jobs/job_control.md b/doc/ci/jobs/job_control.md index 24133fe9a9b..0d5357e63ad 100644 --- a/doc/ci/jobs/job_control.md +++ b/doc/ci/jobs/job_control.md @@ -304,7 +304,7 @@ to add jobs to a pipeline: ```yaml docker build: variables: - DOCKERFILES_DIR: 'path/to/files/' + DOCKERFILES_DIR: 'path/to/files' script: docker build -t my-image:$CI_COMMIT_REF_SLUG . rules: - changes: @@ -567,6 +567,9 @@ In blocking manual jobs: enabled can't be merged with a blocked pipeline. - The pipeline shows a status of **blocked**. +When using manual jobs in triggered pipelines with [`strategy: depend`](../yaml/index.md#triggerstrategy), +the type of manual job can affect the trigger job's status while the pipeline runs. + ### Run a manual job To run a manual job, you must have permission to merge to the assigned branch: @@ -1005,6 +1008,26 @@ Additionally, rules with `changes` always evaluate as true in [scheduled pipelin All files are considered to have changed when a scheduled pipeline runs, so jobs might always be added to scheduled pipelines that use `changes`. +### File paths in CI/CD variables + +Be careful when using file paths in CI/CD variables. A trailing slash can appear correct +in the variable definition, but can become invalid when expanded in `script:`, `changes:`, +or other keywords. For example: + +```yaml +docker_build: + variables: + DOCKERFILES_DIR: 'path/to/files/' # This variable should not have a trailing '/' character + script: echo "A docker job" + rules: + - changes: + - $DOCKERFILES_DIR/* +``` + +When the `DOCKERFILES_DIR` variable is expanded in the `changes:` section, the full +path becomes `path/to/files//*`. The double slashes might cause unexpected behavior +depending on the keyword used, shell and OS of the runner, and so on. + ### `You are not allowed to download code from this project.` error message You might see pipelines fail when a GitLab administrator runs a protected manual job diff --git a/doc/ci/large_repositories/index.md b/doc/ci/large_repositories/index.md index f3044a03e04..3904a527a8f 100644 --- a/doc/ci/large_repositories/index.md +++ b/doc/ci/large_repositories/index.md @@ -116,8 +116,7 @@ you could add [`--no-tags`](https://git-scm.com/docs/git-fetch#Documentation/git to the extra flags to make your fetches faster and more compact. Also in the case where you repository does _not_ contain a lot of -tags, `--no-tags` can [make a big difference in some -cases](https://gitlab.com/gitlab-com/gl-infra/scalability/-/issues/746). +tags, `--no-tags` can [make a big difference in some cases](https://gitlab.com/gitlab-com/gl-infra/scalability/-/issues/746). If your CI builds do not depend on Git tags it is worth trying. See the [`GIT_FETCH_EXTRA_FLAGS` documentation](../runners/configure_runners.md#git-fetch-extra-flags) diff --git a/doc/ci/migration/circleci.md b/doc/ci/migration/circleci.md index 3b890458e56..7255d9aec82 100644 --- a/doc/ci/migration/circleci.md +++ b/doc/ci/migration/circleci.md @@ -265,7 +265,7 @@ test_async: ## Contexts and variables -CircleCI provides [Contexts](https://circleci.com/docs/2.0/contexts/) to securely pass environment variables across project pipelines. In GitLab, a [Group](../../user/group/index.md) can be created to assemble related projects together. At the group level, [CI/CD variables](../variables/index.md#add-a-cicd-variable-to-a-group) can be stored outside the individual projects, and securely passed into pipelines across multiple projects. +CircleCI provides [Contexts](https://circleci.com/docs/contexts) to securely pass environment variables across project pipelines. In GitLab, a [Group](../../user/group/index.md) can be created to assemble related projects together. At the group level, [CI/CD variables](../variables/index.md#add-a-cicd-variable-to-a-group) can be stored outside the individual projects, and securely passed into pipelines across multiple projects. ## Orbs diff --git a/doc/ci/pipeline_editor/index.md b/doc/ci/pipeline_editor/index.md index 4fb2ec94d60..0fd8fac7741 100644 --- a/doc/ci/pipeline_editor/index.md +++ b/doc/ci/pipeline_editor/index.md @@ -40,6 +40,10 @@ is invalid, a tip is shown to help you fix the problem: ## Lint CI configuration +NOTE: +The **Lint** tab is replaced with the **Validate** tab in GitLab 15.3. The lint results are included +in a successful [pipeline simulation](#simulate-a-cicd-pipeline). + To test the validity of your GitLab CI/CD configuration before committing the changes, you can use the CI lint tool. To access it, go to **CI/CD > Editor** and select the **Lint** tab. @@ -51,6 +55,15 @@ reflected in the CI lint. It displays the same results as the existing [CI Lint  +## Simulate a CI/CD pipeline + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/337282) in GitLab 15.3. + +To look for pipeline syntax and logic issues, you can simulate the creation of a +GitLab CI/CD pipeline in the **Validate** tab. A pipeline simulation can help find +problems such as incorrect `rules` and `needs` job dependencies, and is similar to +simulations in the [CI Lint tool](../lint.md#simulate-a-pipeline). + ## View included CI/CD configuration > - [Introduced](https://gitlab.com/groups/gitlab-org/-/epics/7064) in GitLab 15.0 [with a flag](../../administration/feature_flags.md) named `pipeline_editor_file_tree`. Disabled by default. diff --git a/doc/ci/pipelines/cicd_minutes.md b/doc/ci/pipelines/cicd_minutes.md index 1e862c87035..4b7d3845361 100644 --- a/doc/ci/pipelines/cicd_minutes.md +++ b/doc/ci/pipelines/cicd_minutes.md @@ -21,7 +21,7 @@ On GitLab.com: - CI/CD minutes quotas are enabled for both public and private projects, but public projects [consume CI/CD minutes at a slower rate](#cost-factor). -- The base monthly CI/CD minutes quota for a GitLab.com [namespace](../../user/group/index.md#namespaces) +- The base monthly CI/CD minutes quota for a GitLab.com [namespace](../../user/namespace/index.md) is determined by its [license tier](https://about.gitlab.com/pricing/). - You can [purchase additional CI/CD minutes](#purchase-additional-cicd-minutes) if you need more than the number of CI/CD minutes in your monthly quota. @@ -204,12 +204,36 @@ The cost factors for jobs running on shared runners on GitLab.com are: - `0.008` for public projects, and projects in the [GitLab for Open Source program](../../subscriptions/index.md#gitlab-for-open-source). For every 125 minutes of job execution time, you use 1 CI/CD minute. - `1` for internal and private projects. +- Calculated differently for [community contributions to GitLab projects](#cost-factor-for-community-contributions-to-gitlab-projects). The cost factors on self-managed instances are: - `0` for public projects, so they do not consume CI/CD minutes. - `1` for internal and private projects. +#### Cost factor for community contributions to GitLab projects + +Community contributors can use up to 300,000 minutes on shared runners when +contributing to open source projects maintained by GitLab. The 300,000 +minutes applies to all SaaS tiers, and the cost factor calculation is: + +- `Monthly minute quota / 300,000 job duration minutes = Cost factor` + +For example, with the 10,000 CI/CD minutes per month in the Premium tier: + +- 10,000 / 300,000 = 0.03333333333 cost factor. + +For this reduced cost factor: + +- The merge request source project must be a fork of a GitLab-maintained project, + such as [`gitlab-com/www-gitlab-com`](https://gitlab.com/gitlab-com/www-gitlab-com), + [`gitlab-org/gitlab`](https://gitlab.com/gitlab-org/gitlab), and so on. +- The merge request target project must be the fork's parent project. +- The pipeline must be a merge request, merged results, or merge train pipeline. + +GitLab administrators can add a namespace to the reduced cost factor +[with a flag](../../administration/feature_flags.md) named `ci_minimal_cost_factor_for_gitlab_namespaces`. + ### Additional costs on GitLab SaaS GitLab SaaS shared runners have different cost factors, depending on the runner type (Linux, Windows, macOS) and the virtual machine configuration. diff --git a/doc/ci/pipelines/img/pipeline_mini_graph_v15_0.png b/doc/ci/pipelines/img/pipeline_mini_graph_v15_0.png Binary files differnew file mode 100644 index 00000000000..48a0ca9d84f --- /dev/null +++ b/doc/ci/pipelines/img/pipeline_mini_graph_v15_0.png diff --git a/doc/ci/pipelines/merge_request_pipelines.md b/doc/ci/pipelines/merge_request_pipelines.md index 89839de718b..5ba489c9830 100644 --- a/doc/ci/pipelines/merge_request_pipelines.md +++ b/doc/ci/pipelines/merge_request_pipelines.md @@ -151,7 +151,10 @@ or [**Rebase** option](../../user/project/merge_requests/methods/index.md#rebasi Prerequisites: -- You must be a member of the parent project and have at least the [Developer role](../../user/permissions.md). +- The parent project's [CI/CD configuration file](../yaml/index.md) must be configured to + [run jobs in merge request pipelines](#prerequisites). +- You must be a member of the parent project with [permissions to run CI/CD pipelines](../../user/permissions.md#gitlab-cicd-permissions). + You might need additional permissions if the branch is protected. - The fork project must be [visible](../../user/public_access.md) to the user running the pipeline. Otherwise, the **Pipelines** tab does not display in the merge request. @@ -161,6 +164,10 @@ To use the UI to run a pipeline in the parent project for a merge request from a 1. In the merge request, go to the **Pipelines** tab. 1. Select **Run pipeline**. You must read and accept the warning, or the pipeline does not run. +You can disable this feature by using [the projects API](../../api/projects.md#edit-project) +to disable the `ci_allow_fork_pipelines_to_run_in_parent_project` setting. +The setting is `enabled` by default. + ## Available predefined variables When you use merge request pipelines, you can use: diff --git a/doc/ci/pipelines/merge_trains.md b/doc/ci/pipelines/merge_trains.md index ac644628f3a..2882cd378aa 100644 --- a/doc/ci/pipelines/merge_trains.md +++ b/doc/ci/pipelines/merge_trains.md @@ -59,9 +59,8 @@ changes that are included in the target branch, and the `C` changes that are fro the merge request already in the train. <i class="fa fa-youtube-play youtube" aria-hidden="true"></i> -Watch this video for a demonstration on [how parallel execution -of merge trains can prevent commits from breaking the default -branch](https://www.youtube.com/watch?v=D4qCqXgZkHQ). +Watch this video for a demonstration on +[how parallel execution of merge trains can prevent commits from breaking the default branch](https://www.youtube.com/watch?v=D4qCqXgZkHQ). ## Prerequisites diff --git a/doc/ci/pipelines/multi_project_pipelines.md b/doc/ci/pipelines/multi_project_pipelines.md index dfbd2708d8d..a71af78f410 100644 --- a/doc/ci/pipelines/multi_project_pipelines.md +++ b/doc/ci/pipelines/multi_project_pipelines.md @@ -45,8 +45,8 @@ To create multi-project pipelines, you can: > [Moved](https://gitlab.com/gitlab-org/gitlab/-/issues/199224) to GitLab Free in 12.8. -When you create a multi-project pipeline in your `.gitlab-ci.yml` file, -you create what is called a *trigger job*. For example: +When you use the [`trigger`](../yaml/index.md#trigger) keyword to create a multi-project +pipeline in your `.gitlab-ci.yml` file, you create what is called a *trigger job*. For example: ```yaml rspec: @@ -76,7 +76,7 @@ downstream project (`my/deployment`) too. If the downstream project is not found or the user does not have [permission](../../user/permissions.md) to create a pipeline there, the `staging` job is marked as _failed_. -#### Trigger job configuration keywords +#### Trigger job configuration limitations Trigger jobs can use only a limited set of the GitLab CI/CD [configuration keywords](../yaml/index.md). The keywords available for use in trigger jobs are: @@ -90,6 +90,8 @@ The keywords available for use in trigger jobs are: - [`extends`](../yaml/index.md#extends) - [`needs`](../yaml/index.md#needs), but not [`needs:project`](../yaml/index.md#needsproject) +Trigger jobs cannot use [job-level persisted variables](../variables/where_variables_can_be_used.md#persisted-variables). + #### Specify a downstream pipeline branch You can specify a branch name for the downstream pipeline to use. @@ -111,6 +113,8 @@ staging: Use: - The `project` keyword to specify the full path to a downstream project. + In [GitLab 15.3 and later](https://gitlab.com/gitlab-org/gitlab/-/issues/367660), variable expansion is + supported. - The `branch` keyword to specify the name of a branch in the project specified by `project`. In [GitLab 12.4 and later](https://gitlab.com/gitlab-org/gitlab/-/issues/10126), variable expansion is supported. @@ -180,9 +184,12 @@ downstream-job: trigger: my/project ``` -In this scenario, the `UPSTREAM_BRANCH` variable with a value related to the -upstream pipeline is passed to the `downstream-job` job. It is available -in the context of all downstream builds. +In this scenario, the `UPSTREAM_BRANCH` variable with the value of the upstream pipeline's +`$CI_COMMIT_REF_NAME` is passed to `downstream-job`. It is available in the +context of all downstream builds. + +You cannot use this method to forward [job-level persisted variables](../variables/where_variables_can_be_used.md#persisted-variables) +to a downstream pipeline, as they are not available in trigger jobs. Upstream pipelines take precedence over downstream ones. If there are two variables with the same name defined in both upstream and downstream projects, @@ -228,6 +235,96 @@ In the upstream pipeline: artifacts: true ``` +#### Pass artifacts to a downstream pipeline + +You can pass artifacts to a downstream pipeline by using [`needs:project`](../yaml/index.md#needsproject). + +1. In a job in the upstream pipeline, save the artifacts using the [`artifacts`](../yaml/index.md#artifacts) keyword. +1. Trigger the downstream pipeline with a trigger job: + + ```yaml + build_artifacts: + stage: build + script: + - echo "This is a test artifact!" >> artifact.txt + artifacts: + paths: + - artifact.txt + + deploy: + stage: deploy + trigger: my/downstream_project + ``` + +1. In a job in the downstream pipeline, fetch the artifacts from the upstream pipeline + by using `needs:project`. Set `job` to the job in the upstream pipeline to fetch artifacts from, + `ref` to the branch, and `artifacts: true`. + + ```yaml + test: + stage: test + script: + - cat artifact.txt + needs: + - project: my/upstream_project + job: build_artifacts + ref: main + artifacts: true + ``` + +#### Pass artifacts to a downstream pipeline from a Merge Request pipeline + +When you use `needs:project` to [pass artifacts to a downstream pipeline](#pass-artifacts-to-a-downstream-pipeline), +the `ref` value is usually a branch name, like `main` or `development`. + +For merge request pipelines, the `ref` value is in the form of `refs/merge-requests/<id>/head`, +where `id` is the merge request ID. You can retrieve this ref with the [`CI_MERGE_REQUEST_REF_PATH`](../variables/predefined_variables.md#predefined-variables-for-merge-request-pipelines) +CI/CD variable. Do not use a branch name as the `ref` with merge request pipelines, +because the downstream pipeline attempts to fetch artifacts from the latest branch pipeline. + +To fetch the artifacts from the upstream `merge request` pipeline instead of the `branch` pipeline, +pass this variable to the downstream pipeline using variable inheritance: + +1. In a job in the upstream pipeline, save the artifacts using the [`artifacts`](../yaml/index.md#artifacts) keyword. +1. In the job that triggers the downstream pipeline, pass the `$CI_MERGE_REQUEST_REF_PATH` variable by using + [variable inheritance](#pass-cicd-variables-to-a-downstream-pipeline-by-using-the-variables-keyword): + + ```yaml + build_artifacts: + stage: build + script: + - echo "This is a test artifact!" >> artifact.txt + artifacts: + paths: + - artifact.txt + + upstream_job: + variables: + UPSTREAM_REF: $CI_MERGE_REQUEST_REF_PATH + trigger: + project: my/downstream_project + branch: my-branch + ``` + +1. In a job in the downstream pipeline, fetch the artifacts from the upstream pipeline + by using `needs:project`. Set the `ref` to the `UPSTREAM_REF` variable, and `job` + to the job in the upstream pipeline to fetch artifacts from: + + ```yaml + test: + stage: test + script: + - cat artifact.txt + needs: + - project: my/upstream_project + job: build_artifacts + ref: UPSTREAM_REF + artifacts: true + ``` + +This method works for fetching artifacts from a regular merge request parent pipeline, +but fetching artifacts from [merge results](merged_results_pipelines.md) pipelines is not supported. + #### Use `rules` or `only`/`except` with multi-project pipelines You can use CI/CD variables or the [`rules`](../yaml/index.md#rulesif) keyword to @@ -244,8 +341,8 @@ If you use [`only/except`](../yaml/index.md#only--except) to control job behavio > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/11238) in GitLab Premium 12.3. > - [Moved](https://gitlab.com/gitlab-org/gitlab/-/issues/199224) to GitLab Free in 12.8. -You can mirror the pipeline status from the triggered pipeline to the source -trigger job by using `strategy: depend`. For example: +You can mirror the pipeline status from the triggered pipeline to the source trigger job +by using [`strategy: depend`](../yaml/index.md#triggerstrategy). For example: ```yaml trigger_job: @@ -303,11 +400,16 @@ downstream projects. On self-managed instances, an administrator can change this ## Multi-project pipeline visualization **(PREMIUM)** -When you configure GitLab CI/CD for your project, you can visualize the stages of your -[jobs](index.md#configure-a-pipeline) on a [pipeline graph](index.md#visualize-pipelines). +When your pipeline triggers a downstream pipeline, the downstream pipeline displays +to the right of the [pipeline graph](index.md#visualize-pipelines).  +In [pipeline mini graphs](index.md#pipeline-mini-graphs), the downstream pipeline +displays to the right of the mini graph. + + + ## Retry or cancel multi-project pipelines If you have permission to trigger pipelines in the downstream project, you can diff --git a/doc/ci/pipelines/settings.md b/doc/ci/pipelines/settings.md index 43f20bfa9ea..34eae9828dd 100644 --- a/doc/ci/pipelines/settings.md +++ b/doc/ci/pipelines/settings.md @@ -250,8 +250,8 @@ using the [`coverage`](../yaml/index.md#coverage) keyword. ### Add test coverage results using project settings (removed) -> [Deprecated](https://gitlab.com/gitlab-org/gitlab/-/issues/17633) in GitLab 14.8. Replaced by [`coverage` keyword](../yaml/index.md#coverage). -> [Removed](https://gitlab.com/gitlab-org/gitlab/-/issues/17633) in GitLab 15.0. +> - [Deprecated](https://gitlab.com/gitlab-org/gitlab/-/issues/17633) in GitLab 14.8. Replaced by [`coverage` keyword](../yaml/index.md#coverage). +> - [Removed](https://gitlab.com/gitlab-org/gitlab/-/issues/17633) in GitLab 15.0. This feature is in its end-of-life process. It was [deprecated](https://gitlab.com/gitlab-org/gitlab/-/issues/17633) in GitLab 14.8. The feature is [removed](https://gitlab.com/gitlab-org/gitlab/-/issues/17633) in GitLab 15.0. @@ -299,22 +299,22 @@ Use this regex for commonly used test tools. <!-- vale gitlab.Spelling = NO --> -- Simplecov (Ruby). Example: `\(\d+.\d+\%\) covered`. -- pytest-cov (Python). Example: `(?i)total.*? (100(?:\.0+)?\%|[1-9]?\d(?:\.\d+)?\%)$`. -- Scoverage (Scala). Example: `Statement coverage[A-Za-z\.*]\s*:\s*([^%]+)`. -- `phpunit --coverage-text --colors=never` (PHP). Example: `^\s*Lines:\s*\d+.\d+\%`. -- gcovr (C/C++). Example: `^TOTAL.*\s+(\d+\%)$`. -- `tap --coverage-report=text-summary` (NodeJS). Example: `^Statements\s*:\s*([^%]+)`. -- `nyc npm test` (NodeJS). Example: `All files[^|]*\|[^|]*\s+([\d\.]+)`. -- `jest --ci --coverage` (NodeJS). Example: `All files[^|]*\|[^|]*\s+([\d\.]+)`. -- excoveralls (Elixir). Example: `\[TOTAL\]\s+(\d+\.\d+)%`. -- `mix test --cover` (Elixir). Example: `\d+.\d+\%\s+\|\s+Total`. -- JaCoCo (Java/Kotlin). Example: `Total.*?([0-9]{1,3})%`. -- `go test -cover` (Go). Example: `coverage: \d+.\d+% of statements`. -- .NET (OpenCover). Example: `(Visited Points).*\((.*)\)`. -- .NET (`dotnet test` line coverage). Example: `Total\s*\|\s*(\d+(?:\.\d+)?)`. -- tarpaulin (Rust). Example: `^\d+.\d+% coverage`. -- Pester (PowerShell). Example: `Covered (\d+\.\d+%)`. +- Simplecov (Ruby). Example: `/\(\d+.\d+\%\) covered/`. +- pytest-cov (Python). Example: `/(?i)total.*? (100(?:\.0+)?\%|[1-9]?\d(?:\.\d+)?\%)$/`. +- Scoverage (Scala). Example: `/Statement coverage[A-Za-z\.*]\s*:\s*([^%]+)/`. +- `phpunit --coverage-text --colors=never` (PHP). Example: `/^\s*Lines:\s*\d+.\d+\%/`. +- gcovr (C/C++). Example: `/^TOTAL.*\s+(\d+\%)$/`. +- `tap --coverage-report=text-summary` (NodeJS). Example: `/^Statements\s*:\s*([^%]+)/`. +- `nyc npm test` (NodeJS). Example: `/All files[^|]*\|[^|]*\s+([\d\.]+)/`. +- `jest --ci --coverage` (NodeJS). Example: `/All files[^|]*\|[^|]*\s+([\d\.]+)/`. +- excoveralls (Elixir). Example: `/\[TOTAL\]\s+(\d+\.\d+)%/`. +- `mix test --cover` (Elixir). Example: `/\d+.\d+\%\s+\|\s+Total/`. +- JaCoCo (Java/Kotlin). Example: `/Total.*?([0-9]{1,3})%/`. +- `go test -cover` (Go). Example: `/coverage: \d+.\d+% of statements/`. +- .NET (OpenCover). Example: `/(Visited Points).*\((.*)\)/`. +- .NET (`dotnet test` line coverage). Example: `/Total\s*\|\s*(\d+(?:\.\d+)?)/`. +- tarpaulin (Rust). Example: `/^\d+.\d+% coverage/`. +- Pester (PowerShell). Example: `/Covered (\d+\.\d+%)/`. <!-- vale gitlab.Spelling = YES --> diff --git a/doc/ci/runners/configure_runners.md b/doc/ci/runners/configure_runners.md index efd78fac2c6..3efa697bf2f 100644 --- a/doc/ci/runners/configure_runners.md +++ b/doc/ci/runners/configure_runners.md @@ -722,11 +722,14 @@ variables: > [Introduced](https://gitlab.com/gitlab-org/gitlab-runner/-/issues/28940) in GitLab Runner 15.1. +NOTE: +Zip archives are the only supported artifact type. Follow [the issue for details](https://gitlab.com/gitlab-org/gitlab/-/issues/367203). + GitLab Runner can generate and produce attestation metadata for all build artifacts. To enable this feature, you must set the `RUNNER_GENERATE_ARTIFACTS_METADATA` environment variable to `true`. This variable can either be set globally or it can be set for individual jobs. The metadata is in rendered in a plain text `.json` file that's stored with the artifact. The file name is as follows: `{JOB_ID}-artifacts-metadata.json`. ### Attestation format -The attestation metadata is generated in the [in-toto attestation format](https://github.com/in-toto/attestation) for spec version [v0.1](https://in-toto.io/Statement/v0.1). The following fields are populated by default: +The attestation metadata is generated in the [in-toto attestation format](https://github.com/in-toto/attestation) for spec version [v0.1](https://github.com/in-toto/attestation/tree/v0.1.0/spec). The following fields are populated by default: | Field | Value | | ------ | ------ | @@ -863,7 +866,7 @@ Group runners are those that were created at the group level. ### View stale runner cleanup logs -You can check the [Sidekiq logs](../../administration/logs.md#sidekiq-logs) to see the cleanup result. In Kibana you can use the following query: +You can check the [Sidekiq logs](../../administration/logs/index.md#sidekiq-logs) to see the cleanup result. In Kibana you can use the following query: ```json { @@ -889,3 +892,23 @@ Filter entries where stale runners were removed: } } ``` + +## Determine which runners need to be upgraded **(ULTIMATE)** + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/365078) in GitLab 15.3. + +The version of GitLab Runner used by your runners should be +[kept up-to-date](https://docs.gitlab.com/runner/index.html#gitlab-runner-versions). + +To determine which runners need to be upgraded: + +1. View the list of runners: + - For a group, on the top bar, select **Menu > Groups** and on the left sidebar, select **CI/CD > Runners**. + - For the instance, select **Menu > Admin** and on the left sidebar, select **Runners**. + +1. Above the list of runners, view the status: + - **Outdated - recommended**: The runner does not have the latest `PATCH` version, which may make it vulnerable + to security or high severity bugs. Or, the runner is one or more `MAJOR` versions behind your GitLab instance, so some features may not be available or work properly. + - **Outdated - available**: Newer versions are available but upgrading is not critical. + +1. Filter the list by status to view which individual runners need to be upgraded. diff --git a/doc/ci/runners/img/build_isolation.png b/doc/ci/runners/img/build_isolation.png Binary files differnew file mode 100644 index 00000000000..a363ef4709b --- /dev/null +++ b/doc/ci/runners/img/build_isolation.png diff --git a/doc/ci/runners/index.md b/doc/ci/runners/index.md index 038bda4ab09..f69d1f0f730 100644 --- a/doc/ci/runners/index.md +++ b/doc/ci/runners/index.md @@ -18,5 +18,25 @@ The number of minutes you can use on these runners depends on the [maximum number of CI/CD minutes](../pipelines/cicd_minutes.md) in your [subscription plan](https://about.gitlab.com/pricing/). -If you use self-managed GitLab or you use GitLab.com but want to use your own runners, you can -[install and configure your own runners](https://docs.gitlab.com/runner/install/). +## Security for GitLab SaaS runners + +GitLab SaaS runners on Linux and Windows run on Google Compute Platform. The [Google Infrastructure Security Design Overview whitepaper](https://cloud.google.com/docs/security/infrastructure/design/resources/google_infrastructure_whitepaper_fa.pdf) provides an overview of how Google designs security into its technical infrastructure. The GitLab [Trust Center](https://about.gitlab.com/security/) and [GitLab Security Compliance Controls](https://about.staging.gitlab.com/handbook/engineering/security/security-assurance/security-compliance/sec-controls.html) pages provide an overview of the Security and Compliance controls that govern the GitLab SaaS runners. + +The runner that serves as a Runner Manager automatically initiates the creation and deletion of the virtual machines (VMs) used for CI jobs. When the Runner Manager picks up a GitLab SaaS CI job, it automatically executes that job on a new VM. There is no human or manual intervention in this process. The following section provides an overview of the additional built-in layers that harden the security of the GitLab Runner SaaS CI build environment. + +### Security of CI job execution on GitLab Runner SaaS (Linux, Windows) + +A dedicated temporary runner VM hosts and runs each CI job. On GitLab SaaS, two CI jobs never run on the same VM. + + + +In this example, there are three jobs in the project's pipeline. Therefore, there are three temporary VMs used to run that pipeline, or one VM per job. + +GitLab sends the command to remove the temporary runner VM to the Google Compute API immediately after the CI job completes. The [Google Compute Engine hypervisor](https://cloud.google.com/blog/products/gcp/7-ways-we-harden-our-kvm-hypervisor-at-google-cloud-security-in-plaintext) takes over the task of securely deleting the virtual machine and associated data. + +### Network security of CI job virtual machines on GitLab Runner SaaS (Linux, Windows) + +- Firewall rules only allow outbound communication from the temporary VM to the public internet. +- Inbound communication from the public internet to the temporary VM is not allowed. +- Firewall rules do not permit communication between VMs. +- The only internal communication allowed to the temporary VMs is from the Runner Manager. diff --git a/doc/ci/runners/runners_scope.md b/doc/ci/runners/runners_scope.md index 8b2753d26f1..9bd0b52f423 100644 --- a/doc/ci/runners/runners_scope.md +++ b/doc/ci/runners/runners_scope.md @@ -25,11 +25,11 @@ multiple projects. If you are using a self-managed instance of GitLab: - Your administrator can install and register shared runners by - going to your project's **Settings > CI/CD**, expanding the **Runners** section, - and clicking **Show runner installation instructions**. + going to your project's **Settings > CI/CD**, expanding **Runners**, + and selecting **Show runner installation instructions**. These instructions are also available [in the documentation](https://docs.gitlab.com/runner/install/index.html). -- The administrator can also configure a maximum number of shared runner [CI/CD minutes for - each group](../pipelines/cicd_minutes.md#set-the-quota-of-cicd-minutes-for-a-specific-namespace). +- The administrator can also configure a maximum number of shared runner + [CI/CD minutes for each group](../pipelines/cicd_minutes.md#set-the-quota-of-cicd-minutes-for-a-specific-namespace). If you are using GitLab.com: @@ -51,15 +51,19 @@ For existing projects, an administrator must To enable shared runners for a project: -1. Go to the project's **Settings > CI/CD** and expand the **Runners** section. -1. Select **Enable shared runners for this project**. +1. On the top bar, select **Menu > Projects** and find your project. +1. On the left sidebar, select **Settings > CI/CD**. +1. Expand **Runners**. +1. Turn on the **Enable shared runners for this project** toggle. ### Enable shared runners for a group To enable shared runners for a group: -1. Go to the group's **Settings > CI/CD** and expand the **Runners** section. -1. Select **Enable shared runners for this group**. +1. On the top bar, select **Menu > Groups** and find your group. +1. On the left sidebar, select **Settings > CI/CD**. +1. Expand **Runners**. +1. Turn on the **Enable shared runners for this group** toggle. ### Disable shared runners for a project @@ -69,8 +73,10 @@ or group. To disable shared runners for a project: -1. Go to the project's **Settings > CI/CD** and expand the **Runners** section. -1. In the **Shared runners** area, select **Enable shared runners for this project** so the toggle is grayed-out. +1. On the top bar, select **Menu > Projects** and find your project. +1. On the left sidebar, select **Settings > CI/CD**. +1. Expand **Runners**. +1. In the **Shared runners** area, turn off the **Enable shared runners for this project** toggle. Shared runners are automatically disabled for a project: @@ -81,9 +87,11 @@ Shared runners are automatically disabled for a project: To disable shared runners for a group: -1. Go to the group's **Settings > CI/CD** and expand the **Runners** section. -1. In the **Shared runners** area, turn off the **Enable shared runners for this group** toggle. -1. Optionally, to allow shared runners to be enabled for individual projects or subgroups, +1. On the top bar, select **Menu > Groups** and find your group. +1. On the left sidebar, select **Settings > CI/CD**. +1. Expand **Runners**. +1. Turn off the **Enable shared runners for this group** toggle. +1. Optional. To allow shared runners to be enabled for individual projects or subgroups, select **Allow projects and subgroups to override the group setting**. NOTE: @@ -147,7 +155,7 @@ The fair usage algorithm assigns jobs in this order: ## Group runners -Use *Group runners* when you want all projects in a group +Use _group runners_ when you want all projects in a group to have access to a set of runners. Group runners process jobs by using a first in, first out ([FIFO](https://en.wikipedia.org/wiki/FIFO_(computing_and_electronics))) queue. @@ -162,7 +170,7 @@ You must have the Owner role for the group. To create a group runner: 1. [Install GitLab Runner](https://docs.gitlab.com/runner/install/). -1. Go to the group you want to make the runner work for. +1. On the top bar, select **Menu > Groups** and find your group. 1. On the left sidebar, select **CI/CD > Runners**. 1. Note the URL and token. 1. [Register the runner](https://docs.gitlab.com/runner/register/). @@ -175,21 +183,8 @@ You can view and manage all runners for a group, its subgroups, and projects. You can do this for your self-managed GitLab instance or for GitLab.com. You must have the Owner role for the group. -1. Go to the group where you want to view the runners. +1. On the top bar, select **Menu > Groups** and find your group. 1. On the left sidebar, select **CI/CD > Runners**. -1. The following fields are displayed. - - | Attribute | Description | - | ------------ | ----------- | - | Type | Displays the runner type: `group` or `specific`, and the optional state `paused` | - | Runner token | Token used to identify the runner, and that the runner uses to communicate with the GitLab instance | - | Description | Description given to the runner when it was created | - | Version | GitLab Runner version | - | IP address | IP address of the host on which the runner is registered | - | Projects | The count of projects to which the runner is assigned | - | Jobs | Total of jobs run by the runner | - | Tags | Tags associated with the runner | - | Last contact | Timestamp indicating when the GitLab instance last contacted the runner | From this page, you can edit, pause, and remove runners from the group, its subgroups, and projects. @@ -198,7 +193,7 @@ From this page, you can edit, pause, and remove runners from the group, its subg You can pause or remove a group runner for your self-managed GitLab instance or for GitLab.com. You must have the Owner role for the group. -1. Go to the group you want to remove or pause the runner for. +1. On the top bar, select **Menu > Groups** and find your group. 1. On the left sidebar, select **CI/CD > Runners**. 1. Select **Pause** or **Remove runner**. - If you pause a group runner that is used by multiple projects, the runner pauses for all projects. @@ -208,7 +203,7 @@ You must have the Owner role for the group. ## Specific runners -Use *Specific runners* when you want to use runners for specific projects. For example, +Use _specific runners_ when you want to use runners for specific projects. For example, when you have: - Jobs with specific requirements, like a deploy job that requires credentials. @@ -257,9 +252,8 @@ To enable a specific runner for a project: 1. On the top bar, select **Menu > Projects** and find the project where you want to enable the runner. 1. On the left sidebar, select **Settings > CI/CD**. -1. Expand **General pipelines**. 1. Expand **Runners**. -1. By the runner you want, select **Enable for this project**. +1. In the **Specific runners** area, by the runner you want, select **Enable for this project**. You can edit a specific runner from any of the projects it's enabled for. The modifications, which include unlocking and editing tags and the description, @@ -275,9 +269,10 @@ but can also be changed later. To lock or unlock a specific runner: -1. Go to the project's **Settings > CI/CD**. -1. Expand the **Runners** section. +1. On the top bar, select **Menu > Projects** and find the project where you want to enable the runner. +1. On the left sidebar, select **Settings > CI/CD**. +1. Expand **Runners**. 1. Find the specific runner you want to lock or unlock. Make sure it's enabled. You cannot lock shared or group runners. 1. Select **Edit** (**{pencil}**). -1. Check the **Lock to current projects** option. +1. Select the **Lock to current projects** checkbox. 1. Select **Save changes**. diff --git a/doc/ci/runners/saas/linux_saas_runner.md b/doc/ci/runners/saas/linux_saas_runner.md index f1e28f7e74d..e96e89b47e5 100644 --- a/doc/ci/runners/saas/linux_saas_runner.md +++ b/doc/ci/runners/saas/linux_saas_runner.md @@ -24,7 +24,7 @@ The `gitlab-shared-runners-manager-X.gitlab.com` fleet of runners are dedicated Jobs handled by shared runners on GitLab.com (`shared-runners-manager-X.gitlab.com`) **time out after 3 hours**, regardless of the timeout configured in a -project. Check issue [#4010](https://gitlab.com/gitlab-com/infrastructure/-/issues/4010) and [#4070](https://gitlab.com/gitlab-com/infrastructure/-/issues/4070) for the reference. +project. Check issue [#4010](https://gitlab.com/gitlab-com/gl-infra/reliability/-/issues/4010) and [#4070](https://gitlab.com/gitlab-com/gl-infra/reliability/-/issues/4070) for the reference. Jobs handled by shared runners on Windows and macOS on GitLab.com **time out after 1 hour** while this service is in the [Beta](../../../policy/alpha-beta-support.md#beta-features) stage. diff --git a/doc/ci/runners/saas/windows_saas_runner.md b/doc/ci/runners/saas/windows_saas_runner.md index dddb3afee7c..f9fe6290220 100644 --- a/doc/ci/runners/saas/windows_saas_runner.md +++ b/doc/ci/runners/saas/windows_saas_runner.md @@ -126,8 +126,7 @@ test: ## Limitations and known issues -- All the limitations mentioned in our [beta - definition](../../../policy/alpha-beta-support.md#beta-features). +- All the limitations mentioned in our [beta definition](../../../policy/alpha-beta-support.md#beta-features). - The average provisioning time for a new Windows VM is 5 minutes. This means that you may notice slower build start times on the Windows runner fleet during the beta. In a future diff --git a/doc/ci/secrets/index.md b/doc/ci/secrets/index.md index ba395108966..fb91aeb6240 100644 --- a/doc/ci/secrets/index.md +++ b/doc/ci/secrets/index.md @@ -88,7 +88,7 @@ To configure your Vault server: - `VAULT_SERVER_URL` - The URL of your Vault server, such as `https://vault.example.com:8200`. Required. - `VAULT_AUTH_ROLE` - Optional. The role to use when attempting to authenticate. - If no role is specified, Vault uses the [default role](https://www.vaultproject.io/api/auth/jwt#default_role) + If no role is specified, Vault uses the [default role](https://www.vaultproject.io/api-docs/auth/jwt#default_role) specified when the authentication method was configured. - `VAULT_AUTH_PATH` - Optional. The path where the authentication method is mounted, default is `jwt`. - `VAULT_NAMESPACE` - Optional. The [Vault Enterprise namespace](https://www.vaultproject.io/docs/enterprise/namespaces) to use for reading secrets and authentication. @@ -183,7 +183,7 @@ For a full list of `CI_JOB_JWT` claims, read the You can also specify some attributes for the resulting Vault tokens, such as time-to-live, IP address range, and number of uses. The full list of options is available in -[Vault's documentation on creating roles](https://www.vaultproject.io/api/auth/jwt#create-role) +[Vault's documentation on creating roles](https://www.vaultproject.io/api-docs/auth/jwt#create-role) for the JSON web token method. ## Using a self-signed Vault server diff --git a/doc/ci/secure_files/index.md b/doc/ci/secure_files/index.md index fb421ab8944..8e141c62ef6 100644 --- a/doc/ci/secure_files/index.md +++ b/doc/ci/secure_files/index.md @@ -36,7 +36,7 @@ To add a secure file to a project: 1. On the top bar, select **Menu > Projects** and find your project. 1. On the left sidebar, select **Settings > CI/CD**. -1. In the **Secure Files** section, select **Manage**. +1. In the **Secure Files** section, select **Expand**. 1. Select **Upload File**. 1. Find the file to upload, select **Open**, and the file upload begins immediately. The file shows up in the list when the upload is complete. diff --git a/doc/ci/testing/load_performance_testing.md b/doc/ci/testing/load_performance_testing.md index ae177958beb..e15b3944c2b 100644 --- a/doc/ci/testing/load_performance_testing.md +++ b/doc/ci/testing/load_performance_testing.md @@ -121,7 +121,7 @@ the k6 test. NOTE: For Kubernetes setups a different template should be used: [`Jobs/Load-Performance-Testing.gitlab-ci.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/ci/templates/Jobs/Load-Performance-Testing.gitlab-ci.yml). -k6 has [various options](https://k6.io/docs/using-k6/options) to configure how it will run tests, such as what throughput (RPS) to run with, +k6 has [various options](https://k6.io/docs/using-k6/k6-options/reference/) to configure how it will run tests, such as what throughput (RPS) to run with, how long the test should run, and so on. Almost all options can be configured in the test itself, but as you can also pass command line options via the `K6_OPTIONS` variable. diff --git a/doc/ci/testing/unit_test_report_examples.md b/doc/ci/testing/unit_test_report_examples.md index a54deb254b7..b49ac29be65 100644 --- a/doc/ci/testing/unit_test_report_examples.md +++ b/doc/ci/testing/unit_test_report_examples.md @@ -250,7 +250,7 @@ test: This example uses [PHPUnit](https://phpunit.de/) with the `--log-junit` flag. You can also add this option using -[XML](https://phpunit.readthedocs.io/en/stable/configuration.html#the-junit-element) +[XML](https://phpunit.readthedocs.io/en/9.5/configuration.html#the-junit-element) in the `phpunit.xml` configuration file. ```yaml diff --git a/doc/ci/testing/unit_test_reports.md b/doc/ci/testing/unit_test_reports.md index c8e0d6135df..6294996c703 100644 --- a/doc/ci/testing/unit_test_reports.md +++ b/doc/ci/testing/unit_test_reports.md @@ -46,24 +46,44 @@ comparing the head and base branch's JUnit report format XML files, where: - The base branch is the target branch (usually the default branch). - The head branch is the source branch (the latest pipeline in each merge request). -The reports panel has a summary showing how many tests failed, how many had errors +The **Test summary** panel shows how many tests failed, how many had errors, and how many were fixed. If no comparison can be done because data for the base branch -is not available, the panel just shows the list of failed tests for head. +is not available, the panel shows only the list of failed tests for the source branch. -There are four types of results: +The types of results are: -1. **Newly failed tests:** Test cases which passed on base branch and failed on head branch -1. **Newly encountered errors:** Test cases which passed on base branch and failed due to a - test error on head branch -1. **Existing failures:** Test cases which failed on base branch and failed on head branch -1. **Resolved failures:** Test cases which failed on base branch and passed on head branch +- **Newly failed tests:** Test cases which passed on the base branch and failed on the head branch. +- **Newly encountered errors:** Test cases which passed on the base branch and failed due to a + test error on the head branch. +- **Existing failures:** Test cases which failed on the base branch and failed on the head branch. +- **Resolved failures:** Test cases which failed on the base branch and passed on the head branch. -Each entry in the panel shows the test name and its type from the list -above. Clicking on the test name opens a modal window with details of its -execution time and the error output. +### View failed tests + +Each entry in the **Test summary** panel shows the test name and result type. +Select the test name to open a modal window with details of its execution time and +the error output.  +#### Copy failed test names + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/91552) in GitLab 15.2. + +You can copy the name and path of failed tests when there are failed tests listed +in the **Test summary** panel. Use name and path to find and rerun the +test locally for verification. + +To copy the name of all failed tests, at the top of the **Test summary** panel, +select **Copy failed tests**. The failed tests are listed as a string with the tests +separated by spaces. + +To copy the name of a single failed test: + +1. Expand the **Test summary** panel by selecting **Show test summary details** (**{chevron-lg-down}**). +1. Select the test you want to review. +1. Select **Copy test name to rerun locally** (**{copy-to-clipboard}**). + ### Number of recent failures > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/241759) in merge requests in GitLab 13.7. diff --git a/doc/ci/variables/predefined_variables.md b/doc/ci/variables/predefined_variables.md index 6df614c1cda..f8900501244 100644 --- a/doc/ci/variables/predefined_variables.md +++ b/doc/ci/variables/predefined_variables.md @@ -82,7 +82,7 @@ as it can cause the pipeline to behave unexpectedly. | `CI_PAGES_URL` | 11.8 | all | The URL for a GitLab Pages site. Always a subdomain of `CI_PAGES_DOMAIN`. | | `CI_PIPELINE_ID` | 8.10 | all | The instance-level ID of the current pipeline. This ID is unique across all projects on the GitLab instance. | | `CI_PIPELINE_IID` | 11.0 | all | The project-level IID (internal ID) of the current pipeline. This ID is unique only within the current project. | -| `CI_PIPELINE_SOURCE` | 10.0 | all | How the pipeline was triggered. Can be `push`, `web`, `schedule`, `api`, `external`, `chat`, `webide`, `merge_request_event`, `external_pull_request_event`, `parent_pipeline`, [`trigger`, or `pipeline`](../triggers/index.md#configure-cicd-jobs-to-run-in-triggered-pipelines). | +| `CI_PIPELINE_SOURCE` | 10.0 | all | How the pipeline was triggered. Can be `push`, `web`, `schedule`, `api`, `external`, `chat`, `webide`, `merge_request_event`, `external_pull_request_event`, `parent_pipeline`, [`trigger`, or `pipeline`](../triggers/index.md#configure-cicd-jobs-to-run-in-triggered-pipelines). For a description of each value, see [Common `if` clauses for `rules`](../jobs/job_control.md#common-if-clauses-for-rules), which uses this variable to control when jobs run. | | `CI_PIPELINE_TRIGGERED` | all | all | `true` if the job was [triggered](../triggers/index.md). | | `CI_PIPELINE_URL` | 11.1 | 0.5 | The URL for the pipeline details. | | `CI_PIPELINE_CREATED_AT` | 13.10 | all | The UTC datetime when the pipeline was created, in [ISO 8601](https://tools.ietf.org/html/rfc3339#appendix-A) format. | @@ -93,7 +93,7 @@ as it can cause the pipeline to behave unexpectedly. | `CI_PROJECT_NAMESPACE` | 8.10 | 0.5 | The project namespace (username or group name) of the job. | | `CI_PROJECT_PATH_SLUG` | 9.3 | all | `$CI_PROJECT_PATH` in lowercase with characters that are not `a-z` or `0-9` replaced with `-` and shortened to 63 bytes. Use in URLs and domain names. | | `CI_PROJECT_PATH` | 8.10 | 0.5 | The project namespace with the project name included. | -| `CI_PROJECT_REPOSITORY_LANGUAGES` | 12.3 | all | A comma-separated, lowercase list of the languages used in the repository. For example `ruby,javascript,html,css`. | +| `CI_PROJECT_REPOSITORY_LANGUAGES` | 12.3 | all | A comma-separated, lowercase list of the languages used in the repository. For example `ruby,javascript,html,css`. The maximum number of languages is limited to 5. An issue [proposes to increase the limit](https://gitlab.com/gitlab-org/gitlab/-/issues/368925). | | `CI_PROJECT_ROOT_NAMESPACE` | 13.2 | 0.5 | The root project namespace (username or group name) of the job. For example, if `CI_PROJECT_NAMESPACE` is `root-group/child-group/grandchild-group`, `CI_PROJECT_ROOT_NAMESPACE` is `root-group`. | | `CI_PROJECT_TITLE` | 12.4 | all | The human-readable project name as displayed in the GitLab web interface. | | `CI_PROJECT_DESCRIPTION` | 15.1 | all | The project description as displayed in the GitLab web interface. | @@ -124,6 +124,7 @@ as it can cause the pipeline to behave unexpectedly. | `CI_SERVER_VERSION` | all | all | The full version of the GitLab instance. | | `CI_SERVER` | all | all | Available for all jobs executed in CI/CD. `yes` when available. | | `CI_SHARED_ENVIRONMENT` | all | 10.1 | Only available if the job is executed in a shared environment (something that is persisted across CI/CD invocations, like the `shell` or `ssh` executor). `true` when available. | +| `CI_TEMPLATE_REGISTRY_HOST` | 15.3 | all | The host of the registry used by CI/CD templates. Defaults to `registry.gitlab.com`. | | `GITLAB_CI` | all | all | Available for all jobs executed in CI/CD. `true` when available. | | `GITLAB_FEATURES` | 10.6 | all | The comma-separated list of licensed features available for the GitLab instance and license. | | `GITLAB_USER_EMAIL` | 8.12 | all | The email of the user who started the job. | diff --git a/doc/ci/variables/where_variables_can_be_used.md b/doc/ci/variables/where_variables_can_be_used.md index 7a1ef61f109..db83f2a14f3 100644 --- a/doc/ci/variables/where_variables_can_be_used.md +++ b/doc/ci/variables/where_variables_can_be_used.md @@ -40,6 +40,7 @@ There are two places defined variables can be used. On the: | [`services:name`](../yaml/index.md#services) | yes | Runner | The variable expansion is made by GitLab Runner's [internal variable expansion mechanism](#gitlab-runner-internal-variable-expansion-mechanism). | | [`services`](../yaml/index.md#services) | yes | Runner | The variable expansion is made by GitLab Runner's [internal variable expansion mechanism](#gitlab-runner-internal-variable-expansion-mechanism). | | [`tags`](../yaml/index.md#tags) | yes | GitLab | The variable expansion is made by the [internal variable expansion mechanism](#gitlab-internal-variable-expansion-mechanism) in GitLab. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/35742) in GitLab 14.1. | +| [`trigger` and `trigger:project`](../yaml/index.md#trigger) | yes | GitLab | The variable expansion is made by the [internal variable expansion mechanism](#gitlab-internal-variable-expansion-mechanism) in GitLab. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/367660) in GitLab 15.3. | | [`variables`](../yaml/index.md#variables) | yes | GitLab/Runner | The variable expansion is first made by the [internal variable expansion mechanism](#gitlab-internal-variable-expansion-mechanism) in GitLab, and then any unrecognized or unavailable variables are expanded by GitLab Runner's [internal variable expansion mechanism](#gitlab-runner-internal-variable-expansion-mechanism). | ### `config.toml` file @@ -131,10 +132,17 @@ These restrictions exist because `after_script` scripts are executed in a ## Persisted variables -The following variables are known as "persisted": +Some predefined variables are called "persisted". + +Pipeline-level persisted variables: - `CI_PIPELINE_ID` +- `CI_PIPELINE_URL` + +Job-level persisted variables: + - `CI_JOB_ID` +- `CI_JOB_URL` - `CI_JOB_TOKEN` - `CI_JOB_STARTED_AT` - `CI_REGISTRY_USER` @@ -143,7 +151,7 @@ The following variables are known as "persisted": - `CI_DEPLOY_USER` - `CI_DEPLOY_PASSWORD` -They are: +Persisted variables are: - Supported for definitions where the ["Expansion place"](#gitlab-ciyml-file) is: - Runner. @@ -152,6 +160,9 @@ They are: - For definitions where the ["Expansion place"](#gitlab-ciyml-file) is GitLab. - In the `only`, `except`, and `rules` [variables expressions](../jobs/job_control.md#cicd-variable-expressions). +[Pipeline trigger jobs](../yaml/index.md#trigger) cannot use job-level persisted variables, +but can use pipeline-level persisted variables. + Some of the persisted variables contain tokens and cannot be used by some definitions due to security reasons. diff --git a/doc/ci/yaml/artifacts_reports.md b/doc/ci/yaml/artifacts_reports.md index 379a4b3224a..61ef8bbfab7 100644 --- a/doc/ci/yaml/artifacts_reports.md +++ b/doc/ci/yaml/artifacts_reports.md @@ -329,3 +329,27 @@ GitLab can display the results of one or more reports in the merge request [terraform widget](../../user/infrastructure/iac/mr_integration.md#output-terraform-plan-information-into-a-merge-request). For more information, see [Output `terraform plan` information into a merge request](../../user/infrastructure/iac/mr_integration.md). + +## `artifacts:reports:cyclonedx` + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/360766) in GitLab 15.3 + +This report is a Software Bill of Materials describing the components of a project +following the [cyclonedx](https://cyclonedx.org/docs/1.4) protocol format. + +You can specify multiple cyclonedx reports per job. These can be either supplied +as a list of filenames, a filename pattern, or both: + +- List of filenames: `cyclonedx: [gl-sbom-npm-npm.cdx.json, gl-sbom-bundler-gem.cdx.json]`. +- A filename pattern: `cyclonedx: gl-sbom-*.json`. +- Combination of both of the above: `cyclonedx: [gl-sbom-*.json, my-cyclonedx.json]`. + +Below is an example of a job exposing cyclonedx artifacts: + +```yaml +artifacts: + reports: + cyclonedx: + - gl-sbom-npm-npm.cdx.json + - gl-sbom-bundler-gem.cdx.json +``` diff --git a/doc/ci/yaml/includes.md b/doc/ci/yaml/includes.md index 838710834ba..5158f80ea64 100644 --- a/doc/ci/yaml/includes.md +++ b/doc/ci/yaml/includes.md @@ -289,10 +289,10 @@ smoke-test-job: In `include` sections in your `.gitlab-ci.yml` file, you can use: -- [Project variables](../variables/index.md#add-a-cicd-variable-to-a-project) -- [Group variables](../variables/index.md#add-a-cicd-variable-to-a-group) -- [Instance variables](../variables/index.md#add-a-cicd-variable-to-an-instance) -- Project [predefined variables](../variables/predefined_variables.md) +- [Project variables](../variables/index.md#add-a-cicd-variable-to-a-project). +- [Group variables](../variables/index.md#add-a-cicd-variable-to-a-group). +- [Instance variables](../variables/index.md#add-a-cicd-variable-to-an-instance). +- Project [predefined variables](../variables/predefined_variables.md). - In GitLab 14.2 and later, the `$CI_COMMIT_REF_NAME` [predefined variable](../variables/predefined_variables.md). When used in `include`, the `CI_COMMIT_REF_NAME` variable returns the full @@ -322,7 +322,11 @@ include: file: '.compliance-gitlab-ci.yml' ``` -For an example of how you can include these predefined variables, and the variables' impact on CI/CD jobs, +You cannot use variables defined in jobs, or in a global [`variables`](../yaml/index.md#variables) +section which defines the default variables for all jobs. Includes are evaluated before jobs, +so these variables cannot be used with `include`. + +For an example of how you can include predefined variables, and the variables' impact on CI/CD jobs, see this [CI/CD variable demo](https://youtu.be/4XR8gw3Pkos). ## Use `rules` with `include` diff --git a/doc/ci/yaml/index.md b/doc/ci/yaml/index.md index 3bb2007d6e3..0b434e7013c 100644 --- a/doc/ci/yaml/index.md +++ b/doc/ci/yaml/index.md @@ -171,11 +171,11 @@ Use `include:local` instead of symbolic links. **Possible inputs**: -- A full path relative to the root directory (`/`). +A full path relative to the root directory (`/`): + - The YAML file must have the extension `.yml` or `.yaml`. - You can [use `*` and `**` wildcards in the file path](includes.md#use-includelocal-with-wildcard-file-paths). - -CI/CD variables [are supported](../variables/where_variables_can_be_used.md#gitlab-ciyml-file). +- You can use [certain CI/CD variables](includes.md#use-variables-with-include). **Example of `include:local`**: @@ -208,10 +208,10 @@ use `include:file`. You can use `include:file` in combination with `include:proj **Possible inputs**: -- A full path, relative to the root directory (`/`). The YAML file must have the - extension `.yml` or `.yaml`. +A full path, relative to the root directory (`/`): -CI/CD variables [are supported](../variables/where_variables_can_be_used.md#gitlab-ciyml-file). +- The YAML file must have the extension `.yml` or `.yaml`. +- You can use [certain CI/CD variables](includes.md#use-variables-with-include). **Example of `include:file`**: @@ -268,10 +268,11 @@ Use `include:remote` with a full URL to include a file from a different location **Possible inputs**: -- A public URL accessible by an HTTP/HTTPS `GET` request. Authentication with the - remote URL is not supported. The YAML file must have the extension `.yml` or `.yaml`. +A public URL accessible by an HTTP/HTTPS `GET` request: -CI/CD variables [are supported](../variables/where_variables_can_be_used.md#gitlab-ciyml-file). +- Authentication with the remote URL is not supported. +- The YAML file must have the extension `.yml` or `.yaml`. +- You can use [certain CI/CD variables](includes.md#use-variables-with-include). **Example of `include:remote`**: @@ -296,9 +297,12 @@ Use `include:template` to include [`.gitlab-ci.yml` templates](https://gitlab.co **Possible inputs**: -- [`.gitlab-ci.yml` templates](https://gitlab.com/gitlab-org/gitlab/-/tree/master/lib/gitlab/ci/templates). +A [CI/CD template](../examples/index.md#cicd-templates): -CI/CD variables [are supported](../variables/where_variables_can_be_used.md#gitlab-ciyml-file). +- Templates are stored in [`lib/gitlab/ci/templates`](https://gitlab.com/gitlab-org/gitlab/-/tree/master/lib/gitlab/ci/templates). + Not all templates are designed to be used with `include:template`, so check template + comments before using one. +- You can use [certain CI/CD variables](includes.md#use-variables-with-include). **Example of `include:template`**: @@ -2136,7 +2140,7 @@ This example creates four paths of execution: - The maximum number of jobs that a single job can have in the `needs` array is limited: - For GitLab.com, the limit is 50. For more information, see our - [infrastructure issue](https://gitlab.com/gitlab-com/gl-infra/infrastructure/-/issues/7541). + [infrastructure issue](https://gitlab.com/gitlab-com/gl-infra/reliability/-/issues/7541). - For self-managed instances, the default limit is 50. This limit [can be changed](../../administration/cicd.md#set-the-needs-job-limit). - If `needs` refers to a job that uses the [`parallel`](#parallel) keyword, it depends on all jobs created in parallel, not just one job. It also downloads @@ -2784,11 +2788,15 @@ which must be in the `$PATH`. If you use the [Docker executor](https://docs.gitlab.com/runner/executors/docker.html), you can use this image from the GitLab Container Registry: `registry.gitlab.com/gitlab-org/release-cli:latest` +If you use the [Shell executor](https://docs.gitlab.com/runner/executors/shell.html) or similar, +[install `release-cli`](../../user/project/releases/release_cli.md) on the server where the runner is registered. + **Keyword type**: Job keyword. You can use it only as part of a job. **Possible inputs**: The `release` subkeys: - [`tag_name`](#releasetag_name) +- [`tag_message`](#releasetag_message) (optional) - [`name`](#releasename) (optional) - [`description`](#releasedescription) - [`ref`](#releaseref) (optional) @@ -2832,8 +2840,6 @@ This example creates a release: - The `release` section executes after the `script` keyword and before the `after_script`. - A release is created only if the job's main script succeeds. - If the release already exists, it is not updated and the job with the `release` keyword fails. -- If you use the [Shell executor](https://docs.gitlab.com/runner/executors/shell.html) or similar, - [install `release-cli`](../../user/project/releases/release_cli.md) on the server where the runner is registered. **Related topics**: @@ -2887,6 +2893,30 @@ job: - if: $CI_PIPELINE_SOURCE == "schedule" ``` +#### `release:tag_message` + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/363024) in GitLab 15.3. Supported by `release-cli` v0.12.0 or later. + +If the tag does not exist, the newly created tag is annotated with the message specifed by `tag_message`. +If omitted, a lightweight tag is created. + +**Keyword type**: Job keyword. You can use it only as part of a job. + +**Possible inputs**: + +- A text string. + +**Example of `release:tag_message`**: + +```yaml + release_job: + stage: release + release: + tag_name: $CI_COMMIT_TAG + description: 'Release description' + tag_message: 'Annotated tag message' +``` + #### `release:name` The release name. If omitted, it is populated with the value of `release: tag_name`. @@ -2965,7 +2995,7 @@ released_at: '2021-03-15T08:00:00Z' > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/271454) in GitLab 13.12. -Use `release:assets:links` to include [asset links](../../user/project/releases/index.md#release-assets) in the release. +Use `release:assets:links` to include [asset links](../../user/project/releases/release_fields.md#release-assets) in the release. Requires `release-cli` version v0.4.0 or later. @@ -3221,6 +3251,7 @@ branch or merge request pipelines. **Possible inputs**: - An array of file paths. In GitLab 13.6 and later, [file paths can include variables](../jobs/job_control.md#variables-in-ruleschanges). +- Alternatively, the array of file paths can be in [`rules:changes:paths`](#ruleschangespaths). **Example of `rules:changes`**: @@ -3239,6 +3270,8 @@ docker build: - If `Dockerfile` has changed, add the job to the pipeline as a manual job, and the pipeline continues running even if the job is not triggered (`allow_failure: true`). - If `Dockerfile` has not changed, do not add job to any pipeline (same as `when: never`). +- [`rules:changes:paths`](#ruleschangespaths) is the same as `rules:changes` without + any subkeys. **Additional details**: @@ -3250,6 +3283,74 @@ docker build: - [Jobs or pipelines can run unexpectedly when using `rules: changes`](../jobs/job_control.md#jobs-or-pipelines-run-unexpectedly-when-using-changes). +##### `rules:changes:paths` + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/90171) in GitLab 15.2. + +Use `rules:changes` to specify that a job only be added to a pipeline when specific +files are changed, and use `rules:changes:paths` to specify the files. + +`rules:changes:paths` is the same as using [`rules:changes`](#ruleschanges) without +any subkeys. All additional details and related topics are the same. + +**Keyword type**: Job keyword. You can use it only as part of a job. + +**Possible inputs**: + +- An array of file paths. In GitLab 13.6 and later, [file paths can include variables](../jobs/job_control.md#variables-in-ruleschanges). + +**Example of `rules:changes:paths`**: + +```yaml +docker-build-1: + script: docker build -t my-image:$CI_COMMIT_REF_SLUG . + rules: + - if: $CI_PIPELINE_SOURCE == "merge_request_event" + changes: + - Dockerfile + +docker-build-2: + script: docker build -t my-image:$CI_COMMIT_REF_SLUG . + rules: + - if: $CI_PIPELINE_SOURCE == "merge_request_event" + changes: + paths: + - Dockerfile +``` + +In this example, both jobs have the same behavior. + +##### `rules:changes:compare_to` + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/293645) in GitLab 15.3 [with a flag](../../administration/feature_flags.md) named `ci_rules_changes_compare`. Enabled by default. + +Use `rules:changes:compare_to` to specify which ref to compare against for changes to the files +listed under [`rules:changes:paths`](#ruleschangespaths). + +**Keyword type**: Job keyword. You can use it only as part of a job, and it must be combined with `rules:changes:paths`. + +**Possible inputs**: + +- A branch name, like `main`, `branch1`, or `refs/heads/branch1`. +- A tag name, like `tag1` or `refs/tags/tag1`. +- A commit SHA, like `2fg31ga14b`. + +**Example of `rules:changes:compare_to`**: + +```yaml +docker build: + script: docker build -t my-image:$CI_COMMIT_REF_SLUG . + rules: + - if: $CI_PIPELINE_SOURCE == "merge_request_event" + changes: + paths: + - Dockerfile + compare_to: 'refs/heads/branch1' +``` + +In this example, the `docker build` job is only included when the `Dockerfile` has changed +relative to `refs/heads/branch1` and the pipeline source is a merge request event. + #### `rules:exists` > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/24021) in GitLab 12.4. @@ -3792,7 +3893,9 @@ Use `trigger` to start a downstream pipeline that is either: **Possible inputs**: -- For multi-project pipelines, path to the downstream project. +- For multi-project pipelines, path to the downstream project. CI/CD variables + [are supported](../variables/where_variables_can_be_used.md#gitlab-ciyml-file) + in GitLab 15.3 and later. - For child pipelines, path to the child pipeline CI/CD configuration file. **Example of `trigger` for multi-project pipeline**: @@ -3830,6 +3933,8 @@ trigger_job: and [scheduled pipeline variables](../pipelines/schedules.md#add-a-pipeline-schedule) are not passed to downstream pipelines by default. Use [trigger:forward](#triggerforward) to forward these variables to downstream pipelines. +- [Job-level persisted variables](../variables/where_variables_can_be_used.md#persisted-variables) + are not available in trigger jobs. **Related topics**: @@ -3861,6 +3966,17 @@ trigger_job: In this example, jobs from subsequent stages wait for the triggered pipeline to successfully complete before starting. +**Additional details**: + +- [Optional manual jobs](../jobs/job_control.md#types-of-manual-jobs) in the downstream pipeline + do not affect the status of the downstream pipeline or the upstream trigger job. + The downstream pipeline can complete successfully without running any optional manual jobs. +- [Blocking manual jobs](../jobs/job_control.md#types-of-manual-jobs) in the downstream pipeline + must run before the trigger job is marked as successful or failed. The trigger job + shows **pending** (**{status_pending}**) if the downstream pipeline status is + **waiting for manual action** (**{status_manual}**) due to manual jobs. By default, + jobs in later stages do not start until the trigger job completes. + #### `trigger:forward` > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/213729) in GitLab 14.9 [with a flag](../../administration/feature_flags.md) named `ci_trigger_forward_variables`. Disabled by default. diff --git a/doc/ci/yaml/script.md b/doc/ci/yaml/script.md index 4bffcbca1cc..f1cdcf57e64 100644 --- a/doc/ci/yaml/script.md +++ b/doc/ci/yaml/script.md @@ -258,3 +258,27 @@ pages-job: script: - 'curl --header "PRIVATE-TOKEN: ${PRIVATE_TOKEN}" "https://gitlab.example.com/api/v4/projects"' ``` + +### Job does not fail when using `&&` in a script + +If you use `&&` to combine two commands together in a single script line, the job +might return as successful, even if one of the commands failed. For example: + +```yaml +job-does-not-fail: + script: + - invalid-command xyz && invalid-command abc + - echo $? + - echo "The job should have failed already, but this is executed unexpectedly." +``` + +The `&&` operator returns an exit code of `0` even though the two commands failed, +and the job continues to run. To force the script to exit when either command fails, +enclose the entire line in parentheses: + +```yaml +job-fails: + script: + - (invalid-command xyz && invalid-command abc) + - echo "The job failed already, and this is not executed." +``` |
