diff options
Diffstat (limited to 'doc/ci')
63 files changed, 1053 insertions, 828 deletions
diff --git a/doc/ci/chatops/index.md b/doc/ci/chatops/index.md index 3354c94aff2..f0efb5fc884 100644 --- a/doc/ci/chatops/index.md +++ b/doc/ci/chatops/index.md @@ -121,6 +121,6 @@ important to describe those, too. Think of things that may go wrong and include This is important to minimize requests for support, and to avoid doc comments with questions that you know someone might ask. -Each scenario can be a third-level heading, e.g. `### Getting error message X`. +Each scenario can be a third-level heading, for example `### Getting error message X`. If you have none to add when creating a doc, leave this section in place but commented out to help encourage others to add to it in the future. --> diff --git a/doc/ci/ci_cd_for_external_repos/bitbucket_integration.md b/doc/ci/ci_cd_for_external_repos/bitbucket_integration.md index 6256f11f1ea..a0665e1c054 100644 --- a/doc/ci/ci_cd_for_external_repos/bitbucket_integration.md +++ b/doc/ci/ci_cd_for_external_repos/bitbucket_integration.md @@ -25,11 +25,11 @@ To use GitLab CI/CD with a Bitbucket Cloud repository: - You can generate and use a [Bitbucket App Password](https://support.atlassian.com/bitbucket-cloud/docs/app-passwords/) for the password field. GitLab imports the repository and enables [Pull Mirroring](../../user/project/repository/mirror/pull.md). - You can check that mirroring is working in the project by going to **Settings > Repository > Mirroring repositories**. + You can check that mirroring is working in the project in **Settings > Repository > Mirroring repositories**. 1. In GitLab, create a [Personal Access Token](../../user/profile/personal_access_tokens.md) - with `api` scope. This is used to authenticate requests from the web + with `api` scope. The token is used to authenticate requests from the web hook that is created in Bitbucket to notify GitLab of new commits. 1. In Bitbucket, from **Settings > Webhooks**, create a new web hook to notify @@ -58,18 +58,14 @@ To use GitLab CI/CD with a Bitbucket Cloud repository: 1. In GitLab, from **Settings > CI/CD > Variables**, add variables to allow communication with Bitbucket via the Bitbucket API: - `BITBUCKET_ACCESS_TOKEN`: the Bitbucket app password created above. + - `BITBUCKET_ACCESS_TOKEN`: The Bitbucket app password created above. This variable should be [masked](../variables/index.md#mask-a-cicd-variable). + - `BITBUCKET_USERNAME`: The username of the Bitbucket account. + - `BITBUCKET_NAMESPACE`: Set this variable if your GitLab and Bitbucket namespaces differ. + - `BITBUCKET_REPOSITORY`: Set this variable if your GitLab and Bitbucket project names differ. - `BITBUCKET_USERNAME`: the username of the Bitbucket account. - - `BITBUCKET_NAMESPACE`: set this if your GitLab and Bitbucket namespaces differ. - - `BITBUCKET_REPOSITORY`: set this if your GitLab and Bitbucket project names differ. - -1. In Bitbucket, add a script to push the pipeline status to Bitbucket. - - NOTE: - The changes must be made in Bitbucket as any changes in the GitLab repository are overwritten by Bitbucket when GitLab next mirrors the repository. +1. In Bitbucket, add a script that pushes the pipeline status to Bitbucket. The script + is created in Bitbucket, but the mirroring process copies it to the GitLab mirror. The GitLab + CI/CD pipeline runs the script, and pushes the status back to Bitbucket. Create a file `build_status` and insert the script below and run `chmod +x build_status` in your terminal to make the script executable. @@ -125,7 +121,8 @@ To use GitLab CI/CD with a Bitbucket Cloud repository: ``` 1. In Bitbucket, create a `.gitlab-ci.yml` file to use the script to push - pipeline success and failures to Bitbucket. + pipeline success and failures to Bitbucket. Similar to the script added above, + this file is copied to the GitLab repo as part of the mirroring process. ```yaml stages: @@ -168,6 +165,6 @@ important to describe those, too. Think of things that may go wrong and include This is important to minimize requests for support, and to avoid doc comments with questions that you know someone might ask. -Each scenario can be a third-level heading, e.g. `### Getting error message X`. +Each scenario can be a third-level heading, for example `### Getting error message X`. If you have none to add when creating a doc, leave this section in place but commented out to help encourage others to add to it in the future. --> diff --git a/doc/ci/ci_cd_for_external_repos/github_integration.md b/doc/ci/ci_cd_for_external_repos/github_integration.md index 18cc430c225..9933fafcb69 100644 --- a/doc/ci/ci_cd_for_external_repos/github_integration.md +++ b/doc/ci/ci_cd_for_external_repos/github_integration.md @@ -98,6 +98,6 @@ important to describe those, too. Think of things that may go wrong and include This is important to minimize requests for support, and to avoid doc comments with questions that you know someone might ask. -Each scenario can be a third-level heading, e.g. `### Getting error message X`. +Each scenario can be a third-level heading, for example `### Getting error message X`. If you have none to add when creating a doc, leave this section in place but commented out to help encourage others to add to it in the future. --> diff --git a/doc/ci/cloud_deployment/index.md b/doc/ci/cloud_deployment/index.md index 83774f63566..bd9276275a7 100644 --- a/doc/ci/cloud_deployment/index.md +++ b/doc/ci/cloud_deployment/index.md @@ -76,7 +76,7 @@ Prerequisites: - [Authenticate AWS with GitLab](#authenticate-gitlab-with-aws). - Create a cluster on Amazon ECS. -- Create related components, like an ECS service, a database on Amazon RDS, and so on. +- Create related components, like an ECS service or a database on Amazon RDS. - Create an ECS task definition, where the value for the `containerDefinitions[].name` attribute is the same as the `Container name` defined in your targeted ECS service. The task definition can be: - An existing task definition in ECS. diff --git a/doc/ci/cloud_services/azure/index.md b/doc/ci/cloud_services/azure/index.md index 944f95c03e2..b2f78648be9 100644 --- a/doc/ci/cloud_services/azure/index.md +++ b/doc/ci/cloud_services/azure/index.md @@ -16,7 +16,7 @@ Prerequisites: - Access to an existing Azure Subscription with `Owner` access level. - Access to the corresponding Azure Active Directory Tenant with at least the `Application Developer` access level. -- A local installation of the [Azure CLI](https://learn.microsoft.com/cli/azure/install-azure-cli). +- A local installation of the [Azure CLI](https://learn.microsoft.com/en-us/cli/azure/install-azure-cli). Alternatively, you can follow all the steps below with the [Azure Cloud Shell](https://shell.azure.com/). - A GitLab project. @@ -27,11 +27,11 @@ To complete this tutorial: 1. [Grant permissions for the service principal](#grant-permissions-for-the-service-principal). 1. [Retrieve a temporary credential](#retrieve-a-temporary-credential). -For more information, review Azure's documentation on [Workload identity federation](https://learn.microsoft.com/azure/active-directory/develop/workload-identity-federation). +For more information, review Azure's documentation on [Workload identity federation](https://learn.microsoft.com/en-us/azure/active-directory/develop/workload-identity-federation). ## Create Azure AD application and service principal -To create an [Azure AD application](https://learn.microsoft.com/cli/azure/ad/app?view=azure-cli-latest#az-ad-app-create) +To create an [Azure AD application](https://learn.microsoft.com/en-us/cli/azure/ad/app?view=azure-cli-latest#az-ad-app-create) and service principal: 1. In the Azure CLI, create the AD application: @@ -43,13 +43,13 @@ and service principal: Save the `appId` (Application client ID) output, as you need it later to configure your GitLab CI/CD pipeline. -1. Create a corresponding [Service Principal](https://learn.microsoft.com/cli/azure/ad/sp?view=azure-cli-latest#az-ad-sp-create): +1. Create a corresponding [Service Principal](https://learn.microsoft.com/en-us/cli/azure/ad/sp?view=azure-cli-latest#az-ad-sp-create): ```shell az ad sp create --id $appId --query appId -otsv ``` -Instead of the Azure CLI, you can [use the Azure Portal to create these resources](https://learn.microsoft.com/azure/active-directory/develop/howto-create-service-principal-portal). +Instead of the Azure CLI, you can [use the Azure Portal to create these resources](https://learn.microsoft.com/en-us/azure/active-directory/develop/howto-create-service-principal-portal). ## Create Azure AD federated identity credentials @@ -88,7 +88,7 @@ identity credentials from the Azure Portal: ## Grant permissions for the service principal -After you create the credentials, use [`role assignment`](https://learn.microsoft.com/cli/azure/role/assignment?view=azure-cli-latest#az-role-assignment-create) +After you create the credentials, use [`role assignment`](https://learn.microsoft.com/en-us/cli/azure/role/assignment?view=azure-cli-latest#az-role-assignment-create) to grant permissions to the above service principal to access to Azure resources: ```shell @@ -97,13 +97,13 @@ az role assignment create --assignee $appId --role Reader --scope /subscriptions You can find your subscription ID in: -- The [Azure Portal](https://learn.microsoft.com/azure/azure-portal/get-subscription-tenant-id#find-your-azure-subscription). -- The [Azure CLI](https://learn.microsoft.com/cli/azure/manage-azure-subscriptions-azure-cli#get-the-active-subscription). +- The [Azure Portal](https://learn.microsoft.com/en-us/azure/azure-portal/get-subscription-tenant-id#find-your-azure-subscription). +- The [Azure CLI](https://learn.microsoft.com/en-us/cli/azure/manage-azure-subscriptions-azure-cli#get-the-active-subscription). ## Retrieve a temporary credential After you configure the Azure AD application and federated identity credentials, -the CI/CD job can retrieve a temporary credential by using the [Azure CLI](https://learn.microsoft.com/cli/azure/reference-index?view=azure-cli-latest#az-login): +the CI/CD job can retrieve a temporary credential by using the [Azure CLI](https://learn.microsoft.com/en-us/cli/azure/reference-index?view=azure-cli-latest#az-login): ```yaml default: @@ -123,7 +123,7 @@ The CI/CD variables are: - `AZURE_CLIENT_ID`: The [application client ID you saved earlier](#create-azure-ad-application-and-service-principal). - `AZURE_TENANT_ID`: Your Azure Active Directory. You can - [find it by using the Azure CLI or Azure Portal](https://learn.microsoft.com/azure/active-directory/fundamentals/active-directory-how-to-find-tenant). + [find it by using the Azure CLI or Azure Portal](https://learn.microsoft.com/en-us/azure/active-directory/fundamentals/active-directory-how-to-find-tenant). - `CI_JOB_JWT_V2`: The JSON web token is a [predefined CI/CD variable](../../variables/predefined_variables.md). ## Troubleshooting diff --git a/doc/ci/directed_acyclic_graph/index.md b/doc/ci/directed_acyclic_graph/index.md index ded91edeff5..39f45471021 100644 --- a/doc/ci/directed_acyclic_graph/index.md +++ b/doc/ci/directed_acyclic_graph/index.md @@ -91,7 +91,7 @@ To see the needs visualization, select **Needs** when viewing a pipeline that us  -Clicking a node highlights all the job paths it depends on. +Selecting a node highlights all the job paths it depends on.  diff --git a/doc/ci/docker/using_docker_build.md b/doc/ci/docker/using_docker_build.md index 06197d71690..ea62133cb7f 100644 --- a/doc/ci/docker/using_docker_build.md +++ b/doc/ci/docker/using_docker_build.md @@ -47,7 +47,7 @@ the Docker commands, but needs permission to do so. ``` 1. On the server where GitLab Runner is installed, install Docker Engine. - View a list of [supported platforms](https://docs.docker.com/engine/installation/). + View a list of [supported platforms](https://docs.docker.com/engine/install/). 1. Add the `gitlab-runner` user to the `docker` group: @@ -829,7 +829,7 @@ environment = ["DOCKER_DRIVER=overlay2"] If you're running multiple runners, you have to modify all configuration files. Read more about the [runner configuration](https://docs.gitlab.com/runner/configuration/) -and [using the OverlayFS storage driver](https://docs.docker.com/engine/userguide/storagedriver/overlayfs-driver/). +and [using the OverlayFS storage driver](https://docs.docker.com/storage/storagedriver/overlayfs-driver/). ## Docker alternatives diff --git a/doc/ci/docker/using_docker_images.md b/doc/ci/docker/using_docker_images.md index 70680a44ed2..0ba510acdbc 100644 --- a/doc/ci/docker/using_docker_images.md +++ b/doc/ci/docker/using_docker_images.md @@ -64,7 +64,7 @@ For example, you can set the [Docker pull policy](https://docs.gitlab.com/runner to use local images. For more information about images and Docker Hub, see -the [Docker Fundamentals](https://docs.docker.com/engine/understanding-docker/) documentation. +the [Docker overview](https://docs.docker.com/get-started/overview/). ## Define `image` in the `.gitlab-ci.yml` file diff --git a/doc/ci/enable_or_disable_ci.md b/doc/ci/enable_or_disable_ci.md index adb9b5a87d5..e75f902c153 100644 --- a/doc/ci/enable_or_disable_ci.md +++ b/doc/ci/enable_or_disable_ci.md @@ -54,6 +54,6 @@ important to describe those, too. Think of things that may go wrong and include This is important to minimize requests for support, and to avoid doc comments with questions that you know someone might ask. -Each scenario can be a third-level heading, e.g. `### Getting error message X`. +Each scenario can be a third-level heading, for example `### Getting error message X`. If you have none to add when creating a doc, leave this section in place but commented out to help encourage others to add to it in the future. --> diff --git a/doc/ci/environments/deployment_approvals.md b/doc/ci/environments/deployment_approvals.md index 5f5ec027827..d7fa31b583b 100644 --- a/doc/ci/environments/deployment_approvals.md +++ b/doc/ci/environments/deployment_approvals.md @@ -64,10 +64,9 @@ co-exist and multiple approval rules takes the precedence over the unified appro There are two ways to configure approvals for a protected environment: -1. Using the [UI](protected_environments.md#protecting-environments) - 1. Set the **Required approvals** field to 1 or more. -1. Using the [REST API](../../api/protected_environments.md#protect-repository-environments) - 2. Set the `required_approval_count` field to 1 or more. +- Using the [UI](protected_environments.md#protecting-environments), set the **Required approvals** field to 1 or more. +- Using the [REST API](../../api/protected_environments.md#protect-a-single-environment), + set the `required_approval_count` field to 1 or more. After this is configured, all jobs deploying to this environment automatically go into a blocked state and wait for approvals before running. Ensure that the number of required approvals is less than the number of users allowed to deploy. @@ -89,9 +88,9 @@ Maintainer role. > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/345678) in GitLab 14.10 with a flag named `deployment_approval_rules`. Disabled by default. > - [Generally available](https://gitlab.com/gitlab-org/gitlab/-/issues/345678) in GitLab 15.0. [Feature flag `deployment_approval_rules`](https://gitlab.com/gitlab-org/gitlab/-/issues/345678) removed. -1. Using the [REST API](../../api/group_protected_environments.md#protect-an-environment). - 1. `deploy_access_levels` represents which entity can execute the deployment job. - 1. `approval_rules` represents which entity can approve the deployment job. +- Using the [REST API](../../api/group_protected_environments.md#protect-a-single-environment). + - `deploy_access_levels` represents which entity can execute the deployment job. + - `approval_rules` represents which entity can approve the deployment job. After this is configured, all jobs deploying to this environment automatically go into a blocked state and wait for approvals before running. Ensure that the number of required approvals is less than the number of users allowed to deploy. @@ -138,10 +137,6 @@ To approve or reject a deployment to a protected environment using the UI: 1. Optional. Add a comment which describes your reason for approving or rejecting the deployment. 1. Select **Approve** or **Reject**. -NOTE: -This feature might not work as expected when [Multiple approval rules](#multiple-approval-rules) is configured. -See the [issue](https://gitlab.com/gitlab-org/gitlab/-/issues/355708) for planned improvement. - ### Approve or reject a deployment using the API Prerequisites: @@ -191,6 +186,6 @@ important to describe those, too. Think of things that may go wrong and include This is important to minimize requests for support, and to avoid doc comments with questions that you know someone might ask. -Each scenario can be a third-level heading, e.g. `### Getting error message X`. +Each scenario can be a third-level heading, for example `### Getting error message X`. If you have none to add when creating a doc, leave this section in place but commented out to help encourage others to add to it in the future. --> diff --git a/doc/ci/environments/deployment_safety.md b/doc/ci/environments/deployment_safety.md index 665aeb23d28..1e4eb54c559 100644 --- a/doc/ci/environments/deployment_safety.md +++ b/doc/ci/environments/deployment_safety.md @@ -13,16 +13,20 @@ that help maintain deployment security and stability. You can: +- Set appropriate roles to your project. See [Project members permissions](../../user/permissions.md#project-members-permissions) + for the different user roles GitLab supports and the permissions of each. - [Restrict write-access to a critical environment](#restrict-write-access-to-a-critical-environment) - [Prevent deployments during deploy freeze windows](#prevent-deployments-during-deploy-freeze-windows) -- [Set appropriate roles to your project](#setting-appropriate-roles-to-your-project) - [Protect production secrets](#protect-production-secrets) - [Separate project for deployments](#separate-project-for-deployments) If you are using a continuous deployment workflow and want to ensure that concurrent deployments to the same environment do not happen, you should enable the following options: - [Ensure only one deployment job runs at a time](#ensure-only-one-deployment-job-runs-at-a-time) -- [Skip outdated deployment jobs](#skip-outdated-deployment-jobs) +- [Prevent outdated deployment jobs](#prevent-outdated-deployment-jobs) + +<i class="fa fa-youtube-play youtube" aria-hidden="true"></i> +For an overview, see [How to secure your CD pipelines/workflow](https://www.youtube.com/watch?v=Mq3C1KveDc0). ## Restrict write access to a critical environment @@ -63,7 +67,10 @@ The improved pipeline flow **after** using the resource group: For more information, see [Resource Group documentation](../resource_groups/index.md). -## Skip outdated deployment jobs +## Prevent outdated deployment jobs + +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/25276) in GitLab 12.9. +> - In GitLab 15.5, the behavior was [changed](https://gitlab.com/gitlab-org/gitlab/-/issues/363328) to prevent outdated job runs. The effective execution order of pipeline jobs can vary from run to run, which could cause undesired behavior. For example, a [deployment job](../jobs/index.md#deployment-jobs) @@ -71,22 +78,46 @@ in a newer pipeline could finish before a deployment job in an older pipeline. This creates a race condition where the older deployment finishes later, overwriting the "newer" deployment. -You can ensure that older deployment jobs are cancelled automatically when a newer deployment -job is started by enabling the [Skip outdated deployment jobs](../pipelines/settings.md#skip-outdated-deployment-jobs) feature. +You can prevent older deployment jobs from running when a newer deployment +job is started by enabling the [Prevent outdated deployment jobs](../pipelines/settings.md#prevent-outdated-deployment-jobs) feature. + +When an older deployment job starts, it fails and is labeled: + +- `failed outdated deployment job` in the pipeline view. +- `The deployment job is older than the latest deployment, and therefore failed.` + when viewing the completed job. + +When an older deployment job is manual, the play button is disabled with a message +`This deployment job does not run automatically and must be started manually, but it's older than the latest deployment, and therefore can't run.`. + +Job age is determined by the job start time, not the commit time, so a newer commit +can be prevented in some circumstances. + +### How to rollback to an outdated deployment -Example of a problematic pipeline flow **before** enabling Skip outdated deployment jobs: +> In GitLab 15.6, [rollback via job retry was introduced back](https://gitlab.com/gitlab-org/gitlab/-/issues/378359). + +In some cases, you need to rollback to an outdated deployment. +This feature explicitly allows rollback via [Environment Rollback](index.md#environment-rollback), +so that you can quickly rollback in an urgent case. + +Alternatively, you can run a new pipeline with a previous commit. It contains newer deployment jobs than the latest deployment. + +### Example + +Example of a problematic pipeline flow **before** enabling Prevent outdated deployment jobs: 1. Pipeline-A is created on the default branch. 1. Later, Pipeline-B is created on the default branch (with a newer commit SHA). 1. The `deploy` job in Pipeline-B finishes first, and deploys the newer code. 1. The `deploy` job in Pipeline-A finished later, and deploys the older code, **overwriting** the newer (latest) deployment. -The improved pipeline flow **after** enabling Skip outdated deployment jobs: +The improved pipeline flow **after** enabling Prevent outdated deployment jobs: 1. Pipeline-A is created on the default branch. 1. Later, Pipeline-B is created on the default branch (with a newer SHA). 1. The `deploy` job in Pipeline-B finishes first, and deploys the newer code. -1. The `deploy` job in Pipeline-A was automatically cancelled, so that it doesn't overwrite the deployment from the newer pipeline. +1. The `deploy` job in Pipeline-A fails, so that it doesn't overwrite the deployment from the newer pipeline. ## Prevent deployments during deploy freeze windows @@ -95,19 +126,6 @@ vacation period when most employees are out, you can set up a [Deploy Freeze](.. During a deploy freeze period, no deployment can be executed. This is helpful to ensure that deployments do not happen unexpectedly. -## Setting appropriate roles to your project - -GitLab supports several different roles that can be assigned to your project members. See -[Project members permissions](../../user/permissions.md#project-members-permissions) -for an explanation of these roles and the permissions of each. - -<div class="video-fallback"> - See the video: <a href="https://www.youtube.com/watch?v=Mq3C1KveDc0">How to secure your CD pipelines</a>. -</div> -<figure class="video-container"> - <iframe src="https://www.youtube.com/embed/Mq3C1KveDc0" frameborder="0" allowfullscreen="true"> </iframe> -</figure> - ## Protect production secrets Production secrets are needed to deploy successfully. For example, when deploying to the cloud, @@ -153,7 +171,7 @@ Before promoting a deployment to a production environment, cross-verifying it wi ### Pipelines jobs fail with `The deployment job is older than the previously succeeded deployment job...` -This is caused by the [Skip outdated deployment jobs](../pipelines/settings.md#skip-outdated-deployment-jobs) feature. +This is caused by the [Prevent outdated deployment jobs](../pipelines/settings.md#prevent-outdated-deployment-jobs) feature. If you have multiple jobs for the same environment (including non-deployment jobs), you might encounter this problem, for example: ```yaml @@ -166,5 +184,5 @@ build:service-b: name: production ``` -The [Skip outdated deployment jobs](../pipelines/settings.md#skip-outdated-deployment-jobs) might +The [Prevent outdated deployment jobs](../pipelines/settings.md#prevent-outdated-deployment-jobs) might not work well with this configuration, and must be disabled. diff --git a/doc/ci/environments/environments_dashboard.md b/doc/ci/environments/environments_dashboard.md index f662c156f55..479b783202d 100644 --- a/doc/ci/environments/environments_dashboard.md +++ b/doc/ci/environments/environments_dashboard.md @@ -29,7 +29,7 @@ The Environments dashboard displays a paginated list of projects that includes up to three environments per project. The listed environments for each project are unique, such as -"production", "staging", and so on. Review apps and other grouped +"production" and "staging". Review apps and other grouped environments are not displayed. ## Adding a project to the dashboard diff --git a/doc/ci/environments/incremental_rollouts.md b/doc/ci/environments/incremental_rollouts.md index af431a9812e..10cda68c4b5 100644 --- a/doc/ci/environments/incremental_rollouts.md +++ b/doc/ci/environments/incremental_rollouts.md @@ -76,7 +76,7 @@ available, demonstrating manually triggered incremental rollouts. ## Timed Rollouts Timed rollouts behave in the same way as manual rollouts, except that each job is defined with a -delay in minutes before it deploys. Clicking the job reveals the countdown. +delay in minutes before it deploys. Selecting the job reveals the countdown.  diff --git a/doc/ci/environments/index.md b/doc/ci/environments/index.md index 63641a9b7c3..c4672b9dc7e 100644 --- a/doc/ci/environments/index.md +++ b/doc/ci/environments/index.md @@ -178,7 +178,7 @@ deploy_prod: The `when: manual` action: - Exposes a play button for the job in the GitLab UI, with the text **Can be manually deployed to <environment>**. -- Means the `deploy_prod` job is only triggered when the play button is clicked. +- Means the `deploy_prod` job is only triggered when the play button is selected. You can find the play button in the pipelines, environments, deployments, and jobs views. @@ -383,6 +383,11 @@ To retry or rollback a deployment: - To retry a deployment, select **Re-deploy to environment**. - To roll back to a deployment, next to a previously successful deployment, select **Rollback environment**. +NOTE: +If you have [prevented outdated deployment jobs](deployment_safety.md#prevent-outdated-deployment-jobs) in your project, +the rollback buttons might be hidden or disabled. +In this case, see [how to rollback to an outdated deployment](deployment_safety.md#how-to-rollback-to-an-outdated-deployment). + ### Environment URL > - [Fixed](https://gitlab.com/gitlab-org/gitlab/-/issues/337417) to persist arbitrary URLs in GitLab 15.2 [with a flag](../../administration/feature_flags.md) named `soft_validation_on_external_url`. Disabled by default. @@ -721,7 +726,7 @@ build: ``` This gives you access to environment-scoped variables, and can be used to protect builds from unauthorized access. Also, -it's effective to avoid the [skip outdated deployment jobs](deployment_safety.md#skip-outdated-deployment-jobs) feature. +it's effective to avoid the [prevent outdated deployment jobs](deployment_safety.md#prevent-outdated-deployment-jobs) feature. ### Group similar environments @@ -1063,8 +1068,8 @@ To fix this, use one of the following solutions: - Remove `environment` keyword from the deployment job. GitLab has already been ignoring the invalid keyword, therefore your deployment pipelines stay intact even after the keyword removal. -- Ensure the variable exists in the pipeline. Please note that there is - [a limitation on supported variables](../variables/where_variables_can_be_used.md#gitlab-ciyml-file). +- Ensure the variable exists in the pipeline. Review the + [limitation on supported variables](../variables/where_variables_can_be_used.md#gitlab-ciyml-file). #### If you get this error on Review Apps @@ -1107,6 +1112,6 @@ to execute the following command on Rails console: Project.find_by_full_path(<your-project-full-path>).deployments.where(archived: true).each(&:create_ref) ``` -Please note that GitLab could drop this support in the future for the performance concern. +GitLab might drop this support in the future for the performance concern. You can open an issue in [GitLab Issue Tracker](https://gitlab.com/gitlab-org/gitlab/-/issues/new) to discuss the behavior of this feature. diff --git a/doc/ci/environments/protected_environments.md b/doc/ci/environments/protected_environments.md index 0f943679c07..5c120da32a0 100644 --- a/doc/ci/environments/protected_environments.md +++ b/doc/ci/environments/protected_environments.md @@ -26,7 +26,7 @@ Maintainer role. Prerequisites: -- When granting the **Allowed to deploy** permission to a group or subgroup, the user configuring the protected environment must be a **direct member** of the group or subgroup to be added. Otherwise, the group or subgroup will not show up in the dropdown. For more information see [issue #345140](https://gitlab.com/gitlab-org/gitlab/-/issues/345140). +- When granting the **Allowed to deploy** permission to a group or subgroup, the user configuring the protected environment must be a **direct member** of the group or subgroup to be added. Otherwise, the group or subgroup will not show up in the dropdown list. For more information see [issue #345140](https://gitlab.com/gitlab-org/gitlab/-/issues/345140). To protect an environment: @@ -147,7 +147,7 @@ Maintainers can: - Update existing protected environments at any time by changing the access in the **Allowed to Deploy** dropdown list. -- Unprotect a protected environment by clicking the **Unprotect** button for that environment. +- Unprotect a protected environment by selecting the **Unprotect** button for that environment. After an environment is unprotected, all access entries are deleted and must be re-entered if the environment is re-protected. diff --git a/doc/ci/examples/authenticating-with-hashicorp-vault/index.md b/doc/ci/examples/authenticating-with-hashicorp-vault/index.md index 59e377dbb09..7208caaccae 100644 --- a/doc/ci/examples/authenticating-with-hashicorp-vault/index.md +++ b/doc/ci/examples/authenticating-with-hashicorp-vault/index.md @@ -29,7 +29,7 @@ You must replace the `vault.example.com` URL below with the URL of your Vault se ## How it works -Each job has JSON Web Token (JWT) provided as CI/CD variable named `CI_JOB_JWT`. This JWT can be used to authenticate with Vault using the [JWT Auth](https://www.vaultproject.io/docs/auth/jwt#jwt-authentication) method. +Each job has JSON Web Token (JWT) provided as CI/CD variable named `CI_JOB_JWT`. This JWT can be used to authenticate with Vault using the [JWT Auth](https://developer.hashicorp.com/vault/docs/auth/jwt#jwt-authentication) method. The following fields are included in the JWT: @@ -90,7 +90,7 @@ The JWT is encoded by using RS256 and signed with a dedicated private key. The e You can use this JWT and your instance's JWKS endpoint (`https://gitlab.example.com/-/jwks`) to authenticate with a Vault server that is configured to allow the JWT Authentication method for authentication. -When configuring roles in Vault, you can use [bound_claims](https://www.vaultproject.io/docs/auth/jwt#bound-claims) to match against the JWT's claims and restrict which secrets each CI job has access to. +When configuring roles in Vault, you can use [bound_claims](https://developer.hashicorp.com/vault/docs/auth/jwt#bound-claims) to match against the JWT's claims and restrict which secrets each CI job has access to. To communicate with Vault, you can use either its CLI client or perform API requests (using `curl` or another client). @@ -109,7 +109,7 @@ $ vault kv get -field=password secret/myproject/production/db real-pa$$w0rd ``` -To configure your Vault server, start by enabling the [JWT Auth](https://www.vaultproject.io/docs/auth/jwt) method: +To configure your Vault server, start by enabling the [JWT Auth](https://developer.hashicorp.com/vault/docs/auth/jwt) method: ```shell $ vault auth enable jwt @@ -180,17 +180,17 @@ $ vault write auth/jwt/role/myproject-production - <<EOF EOF ``` -This example uses [bound_claims](https://www.vaultproject.io/api-docs/auth/jwt#bound_claims) to specify that only a JWT with matching values for the specified claims is allowed to authenticate. +This example uses [bound_claims](https://developer.hashicorp.com/vault/api-docs/auth/jwt#bound_claims) to specify that only a JWT with matching values for the specified claims is allowed to authenticate. Combined with [protected branches](../../../user/project/protected_branches.md), you can restrict who is able to authenticate and read the secrets. -[`token_explicit_max_ttl`](https://www.vaultproject.io/api-docs/auth/jwt#token_explicit_max_ttl) specifies that the token issued by Vault, upon successful authentication, has a hard lifetime limit of 60 seconds. +[`token_explicit_max_ttl`](https://developer.hashicorp.com/vault/api-docs/auth/jwt#token_explicit_max_ttl) specifies that the token issued by Vault, upon successful authentication, has a hard lifetime limit of 60 seconds. -[`user_claim`](https://www.vaultproject.io/api-docs/auth/jwt#user_claim) specifies the name for the Identity alias created by Vault upon a successful login. +[`user_claim`](https://developer.hashicorp.com/vault/api-docs/auth/jwt#user_claim) specifies the name for the Identity alias created by Vault upon a successful login. -[`bound_claims_type`](https://www.vaultproject.io/api-docs/auth/jwt#bound_claims_type) configures the interpretation of the `bound_claims` values. If set to `glob`, the values are interpreted as globs, with `*` matching any number of characters. +[`bound_claims_type`](https://developer.hashicorp.com/vault/api-docs/auth/jwt#bound_claims_type) configures the interpretation of the `bound_claims` values. If set to `glob`, the values are interpreted as globs, with `*` matching any number of characters. -The claim fields listed in [the table above](#how-it-works) can also be accessed for [Vault's policy path templating](https://learn.hashicorp.com/tutorials/vault/policy-templating?in=vault/policies) purposes by using the accessor name of the JWT auth within Vault. The [mount accessor name](https://learn.hashicorp.com/tutorials/vault/identity#step-1-create-an-entity-with-alias) (`ACCESSOR_NAME` in the example below) can be retrieved by running `vault auth list`. +The claim fields listed in [the table above](#how-it-works) can also be accessed for [Vault's policy path templating](https://developer.hashicorp.com/vault/tutorials/policies/policy-templating?in=vault%2Fpolicies) purposes by using the accessor name of the JWT auth within Vault. The [mount accessor name](https://developer.hashicorp.com/vault/tutorials/auth-methods/identity#step-1-create-an-entity-with-alias) (`ACCESSOR_NAME` in the example below) can be retrieved by running `vault auth list`. Policy template example making use of a named metadata field named `project_path`: @@ -200,7 +200,7 @@ path "secret/data/{{identity.entity.aliases.ACCESSOR_NAME.metadata.project_path} } ``` -Role example to support the templated policy above, mapping the claim field `project_path` as a metadata field through use of [`claim_mappings`](https://www.vaultproject.io/api-docs/auth/jwt#claim_mappings) configuration: +Role example to support the templated policy above, mapping the claim field `project_path` as a metadata field through use of [`claim_mappings`](https://developer.hashicorp.com/vault/api-docs/auth/jwt#claim_mappings) configuration: ```plaintext { @@ -212,7 +212,7 @@ Role example to support the templated policy above, mapping the claim field `pro } ``` -For the full list of options, see Vault's [Create Role documentation](https://www.vaultproject.io/api-docs/auth/jwt#create-role). +For the full list of options, see Vault's [Create Role documentation](https://developer.hashicorp.com/vault/api-docs/auth/jwt#create-role). WARNING: Always restrict your roles to project or namespace by using one of the provided claims (for example, `project_id` or `namespace_id`). Otherwise any JWT generated by this instance may be allowed to authenticate using this role. @@ -225,9 +225,9 @@ $ vault write auth/jwt/config \ bound_issuer="gitlab.example.com" ``` -[bound_issuer](https://www.vaultproject.io/api-docs/auth/jwt#bound_issuer) specifies that only a JWT with the issuer (that is, the `iss` claim) set to `gitlab.example.com` can use this method to authenticate, and that the JWKS endpoint (`https://gitlab.example.com/-/jwks`) should be used to validate the token. +[bound_issuer](https://developer.hashicorp.com/vault/api-docs/auth/jwt#bound_issuer) specifies that only a JWT with the issuer (that is, the `iss` claim) set to `gitlab.example.com` can use this method to authenticate, and that the JWKS endpoint (`https://gitlab.example.com/-/jwks`) should be used to validate the token. -For the full list of available configuration options, see Vault's [API documentation](https://www.vaultproject.io/api-docs/auth/jwt#configure). +For the full list of available configuration options, see Vault's [API documentation](https://developer.hashicorp.com/vault/api-docs/auth/jwt#configure). The following job, when run for the default branch, is able to read secrets under `secret/myproject/staging/`, but not the secrets under `secret/myproject/production/`: @@ -242,7 +242,7 @@ read_secrets: # Vault's address can be provided here or as CI/CD variable - export VAULT_ADDR=http://vault.example.com:8200 # Authenticate and get token. Token expiry time and other properties can be configured - # when configuring JWT Auth - https://www.vaultproject.io/api-docs/auth/jwt#parameters-1 + # when configuring JWT Auth - https://developer.hashicorp.com/vault/api-docs/auth/jwt#parameters-1 - export VAULT_TOKEN="$(vault write -field=token auth/jwt/login role=myproject-staging jwt=$CI_JOB_JWT)" # Now use the VAULT_TOKEN to read the secret and store it in an environment variable - export PASSWORD="$(vault kv get -field=password secret/myproject/staging/db)" @@ -271,7 +271,7 @@ read_secrets: # Vault's address can be provided here or as CI/CD variable - export VAULT_ADDR=http://vault.example.com:8200 # Authenticate and get token. Token expiry time and other properties can be configured - # when configuring JWT Auth - https://www.vaultproject.io/api-docs/auth/jwt#parameters-1 + # when configuring JWT Auth - https://developer.hashicorp.com/vault/api-docs/auth/jwt#parameters-1 - export VAULT_TOKEN="$(vault write -field=token auth/jwt/login role=myproject-production jwt=$CI_JOB_JWT)" # Now use the VAULT_TOKEN to read the secret and store it in environment variable - export PASSWORD="$(vault kv get -field=password secret/myproject/production/db)" @@ -286,11 +286,11 @@ read_secrets: You can control `CI_JOB_JWT` access to Vault secrets by using Vault protections and GitLab features. For example, restrict the token by: -- Using Vault [bound_claims](https://www.vaultproject.io/docs/auth/jwt#bound-claims) +- Using Vault [bound_claims](https://developer.hashicorp.com/vault/docs/auth/jwt#bound-claims) for specific groups using `group_claim`. - Hard coding values for Vault bound claims based on the `user_login` and `user_email` of specific users. -- Setting Vault time limits for TTL of the token as specified in [`token_explicit_max_ttl`](https://www.vaultproject.io/api-docs/auth/jwt#token_explicit_max_ttl), +- Setting Vault time limits for TTL of the token as specified in [`token_explicit_max_ttl`](https://developer.hashicorp.com/vault/api-docs/auth/jwt#token_explicit_max_ttl), where the token expires after authentication. - Scoping the JWT to [GitLab protected branches](../../../user/project/protected_branches.md) that are restricted to a subset of project users. diff --git a/doc/ci/examples/deployment/composer-npm-deploy.md b/doc/ci/examples/deployment/composer-npm-deploy.md index f14372757a9..a603207aef7 100644 --- a/doc/ci/examples/deployment/composer-npm-deploy.md +++ b/doc/ci/examples/deployment/composer-npm-deploy.md @@ -47,7 +47,7 @@ All these operations put all files into a `build` folder, which is ready to be d ## How to transfer files to a live server -You have multiple options: rsync, SCP, SFTP, and so on. For now, use SCP. +You have multiple options such as rsync, SCP, or SFTP. For now, use SCP. To make this work, you must add a GitLab CI/CD Variable (accessible on `gitlab.example/your-project-name/variables`). Name this variable `STAGING_PRIVATE_KEY` and set it to the **private** SSH key of your server. diff --git a/doc/ci/examples/end_to_end_testing_webdriverio/index.md b/doc/ci/examples/end_to_end_testing_webdriverio/index.md index e9f126b0409..7b5690e2d3d 100644 --- a/doc/ci/examples/end_to_end_testing_webdriverio/index.md +++ b/doc/ci/examples/end_to_end_testing_webdriverio/index.md @@ -88,7 +88,7 @@ hit our 404 page. We can then use [`browser.getUrl`](http://v4.webdriver.io/api/ to verify that the current page is indeed at the location we specified. To interact with the page, we can pass CSS selectors to [`browser.element`](http://v4.webdriver.io/api/protocol/element.html) to get access to elements on the -page and to interact with them - for example, to click on the link back to the home page. +page and to interact with them - for example, to select the link back to the home page. The simple test shown above can already give us a lot of confidence if it passes: we know our deployment has succeeded, that the diff --git a/doc/ci/examples/laravel_with_gitlab_and_envoy/index.md b/doc/ci/examples/laravel_with_gitlab_and_envoy/index.md index 80e476f2a87..28016216dbb 100644 --- a/doc/ci/examples/laravel_with_gitlab_and_envoy/index.md +++ b/doc/ci/examples/laravel_with_gitlab_and_envoy/index.md @@ -444,7 +444,7 @@ On your GitLab project repository navigate to the **Registry** tab. You may need to enable the Container Registry for your project to see this tab. You'll find it under your project's **Settings > General > Visibility, project features, permissions**. To start using Container Registry on our machine, we first need to sign in to the GitLab registry using our GitLab username and password. -Make sure you have [Docker](https://docs.docker.com/engine/installation/) installed on our machine, +Make sure you have [Docker](https://docs.docker.com/engine/install/) installed on our machine, then run the following commands: ```shell @@ -628,11 +628,11 @@ To do that, commit and push `.gitlab-ci.yml` to the `main` branch. It will trigg Here we see our **Test** and **Deploy** stages. The **Test** stage has the `unit_test` build running. -click on it to see the runner's output. +Select it to see the runner's output.  -After our code passed through the pipeline successfully, we can deploy to our production server by clicking the **play** button on the right side. +After our code passed through the pipeline successfully, we can deploy to our production server by selecting the **play** button on the right side.  @@ -644,7 +644,7 @@ If something doesn't work as expected, you can roll back to the latest working v  -By clicking on the external link icon specified on the right side, GitLab opens the production website. +By selecting the external link icon specified on the right side, GitLab opens the production website. Our deployment successfully was done and we can see the application is live.  diff --git a/doc/ci/examples/semantic-release.md b/doc/ci/examples/semantic-release.md index 88e63a7f36f..8f0321517ab 100644 --- a/doc/ci/examples/semantic-release.md +++ b/doc/ci/examples/semantic-release.md @@ -1,6 +1,6 @@ --- stage: Package -group: Package +group: Package Registry info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- diff --git a/doc/ci/git_submodules.md b/doc/ci/git_submodules.md index ee9d15894fe..68cbff21fae 100644 --- a/doc/ci/git_submodules.md +++ b/doc/ci/git_submodules.md @@ -60,6 +60,13 @@ To make submodules work correctly in CI/CD jobs: GIT_SUBMODULE_STRATEGY: recursive ``` +1. Use `GIT_SUBMODULE_DEPTH` to configure the cloning depth of submodules independently of the [`GIT_DEPTH`](runners/configure_runners.md#shallow-cloning) variable: + + ```yaml + variables: + GIT_SUBMODULE_DEPTH: 1 + ``` + 1. You can filter or exclude specific submodules to control which submodules will be synced using [`GIT_SUBMODULE_PATHS`](runners/configure_runners.md#git-submodule-paths). @@ -76,6 +83,14 @@ To make submodules work correctly in CI/CD jobs: GIT_SUBMODULE_STRATEGY: recursive GIT_SUBMODULE_UPDATE_FLAGS: --jobs 4 ``` + +1. You can set the [GIT_SUBMODULE_PATHS](runners/configure_runners.md#sync-or-exclude-specific-submodules-from-ci-jobs) to explicitly ignore submodules during cloning: + + ```yaml + variables: + GIT_SUBMODULE_STRATEGY: recursive + GIT_SUBMODULE_PATHS: ':(exclude)submodule' + ``` If you use the [`CI_JOB_TOKEN`](jobs/ci_job_token.md) to clone a submodule in a pipeline job, the user executing the job must be assigned to a role that has diff --git a/doc/ci/interactive_web_terminal/index.md b/doc/ci/interactive_web_terminal/index.md index a1df55cfc38..44c081b9d7f 100644 --- a/doc/ci/interactive_web_terminal/index.md +++ b/doc/ci/interactive_web_terminal/index.md @@ -65,7 +65,7 @@ for the current job.  -When clicked, a new tab opens to the terminal page where you can access +When selected, a new tab opens to the terminal page where you can access the terminal and type commands like a normal shell.  diff --git a/doc/ci/jobs/ci_job_token.md b/doc/ci/jobs/ci_job_token.md index 7282ebb0909..1e7b389c84a 100644 --- a/doc/ci/jobs/ci_job_token.md +++ b/doc/ci/jobs/ci_job_token.md @@ -26,8 +26,8 @@ You can use a GitLab CI/CD job token to authenticate with specific API endpoints - [Terraform plan](../../user/infrastructure/index.md). The token has the same permissions to access the API as the user that caused the -job to run. A user can cause a job to run by pushing a commit, triggering a manual job, -being the owner of a scheduled pipeline, and so on. Therefore, this user must be assigned to +job to run. A user can cause a job to run by taking action like pushing a commit, +triggering a manual job, or being the owner of a scheduled pipeline. Therefore, this user must be assigned to [a role that has the required privileges](../../user/permissions.md#gitlab-cicd-permissions). The token is valid only while the pipeline job runs. After the job finishes, you can't diff --git a/doc/ci/jobs/index.md b/doc/ci/jobs/index.md index d1a4f1e35bf..4eb8952dd73 100644 --- a/doc/ci/jobs/index.md +++ b/doc/ci/jobs/index.md @@ -37,7 +37,7 @@ independently from each other. When you access a pipeline, you can see the related jobs for that pipeline. -Clicking an individual job shows you its job log, and allows you to: +Selecting an individual job shows you its job log, and allows you to: - Cancel the job. - Retry the job. @@ -389,5 +389,5 @@ deploy me: The behavior of deployment jobs can be controlled with [deployment safety](../environments/deployment_safety.md) settings like -[skipping outdated deployment jobs](../environments/deployment_safety.md#skip-outdated-deployment-jobs) +[preventing outdated deployment jobs](../environments/deployment_safety.md#prevent-outdated-deployment-jobs) and [ensuring only one deployment job runs at a time](../environments/deployment_safety.md#ensure-only-one-deployment-job-runs-at-a-time). diff --git a/doc/ci/jobs/job_control.md b/doc/ci/jobs/job_control.md index 39ab0998291..d26c698af89 100644 --- a/doc/ci/jobs/job_control.md +++ b/doc/ci/jobs/job_control.md @@ -8,7 +8,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w When a new pipeline starts, GitLab checks the pipeline configuration to determine which jobs should run in that pipeline. You can configure jobs to run depending on -the status of variables, the pipeline type, and so on. +factors like the status of variables, or the pipeline type. To configure a job to be included or excluded from certain pipelines, you can use: @@ -107,6 +107,33 @@ job: - make build ``` +#### Skip job if the branch is empty + +Use [`rules:changes:compare_to`](../yaml/index.md#ruleschangescompare_to) to avoid +running a job when the branch is empty, which saves CI/CD resources. Compare the +branch to the default branch, and if the branch: + +- Doesn't have changed files, the job doesn't run. +- Has changed files, the job runs. + +For example, in a project with `main` as the default branch: + +```yaml +job: + script: + - echo "This job only runs for branches that are not empty" + rules: + - if: $CI_COMMIT_BRANCH + changes: + compare_to: refs/heads/main + paths: + - '*' +``` + +The rule for this job compares all files and paths (`*`) in the current branch against +the default branch `main`. The rule matches and the job runs only when there are +changes to the files in the branch. + ### Complex rules You can use all `rules` keywords, like `if`, `changes`, and `exists`, in the same @@ -1061,7 +1088,7 @@ docker_build: When the `DOCKERFILES_DIR` variable is expanded in the `changes:` section, the full path becomes `path/to/files//*`. The double slashes might cause unexpected behavior -depending on the keyword used, shell and OS of the runner, and so on. +depending on factors like the keyword used, or the shell and OS of the runner. ### `You are not allowed to download code from this project.` error message diff --git a/doc/ci/migration/circleci.md b/doc/ci/migration/circleci.md index 5354e406e34..034c8ba5ad6 100644 --- a/doc/ci/migration/circleci.md +++ b/doc/ci/migration/circleci.md @@ -268,7 +268,7 @@ test_async: ## Contexts and variables -CircleCI provides [Contexts](https://circleci.com/docs/contexts) to securely pass environment variables across project pipelines. In GitLab, a [Group](../../user/group/index.md) can be created to assemble related projects together. At the group level, [CI/CD variables](../variables/index.md#add-a-cicd-variable-to-a-group) can be stored outside the individual projects, and securely passed into pipelines across multiple projects. +CircleCI provides [Contexts](https://circleci.com/docs/contexts/) to securely pass environment variables across project pipelines. In GitLab, a [Group](../../user/group/index.md) can be created to assemble related projects together. At the group level, [CI/CD variables](../variables/index.md#add-a-cicd-variable-to-a-group) can be stored outside the individual projects, and securely passed into pipelines across multiple projects. ## Orbs diff --git a/doc/ci/migration/jenkins.md b/doc/ci/migration/jenkins.md index 300de610aa7..4ba59e14811 100644 --- a/doc/ci/migration/jenkins.md +++ b/doc/ci/migration/jenkins.md @@ -49,8 +49,8 @@ things we have found that help this: your users understand why the effort is worth it. The value is clear when the work is done, but people need to be aware while it's in progress too. - Sponsorship and alignment from the relevant leadership team helps with the point above. -- Spending time educating your users on what's different, sharing this document with them, - and so on helps ensure you are successful. +- Spending time educating your users on what's different and sharing this document + with them helps ensure you are successful. - Finding ways to sequence or delay parts of the migration can help a lot, but you don't want to leave things in a non-migrated (or partially-migrated) state for too long. To gain all the benefits of GitLab, moving your existing Jenkins setup over @@ -67,7 +67,7 @@ of transition, by letting you delay the migration of less urgent pipelines for a If you are interested in helping GitLab test the wrapper, join our [public testing issue](https://gitlab.com/gitlab-org/gitlab/-/issues/215675) for instructions and to provide your feedback. NOTE: -If you have a paid GitLab subscription, note that the JenkinsFile Wrapper is not packaged as part of GitLab, and falls outside of the scope of support. For more information, see the [Statement of Support](https://about.gitlab.com/support/statement-of-support.html). +If you have a paid GitLab subscription, note that the JenkinsFile Wrapper is not packaged as part of GitLab, and falls outside of the scope of support. For more information, see the [Statement of Support](https://about.gitlab.com/support/statement-of-support/). ## Important product differences @@ -197,7 +197,7 @@ can leverage. You can see the complete list of packaging features in the ## Integrated features -Where you may have used plugins to get things like code quality, unit tests, security scanning, and so on working in Jenkins, +Where you may have used plugins to get things like code quality, unit tests, and security scanning working in Jenkins, GitLab takes advantage of our connected ecosystem to automatically pull these kinds of results into your merge requests, pipeline details pages, and other locations. You may find that you actually don't need to configure anything to have these appear. diff --git a/doc/ci/pipelines/cicd_minutes.md b/doc/ci/pipelines/cicd_minutes.md index a5484fcdf5a..772a06980af 100644 --- a/doc/ci/pipelines/cicd_minutes.md +++ b/doc/ci/pipelines/cicd_minutes.md @@ -128,9 +128,7 @@ These additional CI/CD minutes: - Are carried over to the next month, if any remain at the end of the month. - Are valid for 12 months from date of purchase or until all minutes are consumed, whichever comes first. Expiry of minutes is not currently enforced. -If you use more CI/CD minutes than your monthly quota, when you purchase more, -those CI/CD minutes are deducted from your quota. For example, with a GitLab SaaS -Premium license: +For example, with a GitLab SaaS Premium license: - You have `10,000` monthly minutes. - You purchase an additional `5,000` minutes. @@ -232,7 +230,7 @@ For this reduced cost factor: - The merge request source project must be a fork of a GitLab-maintained project, such as [`gitlab-com/www-gitlab-com`](https://gitlab.com/gitlab-com/www-gitlab-com), - [`gitlab-org/gitlab`](https://gitlab.com/gitlab-org/gitlab), and so on. + or [`gitlab-org/gitlab`](https://gitlab.com/gitlab-org/gitlab). - The merge request target project must be the fork's parent project. - The pipeline must be a merge request, merged results, or merge train pipeline. @@ -271,9 +269,9 @@ the next month. For example: -- On **1st April**, you purchase `5,000` CI/CD minutes. -- During April, you use only `3,000` of the `5,000` minutes. -- On **1st May**, the remaining `2,000` minutes roll over and are added to your monthly quota. +- On **1st April**, you purchase `5,000` additional CI/CD minutes. +- During April, you use only `3,000` of the `5,000` additional minutes. +- On **1st May**, the unused minute roll over, so you have `2,000` additional minutes available for May. Additional CI/CD minutes are a one-time purchase and do not renew or refresh each month. diff --git a/doc/ci/pipelines/downstream_pipelines.md b/doc/ci/pipelines/downstream_pipelines.md index 0b1963e1874..c8fcd06da07 100644 --- a/doc/ci/pipelines/downstream_pipelines.md +++ b/doc/ci/pipelines/downstream_pipelines.md @@ -20,8 +20,10 @@ but there are [key differences](pipeline_architectures.md). ## Parent-child pipelines -A parent pipeline is one that triggers a downstream pipeline in the same project. -The downstream pipeline is called a child pipeline. Child pipelines: +A parent pipeline is a pipeline that triggers a downstream pipeline in the same project. +The downstream pipeline is called a child pipeline. + +Child pipelines: - Run under the same project, ref, and commit SHA as the parent pipeline. - Do not directly affect the overall status of the ref the pipeline runs against. For example, @@ -30,18 +32,18 @@ The downstream pipeline is called a child pipeline. Child pipelines: pipeline is triggered with [`strategy:depend`](../yaml/index.md#triggerstrategy). - Are automatically canceled if the pipeline is configured with [`interruptible`](../yaml/index.md#interruptible) when a new pipeline is created for the same ref. -- Are not displayed in the pipeline index page. You can only view child pipelines on - their parent pipeline's page. +- Are not displayed in the project's pipeline list. You can only view child pipelines on + their parent pipeline's details page. ### Nested child pipelines > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/29651) in GitLab 13.4. > - [Feature flag removed](https://gitlab.com/gitlab-org/gitlab/-/issues/243747) in GitLab 13.5. -Parent and child pipelines were introduced with a maximum depth of one level of child -pipelines, which was later increased to two. A parent pipeline can trigger many child -pipelines, and these child pipelines can trigger their own child pipelines. It's not -possible to trigger another level of child pipelines. +Parent and child pipelines have a maximum depth of two levels of child pipelines. + +A parent pipeline can trigger many child pipelines, and these child pipelines can trigger +their own child pipelines. You cannot trigger another level of child pipelines. <i class="fa fa-youtube-play youtube" aria-hidden="true"></i> For an overview, see [Nested Dynamic Pipelines](https://youtu.be/C5j3ju9je2M). @@ -52,11 +54,6 @@ A pipeline in one project can trigger downstream pipelines in another project, called multi-project pipelines. The user triggering the upstream pipeline must be able to start pipelines in the downstream project, otherwise [the downstream pipeline fails to start](#trigger-job-fails-and-does-not-create-multi-project-pipeline). -For example, you might deploy your web application from three different GitLab projects. -With multi-project pipelines you can trigger a pipeline in each project, where each -has its own build, test, and deploy process. You can visualize the connected pipelines -in one place, including all cross-project interdependencies. - Multi-project pipelines: - Are triggered from another project's pipeline, but the upstream (triggering) pipeline does @@ -68,8 +65,7 @@ Multi-project pipelines: - Are not automatically canceled in the downstream project when using [`interruptible`](../yaml/index.md#interruptible) if a new pipeline runs for the same ref in the upstream pipeline. They can be automatically canceled if a new pipeline is triggered for the same ref on the downstream project. -- Multi-project pipelines are standalone pipelines because they are normal pipelines - that happened to be triggered by an external project. They are all visible on the pipeline index page. +- Are visible in the downstream project's pipeline list. - Are independent, so there are no nesting limits. Learn more in the "Cross-project Pipeline Triggering and Visualization" demo at @@ -87,48 +83,49 @@ always displays: Use the [`trigger`](../yaml/index.md#trigger) keyword in your `.gitlab-ci.yml` file to create a job that triggers a downstream pipeline. This job is called a trigger job. -After the trigger job starts, the initial status of the job is `pending` while GitLab -attempts to create the downstream pipeline. If the downstream pipeline is created, -GitLab marks the job as passed, otherwise the job failed. Alternatively, -you can [set the trigger job to show the downstream pipeline's status](#mirror-the-status-of-a-downstream-pipeline-in-the-trigger-job) -instead. - For example: ::Tabs -:::TabTitle Multi-project pipeline +:::TabTitle Parent-child pipeline ```yaml trigger_job: trigger: - project: project-group/my-downstream-project + include: + - local: path/to/child-pipeline.yml ``` -:::TabTitle Parent-child pipeline +:::TabTitle Multi-project pipeline ```yaml trigger_job: trigger: - include: - - local: path/to/child-pipeline.yml + project: project-group/my-downstream-project ``` ::EndTabs +After the trigger job starts, the initial status of the job is `pending` while GitLab +attempts to create the downstream pipeline. The trigger job shows `passed` if the +downstream pipeline is created successfully, otherwise it shows `failed`. Alternatively, +you can [set the trigger job to show the downstream pipeline's status](#mirror-the-status-of-a-downstream-pipeline-in-the-trigger-job) +instead. + ### Use `rules` to control downstream pipeline jobs -You can use CI/CD variables or the [`rules`](../yaml/index.md#rulesif) keyword to -[control job behavior](../jobs/job_control.md) for downstream pipelines. +Use CI/CD variables or the [`rules`](../yaml/index.md#rulesif) keyword to +[control job behavior](../jobs/job_control.md) in downstream pipelines. -When a downstream pipeline is triggered with the [`trigger`](../yaml/index.md#trigger) keyword, +When you trigger a downstream pipeline with the [`trigger`](../yaml/index.md#trigger) keyword, the value of the [`$CI_PIPELINE_SOURCE` predefined variable](../variables/predefined_variables.md) for all jobs is: - `pipeline` for multi-project pipelines. -- `parent` for parent-child pipelines. +- `parent_pipeline` for parent-child pipelines. -For example, with a multi-project pipeline: +For example, to control jobs in multi-project pipelines in a project that also runs +merge request pipelines: ```yaml job1: @@ -148,38 +145,12 @@ job3: script: echo "This job runs in both multi-project and merge request pipelines" ``` -### Specify a branch for multi-project pipelines - -You can specify a branch name for a multi-project pipeline to use. GitLab uses -the commit on the head of the branch to create the downstream pipeline: - -```yaml -rspec: - stage: test - script: bundle exec rspec - -staging: - stage: deploy - trigger: - project: my/deployment - branch: stable-11-2 -``` - -Use: - -- The `project` keyword to specify the full path to a downstream project. - In [GitLab 15.3 and later](https://gitlab.com/gitlab-org/gitlab/-/issues/367660), variable expansion is - supported. -- The `branch` keyword to specify the name of a branch in the project specified by `project`. - In [GitLab 12.4 and later](https://gitlab.com/gitlab-org/gitlab/-/issues/10126), variable expansion is - supported. - ### Use a child pipeline configuration file in a different project > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/205157) in GitLab 13.5. -You can use [`include:file`](../yaml/index.md#includefile) to trigger child pipelines -with a configuration file in a different project: +You can use [`include:project`](../yaml/index.md#includeproject) in a trigger job +to trigger child pipelines with a configuration file in a different project: ```yaml microservice_a: @@ -208,8 +179,6 @@ microservice_a: ### Dynamic child pipelines -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/35632) in GitLab 12.9. - You can trigger a child pipeline from a YAML file generated in a job, instead of a static file saved in your project. This technique can be very powerful for generating pipelines targeting content that changed or to build a matrix of targets and architectures. @@ -231,43 +200,42 @@ To trigger a child pipeline from a dynamically generated configuration file: 1. Generate the configuration file in a job and save it as an [artifact](../yaml/index.md#artifactspaths): - ```yaml - generate-config: - stage: build - script: generate-ci-config > generated-config.yml - artifacts: - paths: - - generated-config.yml - ``` + ```yaml + generate-config: + stage: build + script: generate-ci-config > generated-config.yml + artifacts: + paths: + - generated-config.yml + ``` 1. Configure the trigger job to run after the job that generated the configuration file, and set `include: artifact` to the generated artifact: - ```yaml - child-pipeline: - stage: test - trigger: - include: - - artifact: generated-config.yml - job: generate-config - ``` + ```yaml + child-pipeline: + stage: test + trigger: + include: + - artifact: generated-config.yml + job: generate-config + ``` -In this example, `generated-config.yml` is extracted from the artifacts and used as the configuration -for triggering the child pipeline. +In this example, GitLab retrieves `generated-config.yml` and triggers a child pipeline +with the CI/CD configuration in that file. The artifact path is parsed by GitLab, not the runner, so the path must match the syntax for the OS running GitLab. If GitLab is running on Linux but using a Windows runner for testing, the path separator for the trigger job is `/`. Other CI/CD -configuration for jobs that use the Windows runner, like scripts, use `\`. +configuration for jobs that use the Windows runner, like scripts, use <code>\</code>. ### Run child pipelines with merge request pipelines To trigger a child pipeline as a [merge request pipeline](merge_request_pipelines.md): -1. Set the trigger job to run on merge requests: +1. Set the trigger job to run on merge requests in the parent pipeline's configuration file: ```yaml - # parent .gitlab-ci.yml microservice_a: trigger: include: path/to/microservice_a.yml @@ -275,45 +243,50 @@ To trigger a child pipeline as a [merge request pipeline](merge_request_pipeline - if: $CI_PIPELINE_SOURCE == "merge_request_event" ``` -1. Configure the child pipeline jobs to run in merge request pipelines: +1. Configure the child pipeline jobs to run in merge request pipelines with [`rules`](../yaml/index.md#rules) + or [`workflow:rules`](../yaml/index.md#workflowrules). For example, with `rules` + in a child pipeline's configuration file: - - With [`workflow:rules`](../yaml/index.md#workflowrules): + ```yaml + job1: + script: ... + rules: + - if: $CI_PIPELINE_SOURCE == "merge_request_event" - ```yaml - # child path/to/microservice_a.yml - workflow: - rules: - - if: $CI_PIPELINE_SOURCE == "merge_request_event" + job2: + script: ... + rules: + - if: $CI_PIPELINE_SOURCE == "merge_request_event" + ``` - job1: - script: ... +### Specify a branch for multi-project pipelines - job2: - script: ... - ``` +You can specify the branch to use when triggering a multi-project pipeline. GitLab uses +the commit on the head of the branch to create the downstream pipeline. For example: - - By configuring [rules](../yaml/index.md#rules) for each job: +```yaml +staging: + stage: deploy + trigger: + project: my/deployment + branch: stable-11-2 +``` - ```yaml - # child path/to/microservice_a.yml - job1: - script: ... - rules: - - if: $CI_PIPELINE_SOURCE == "merge_request_event" +Use: - job2: - script: ... - rules: - - if: $CI_PIPELINE_SOURCE == "merge_request_event" - ``` +- The `project` keyword to specify the full path to the downstream project. + In [GitLab 15.3 and later](https://gitlab.com/gitlab-org/gitlab/-/issues/367660), + you can use [variable expansion](../variables/where_variables_can_be_used.md#gitlab-ciyml-file). +- The `branch` keyword to specify the name of a branch or [tag](../../topics/git/tags.md) + in the project specified by `project`. You can use variable expansion. ## Trigger a multi-project pipeline by using the API You can use the [CI/CD job token (`CI_JOB_TOKEN`)](../jobs/ci_job_token.md) with the [pipeline trigger API endpoint](../../api/pipeline_triggers.md#trigger-a-pipeline-with-a-token) -to trigger multi-project pipelines from a CI/CD job. GitLab recognizes the source of the job token -and marks the pipelines as related. In the pipeline graph, the relationships are displayed -as inbound and outbound connections for upstream and downstream pipeline dependencies. +to trigger multi-project pipelines from inside a CI/CD job. GitLab sets pipelines triggered +with a job token as downstream pipelines of the pipeline that contains the job that +made the API call. For example: @@ -357,27 +330,27 @@ To cancel a downstream pipeline that is still running, select **Cancel** (**{can ### Mirror the status of a downstream pipeline in the trigger job -You can mirror the pipeline status from the triggered pipeline to the source trigger job +You can mirror the status of the downstream pipeline in the trigger job by using [`strategy: depend`](../yaml/index.md#triggerstrategy): ::Tabs -:::TabTitle Multi-project pipeline +:::TabTitle Parent-child pipeline ```yaml trigger_job: trigger: - project: my/project + include: + - local: path/to/child-pipeline.yml strategy: depend ``` -:::TabTitle Parent-child pipeline +:::TabTitle Multi-project pipeline ```yaml trigger_job: trigger: - include: - - local: path/to/child-pipeline.yml + project: my/project strategy: depend ``` @@ -385,7 +358,7 @@ trigger_job: ### View multi-project pipelines in pipeline graphs **(PREMIUM)** -When you trigger a multi-project pipeline, the downstream pipeline displays +After you trigger a multi-project pipeline, the downstream pipeline displays to the right of the [pipeline graph](index.md#visualize-pipelines).  @@ -395,12 +368,13 @@ displays to the right of the mini graph.  -## Pass artifacts to a downstream pipeline +## Fetch artifacts from an upstream pipeline -You can pass artifacts to a downstream pipeline by using [`needs:project`](../yaml/index.md#needsproject). +Use [`needs:project`](../yaml/index.md#needsproject) to fetch artifacts from an +upstream pipeline: -1. In a job in the upstream pipeline, save the artifacts using the [`artifacts`](../yaml/index.md#artifacts) keyword. -1. Trigger the downstream pipeline with a trigger job: +1. In the upstream pipeline, save the artifacts in a job with the [`artifacts`](../yaml/index.md#artifacts) + keyword, then trigger the downstream pipeline with a trigger job: ```yaml build_artifacts: @@ -416,9 +390,7 @@ You can pass artifacts to a downstream pipeline by using [`needs:project`](../ya trigger: my/downstream_project ``` -1. In a job in the downstream pipeline, fetch the artifacts from the upstream pipeline - by using `needs:project`. Set `job` to the job in the upstream pipeline to fetch artifacts from, - `ref` to the branch, and `artifacts: true`. +1. Use `needs:project` in a job in the downstream pipeline to fetch the artifacts. ```yaml test: @@ -432,22 +404,27 @@ You can pass artifacts to a downstream pipeline by using [`needs:project`](../ya artifacts: true ``` -### Pass artifacts from a Merge Request pipeline + Set: -When you use `needs:project` to [pass artifacts to a downstream pipeline](#pass-artifacts-to-a-downstream-pipeline), + - `job` to the job in the upstream pipeline that created the artifacts. + - `ref` to the branch. + - `artifacts` to `true`. + +### Fetch artifacts from an upstream merge request pipeline + +When you use `needs:project` to [pass artifacts to a downstream pipeline](#fetch-artifacts-from-an-upstream-pipeline), the `ref` value is usually a branch name, like `main` or `development`. -For merge request pipelines, the `ref` value is in the form of `refs/merge-requests/<id>/head`, +For [merge request pipelines](merge_request_pipelines.md), the `ref` value is in the form of `refs/merge-requests/<id>/head`, where `id` is the merge request ID. You can retrieve this ref with the [`CI_MERGE_REQUEST_REF_PATH`](../variables/predefined_variables.md#predefined-variables-for-merge-request-pipelines) CI/CD variable. Do not use a branch name as the `ref` with merge request pipelines, because the downstream pipeline attempts to fetch artifacts from the latest branch pipeline. To fetch the artifacts from the upstream `merge request` pipeline instead of the `branch` pipeline, -pass this variable to the downstream pipeline using variable inheritance: +pass `CI_MERGE_REQUEST_REF_PATH` to the downstream pipeline using [variable inheritance](#pass-yaml-defined-cicd-variables): 1. In a job in the upstream pipeline, save the artifacts using the [`artifacts`](../yaml/index.md#artifacts) keyword. -1. In the job that triggers the downstream pipeline, pass the `$CI_MERGE_REQUEST_REF_PATH` variable by using - [variable inheritance](#pass-yaml-defined-cicd-variables): +1. In the job that triggers the downstream pipeline, pass the `$CI_MERGE_REQUEST_REF_PATH` variable: ```yaml build_artifacts: @@ -467,8 +444,7 @@ pass this variable to the downstream pipeline using variable inheritance: ``` 1. In a job in the downstream pipeline, fetch the artifacts from the upstream pipeline - by using `needs:project`. Set the `ref` to the `UPSTREAM_REF` variable, and `job` - to the job in the upstream pipeline to fetch artifacts from: + by using `needs:project` and the passed variable as the `ref`: ```yaml test: @@ -482,86 +458,137 @@ pass this variable to the downstream pipeline using variable inheritance: artifacts: true ``` -This method works for fetching artifacts from a regular merge request parent pipeline, -but fetching artifacts from [merge results](merged_results_pipelines.md) pipelines is not supported. +You can use this method to fetch artifacts from upstream merge request pipeline, +but not from [merge results pipelines](merged_results_pipelines.md). ## Pass CI/CD variables to a downstream pipeline -You can pass CI/CD variables to a downstream pipeline with a few different methods, -based on where the variable is created or defined. +You can pass [CI/CD variables](../variables/index.md) to a downstream pipeline with +a few different methods, based on where the variable is created or defined. ### Pass YAML-defined CI/CD variables -You can use the `variables` keyword to pass CI/CD variables to a downstream pipeline, -just like you would for any other job. +You can use the `variables` keyword to pass CI/CD variables to a downstream pipeline. -For example, in a [multi-project pipeline](#multi-project-pipelines): +For example: + +::Tabs + +:::TabTitle Parent-child pipeline ```yaml -rspec: - stage: test - script: bundle exec rspec +variables: + VERSION: "1.0.0" staging: variables: ENVIRONMENT: staging stage: deploy - trigger: my/deployment + trigger: + include: + - local: path/to/child-pipeline.yml ``` -The `ENVIRONMENT` variable is passed to every job defined in a downstream -pipeline. It is available as a variable when GitLab Runner picks a job. - -In the following configuration, the `MY_VARIABLE` variable is passed to the downstream pipeline -that is created when the `trigger-downstream` job is queued. This behavior is because `trigger-downstream` -job inherits variables declared in [global `variables`](../yaml/index.md#variables) blocks, -and then GitLab passes these variables to the downstream pipeline. +:::TabTitle Multi-project pipeline ```yaml variables: - MY_VARIABLE: my-value + VERSION: "1.0.0" -trigger-downstream: +staging: variables: - ENVIRONMENT: something - trigger: my/project + ENVIRONMENT: staging + stage: deploy + trigger: my-group/my-deployment-project ``` +::EndTabs + +The `ENVIRONMENT` variable is available in every job defined in the downstream pipeline. + +The `VERSION` global variable is also available in the downstream pipeline, because +all jobs in a pipeline, including trigger jobs, inherit [global `variables`](../yaml/index.md#variables). + #### Prevent global variables from being passed -You can stop global variables from reaching the downstream pipeline by using the [`inherit:variables` keyword](../yaml/index.md#inheritvariables). -For example, in a [multi-project pipeline](#multi-project-pipelines): +You can stop global CI/CD variables from reaching the downstream pipeline with +[`inherit:variables:false`](../yaml/index.md#inheritvariables). + +For example: + +::Tabs + +:::TabTitle Parent-child pipeline ```yaml variables: - MY_GLOBAL_VAR: value + GLOBAL_VAR: value -trigger-downstream: +trigger-job: inherit: variables: false variables: - MY_LOCAL_VAR: value - trigger: my/project + JOB_VAR: value + trigger: + include: + - local: path/to/child-pipeline.yml ``` -In this example, the `MY_GLOBAL_VAR` variable is not available in the triggered pipeline. +:::TabTitle Multi-project pipeline + +```yaml +variables: + GLOBAL_VAR: value + +trigger-job: + inherit: + variables: false + variables: + JOB_VAR: value + trigger: my-group/my-project +``` + +::EndTabs + +The `GLOBAL_VAR` variable is not available in the triggered pipeline, but `JOB_VAR` +is available. ### Pass a predefined variable -You might want to pass some information about the upstream pipeline using predefined variables. -To do that, you can use interpolation to pass any variable. For example, -in a [multi-project pipeline](#multi-project-pipelines): +To pass information about the upstream pipeline using [predefined CI/CD variables](../variables/predefined_variables.md). +use interpolation. Save the predefined variable as a new job variable in the trigger +job, which is passed to the downstream pipeline. For example: + +::Tabs + +:::TabTitle Parent-child pipeline + +```yaml +trigger-job: + variables: + PARENT_BRANCH: $CI_COMMIT_REF_NAME + trigger: + include: + - local: path/to/child-pipeline.yml +``` + +:::TabTitle Multi-project pipeline ```yaml -downstream-job: +trigger-job: variables: UPSTREAM_BRANCH: $CI_COMMIT_REF_NAME - trigger: my/project + trigger: my-group/my-project ``` -In this scenario, the `UPSTREAM_BRANCH` variable with the value of the upstream pipeline's -`$CI_COMMIT_REF_NAME` is passed to `downstream-job`. It is available in the -context of all downstream builds. +::EndTabs + +The `UPSTREAM_BRANCH` variable, which contains the value of the upstream pipeline's `$CI_COMMIT_REF_NAME` +predefined CI/CD variable, is available in the downstream pipeline. + +Do not use this method to pass [masked variables](../variables/index.md#mask-a-cicd-variable) +to a multi-project pipeline. The CI/CD masking configuration is not passed to the +downstream pipeline and the variable could be unmasked in job logs in the downstream project. You cannot use this method to forward [job-level persisted variables](../variables/where_variables_can_be_used.md#persisted-variables) to a downstream pipeline, as they are not available in trigger jobs. @@ -623,3 +650,10 @@ With multi-project pipelines, the trigger job fails and does not create the down - The downstream pipeline targets a protected branch and the user does not have permission to run pipelines against the protected branch. See [pipeline security for protected branches](index.md#pipeline-security-on-protected-branches) for more information. + +### `Ref is ambiguous` + +You cannot trigger a multi-project pipeline with a tag when a branch exists with the same +name. The downstream pipeline fails to create with the error: `downstream pipeline can not be created, Ref is ambiguous`. + +Only trigger multi-project pipelines with tag names that do not match branch names. diff --git a/doc/ci/pipelines/img/merge_train_cancel_v12_0.png b/doc/ci/pipelines/img/merge_train_cancel_v12_0.png Binary files differdeleted file mode 100644 index d7720ac1143..00000000000 --- a/doc/ci/pipelines/img/merge_train_cancel_v12_0.png +++ /dev/null diff --git a/doc/ci/pipelines/img/merge_train_failure.png b/doc/ci/pipelines/img/merge_train_failure.png Binary files differdeleted file mode 100644 index 8a795fff432..00000000000 --- a/doc/ci/pipelines/img/merge_train_failure.png +++ /dev/null diff --git a/doc/ci/pipelines/img/merge_train_immediate_merge_v12_6.png b/doc/ci/pipelines/img/merge_train_immediate_merge_v12_6.png Binary files differdeleted file mode 100644 index 7b903716a3d..00000000000 --- a/doc/ci/pipelines/img/merge_train_immediate_merge_v12_6.png +++ /dev/null diff --git a/doc/ci/pipelines/img/merge_train_position_v12_0.png b/doc/ci/pipelines/img/merge_train_position_v12_0.png Binary files differdeleted file mode 100644 index ec4b157d428..00000000000 --- a/doc/ci/pipelines/img/merge_train_position_v12_0.png +++ /dev/null diff --git a/doc/ci/pipelines/img/merge_train_start_v12_0.png b/doc/ci/pipelines/img/merge_train_start_v12_0.png Binary files differdeleted file mode 100644 index a4d0c8cf0e6..00000000000 --- a/doc/ci/pipelines/img/merge_train_start_v12_0.png +++ /dev/null diff --git a/doc/ci/pipelines/img/merge_train_start_when_pipeline_succeeds_v12_0.png b/doc/ci/pipelines/img/merge_train_start_when_pipeline_succeeds_v12_0.png Binary files differdeleted file mode 100644 index 45762b8e85e..00000000000 --- a/doc/ci/pipelines/img/merge_train_start_when_pipeline_succeeds_v12_0.png +++ /dev/null diff --git a/doc/ci/pipelines/index.md b/doc/ci/pipelines/index.md index ed0583e0872..f1ca8afa62c 100644 --- a/doc/ci/pipelines/index.md +++ b/doc/ci/pipelines/index.md @@ -80,7 +80,7 @@ You can also configure specific aspects of your pipelines through the GitLab UI. ### Ref specs for runners When a runner picks a pipeline job, GitLab provides that job's metadata. This includes the [Git refspecs](https://git-scm.com/book/en/v2/Git-Internals-The-Refspec), -which indicate which ref (branch, tag, and so on) and commit (SHA1) are checked out from your +which indicate which ref (such as branch or tag) and commit (SHA1) are checked out from your project repository. This table lists the refspecs injected for each pipeline type: @@ -158,7 +158,7 @@ The pipeline now executes the jobs as configured. You can use the [`description` and `value`](../yaml/index.md#variablesdescription) keywords to define [pipeline-level (global) variables](../variables/index.md#create-a-custom-cicd-variable-in-the-gitlab-ciyml-file) that are prefilled when running a pipeline manually. Use the description to explain -what the variable is used for, what the acceptable values are, and so on. +information such as what the variable is used for, and what the acceptable values are. Job-level variables cannot be pre-filled. @@ -260,7 +260,7 @@ pipelines. > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/24851) in GitLab 12.7. Users with the Owner role for a project can delete a pipeline -by clicking on the pipeline in the **CI/CD > Pipelines** to get to the **Pipeline Details** +by selecting the pipeline in the **CI/CD > Pipelines** to get to the **Pipeline Details** page, then selecting **Delete**.  @@ -301,6 +301,9 @@ preserving deployment keys and other credentials from being unintentionally accessed. To ensure that jobs intended to be executed on protected runners do not use regular runners, they must be tagged accordingly. +Review the [deployment safety](../environments/deployment_safety.md) +page for additional security recommendations for securing your pipelines. + ## Trigger a pipeline when an upstream project is rebuilt **(PREMIUM)** > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/9045) in GitLab 12.8. diff --git a/doc/ci/pipelines/job_artifacts.md b/doc/ci/pipelines/job_artifacts.md index 33a9069240b..cc26ede5f6e 100644 --- a/doc/ci/pipelines/job_artifacts.md +++ b/doc/ci/pipelines/job_artifacts.md @@ -309,61 +309,50 @@ for [parent and child pipelines](downstream_pipelines.md#parent-child-pipelines) order from parent to child. For example, if both parent and child pipelines have a job with the same name, the job artifact from the parent pipeline is returned. -## Access the latest job artifacts by URL +## Access the latest job artifacts -You can download job artifacts from the latest successful pipeline by using a URL. +You can download job artifacts from the latest successful pipeline by using [the job artifacts API](../../api/job_artifacts.md). +You cannot download [artifact reports](../yaml/artifacts_reports.md) with the job artifacts API, +unless the report is added as a regular artifact with `artifacts:paths`. -To download the whole artifacts archive: +### Download the whole artifacts archive for a specific job -```plaintext -https://example.com/<namespace>/<project>/-/jobs/artifacts/<ref>/download?job=<job_name> -``` +You can download the artifacts archive for a specific job with [the job artifacts API](../../api/job_artifacts.md#download-the-artifacts-archive). -To download a single file from the artifacts: +For example, to download the latest artifacts of a job named `build` in the `main` branch of a project on GitLab.com: ```plaintext -https://example.com/<namespace>/<project>/-/jobs/artifacts/<ref>/raw/<path_to_file>?job=<job_name> +https://gitlab.com/api/v4/projects/<project-id>/jobs/artifacts/main/download?job=build ``` -For example, to download the latest artifacts of the job named `coverage` in -the `main` branch of the `gitlab` project in the `gitlab-org` -namespace: - -```plaintext -https://gitlab.com/gitlab-org/gitlab/-/jobs/artifacts/main/download?job=coverage -``` +Replace `<project-id>` with a valid project ID, found at the top of the project details page. -To download the file `review/index.html` from the same artifacts: +### Download a single file from the artifacts -```plaintext -https://gitlab.com/gitlab-org/gitlab/-/jobs/artifacts/main/raw/review/index.html?job=coverage -``` +You can download a specific file from the artifacts archive for a specific job with [the job artifacts API](../../api/job_artifacts.md#download-a-single-artifact-file-by-job-id). -To browse the latest job artifacts: +For example, to download the file `review/index.html` from the latest job named `build` in the `main` branch of the `gitlab` project in the `gitlab-org` namespace: ```plaintext -https://example.com/<namespace>/<project>/-/jobs/artifacts/<ref>/browse?job=<job_name> +https://gitlab.com/api/v4/projects/27456355/jobs/artifacts/main/raw/review/index.html?job=build ``` -For example: +### Browse job artifacts -```plaintext -https://gitlab.com/gitlab-org/gitlab/-/jobs/artifacts/main/browse?job=coverage -``` - -To download specific files, including HTML files that -are shown in [GitLab Pages](../../administration/pages/index.md): +To browse the job artifacts of the latest successful pipeline for a specific job you can use the following URL: ```plaintext -https://example.com/<namespace>/<project>/-/jobs/artifacts/<ref>/file/<path>?job=<job_name> +https://example.com/<namespace>/<project>/-/jobs/artifacts/<ref>/browse?job=<job_name> ``` -For example, when a job `coverage` creates the artifact `htmlcov/index.html`: +For example, to browse the latest artifacts of a job named `build` in the `main` branch of a project on GitLab.com: ```plaintext -https://gitlab.com/gitlab-org/gitlab/-/jobs/artifacts/main/file/htmlcov/index.html?job=coverage +https://gitlab.com/<full-project-path>/-/jobs/artifacts/main/browse?job=build ``` +Replace `<full-project-path>` with a valid project path, you can find it in the URL for your project. + ## When job artifacts are deleted See the [`expire_in`](../yaml/index.md#artifactsexpire_in) documentation for information on when @@ -396,7 +385,24 @@ a project, you can disable this behavior to save space: You can disable this behavior for all projects on a self-managed instance in the [instance's CI/CD settings](../../user/admin_area/settings/continuous_integration.md#keep-the-latest-artifacts-for-all-jobs-in-the-latest-successful-pipelines). -## Troubleshooting job artifacts +## Troubleshooting + +### Job does not retrieve certain artifacts + +By default, jobs fetch all artifacts from previous stages, but jobs using `dependencies` +or `needs` do not fetch artifacts from all jobs by default. + +If you use these keywords, artifacts are fetched from only a subset of jobs. Review +the keyword reference for information on how to fetch artifacts with these keywords: + +- [`dependencies`](../yaml/index.md#dependencies) +- [`needs`](../yaml/index.md#needs) +- [`needs:artifacts`](../yaml/index.md#needsartifacts) + +### Job artifacts using too much disk space + +There are a number of potential causes for this. +[Read more in the job artifacts administration documentation](../../administration/job_artifacts.md#job-artifacts-using-too-much-disk-space). ### Error message `No files to upload` diff --git a/doc/ci/pipelines/merge_trains.md b/doc/ci/pipelines/merge_trains.md index c501d2a7904..370d81dacc4 100644 --- a/doc/ci/pipelines/merge_trains.md +++ b/doc/ci/pipelines/merge_trains.md @@ -6,253 +6,245 @@ info: To determine the technical writer assigned to the Stage/Group associated w # Merge trains **(PREMIUM)** -> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/9186) in GitLab 12.0. -> - [Squash and merge](../../user/project/merge_requests/squash_and_merge.md) support [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/13001) in GitLab 12.6. +Use merge trains to queue merge requests and verify their changes work together before +they are merged to the target branch. -For more information about why you might want to use merge trains, read [How starting merge trains improve efficiency for DevOps](https://about.gitlab.com/blog/2020/01/30/all-aboard-merge-trains/). +In projects with frequent merges to the default branch, changes in different merge requests +might conflict with each other. [Merged results pipelines](merged_results_pipelines.md) +ensure the changes work with the content in the default branch, but not content +that others are merging at the same time. -When [merged results pipelines](merged_results_pipelines.md) are -enabled, the pipeline jobs run as if the changes from your source branch have already -been merged into the target branch. +Merge trains do not work with [Semi-linear history merge requests](../../user/project/merge_requests/methods/index.md#merge-commit-with-semi-linear-history) +or [fast-forward merge requests](../../user/project/merge_requests/methods/index.md#fast-forward-merge). -However, the target branch may be changing rapidly. When you're ready to merge, -if you haven't run the pipeline in a while, the target branch may have already changed. -Merging now could introduce breaking changes. +For more information about: -*Merge trains* can prevent this from happening. A merge train is a queued list of merge -requests, each waiting to be merged into the target branch. +- How merge trains work, review the [merge train workflow](#merge-train-workflow). +- Why you might want to use merge trains, read [How starting merge trains improve efficiency for DevOps](https://about.gitlab.com/blog/2020/01/30/all-aboard-merge-trains/). -Many merge requests can be added to the train. Each merge request runs its own merged results pipeline, -which includes the changes from all of the other merge requests in *front* of it on the train. -All the pipelines run in parallel, to save time. The author of the internal merged result commit is always the user that initiated the merge. +## Merge train workflow -If the pipeline for a merge request fails, the breaking changes are not merged, and the target -branch is unaffected. The merge request is removed from the train, and all pipelines behind it restart. +A merge train starts when there are no merge requests waiting to merge and you +select [**Start merge train**](#start-a-merge-train). GitLab starts a merge train pipeline +that verifies that the changes can merge into the default branch. This first pipeline +is the same as a [merged results pipeline](merged_results_pipelines.md), which runs on +the changes of the source and target branches combined together. The author of the +internal merged result commit is the user that initiated the merge. -If the pipeline for the merge request at the front of the train completes successfully, -the changes are merged into the target branch, and the other pipelines continue to -run. +To queue a second merge request to merge immediately after the first pipeline completes, select +[**Add to merge train**](#add-a-merge-request-to-a-merge-train) and add it to the train. +This second merge train pipeline runs on the changes of _both_ merge requests combined with the +target branch. Similarly, if you add a third merge request, that pipeline runs on the changes +of all three merge requests merged with the target branch. The pipelines all run in parallel. -To add a merge request to a merge train, you need [permissions](../../user/permissions.md) to merge or push to the -target branch. +Each merge request merges into the target branch only after: -Each merge train can run a maximum of **twenty** pipelines in parallel. -If more than twenty merge requests are added to the merge train, the merge requests -are queued until a slot in the merge train is free. There is no limit to the -number of merge requests that can be queued. +- The merge request's pipeline completes successfully. +- All other merge requests queued before it are merged. -## Merge train example +If a merge train pipeline fails, the merge request is not merged. GitLab +removes that merge request from the merge train, and starts new pipelines for all +the merge requests that were queued after it. -Three merge requests (`A`, `B` and `C`) are added to a merge train in order, which +For example: + +Three merge requests (`A`, `B`, and `C`) are added to a merge train in order, which creates three merged results pipelines that run in parallel: 1. The first pipeline runs on the changes from `A` combined with the target branch. 1. The second pipeline runs on the changes from `A` and `B` combined with the target branch. 1. The third pipeline runs on the changes from `A`, `B`, and `C` combined with the target branch. -If the pipeline for `B` fails, it is removed from the train. The pipeline for -`C` restarts with the `A` and `C` changes, but without the `B` changes. +If the pipeline for `B` fails: + +- The first pipeline (`A`) continues to run. +- `B` is removed from the train. +- The pipeline for `C` [is cancelled](#automatic-pipeline-cancellation), and a new pipeline + starts for the changes from `A` and `C` combined with the target branch (without the `B` changes). If `A` then completes successfully, it merges into the target branch, and `C` continues -to run. If more merge requests are added to the train, they now include the `A` -changes that are included in the target branch, and the `C` changes that are from -the merge request already in the train. +to run. Any new merge requests added to the train include the `A` changes now in +the target branch, and the `C` changes from the merge train. <i class="fa fa-youtube-play youtube" aria-hidden="true"></i> -Watch this video for a demonstration on -[how parallel execution of merge trains can prevent commits from breaking the default branch](https://www.youtube.com/watch?v=D4qCqXgZkHQ). +Watch this video for a demonstration on [how parallel execution of merge trains can prevent commits from breaking the default branch](https://www.youtube.com/watch?v=D4qCqXgZkHQ). -## Prerequisites +### Automatic pipeline cancellation -To enable merge trains: +GitLab CI/CD detects redundant pipelines, and cancels them to conserve resources. -- You must have the Maintainer role. -- You must be using [GitLab Runner](https://gitlab.com/gitlab-org/gitlab-runner) 11.9 or later. -- Your repository must be a GitLab repository, not an - [external repository](../ci_cd_for_external_repos/index.md). +Redundant merge train pipelines happen when: -Merge trains do not work with [Semi-linear history merge requests](../../user/project/merge_requests/methods/index.md#merge-commit-with-semi-linear-history) -or [fast-forward merge requests](../../user/project/merge_requests/methods/index.md#fast-forward-merge). +- The pipeline fails for one of the merge requests in the merge train. +- You [skip the merge train and merge immediately](#skip-the-merge-train-and-merge-immediately). +- You [remove a merge request from a merge train](#remove-a-merge-request-from-a-merge-train). + +In these cases, GitLab must create new merge train pipelines for some or all of the +merge requests on the train. The old pipelines were comparing against the previous +combined changes in the merge train, which are no longer valid, so these old pipelines +are cancelled. ## Enable merge trains -To enable merge trains for your project: +Prerequisites: + +- You must have the Maintainer role. +- Your repository must be a GitLab repository, not an [external repository](../ci_cd_for_external_repos/index.md). +- Your pipeline must be [configured to use merge request pipelines](merge_request_pipelines.md#prerequisites). + Otherwise your merge requests may become stuck in an unresolved state or your pipelines + might be dropped. + +To enable merge trains: -1. If you are on a self-managed GitLab instance, ensure the [feature flag](#merge-trains-feature-flag) is set correctly. -1. [Configure your CI/CD configuration file](merge_request_pipelines.md#prerequisites) - so that the pipeline or individual jobs run for merge requests. 1. On the top bar, select **Main menu > Projects** and find your project. 1. On the left sidebar, select **Settings > Merge requests**. 1. In the **Merge method** section, verify that **Merge commit** is selected. -1. In the **Merge options** section, select **Enable merged results pipelines** (if not already selected) and **Enable merge trains**. +1. In the **Merge options** section: + - In GitLab 13.6 and later, select **Enable merged results pipelines** and **Enable merge trains**. + - In GitLab 13.5 and earlier, select **Enable merge trains and pipelines for merged results**. + Additionally, [a feature flag](#disable-merge-trains-in-gitlab-135-and-earlier) + must be set correctly. 1. Select **Save changes**. -In GitLab 13.5 and earlier, there is only one checkbox, named -**Enable merge trains and pipelines for merged results**. +## Start a merge train -WARNING: -If you select the checkbox but don't configure your CI/CD to use -merge request pipelines, your merge requests may become stuck in an -unresolved state or your pipelines may be dropped. +Prerequisites: -## Start a merge train +- You must have [permissions](../../user/permissions.md) to merge or push to the target branch. To start a merge train: 1. Visit a merge request. -1. Select **Start merge train**. +1. Select: + - When no pipeline is running, **Start merge train**. + - When a pipeline is running, **Start merge train when pipeline succeeds**. - +The merge request's merge train status displays under the pipeline widget with a +message similar to `A new merge train has started and this merge request is the first of the queue.` Other merge requests can now be added to the train. ## Add a merge request to a merge train +Prerequisites: + +- You must have [permissions](../../user/permissions.md) to merge or push to the target branch. + To add a merge request to a merge train: 1. Visit a merge request. -1. Select **Add to merge train**. +1. Select: + - When no pipeline is running, **Add to merge train**. + - When a pipeline is running, **Add to merge train when pipeline succeeds**. -If pipelines are already running for the merge request, you cannot add the merge request -to the train. Instead, you can schedule to add the merge request to a merge train **when the latest -pipeline succeeds**. +The merge request's merge train status displays under the pipeline widget with a +message similar to `Added to the merge train. There are 2 merge requests waiting to be merged.` - +Each merge train can run a maximum of twenty pipelines in parallel. If you add more than +twenty merge requests to the merge train, the extra merge requests are queued, waiting +for pipelines to complete. There is no limit to the number of queued merge requests +waiting to join the merge train. ## Remove a merge request from a merge train -1. Visit a merge request. -1. Select **Remove from merge train**. - - +To remove a merge request from a merge train, select **Remove from merge train**. +You can add the merge request to a merge train again later. -If you want to add the merge request to a merge train again later, you can. +When you remove a merge request from a merge train: -## View a merge request's current position on the merge train +- All pipelines for merge requests queued after the removed merge request restart. +- Redundant pipelines [are cancelled](#automatic-pipeline-cancellation). -After a merge request has been added to the merge train, the merge request's -current position is displayed under the pipeline widget: +## Skip the merge train and merge immediately - +If you have a high-priority merge request, like a critical patch that must +be merged urgently, select **Merge Immediately**. -## Immediately merge a merge request with a merge train +When you merge a merge request immediately: -If you have a high-priority merge request (for example, a critical patch) that must -be merged urgently, you can bypass the merge train by using the **Merge Immediately** option. -This is the fastest option to get the change merged into the target branch. - - +- The current merge train is recreated. +- All pipelines restart. +- Redundant pipelines [are cancelled](#automatic-pipeline-cancellation). WARNING: -Each time you merge a merge request immediately, the current merge train is recreated, -all pipelines restart, and [redundant pipelines are cancelled](#automatic-pipeline-cancellation). +Merging immediately can use a lot of CI/CD resources. Use this option +only in critical situations. -### Automatic pipeline cancellation +## Disable merge trains in GitLab 13.5 and earlier **(PREMIUM SELF)** -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/12996) in GitLab 12.3. +In [GitLab 13.6 and later](https://gitlab.com/gitlab-org/gitlab/-/issues/244831), +you can [enable or disable merge trains in the project settings](#enable-merge-trains). -GitLab CI/CD can detect the presence of redundant pipelines, and cancels them -to conserve CI resources. +In GitLab 13.5 and earlier, merge trains are automatically enabled when +[merged results pipelines](merged_results_pipelines.md) are enabled. +To use merged results pipelines but not merge trains, enable the `disable_merge_trains` +[feature flag](../../user/feature_flags.md). -When a user merges a merge request immediately in an ongoing merge -train, the train is reconstructed, because it recreates the expected -post-merge commit and pipeline. In this case, the merge train may already -have pipelines running against the previous expected post-merge commit. -These pipelines are considered redundant and are automatically -canceled. +[GitLab administrators with access to the GitLab Rails console](../../administration/feature_flags.md) +can enable the feature flag to disable merge trains: -## Troubleshooting +```ruby +Feature.enable(:disable_merge_trains) +``` -### Merge request dropped from the merge train immediately +After you enable this feature flag, GitLab cancels existing merge trains and removes +the **Start/Add to merge train** option from merge requests. -If a merge request is not mergeable (for example, it's a draft merge request or it has a merge -conflict), the merge train drops your merge request automatically. +To disable the feature flag, which enables merge trains again: -In these cases, the reason for dropping the merge request is in the **system notes**. +```ruby +Feature.disable(:disable_merge_trains) +``` -To check the reason: +## Troubleshooting -1. Open the merge request that was dropped from the merge train. -1. Select the **Discussion** tab. -1. Find a system note that includes either: - - **... removed this merge request from the merge train because ...** - - **... aborted this merge request from the merge train because ...** +### Merge request dropped from the merge train -The reason is given in the text after the **because ...** phrase. +If a merge request becomes unmergeable while a merge train pipeline is running, +the merge train drops your merge request automatically. For example, this could be caused by: - +- Changing the merge request to a [draft](../../user/project/merge_requests/drafts.md). +- A merge conflict. +- A new conversation thread that is unresolved, when [all threads must be resolved](../../user/discussions/index.md#prevent-merge-unless-all-threads-are-resolved) + is enabled. -### Merge When Pipeline Succeeds cannot be chosen +You can find reason the merge request was dropped from the merge train in the system +notes. Check the **Activity** section in the **Overview** tab for a message similar to: +`User removed this merge request from the merge train because ...` -[Merge When Pipeline Succeeds](../../user/project/merge_requests/merge_when_pipeline_succeeds.md) -is currently unavailable when merge trains are enabled. +### Cannot use merge when pipeline succeeds -See [the related issue](https://gitlab.com/gitlab-org/gitlab/-/issues/12267) +You cannot use [merge when pipeline succeeds](../../user/project/merge_requests/merge_when_pipeline_succeeds.md) +when merge trains are enabled. See [the related issue](https://gitlab.com/gitlab-org/gitlab/-/issues/12267) for more information. -### Merge train pipeline cannot be retried +### Cannot retry merge train pipeline cannot When a merge train pipeline fails, the merge request is dropped from the train and the pipeline can't be retried. Merge train pipelines run on the merged result of the changes in the merge request and -the changes from other merge requests already on the train. If the merge request is dropped from the train, +changes from other merge requests already on the train. If the merge request is dropped from the train, the merged result is out of date and the pipeline can't be retried. -Instead, you should [add the merge request to the train](#add-a-merge-request-to-a-merge-train) -again, which triggers a new pipeline. - -If a job only fails intermittently, you can try using the [`retry`](../yaml/index.md#retry) -keyword in the `.gitlab-ci.yml` file to have the job retried before the pipeline completes. -If it succeeds after a retry, the merge request is not removed from the merge train. +You can: -### Unable to add to merge train with message "The pipeline for this merge request failed." +- [Add the merge request to the train](#add-a-merge-request-to-a-merge-train) again, + which triggers a new pipeline. +- Add the [`retry`](../yaml/index.md#retry) keyword to the job if it fails intermittently. + If it succeeds after a retry, the merge request is not removed from the merge train. -Sometimes the **Start/Add to merge train** button is not available and the merge request says, -"The pipeline for this merge request failed. Please retry the job or push a new commit to fix the failure." +### Unable to add to the merge train -This issue occurs when [**Pipelines must succeed**](../../user/project/merge_requests/merge_when_pipeline_succeeds.md#require-a-successful-pipeline-for-merge) -is enabled in **Settings > General > Merge requests**. This option requires that you -run a new successful pipeline before you can re-add a merge request to a merge train. +When [**Pipelines must succeed**](../../user/project/merge_requests/merge_when_pipeline_succeeds.md#require-a-successful-pipeline-for-merge) +is enabled, but the latest pipeline failed: -Merge trains ensure that each pipeline has succeeded before a merge happens, so -you can: +- The **Start/Add to merge train** option is not available. +- The merge request displays `The pipeline for this merge request failed. Please retry the job or push a new commit to fix the failure.` -- Clear the **Pipelines must succeed** checkbox. -- Select the **Enable merged results pipelines** and **Enable merge trains** checkboxes. +Before you can re-add a merge request to a merge train, you can try to: - In GitLab 13.5 and earlier, there is only one checkbox, named - **Enable merge trains and pipelines for merged results**. - -If you want to keep the **Pipelines must succeed** option selected along with merge -trains, create a new merged results pipeline when this error occurs: - -1. On the **Pipelines** tab, select **Run pipeline**. -1. Select **Start/Add to merge train when pipeline succeeds**. +- Retry the failed job. If it passes, and no other jobs failed, the pipeline is marked as successful. +- Rerun the whole pipeline. On the **Pipelines** tab, select **Run pipeline**. +- Push a new commit that fixes the issue, which also triggers a new pipeline. See [the related issue](https://gitlab.com/gitlab-org/gitlab/-/issues/35135) for more information. - -### Merge trains feature flag **(PREMIUM SELF)** - -In [GitLab 13.6 and later](https://gitlab.com/gitlab-org/gitlab/-/issues/244831), -you can [enable or disable merge trains in the project settings](#enable-merge-trains). - -In GitLab 13.5 and earlier, merge trains are automatically enabled when -[merged results pipelines](merged_results_pipelines.md) are enabled. -To use merged results pipelines without using merge trains, you can enable a -[feature flag](../../user/feature_flags.md) that blocks the merge trains feature. - -[GitLab administrators with access to the GitLab Rails console](../../administration/feature_flags.md) -can enable the feature flag to disable merge trains: - -```ruby -Feature.enable(:disable_merge_trains) -``` - -After you enable this feature flag, all existing merge trains are cancelled and -the **Start/Add to merge train** button no longer appears in merge requests. - -To disable the feature flag, and enable merge trains again: - -```ruby -Feature.disable(:disable_merge_trains) -``` diff --git a/doc/ci/pipelines/pipeline_architectures.md b/doc/ci/pipelines/pipeline_architectures.md index e36eb24055f..1e4d32fc331 100644 --- a/doc/ci/pipelines/pipeline_architectures.md +++ b/doc/ci/pipelines/pipeline_architectures.md @@ -23,13 +23,18 @@ own advantages. These methods can be mixed and matched if needed: - [Multi-project pipelines](downstream_pipelines.md#multi-project-pipelines): Good for larger products that require cross-project interdependencies, like those with a [microservices architecture](https://about.gitlab.com/blog/2016/08/16/trends-in-version-control-land-microservices/). + For example, you might deploy your web application from three different GitLab projects. + With multi-project pipelines you can trigger a pipeline in each project, where each + has its own build, test, and deploy process. You can visualize the connected pipelines + in one place, including all cross-project interdependencies. + <i class="fa fa-youtube-play youtube" aria-hidden="true"></i> For an overview, see the [Multi-project pipelines demo](https://www.youtube.com/watch?v=g_PIwBM1J84). ## Basic Pipelines This is the simplest pipeline in GitLab. It runs everything in the build stage concurrently, -and once all of those finish, it runs everything in the test stage the same way, and so on. +and once all of those finish, it runs everything in the test and subsequent stages the same way. It's not the most efficient, and if you have lots of steps it can grow quite complex, but it's easier to maintain: diff --git a/doc/ci/pipelines/pipeline_efficiency.md b/doc/ci/pipelines/pipeline_efficiency.md index aa2f074693a..276a03cb480 100644 --- a/doc/ci/pipelines/pipeline_efficiency.md +++ b/doc/ci/pipelines/pipeline_efficiency.md @@ -225,7 +225,7 @@ has more information about building efficient Docker images. Methods to reduce Docker image size: - Use a small base image, for example `debian-slim`. -- Do not install convenience tools like vim, curl, and so on, if they aren't strictly needed. +- Do not install convenience tools such as vim or curl if they aren't strictly needed. - Create a dedicated development image. - Disable man pages and docs installed by packages to save space. - Reduce the `RUN` layers and combine software installation steps. diff --git a/doc/ci/pipelines/schedules.md b/doc/ci/pipelines/schedules.md index 0eeb0eada87..02728d5e1e0 100644 --- a/doc/ci/pipelines/schedules.md +++ b/doc/ci/pipelines/schedules.md @@ -95,6 +95,6 @@ important to describe those, too. Think of things that may go wrong and include This is important to minimize requests for support, and to avoid doc comments with questions that you know someone might ask. -Each scenario can be a third-level heading, e.g. `### Getting error message X`. +Each scenario can be a third-level heading, for example `### Getting error message X`. If you have none to add when creating a doc, leave this section in place but commented out to help encourage others to add to it in the future. --> diff --git a/doc/ci/pipelines/settings.md b/doc/ci/pipelines/settings.md index d7155004359..a7f3cfb59d3 100644 --- a/doc/ci/pipelines/settings.md +++ b/doc/ci/pipelines/settings.md @@ -82,9 +82,10 @@ You can set pending or running pipelines to cancel automatically when a new pipe Use the [`interruptible`](../yaml/index.md#interruptible) keyword to indicate if a running job can be cancelled before it completes. -## Skip outdated deployment jobs +## Prevent outdated deployment jobs -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/25276) in GitLab 12.9. +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/25276) in GitLab 12.9. +> - In GitLab 15.5, the behavior was [changed](https://gitlab.com/gitlab-org/gitlab/-/issues/363328) to prevent outdated job runs. Your project may have multiple concurrent deployment jobs that are scheduled to run in the same time frame. @@ -97,29 +98,10 @@ To avoid this scenario: 1. On the top bar, select **Main menu > Projects** and find your project. 1. On the left sidebar, select **Settings > CI/CD**. 1. Expand **General pipelines**. -1. Select the **Skip outdated deployment jobs** checkbox. +1. Select the **Prevent outdated deployment jobs** checkbox. 1. Select **Save changes**. -When a new deployment starts, older deployment jobs are skipped. Skipped jobs are labeled: - -- `forward deployment failure` in the pipeline view. -- `The deployment job is older than the previously succeeded deployment job, and therefore cannot be run` - when viewing the completed job. - -Job age is determined by the job start time, not the commit time, so a newer commit -can be skipped in some circumstances. - -For more information, see [Deployment safety](../environments/deployment_safety.md). - -## Retry outdated jobs - -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/211339) in GitLab 13.6. - -A deployment job can fail because a newer one has run. If you retry the failed deployment job, the -environment could be overwritten with older source code. If you select **Retry**, a modal warns you -about this and asks for confirmation. - -For more information, see [Deployment safety](../environments/deployment_safety.md). +For more information, see [Deployment safety](../environments/deployment_safety.md#prevent-outdated-deployment-jobs). ## Specify a custom CI/CD configuration file @@ -246,53 +228,6 @@ averaged. To add test coverage results to a merge request using the project's `.gitlab-ci.yml` file, provide a regular expression using the [`coverage`](../yaml/index.md#coverage) keyword. -<!-- start_remove The following content will be removed on remove_date: '2023-08-22' --> - -### Add test coverage results using project settings (removed) - -> - [Deprecated](https://gitlab.com/gitlab-org/gitlab/-/issues/17633) in GitLab 14.8. Replaced by [`coverage` keyword](../yaml/index.md#coverage). -> - [Removed](https://gitlab.com/gitlab-org/gitlab/-/issues/17633) in GitLab 15.0. - -This feature is in its end-of-life process. It was [deprecated](https://gitlab.com/gitlab-org/gitlab/-/issues/17633) -in GitLab 14.8. The feature is [removed](https://gitlab.com/gitlab-org/gitlab/-/issues/17633) in GitLab 15.0. - -To migrate from a project setting to the `coverage` keyword, add the [former project setting](#locate-former-project-setting) -to a CI/CD job. For example: - -- A Go test coverage project setting: `coverage: \d+.\d+% of statements`. -- A CI/CD job with `coverage` keyword setting: - - ```yaml - unit-test: - stage: test - coverage: '/coverage: \d+.\d+% of statements/' - script: - - go test -cover - ``` - -The `.gitlab-ci.yml` job [`coverage`](../yaml/index.md#coverage) keyword must be: - -- A regular expression starts and ends with the `/` character. -- Defined as single-quoted string. - -You can verify correct syntax using the [pipeline editor](../pipeline_editor/index.md). - -#### Locate former project setting - -To migrate from the project coverage setting to the `coverage` keyword, use the -regular expression displayed in the settings. Available in GitLab 14.10 and earlier: - -1. On the top bar, select **Main menu > Projects** and find your project. -1. On the left sidebar, select **Settings > CI/CD**. -1. Expand **General pipelines**. - -The regular expression you need is in the **Test coverage parsing** field. - -If you need to retrieve the project coverage setting from many projects, you can -[use the API to programmatically retrieve the setting](https://gitlab.com/gitlab-org/gitlab/-/issues/17633#note_945941397). - -<!-- end_remove --> - ### Test coverage examples Use this regex for commonly used test tools. @@ -302,6 +237,7 @@ Use this regex for commonly used test tools. - Simplecov (Ruby). Example: `/\(\d+.\d+\%\) covered/`. - pytest-cov (Python). Example: `/(?i)total.*? (100(?:\.0+)?\%|[1-9]?\d(?:\.\d+)?\%)$/`. - Scoverage (Scala). Example: `/Statement coverage[A-Za-z\.*]\s*:\s*([^%]+)/`. +- `pest --coverage --colors=never` (PHP). Example: `/^\s*Cov:\s*\d+\.\d+?%$/`. - `phpunit --coverage-text --colors=never` (PHP). Example: `/^\s*Lines:\s*\d+.\d+\%/`. - gcovr (C/C++). Example: `/^TOTAL.*\s+(\d+\%)$/`. - `tap --coverage-report=text-summary` (NodeJS). Example: `/^Statements\s*:\s*([^%]+)/`. @@ -437,12 +373,12 @@ https://gitlab.example.com/<namespace>/<project>/badges/<branch>/coverage.svg ``` To get the coverage report from a specific job, add -the `job=coverage_job_name` parameter to the URL. For example, the following -Markdown code embeds the test coverage report badge of the `coverage` job -in your `README.md`: +the `job=coverage_job_name` parameter to the URL. For example, you can use code +similar to the following to add the test coverage report badge of the `coverage` job +to a Markdown file: ```markdown - + ``` #### Test coverage report badge colors and limits @@ -527,6 +463,6 @@ important to describe those, too. Think of things that may go wrong and include This is important to minimize requests for support, and to avoid doc comments with questions that you know someone might ask. -Each scenario can be a third-level heading, e.g. `### Getting error message X`. +Each scenario can be a third-level heading, for example `### Getting error message X`. If you have none to add when creating a doc, leave this section in place but commented out to help encourage others to add to it in the future. --> diff --git a/doc/ci/quick_start/index.md b/doc/ci/quick_start/index.md index ad41e4fa88a..8d71f4569e5 100644 --- a/doc/ci/quick_start/index.md +++ b/doc/ci/quick_start/index.md @@ -1,50 +1,41 @@ --- stage: Verify -group: Pipeline Execution +group: Pipeline Authoring info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments type: reference --- -# Get started with GitLab CI/CD **(FREE)** +# Tutorial: Create and run your first GitLab CI/CD pipeline **(FREE)** -Use this document to get started with [GitLab CI/CD](../index.md). +This tutorial shows you how to configure and run your first CI/CD pipeline in GitLab. + +## Prerequisites Before you start, make sure you have: - A project in GitLab that you would like to use CI/CD for. - The Maintainer or Owner role for the project. -If you are migrating from another CI/CD tool, view this documentation: - -- [Migrate from CircleCI](../migration/circleci.md). -- [Migrate from Jenkins](../migration/jenkins.md). +If you don't have a project, you can create a public project for free on <https://gitlab.com>. -> - <i class="fa fa-youtube-play youtube" aria-hidden="true"></i> Watch [First time GitLab & CI/CD](https://www.youtube.com/watch?v=kTNfi5z6Uvk&t=553s). This includes a quick introduction to GitLab, the first steps with CI/CD, building a Go project, running tests, using the CI/CD pipeline editor, detecting secrets and security vulnerabilities and offers more exercises for asynchronous practice. -> - <i class="fa fa-youtube-play youtube" aria-hidden="true"></i> Watch [Intro to GitLab CI](https://www.youtube.com/watch?v=l5705U8s_nQ&t=358s). This workshop uses the Web IDE to quickly get going with building source code using CI/CD, and run unit tests. +## Steps -## CI/CD process overview - -To use GitLab CI/CD: +To create and run your first pipeline: 1. [Ensure you have runners available](#ensure-you-have-runners-available) to run your jobs. - GitLab SaaS provides runners, so if you're using GitLab.com, you can skip this step. - If you don't have a runner, [install GitLab Runner](https://docs.gitlab.com/runner/install/) - and [register a runner](https://docs.gitlab.com/runner/register/) for your instance, project, or group. + If you're using GitLab.com, you can skip this step. GitLab.com provides shared runners for you. + 1. [Create a `.gitlab-ci.yml` file](#create-a-gitlab-ciyml-file) - at the root of your repository. This file is where you define your CI/CD jobs. + at the root of your repository. This file is where you define the CI/CD jobs. When you commit the file to your repository, the runner runs your jobs. The job results [are displayed in a pipeline](#view-the-status-of-your-pipeline-and-jobs). -### Ensure you have runners available +## Ensure you have runners available In GitLab, runners are agents that run your CI/CD jobs. -You might already have runners available for your project, including -[shared runners](../runners/runners_scope.md), which are -available to all projects in your GitLab instance. - To view available runners: - Go to **Settings > CI/CD** and expand **Runners**. @@ -52,34 +43,32 @@ To view available runners: As long as you have at least one runner that's active, with a green circle next to it, you have a runner available to process your jobs. -If no runners are listed on the **Runners** page in the UI, you or an administrator -must [install GitLab Runner](https://docs.gitlab.com/runner/install/) and -[register](https://docs.gitlab.com/runner/register/) at least one runner. +### If you don't have a runner + +If you don't have a runner: + +1. [Install GitLab Runner](https://docs.gitlab.com/runner/install/) on your local machine. +1. [Register the runner](https://docs.gitlab.com/runner/register/) for your project. + Choose the `shell` executor. -If you are testing CI/CD, you can install GitLab Runner and register runners on your local machine. -When your CI/CD jobs run, they run on your local machine. +When your CI/CD jobs run, in a later step, they will run on your local machine. -### Create a `.gitlab-ci.yml` file +## Create a `.gitlab-ci.yml` file -The `.gitlab-ci.yml` file is a [YAML](https://en.wikipedia.org/wiki/YAML) file where -you configure specific instructions for GitLab CI/CD. +Now create a `.gitlab-ci.yml` file. It is a [YAML](https://en.wikipedia.org/wiki/YAML) file where +you specify instructions for GitLab CI/CD. In this file, you define: - The structure and order of jobs that the runner should execute. - The decisions the runner should make when specific conditions are encountered. -For example, you might want to run a suite of tests when you commit to -any branch except the default branch. When you commit to the default branch, you want -to run the same suite, but also publish your application. - -All of this is defined in the `.gitlab-ci.yml` file. - To create a `.gitlab-ci.yml` file: -1. On the left sidebar, select **Project information > Details**. -1. Above the file list, select the branch you want to commit to, - select the plus icon, then select **New file**: +1. On the left sidebar, select **Repository > Files**. +1. Above the file list, select the branch you want to commit to. + If you're not sure, leave `master` or `main`. + Then select the plus icon (**{plus}**) and **New file**:  @@ -112,46 +101,50 @@ To create a `.gitlab-ci.yml` file: environment: production ``` - `$GITLAB_USER_LOGIN` and `$CI_COMMIT_BRANCH` are - [predefined variables](../variables/predefined_variables.md) - that populate when the job runs. + This example shows four jobs: `build-job`, `test-job1`, `test-job2`, and `deploy-prod`. + The comments listed in the `echo` commands are displayed in the UI when you view the jobs. + The values for the [predefined variables](../variables/predefined_variables.md) + `$GITLAB_USER_LOGIN` and `$CI_COMMIT_BRANCH` are populated when the jobs run. 1. Select **Commit changes**. -The pipeline starts when the commit is committed. +The pipeline starts and runs the jobs you defined in the `.gitlab-ci.yml` file. + +## View the status of your pipeline and jobs + +Now take a look at your pipeline and the jobs within. + +1. Go to **CI/CD > Pipelines**. A pipeline with three stages should be displayed: + +  -#### `.gitlab-ci.yml` tips +1. View a visual representation of your pipeline by selecting the pipeline ID: -- After you create your first `.gitlab-ci.yml` file, use the [pipeline editor](../pipeline_editor/index.md) - for all future edits to the file. With the pipeline editor, you can: - - Edit the pipeline configuration with automatic syntax highlighting and validation. - - View the [CI/CD configuration visualization](../pipeline_editor/index.md#visualize-ci-configuration), - a graphical representation of your `.gitlab-ci.yml` file. -- If you want the runner to [use a Docker container to run the jobs](../docker/using_docker_images.md), - edit the `.gitlab-ci.yml` file - to include an image name: +  - ```yaml - default: - image: ruby:2.7.5 - ``` +1. View details of a job by selecting the job name. For example, `deploy-prod`: - This command tells the runner to use a Ruby image from Docker Hub - and to run the jobs in a container that's generated from the image. +  - This process is different than - [building an application as a Docker container](../docker/using_docker_build.md). - Your application does not need to be built as a Docker container to - run CI/CD jobs in Docker containers. +You have successfully created your first CI/CD pipeline in GitLab. Congratulations! -- Each job contains scripts and stages: +Now you can get started customizing your `.gitlab-ci.yml` and defining more advanced jobs. + +## `.gitlab-ci.yml` tips + +Here are some tips to get started working with the `.gitlab-ci.yml` file. + +For the complete `.gitlab-ci.yml` syntax, see [the full `.gitlab-ci.yml` keyword reference](../yaml/index.md). + +- Use the [pipeline editor](../pipeline_editor/index.md) to edit your `.gitlab-ci.yml` file. +- Each job contains a script section and belongs to a stage: - The [`default`](../yaml/index.md#default) keyword is for custom defaults, for example with [`before_script`](../yaml/index.md#before_script) and [`after_script`](../yaml/index.md#after_script). - [`stage`](../yaml/index.md#stage) describes the sequential execution of jobs. Jobs in a single stage run in parallel as long as there are available runners. - - Use [Directed Acyclic Graphs (DAG)](../directed_acyclic_graph/index.md) keywords - to run jobs out of stage order. + - Use the [`needs` keyword](../yaml/index.md#needs) to run jobs out of stage order. + This creates a [Directed Acyclic Graph (DAG)](../directed_acyclic_graph/index.md). - You can set additional configuration to customize how your jobs and stages perform: - Use the [`rules`](../yaml/index.md#rules) keyword to specify when to run or skip jobs. The `only` and `except` legacy keywords are still supported, but can't be used @@ -159,26 +152,10 @@ The pipeline starts when the commit is committed. - Keep information across jobs and stages persistent in a pipeline with [`cache`](../yaml/index.md#cache) and [`artifacts`](../yaml/index.md#artifacts). These keywords are ways to store dependencies and job output, even when using ephemeral runners for each job. -- For the complete `.gitlab-ci.yml` syntax, see [the full `.gitlab-ci.yml` reference topic](../yaml/index.md). - -### View the status of your pipeline and jobs - -When you committed your changes, a pipeline started. - -To view your pipeline: - -- Go to **CI/CD > Pipelines**. - - A pipeline with three stages should be displayed: - -  - -- To view a visual representation of your pipeline, select the pipeline ID. - -  - -- To view details of a job, select the job name, for example, `deploy-prod`. -  +## Related topics -If the job status is `stuck`, check to ensure a runner is properly configured for the project. +- [Follow this guide to migrate from CircleCI](../migration/circleci.md). +- [Follow this guide to migrate from Jenkins](../migration/jenkins.md). +- <i class="fa fa-youtube-play youtube" aria-hidden="true"></i> Watch [First time GitLab & CI/CD](https://www.youtube.com/watch?v=kTNfi5z6Uvk&t=553s). This includes a quick introduction to GitLab, the first steps with CI/CD, building a Go project, running tests, using the CI/CD pipeline editor, detecting secrets and security vulnerabilities and offers more exercises for asynchronous practice. +- <i class="fa fa-youtube-play youtube" aria-hidden="true"></i> Watch [Intro to GitLab CI](https://www.youtube.com/watch?v=l5705U8s_nQ&t=358s). This workshop uses the Web IDE to quickly get going with building source code using CI/CD, and run unit tests. diff --git a/doc/ci/resource_groups/index.md b/doc/ci/resource_groups/index.md index 2803ddba828..b46008abb07 100644 --- a/doc/ci/resource_groups/index.md +++ b/doc/ci/resource_groups/index.md @@ -89,7 +89,7 @@ The following modes are supported: that are sorted by pipeline ID in descending order. This mode is efficient when you want to ensure that the jobs are executed from the newest pipeline and - cancel all of the old deploy jobs with the [skip outdated deployment jobs](../environments/deployment_safety.md#skip-outdated-deployment-jobs) feature. + prevent all of the old deploy jobs with the [prevent outdated deployment jobs](../environments/deployment_safety.md#prevent-outdated-deployment-jobs) feature. This is the most efficient option in terms of the pipeline efficiency, but you must ensure that each deployment job is idempotent. ### Change the process mode @@ -171,7 +171,6 @@ deploy: include: deploy.gitlab-ci.yml strategy: depend resource_group: AWS-production - environment: production ``` ```yaml diff --git a/doc/ci/review_apps/index.md b/doc/ci/review_apps/index.md index 2f23ebbfd9f..7a0f43f9ec5 100644 --- a/doc/ci/review_apps/index.md +++ b/doc/ci/review_apps/index.md @@ -4,96 +4,96 @@ group: Pipeline Insights info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- -# Review Apps **(FREE)** +# Review apps **(FREE)** -Review Apps is a collaboration tool that assists with providing an environment to showcase product changes. +Review apps are a collaboration tool that provide an environment to showcase product changes. NOTE: If you have a Kubernetes cluster, you can automate this feature in your applications by using [Auto DevOps](../../topics/autodevops/index.md). -Review Apps: +Review apps: - Provide an automatic live preview of changes made in a feature branch by spinning up a dynamic environment for your merge requests. - Allow designers and product managers to see your changes without needing to check out your branch and run your changes in a sandbox environment. - Are fully integrated with the [GitLab DevOps LifeCycle](../../index.md#the-entire-devops-lifecycle). - Allow you to deploy your changes wherever you want. - + In the previous example: -- A Review App is built every time a commit is pushed to `topic branch`. +- A review app is built every time a commit is pushed to `topic branch`. - The reviewer fails two reviews before passing the third review. - After the review passes, `topic branch` is merged into the default branch, where it's deployed to staging. - After its approval in staging, the changes that were merged into the default branch are deployed to production. -## How Review Apps work +## How review apps work -A Review App is a mapping of a branch with an [environment](../environments/index.md). -Access to the Review App is made available as a link on the [merge request](../../user/project/merge_requests/index.md) relevant to the branch. +A review app is a mapping of a branch with an [environment](../environments/index.md). +Access to the review app is made available as a link on the [merge request](../../user/project/merge_requests/index.md) relevant to the branch. The following is an example of a merge request with an environment set dynamically. - + In this example, a branch was: - Successfully built. - Deployed under a dynamic environment that can be reached by selecting **View app**. -After adding Review Apps to your workflow, you follow the branched Git flow. That is: +After adding review apps to your workflow, you follow the branched Git flow. That is: -1. Push a branch and let the runner deploy the Review App based on the `script` definition of the dynamic environment job. +1. Push a branch and let the runner deploy the review app based on the `script` definition of the dynamic environment job. 1. Wait for the runner to build and deploy your web application. 1. To view the changes live, select the link in the merge request related to the branch. -## Configuring Review Apps +## Configuring review apps -Review Apps are built on [dynamic environments](../environments/index.md#create-a-dynamic-environment), which allow you to dynamically create a new environment for each branch. +Review apps are built on [dynamic environments](../environments/index.md#create-a-dynamic-environment), which allow you to dynamically create a new environment for each branch. -The process of configuring Review Apps is as follows: +The process of configuring review apps is as follows: -1. Set up the infrastructure to host and deploy the Review Apps (check the [examples](#review-apps-examples) below). +1. Set up the infrastructure to host and deploy the review apps (check the [examples](#review-apps-examples) below). 1. [Install](https://docs.gitlab.com/runner/install/) and [configure](https://docs.gitlab.com/runner/commands/) a runner to do deployment. 1. Set up a job in `.gitlab-ci.yml` that uses the [predefined CI/CD variable](../variables/index.md) `${CI_COMMIT_REF_SLUG}` to create dynamic environments and restrict it to run only on branches. - Alternatively, you can get a YAML template for this job by [enabling review apps](#enable-review-apps-button) for your project. -1. Optionally, set a job that [manually stops](../environments/index.md#stop-an-environment) the Review Apps. + Alternatively, you can get a YAML template for this job by [enabling review apps](#enable-review-app-button) for your project. +1. Optionally, set a job that [manually stops](../environments/index.md#stop-an-environment) the review apps. -### Enable Review Apps button +### Enable review app button > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/118844) in GitLab 12.8. -When configuring Review Apps for a project, you add a new job to the `.gitlab-ci.yml` file, +When configuring review apps for a project, you add a new job to the `.gitlab-ci.yml` file, as mentioned above. To facilitate this, and if you are using Kubernetes, you can select -**Enable Review Apps** and GitLab prompts you with a template code block that +**Enable review app** and GitLab prompts you with a template code block that you can copy and paste into `.gitlab-ci.yml` as a starting point. Prerequisite: - You need at least the Developer role for the project. -To use the Review Apps template: +To use the review apps template: -1. On the top bar, select **Main menu > Projects** and find the project you want to create a Review App job for. +1. On the top bar, select **Main menu > Projects** and find the project you want to create a review app job for. 1. On the left sidebar, select **Deployments > Environments**. -1. Select **Enable Review Apps**. +1. Select **Enable review app**. 1. Copy the provided code snippet and paste it into your `.gitlab-ci.yml` file: -  +  You can edit this template as needed. -## Review Apps auto-stop +## Review apps auto-stop -See how to [configure Review Apps environments to expire and auto-stop](../environments/index.md#stop-an-environment-after-a-certain-time-period) +See how to [configure review apps environments to expire and auto-stop](../environments/index.md#stop-an-environment-after-a-certain-time-period) after a given period of time. -## Review Apps examples +## Review apps examples -The following are example projects that demonstrate Review App configuration: +The following are example projects that demonstrate review app configuration: | Project | Configuration file | |-------------------------------------------------------------------------|--------------------| @@ -104,17 +104,17 @@ The following are example projects that demonstrate Review App configuration: | [`https://about.gitlab.com/`](https://gitlab.com/gitlab-com/www-gitlab-com/) | [`.gitlab-ci.yml`](https://gitlab.com/gitlab-com/www-gitlab-com/-/blob/6ffcdc3cb9af2abed490cbe5b7417df3e83cd76c/.gitlab-ci.yml#L332) | | [GitLab Insights](https://gitlab.com/gitlab-org/gitlab-insights/) | [`.gitlab-ci.yml`](https://gitlab.com/gitlab-org/gitlab-insights/-/blob/9e63f44ac2a5a4defc965d0d61d411a768e20546/.gitlab-ci.yml#L234) | -Other examples of Review Apps: +Other examples of review apps: - <i class="fa fa-youtube-play youtube" aria-hidden="true"></i> [Cloud Native Development with GitLab](https://www.youtube.com/watch?v=jfIyQEwrocw). -- [Review Apps for Android](https://about.gitlab.com/blog/2020/05/06/how-to-create-review-apps-for-android-with-gitlab-fastlane-and-appetize-dot-io/). +- [Review apps for Android](https://about.gitlab.com/blog/2020/05/06/how-to-create-review-apps-for-android-with-gitlab-fastlane-and-appetize-dot-io/). ## Route Maps Route Maps allows you to go directly from source files to public pages on the [environment](../environments/index.md) defined for -Review Apps. +review apps. Once set up, the review app link in the merge request widget can take you directly to the pages changed, making it easier @@ -198,7 +198,7 @@ After you have the route mapping set up, it takes effect in the following locati > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/10761) in GitLab 12.0. > - [Moved](https://about.gitlab.com/blog/2021/01/26/new-gitlab-product-subscription-model/) to GitLab Premium in 13.9. > - It's [deployed behind a feature flag](../../user/feature_flags.md), `anonymous_visual_review_feedback`, disabled by default. -> - It's enabled on GitLab.com. +> - It's disabled on GitLab.com. FLAG: On self-managed GitLab, by default this feature is not available. To make it available, @@ -209,7 +209,7 @@ With Visual Reviews, members of any team (Product, Design, Quality, and so on) c ### Using Visual Reviews After Visual Reviews has been [configured](#configure-review-apps-for-visual-reviews) for the -Review App, the Visual Reviews feedback form is overlaid on the right side of every page. +review app, the Visual Reviews feedback form is overlaid on the right side of every page.  @@ -227,9 +227,9 @@ To use the feedback form to make a comment in the merge request: <i class="fa fa-youtube-play youtube" aria-hidden="true"></i> To see Visual reviews in action, see the [Visual Reviews Walk through](https://youtu.be/1_tvWTlPfM4). -### Configure Review Apps for Visual Reviews +### Configure review apps for Visual Reviews -The feedback form is served through a script you add to pages in your Review App. +The feedback form is served through a script you add to pages in your review app. It should be added to the `<head>` of your application and consists of some project and merge request specific values. Here's how it looks for a project with code hosted in a project on GitLab.com: diff --git a/doc/ci/runners/configure_runners.md b/doc/ci/runners/configure_runners.md index 19e0c1e3c81..c675f7204ec 100644 --- a/doc/ci/runners/configure_runners.md +++ b/doc/ci/runners/configure_runners.md @@ -97,6 +97,13 @@ Whenever a project is forked, it copies the settings of the jobs that relate to it. This means that if you have shared runners set up for a project and someone forks that project, the shared runners serve jobs of this project. +Because of a [known issue](https://gitlab.com/gitlab-org/gitlab/-/issues/364303), you might encounter the message `An error occurred while forking the project. Please try again.` if the runner settings of the project you are forking does not match the new project namespace. + +To work around this issue, you should make sure that the shared runner settings are consistent in the forked project and the new namespace. + +- If shared runners are **enabled** on the forked project, then this should also be **enabled** on the new namespace. +- If shared runners are **disabled** on the forked project, then this should also be **disabled** on the new namespace. + ### Attack vectors in runners Mentioned briefly earlier, but the following things of runners can be exploited. @@ -299,12 +306,14 @@ globally or for individual jobs: - [`GIT_STRATEGY`](#git-strategy) - [`GIT_SUBMODULE_STRATEGY`](#git-submodule-strategy) +- [`GIT_SUBMODULE_PATHS`](#sync-or-exclude-specific-submodules-from-ci-jobs) - [`GIT_CHECKOUT`](#git-checkout) - [`GIT_CLEAN_FLAGS`](#git-clean-flags) - [`GIT_FETCH_EXTRA_FLAGS`](#git-fetch-extra-flags) - [`GIT_SUBMODULE_PATHS`](#git-submodule-paths) - [`GIT_SUBMODULE_UPDATE_FLAGS`](#git-submodule-update-flags) - [`GIT_DEPTH`](#shallow-cloning) (shallow cloning) +- [`GIT_SUBMODULE_DEPTH`](#git-submodule-depth) - [`GIT_CLONE_PATH`](#custom-build-directories) (custom build directories) - [`TRANSFER_METER_FREQUENCY`](#artifact-and-cache-settings) (artifact/cache meter update frequency) - [`ARTIFACT_COMPRESSION_LEVEL`](#artifact-and-cache-settings) (artifact archiver compression level) @@ -556,6 +565,34 @@ You should be aware of the implications for the security, stability, and reprodu your builds when using the `--remote` flag. In most cases, it is better to explicitly track submodule commits as designed, and update them using an auto-remediation/dependency bot. +### Sync or exclude specific submodules from CI jobs + +> [Introduced](https://gitlab.com/gitlab-org/gitlab-runner/-/issues/26495) in GitLab Runner 14.0. + +Some projects have a large number of submodules, and not all of them need to be +synced or updated in all CI jobs. Use the `GIT_SUBMODULE_PATHS` variable to control this behavior. +The path syntax is the same as [`git submodule`](https://git-scm.com/docs/git-submodule#Documentation/git-submodule.txt-ltpathgt82308203): + +- To sync and update specific paths: + + ```yaml + variables: + GIT_SUBMODULE_PATHS: 'submoduleA' + ``` + +- To exclude specific paths: + + ```yaml + variables: + GIT_SUBMODULE_PATHS: ':(exclude)submoduleA' + ``` + +WARNING: +Git ignores nested and multiple submodule paths. To ignore a nested submodule, exclude +the parent submodule and then manually clone it in the job's scripts. For example, + `git clone <repo> --recurse-submodules=':(exclude)nested-submodule'`. Make sure +to wrap the string in single quotes so the YAML can be parsed successfully. + ### Shallow cloning > Introduced in GitLab 8.9 as an experimental feature. @@ -590,6 +627,24 @@ variables: You can set it globally or per-job in the [`variables`](../yaml/index.md#variables) section. +### Git submodule depth + +> [Introduced](https://gitlab.com/gitlab-org/gitlab-runner/-/merge_requests/3651) in GitLab Runner 15.5. + +Use the `GIT_SUBMODULE_DEPTH` variable to specify the depth of fetching and cloning submodules +when [`GIT_SUBMODULE_STRATEGY`](#git-submodule-strategy) is set to either `normal` or `recursive`. +You can set it globally or for a specific job in the [`variables`](../yaml/index.md#variables) section. + +When you set the `GIT_SUBMODULE_DEPTH` variable, it overwrites the [`GIT_DEPTH`](#shallow-cloning) setting +for the submodules only. + +To fetch or clone only the last 3 commits: + +```yaml +variables: + GIT_SUBMODULE_DEPTH: 3 +``` + ### Custom build directories > [Introduced](https://gitlab.com/gitlab-org/gitlab-runner/-/issues/2211) in GitLab Runner 11.10. @@ -752,7 +807,7 @@ variables: NOTE: Zip archives are the only supported artifact type. Follow [the issue for details](https://gitlab.com/gitlab-org/gitlab/-/issues/367203). -GitLab Runner can generate and produce attestation metadata for all build artifacts. To enable this feature, you must set the `RUNNER_GENERATE_ARTIFACTS_METADATA` environment variable to `true`. This variable can either be set globally or it can be set for individual jobs. The metadata is in rendered in a plain text `.json` file that's stored with the artifact. The file name is as follows: `{JOB_ID}-artifacts-metadata.json`. +GitLab Runner can generate and produce attestation metadata for all build artifacts. To enable this feature, you must set the `RUNNER_GENERATE_ARTIFACTS_METADATA` environment variable to `true`. This variable can either be set globally or it can be set for individual jobs. The metadata is in rendered in a plain text `.json` file that's stored with the artifact. The file name is as follows: `{ARTIFACT_NAME}-metadata.json` where `ARTIFACT_NAME` is what was defined as the [name for the artifact](../pipelines/job_artifacts.md#use-cicd-variables-to-define-the-artifacts-name) in the CI file. The file name, however, defaults to `artifacts-metadata.json` if no name was given to the build artifacts. ### Attestation format diff --git a/doc/ci/runners/index.md b/doc/ci/runners/index.md index 405310fb8ba..41cde709143 100644 --- a/doc/ci/runners/index.md +++ b/doc/ci/runners/index.md @@ -1,6 +1,6 @@ --- stage: Verify -group: Runner +group: Runner SaaS info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments type: reference --- diff --git a/doc/ci/runners/runners_scope.md b/doc/ci/runners/runners_scope.md index 4be4ed33feb..6354a519810 100644 --- a/doc/ci/runners/runners_scope.md +++ b/doc/ci/runners/runners_scope.md @@ -192,6 +192,25 @@ You must have the Owner role for the group. From this page, you can edit, pause, and remove runners from the group, its subgroups, and projects. +#### Delete multiple group runners + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/361721/) in GitLab 15.6 + +Prerequisites: + +- You must have either: + - Owner role for the group. + - Access to delete any runners in the group. + +To delete multiple runners in a single action in the group list: + +1. On the top bar, select **Main menu > Groups** and find your group. +1. On the left sidebar, select **CI/CD > Runners**. +1. To delete multiple runners, you can either: + - Select the checkbox next to the runner. + - Select the checkbox at the top of the runner list to select all runners in the list. +1. To delete the runners, select **Delete selected**. + #### Filter group runners to show only inherited > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/337838/) in GitLab 15.5. diff --git a/doc/ci/runners/saas/linux_saas_runner.md b/doc/ci/runners/saas/linux_saas_runner.md index d1c63ab7291..df7d5570953 100644 --- a/doc/ci/runners/saas/linux_saas_runner.md +++ b/doc/ci/runners/saas/linux_saas_runner.md @@ -116,7 +116,7 @@ The `CI_PRE_CLONE_SCRIPT` variable does not work on GitLab SaaS Windows or macOS This example was used in the `gitlab-org/gitlab` project until November 2021. The project no longer uses this optimization because the [pack-objects cache](../../../administration/gitaly/configure_gitaly.md#pack-objects-cache) -lets Gitaly serve the full CI/CD fetch traffic. See [Git fetch caching](../../../development/pipelines.md#git-fetch-caching). +lets Gitaly serve the full CI/CD fetch traffic. See [Git fetch caching](../../../development/pipelines/performance.md#git-fetch-caching). The `CI_PRE_CLONE_SCRIPT` was defined as a project CI/CD variable: diff --git a/doc/ci/runners/saas/macos_saas_runner.md b/doc/ci/runners/saas/macos_saas_runner.md index 50c24349712..cd40aac25bc 100644 --- a/doc/ci/runners/saas/macos_saas_runner.md +++ b/doc/ci/runners/saas/macos_saas_runner.md @@ -14,7 +14,7 @@ Use these runners to build, test, and deploy apps for the Apple ecosystem (macOS of all the capabilities of the GitLab single DevOps platform and not have to manage or operate a build environment. -Jobs handled by macOS shared runners on GitLab.com **time out after 2 hours**, regardless of the timeout configured in a project. +Jobs handled by macOS shared runners on GitLab.com **time out after 3 hours**, regardless of the timeout configured in a project. ## Access request process diff --git a/doc/ci/secrets/index.md b/doc/ci/secrets/index.md index 62350905bd4..a5082af89bc 100644 --- a/doc/ci/secrets/index.md +++ b/doc/ci/secrets/index.md @@ -20,11 +20,11 @@ required by a job. Read [GitLab CI/CD pipeline configuration reference](../yaml/ for more information about the syntax. GitLab has selected [Vault by HashiCorp](https://www.vaultproject.io) as the -first supported provider, and [KV-V2](https://www.vaultproject.io/docs/secrets/kv/kv-v2) +first supported provider, and [KV-V2](https://developer.hashicorp.com/vault/docs/secrets/kv/kv-v2) as the first supported secrets engine. GitLab authenticates using Vault's -[JSON Web Token (JWT) authentication method](https://www.vaultproject.io/docs/auth/jwt#jwt-authentication), using +[JSON Web Token (JWT) authentication method](https://developer.hashicorp.com/vault/docs/auth/jwt#jwt-authentication), using the [JSON Web Token](https://gitlab.com/gitlab-org/gitlab/-/issues/207125) (`CI_JOB_JWT`) introduced in GitLab 12.10. @@ -88,10 +88,10 @@ To configure your Vault server: - `VAULT_SERVER_URL` - The URL of your Vault server, such as `https://vault.example.com:8200`. Required. - `VAULT_AUTH_ROLE` - Optional. The role to use when attempting to authenticate. - If no role is specified, Vault uses the [default role](https://www.vaultproject.io/api-docs/auth/jwt#default_role) + If no role is specified, Vault uses the [default role](https://developer.hashicorp.com/vault/api-docs/auth/jwt#default_role) specified when the authentication method was configured. - `VAULT_AUTH_PATH` - Optional. The path where the authentication method is mounted, default is `jwt`. - - `VAULT_NAMESPACE` - Optional. The [Vault Enterprise namespace](https://www.vaultproject.io/docs/enterprise/namespaces) to use for reading secrets and authentication. + - `VAULT_NAMESPACE` - Optional. The [Vault Enterprise namespace](https://developer.hashicorp.com/vault/docs/enterprise/namespaces) to use for reading secrets and authentication. If no namespace is specified, Vault uses the `root` ("`/`") namespace. The setting is ignored by Vault Open Source. @@ -142,7 +142,7 @@ When a CI job attempts to authenticate, it specifies a role. You can use roles t different policies together. If authentication is successful, these policies are attached to the resulting Vault token. -[Bound claims](https://www.vaultproject.io/docs/auth/jwt#bound-claims) are predefined +[Bound claims](https://developer.hashicorp.com/vault/docs/auth/jwt#bound-claims) are predefined values that are matched to the JWT's claims. With bounded claims, you can restrict access to specific GitLab users, specific projects, or even jobs running for specific Git references. You can have as many bounded claims you need, but they must *all* match @@ -183,7 +183,7 @@ For a full list of `CI_JOB_JWT` claims, read the You can also specify some attributes for the resulting Vault tokens, such as time-to-live, IP address range, and number of uses. The full list of options is available in -[Vault's documentation on creating roles](https://www.vaultproject.io/api-docs/auth/jwt#create-role) +[Vault's documentation on creating roles](https://developer.hashicorp.com/vault/api-docs/auth/jwt#create-role) for the JSON web token method. ## Using a self-signed Vault server diff --git a/doc/ci/services/index.md b/doc/ci/services/index.md index 0f82f2301c7..830b9406c6e 100644 --- a/doc/ci/services/index.md +++ b/doc/ci/services/index.md @@ -37,7 +37,7 @@ the CI container itself. ## How services are linked to the job To better understand how container linking works, read -[Linking containers together](https://docs.docker.com/engine/userguide/networking/default_network/dockerlinks/). +[Linking containers together](https://docs.docker.com/network/links/). If you add `mysql` as service to your application, the image is used to create a container that's linked to the job container. @@ -392,6 +392,46 @@ time. 1. Check the exit status of build script. 1. Remove the build container and all created service containers. +## Capturing service container logs + +Logs generated by applications running in service containers can be captured for subsequent examination and debugging. +You might want to look at service container's logs when the service container has started successfully, but is not +behaving es expected, leading to job failures. The logs can indicate missing or incorrect configuration of the service +within the container. + +`CI_DEBUG_SERVICES` should only by enabled when service containers are being actively debugged as there are both storage +and performance consequences to capturing service container logs. + +To enable service logging, add the `CI_DEBUG_SERVICES` variable to the project's +`.gitlab-ci.yml` file: + +```yaml +variables: + CI_DEBUG_SERVICES: "true" +``` + +Accepted values are: + +- Enabled: `TRUE`, `true`, `True` +- Disabled: `FALSE`, `false`, `False` + +Any other values will result in an error message and effectively disable the feature. + +When enabled, logs for _all_ service containers will be captured and streamed into the jobs trace log concurrently with +other logs. Logs from each container will be prefixed with the container's aliases, and displayed in a different color. + +NOTE: +You may want to adjust the logging level in the service container for which you want to capture logs since the default +logging level may not provide sufficient details to diagnose job failures. + +WARNING: +Enabling `CI_DEBUG_SERVICES` _may_ result in masked variables being revealed. When `CI_DEBUG_SERVICES` is enabled, +service container logs and the CI job's logs are streamed to the job's trace log _concurrently_, which makes it possible +for a service container log to be inserted _inside_ a job's masked log. This would thwart the variable masking mechanism +and result in the masked variable being revealed. + +See [Mask a CI/CD Variable](../variables/index.md#mask-a-cicd-variable) + ## Debug a job locally The following commands are run without root privileges. You should be diff --git a/doc/ci/testing/code_quality.md b/doc/ci/testing/code_quality.md index 7345c7ca5eb..d1ed28b79c0 100644 --- a/doc/ci/testing/code_quality.md +++ b/doc/ci/testing/code_quality.md @@ -339,6 +339,20 @@ This example is specific to GitLab Code Quality. For more general instructions on how to configure DinD with a registry mirror, see the relevant [documentation](../docker/using_docker_build.md#enable-registry-mirror-for-dockerdind-service). +#### List of images to be stored in the private container registry + +The following images are needed for the [default `.codeclimate.yml`](https://gitlab.com/gitlab-org/ci-cd/codequality/-/blob/master/codeclimate_defaults/.codeclimate.yml.template): + +- `codeclimate/codeclimate-structure:latest` +- `codeclimate/codeclimate-csslint:latest` +- `codeclimate/codeclimate-coffeelint:latest` +- `codeclimate/codeclimate-duplication:latest` +- `codeclimate/codeclimate-eslint:latest` +- `codeclimate/codeclimate-fixme:latest` +- `codeclimate/codeclimate-rubocop:rubocop-0-92` + +If you are using a custom `.codeclimate.yml` configuration file, you must add the specified plugins in your private container registry. + #### Configure Code Quality to use the Dependency Proxy Prerequisite: diff --git a/doc/ci/testing/load_performance_testing.md b/doc/ci/testing/load_performance_testing.md index 6e1b440f252..ba7bf14fb59 100644 --- a/doc/ci/testing/load_performance_testing.md +++ b/doc/ci/testing/load_performance_testing.md @@ -138,7 +138,7 @@ For example, you can override the duration of the test with a CLI option: ``` GitLab only displays the key performance metrics in the MR widget if k6's results are saved -via [summary export](https://k6.io/docs/results-visualization/json#summary-export) +via [summary export](https://k6.io/docs/results-output/real-time/json/#summary-export) as a [Load Performance report artifact](../yaml/artifacts_reports.md#artifactsreportsload_performance). The latest Load Performance artifact available is always used, using the summary values from the test. diff --git a/doc/ci/testing/test_coverage_visualization.md b/doc/ci/testing/test_coverage_visualization.md index ee6b47e69a5..9bff8dbf780 100644 --- a/doc/ci/testing/test_coverage_visualization.md +++ b/doc/ci/testing/test_coverage_visualization.md @@ -360,7 +360,7 @@ The following [`.gitlab-ci.yml`](../yaml/index.md) example for Go uses: - [`gocover-cobertura`](https://github.com/boumenot/gocover-cobertura) to convert Go's coverage profile into the Cobertura XML format. This example assumes that [Go modules](https://go.dev/ref/mod) -are being used. Please note that the `-covermode count` option does not work with the `-race` flag. +are being used. The `-covermode count` option does not work with the `-race` flag. If you want to generate code coverage while also using the `-race` flag, you must switch to `-covermode atomic` which is slower than `-covermode count`. See [this blog post](https://go.dev/blog/cover) for more details. diff --git a/doc/ci/testing/unit_test_report_examples.md b/doc/ci/testing/unit_test_report_examples.md index c14e4eedd7c..5d4cfa88d88 100644 --- a/doc/ci/testing/unit_test_report_examples.md +++ b/doc/ci/testing/unit_test_report_examples.md @@ -18,7 +18,10 @@ Use the following job in `.gitlab-ci.yml`. This includes the `artifacts:paths` k ```yaml ## Use https://github.com/sj26/rspec_junit_formatter to generate a JUnit report format XML file with rspec ruby: + image: ruby:3.0.4 stage: test + before_script: + - apt-get update -y && apt-get install -y bundler script: - bundle install - bundle exec rspec --format progress --format RspecJunitFormatter --out rspec.xml diff --git a/doc/ci/triggers/index.md b/doc/ci/triggers/index.md index cafa64c4832..a667836fee4 100644 --- a/doc/ci/triggers/index.md +++ b/doc/ci/triggers/index.md @@ -193,3 +193,11 @@ A response of `{"message":"404 Not Found"}` when triggering a pipeline might be by using a [personal access token](../../user/profile/personal_access_tokens.md) instead of a trigger token. [Create a new trigger token](#create-a-trigger-token) and use it instead of the personal access token. + +### `The requested URL returned error: 400` when triggering a pipeline + +If you attempt to trigger a pipeline by using a `ref` that is a branch name that +doesn't exist, GitLab returns `The requested URL returned error: 400`. + +For example, you might accidentally use `main` for the branch name in a project that +uses a different branch name for its default branch. diff --git a/doc/ci/troubleshooting.md b/doc/ci/troubleshooting.md index 8f78469af18..87ebff74600 100644 --- a/doc/ci/troubleshooting.md +++ b/doc/ci/troubleshooting.md @@ -182,6 +182,14 @@ a branch pipeline instead. It's also possible that your [`workflow: rules`](yaml/index.md#workflow) configuration blocked the pipeline, or allowed the wrong pipeline type. +### Pipeline with many jobs fails to start + +A Pipeline that has more jobs than the instance's defined [CI/CD limits](../user/admin_area/settings/continuous_integration.md#set-cicd-limits) +fails to start. + +To reduce the number of jobs in your pipeline, you can split your `.gitlab-ci.yml` +configuration using [parent-child pipelines](../ci/pipelines/pipeline_architectures.md#parent-child-pipelines). + ### A job runs unexpectedly A common reason a job is added to a pipeline unexpectedly is because the `changes` @@ -285,14 +293,14 @@ has failed or been canceled. If a merge request pipeline or merged result pipeline was canceled or failed, you can: -- Re-run the entire pipeline by clicking **Run pipeline** in the pipeline tab in the merge request. +- Re-run the entire pipeline by selecting **Run pipeline** in the pipeline tab in the merge request. - [Retry only the jobs that failed](pipelines/index.md#view-pipelines). If you re-run the entire pipeline, this is not necessary. - Push a new commit to fix the failure. If the merge train pipeline has failed, you can: - Check the failure and determine if you can use the [`/merge` quick action](../user/project/quick_actions.md) to immediately add the merge request to the train again. -- Re-run the entire pipeline by clicking **Run pipeline** in the pipeline tab in the merge request, then add the merge request to the train again. +- Re-run the entire pipeline by selecting **Run pipeline** in the pipeline tab in the merge request, then add the merge request to the train again. - Push a commit to fix the failure, then add the merge request to the train again. If the merge train pipeline was canceled before the merge request was merged, without a failure, you can: @@ -400,6 +408,77 @@ This flag reduces system resource usage on the `jobs/request` endpoint. When enabled, jobs created in the last hour can run in projects which are out of quota. Earlier jobs are already canceled by a periodic background worker (`StuckCiJobsWorker`). +## CI/CD troubleshooting rails console commands + +The following commands are run in the [rails console](../administration/operations/rails_console.md#starting-a-rails-console-session). + +WARNING: +Any command that changes data directly could be damaging if not run correctly, or under the right conditions. +We highly recommend running them in a test environment with a backup of the instance ready to be restored, just in case. + +### Cancel stuck pending pipelines + +```ruby +project = Project.find_by_full_path('<project_path>') +Ci::Pipeline.where(project_id: project.id).where(status: 'pending').count +Ci::Pipeline.where(project_id: project.id).where(status: 'pending').each {|p| p.cancel if p.stuck?} +Ci::Pipeline.where(project_id: project.id).where(status: 'pending').count +``` + +### Try merge request integration + +```ruby +project = Project.find_by_full_path('<project_path>') +mr = project.merge_requests.find_by(iid: <merge_request_iid>) +mr.project.try(:ci_integration) +``` + +### Validate the `.gitlab-ci.yml` file + +```ruby +project = Project.find_by_full_path('<project_path>') +content = p.repository.gitlab_ci_yml_for(project.repository.root_ref_sha) +Gitlab::Ci::Lint.new(project: project, current_user: User.first).validate(content) +``` + +### Disable AutoDevOps on Existing Projects + +```ruby +Project.all.each do |p| + p.auto_devops_attributes={"enabled"=>"0"} + p.save +end +``` + +### Obtain runners registration token + +```ruby +Gitlab::CurrentSettings.current_application_settings.runners_registration_token +``` + +### Seed runners registration token + +```ruby +appSetting = Gitlab::CurrentSettings.current_application_settings +appSetting.set_runners_registration_token('<new-runners-registration-token>') +appSetting.save! +``` + +### Run pipeline schedules manually + +You can run pipeline schedules manually through the Rails console to reveal any errors that are usually not visible. + +```ruby +# schedule_id can be obtained from Edit Pipeline Schedule page +schedule = Ci::PipelineSchedule.find_by(id: <schedule_id>) + +# Select the user that you want to run the schedule for +user = User.find_by_username('<username>') + +# Run the schedule +ps = Ci::CreatePipelineService.new(schedule.project, user, ref: schedule.ref).execute!(:schedule, ignore_skip_ci: true, save_on_errors: false, schedule: schedule) +``` + ## How to get help If you are unable to resolve pipeline issues, you can get help from: diff --git a/doc/ci/variables/index.md b/doc/ci/variables/index.md index 7ad42aaf96b..97707c603bd 100644 --- a/doc/ci/variables/index.md +++ b/doc/ci/variables/index.md @@ -106,7 +106,7 @@ job1: Variables saved in the `.gitlab-ci.yml` file should store only non-sensitive project configuration, like a `RAILS_ENV` or `DATABASE_URL` variable. These variables are -visible in the repository. Store sensitive variables containing secrets, keys, and so on +visible in the repository. Store sensitive variables containing values like secrets or keys in project settings. Variables saved in the `.gitlab-ci.yml` file are also available in [service containers](../docker/using_docker_images.md). @@ -161,7 +161,7 @@ in the project settings, not in the `.gitlab-ci.yml` file. To add or update variables in the project settings: 1. Go to your project's **Settings > CI/CD** and expand the **Variables** section. -1. Select the **Add Variable** button and fill in the details: +1. Select **Add variable** and fill in the details: - **Key**: Must be one line, with no spaces, using only letters, numbers, or `_`. - **Value**: No limitations. @@ -203,7 +203,7 @@ Use group variables to store secrets like passwords, SSH keys, and credentials, To add a group variable: 1. In the group, go to **Settings > CI/CD**. -1. Select the **Add Variable** button and fill in the details: +1. Select **Add variable** and fill in the details: - **Key**: Must be one line, with no spaces, using only letters, numbers, or `_`. - **Value**: No limitations. @@ -241,7 +241,7 @@ To add an instance variable: 1. On the top bar, select **Main menu > Admin**. 1. On the left sidebar, select **Settings > CI/CD** and expand the **Variables** section. -1. Select the **Add variable** button, and fill in the details: +1. Select **Add variable** and fill in the details: - **Key**: Must be one line, with no spaces, using only letters, numbers, or `_`. - **Value**: In [GitLab 13.3 and later](https://gitlab.com/gitlab-org/gitlab/-/issues/220028), @@ -255,8 +255,6 @@ To add an instance variable: ### CI/CD variable types -> [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/46806) in GitLab 11.11. - All predefined CI/CD variables and variables defined in the `.gitlab-ci.yml` file are `Variable` type. Project, group and instance CI/CD variables can be `Variable` or `File` type. @@ -333,7 +331,7 @@ job1: ### Mask a CI/CD variable -> [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/13784) in GitLab 11.10 +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/330650) in GitLab 13.12, the `~` character can be used in masked variables. You can mask a project, group, or instance CI/CD variable so the value of the variable does not display in job logs. @@ -352,20 +350,24 @@ The value of the variable must: - Be a single line. - Be 8 characters or longer, consisting only of: - Characters from the Base64 alphabet (RFC4648). - - The `@` and `:` characters (In [GitLab 12.2 and later](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/63043)). - - The `.` character (In [GitLab 12.10 and later](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/29022)). - - The `~` character (In [GitLab 13.12 and later](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/61517)). + - The `@`, `:`, `.`, or `~` characters. - Not match the name of an existing predefined or custom CI/CD variable. -NOTE: -Masking a CI/CD variable is not a guaranteed way to prevent malicious users from accessing -variable values. To make variables more secure, you can [use external secrets](../secrets/index.md). - WARNING: -Due to a technical limitation, masked variables that are more than 4 KiB in length are not recommended. Printing such -a large value to the trace log has the potential to be [revealed](https://gitlab.com/gitlab-org/gitlab-runner/-/issues/28128). -When using GitLab Runner 14.2, only the tail of the variable, characters beyond 4KiB in length, have the potential to -be revealed. +Masking a CI/CD variable is not a guaranteed way to prevent malicious users from +accessing variable values. The masking feature is "best-effort" and there to +help when a variable is accidentally revealed. To make variables more secure, +consider using [external secrets](../secrets/index.md) and [file type variables](#cicd-variable-types) +to prevent commands such as `env`/`printenv` from printing secret variables. + +Runner versions implement masking in different ways, some with technical +limitations. Below is a table of such limitations. + +| Version from | Version to | Limitations | +| ------------ | ---------- | ------ | +| v11.9.0 | v14.1.0 | Masking of large secrets (greater than 4 KiB) could potentially be [revealed](https://gitlab.com/gitlab-org/gitlab-runner/-/issues/28128). No sensitive URL parameter masking. | +| v14.2.0 | v15.3.0 | The tail of a large secret (greater than 4 KiB) could potentially be [revealed](https://gitlab.com/gitlab-org/gitlab-runner/-/issues/28128). No sensitive URL parameter masking. | +| v15.7.0 | | Potential for secrets to be revealed when `CI_DEBUG_SERVICES` is enabled. For details, read about [service container logging](../services/index.md#capturing-service-container-logs). | ### Protected CI/CD variables @@ -506,7 +508,7 @@ job_name: # - 'dir env:' # Use this for PowerShell ``` -Example job log output: +Example job log output (truncated): ```shell export CI_JOB_ID="50" @@ -528,31 +530,6 @@ export CI_PROJECT_ID="34" export CI_PROJECT_DIR="/builds/gitlab-org/gitlab-foss" export CI_PROJECT_NAME="gitlab-foss" export CI_PROJECT_TITLE="GitLab FOSS" -export CI_PROJECT_DESCRIPTION="GitLab FOSS is a read-only mirror of GitLab, with all proprietary code removed." -export CI_PROJECT_NAMESPACE="gitlab-org" -export CI_PROJECT_ROOT_NAMESPACE="gitlab-org" -export CI_PROJECT_PATH="gitlab-org/gitlab-foss" -export CI_PROJECT_URL="https://example.com/gitlab-org/gitlab-foss" -export CI_REGISTRY="registry.example.com" -export CI_REGISTRY_IMAGE="registry.example.com/gitlab-org/gitlab-foss" -export CI_REGISTRY_USER="gitlab-ci-token" -export CI_REGISTRY_PASSWORD="[masked]" -export CI_RUNNER_ID="10" -export CI_RUNNER_DESCRIPTION="my runner" -export CI_RUNNER_TAGS="docker, linux" -export CI_SERVER="yes" -export CI_SERVER_URL="https://example.com" -export CI_SERVER_HOST="example.com" -export CI_SERVER_PORT="443" -export CI_SERVER_PROTOCOL="https" -export CI_SERVER_NAME="GitLab" -export CI_SERVER_REVISION="70606bf" -export CI_SERVER_VERSION="8.9.0" -export CI_SERVER_VERSION_MAJOR="8" -export CI_SERVER_VERSION_MINOR="9" -export CI_SERVER_VERSION_PATCH="0" -export GITLAB_USER_EMAIL="user@example.com" -export GITLAB_USER_ID="42" ... ``` @@ -776,8 +753,6 @@ explains if the integration has any deployment variables available. ## Auto DevOps environment variables -> [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/49056) in GitLab 11.7. - You can configure [Auto DevOps](../../topics/autodevops/index.md) to pass CI/CD variables to a running application. @@ -817,7 +792,7 @@ job_name: Example output (truncated): -```shell +```plaintext ... export CI_SERVER_TLS_CA_FILE="/builds/gitlab-examples/ci-debug-trace.tmp/CI_SERVER_TLS_CA_FILE" if [[ -d "/builds/gitlab-examples/ci-debug-trace/.git" ]]; then @@ -885,48 +860,6 @@ if [[ -d "/builds/gitlab-examples/ci-debug-trace/.git" ]]; then ++ CI_PROJECT_ID=17893 ++ export CI_PROJECT_NAME=ci-debug-trace ++ CI_PROJECT_NAME=ci-debug-trace -++ export CI_PROJECT_TITLE='GitLab FOSS' -++ CI_PROJECT_TITLE='GitLab FOSS' -++ export CI_PROJECT_DESCRIPTION='GitLab FOSS is a read-only mirror of GitLab, with all proprietary code removed.' -++ CI_PROJECT_DESCRIPTION='GitLab FOSS is a read-only mirror of GitLab, with all proprietary code removed.' -++ export CI_PROJECT_PATH=gitlab-examples/ci-debug-trace -++ CI_PROJECT_PATH=gitlab-examples/ci-debug-trace -++ export CI_PROJECT_PATH_SLUG=gitlab-examples-ci-debug-trace -++ CI_PROJECT_PATH_SLUG=gitlab-examples-ci-debug-trace -++ export CI_PROJECT_NAMESPACE=gitlab-examples -++ CI_PROJECT_NAMESPACE=gitlab-examples -++ export CI_PROJECT_ROOT_NAMESPACE=gitlab-examples -++ CI_PROJECT_ROOT_NAMESPACE=gitlab-examples -++ export CI_PROJECT_URL=https://gitlab.com/gitlab-examples/ci-debug-trace -++ CI_PROJECT_URL=https://gitlab.com/gitlab-examples/ci-debug-trace -++ export CI_PROJECT_VISIBILITY=public -++ CI_PROJECT_VISIBILITY=public -++ export CI_PROJECT_REPOSITORY_LANGUAGES= -++ CI_PROJECT_REPOSITORY_LANGUAGES= -++ export CI_PROJECT_CLASSIFICATION_LABEL= -++ CI_PROJECT_CLASSIFICATION_LABEL= -++ export CI_DEFAULT_BRANCH=main -++ CI_DEFAULT_BRANCH=main -++ export CI_REGISTRY=registry.gitlab.com -++ CI_REGISTRY=registry.gitlab.com -++ export CI_API_V4_URL=https://gitlab.com/api/v4 -++ CI_API_V4_URL=https://gitlab.com/api/v4 -++ export CI_PIPELINE_IID=123 -++ CI_PIPELINE_IID=123 -++ export CI_PIPELINE_SOURCE=web -++ CI_PIPELINE_SOURCE=web -++ export CI_CONFIG_PATH=.gitlab-ci.yml -++ CI_CONFIG_PATH=.gitlab-ci.yml -++ export CI_COMMIT_SHA=dd648b2e48ce6518303b0bb580b2ee32fadaf045 -++ CI_COMMIT_SHA=dd648b2e48ce6518303b0bb580b2ee32fadaf045 -++ export CI_COMMIT_SHORT_SHA=dd648b2e -++ CI_COMMIT_SHORT_SHA=dd648b2e -++ export CI_COMMIT_BEFORE_SHA=0000000000000000000000000000000000000000 -++ CI_COMMIT_BEFORE_SHA=0000000000000000000000000000000000000000 -++ export CI_COMMIT_REF_NAME=main -++ CI_COMMIT_REF_NAME=main -++ export CI_COMMIT_REF_SLUG=main -++ CI_COMMIT_REF_SLUG=main ... ``` diff --git a/doc/ci/variables/predefined_variables.md b/doc/ci/variables/predefined_variables.md index 606847ee756..852110a1415 100644 --- a/doc/ci/variables/predefined_variables.md +++ b/doc/ci/variables/predefined_variables.md @@ -46,6 +46,7 @@ as it can cause the pipeline to behave unexpectedly. | `CI_CONCURRENT_PROJECT_ID` | all | 11.10 | The unique ID of build execution in a single executor and project. | | `CI_CONFIG_PATH` | 9.4 | 0.5 | The path to the CI/CD configuration file. Defaults to `.gitlab-ci.yml`. Read-only inside a running pipeline. | | `CI_DEBUG_TRACE` | all | 1.7 | `true` if [debug logging (tracing)](index.md#debug-logging) is enabled. | +| `CI_DEBUG_SERVICES` | 15.7 | 15.7 | `true` if [service container logging](../services/index.md#capturing-service-container-logs) is enabled. | | `CI_DEFAULT_BRANCH` | 12.4 | all | The name of the project's default branch. | | `CI_DEPENDENCY_PROXY_GROUP_IMAGE_PREFIX` | 13.7 | all | The top-level group image prefix for pulling images through the Dependency Proxy. | | `CI_DEPENDENCY_PROXY_DIRECT_GROUP_IMAGE_PREFIX` | 14.3 | all | The direct group image prefix for pulling images through the Dependency Proxy. | @@ -89,7 +90,6 @@ as it can cause the pipeline to behave unexpectedly. | `CI_PIPELINE_TRIGGERED` | all | all | `true` if the job was [triggered](../triggers/index.md). | | `CI_PIPELINE_URL` | 11.1 | 0.5 | The URL for the pipeline details. | | `CI_PIPELINE_CREATED_AT` | 13.10 | all | The UTC datetime when the pipeline was created, in [ISO 8601](https://www.rfc-editor.org/rfc/rfc3339#appendix-A) format. | -| `CI_PROJECT_CONFIG_PATH` | 13.8 to 13.12 | all | [Removed](https://gitlab.com/gitlab-org/gitlab/-/issues/322807) in GitLab 14.0. Use `CI_CONFIG_PATH`. | | `CI_PROJECT_DIR` | all | all | The full path the repository is cloned to, and where the job runs from. If the GitLab Runner `builds_dir` parameter is set, this variable is set relative to the value of `builds_dir`. For more information, see the [Advanced GitLab Runner configuration](https://docs.gitlab.com/runner/configuration/advanced-configuration.html#the-runners-section). | | `CI_PROJECT_ID` | all | all | The ID of the current project. This ID is unique across all projects on the GitLab instance. | | `CI_PROJECT_NAME` | 8.10 | 0.5 | The name of the directory for the project. For example if the project URL is `gitlab.example.com/group-name/project-1`, `CI_PROJECT_NAME` is `project-1`. | diff --git a/doc/ci/variables/where_variables_can_be_used.md b/doc/ci/variables/where_variables_can_be_used.md index 7e5451658f2..c8436fc044d 100644 --- a/doc/ci/variables/where_variables_can_be_used.md +++ b/doc/ci/variables/where_variables_can_be_used.md @@ -36,13 +36,15 @@ There are two places defined variables can be used. On the: | [`include`](../yaml/index.md#include) | yes | GitLab | The variable expansion is made by the [internal variable expansion mechanism](#gitlab-internal-variable-expansion-mechanism) in GitLab. <br/><br/>See [Use variables with include](../yaml/includes.md#use-variables-with-include) for more information on supported variables. | | [`only:variables`](../yaml/index.md#onlyvariables--exceptvariables) | no | Not applicable | The variable must be in the form of `$variable`. Not supported are the following:<br/><br/>- Variables that are based on the environment's name (`CI_ENVIRONMENT_NAME`, `CI_ENVIRONMENT_SLUG`).<br/>- Any other variables related to environment (currently only `CI_ENVIRONMENT_URL`).<br/>- [Persisted variables](#persisted-variables). | | [`resource_group`](../yaml/index.md#resource_group) | yes | GitLab | Similar to `environment:url`, but the variables expansion doesn't support the following:<br/>- `CI_ENVIRONMENT_URL`<br/>- [Persisted variables](#persisted-variables). | +| [`rules:exists`](../yaml/index.md#rulesexists) | yes | GitLab | The variable expansion is made by the [internal variable expansion mechanism](#gitlab-internal-variable-expansion-mechanism) in GitLab. | | [`rules:if`](../yaml/index.md#rulesif) | no | Not applicable | The variable must be in the form of `$variable`. Not supported are the following:<br/><br/>- Variables that are based on the environment's name (`CI_ENVIRONMENT_NAME`, `CI_ENVIRONMENT_SLUG`).<br/>- Any other variables related to environment (currently only `CI_ENVIRONMENT_URL`).<br/>- [Persisted variables](#persisted-variables). | | [`script`](../yaml/index.md#script) | yes | Script execution shell | The variable expansion is made by the [execution shell environment](#execution-shell-environment). | | [`services:name`](../yaml/index.md#services) | yes | Runner | The variable expansion is made by GitLab Runner's [internal variable expansion mechanism](#gitlab-runner-internal-variable-expansion-mechanism). | | [`services`](../yaml/index.md#services) | yes | Runner | The variable expansion is made by GitLab Runner's [internal variable expansion mechanism](#gitlab-runner-internal-variable-expansion-mechanism). | | [`tags`](../yaml/index.md#tags) | yes | GitLab | The variable expansion is made by the [internal variable expansion mechanism](#gitlab-internal-variable-expansion-mechanism) in GitLab. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/35742) in GitLab 14.1. | -| [`trigger` and `trigger:project`](../yaml/index.md#trigger) | yes | GitLab | The variable expansion is made by the [internal variable expansion mechanism](#gitlab-internal-variable-expansion-mechanism) in GitLab. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/367660) in GitLab 15.3. | +| [`trigger` and `trigger:project`](../yaml/index.md#trigger) | yes | GitLab | The variable expansion is made by the [internal variable expansion mechanism](#gitlab-internal-variable-expansion-mechanism) in GitLab. Variable expansion for `trigger:project` [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/367660) in GitLab 15.3. | | [`variables`](../yaml/index.md#variables) | yes | GitLab/Runner | The variable expansion is first made by the [internal variable expansion mechanism](#gitlab-internal-variable-expansion-mechanism) in GitLab, and then any unrecognized or unavailable variables are expanded by GitLab Runner's [internal variable expansion mechanism](#gitlab-runner-internal-variable-expansion-mechanism). | +| [`workflow:name`](../yaml/index.md#workflowname) | yes | GitLab | The variable expansion is made by the [internal variable expansion mechanism](#gitlab-internal-variable-expansion-mechanism) in GitLab.<br/><br/>Supported are all variables available in `workflow`:<br/>- Project/Group variables.<br/>- Global `variables` and `workflow:rules:variables` (when matching the rule).<br/>- Variables inherited from parent pipelines.<br/>- Variables from triggers.<br/>- Variables from pipeline schedules.<br/><br/>Not supported are variables defined in the GitLab Runner `config.toml`, variables defined in jobs, or [Persisted variables](#persisted-variables). | ### `config.toml` file diff --git a/doc/ci/yaml/artifacts_reports.md b/doc/ci/yaml/artifacts_reports.md index 67cbe989f74..9f5e4e919b0 100644 --- a/doc/ci/yaml/artifacts_reports.md +++ b/doc/ci/yaml/artifacts_reports.md @@ -178,7 +178,7 @@ report uploads to GitLab as an artifact. GitLab can display the results of one or more reports in: -- The merge request [security widget](../../user/application_security/dast/index.md#view-details-of-a-vulnerability-detected-by-dast). +- The merge request security widget. - The pipeline [**Security** tab](../../user/application_security/vulnerability_report/pipeline.md#view-vulnerabilities-in-a-pipeline). - The [Project Vulnerability report](../../user/application_security/vulnerability_report/index.md). - The [security dashboard](../../user/application_security/security_dashboard/index.md). diff --git a/doc/ci/yaml/index.md b/doc/ci/yaml/index.md index e06abe1dc69..3392304775a 100644 --- a/doc/ci/yaml/index.md +++ b/doc/ci/yaml/index.md @@ -146,7 +146,7 @@ the time limit to resolve all files is 30 seconds. **Possible inputs**: The `include` subkeys: - [`include:local`](#includelocal) -- [`include:file`](#includefile) +- [`include:project`](#includeproject) - [`include:remote`](#includeremote) - [`include:template`](#includetemplate) @@ -203,58 +203,52 @@ include: '.gitlab-ci-production.yml' - All [nested includes](includes.md#use-nested-includes) are executed in the scope of the same project, so you can use local, project, remote, or template includes. -#### `include:file` +#### `include:project` > Including multiple files from the same project [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/26793) in GitLab 13.6. [Feature flag removed](https://gitlab.com/gitlab-org/gitlab/-/issues/271560) in GitLab 13.8. To include files from another private project on the same GitLab instance, -use `include:file`. You can use `include:file` in combination with `include:project` only. +use `include:project` and `include:file`. **Keyword type**: Global keyword. **Possible inputs**: -A full path, relative to the root directory (`/`): +- `include:project`: The full GitLab project path. +- `include:file` A full file path, or array of file paths, relative to the root directory (`/`). + The YAML files must have the `.yml` or `.yaml` extension. +- `include:ref`: Optional. The ref to retrieve the file from. Defaults to the `HEAD` of the project + when not specified. -- The YAML file must have the extension `.yml` or `.yaml`. -- You can use [certain CI/CD variables](includes.md#use-variables-with-include). +You can use [certain CI/CD variables](includes.md#use-variables-with-include). -**Example of `include:file`**: +**Example of `include:project`**: ```yaml include: - project: 'my-group/my-project' file: '/templates/.gitlab-ci-template.yml' + - project: 'my-group/my-subgroup/my-project-2' + file: + - '/templates/.builds.yml' + - '/templates/.tests.yml' ``` -You can also specify a `ref`. If you do not specify a value, the ref defaults to the `HEAD` of the project: +You can also specify a `ref`: ```yaml include: - project: 'my-group/my-project' - ref: main + ref: main # Git branch file: '/templates/.gitlab-ci-template.yml' - - project: 'my-group/my-project' - ref: v1.0.0 # Git Tag + ref: v1.0.0 # Git Tag file: '/templates/.gitlab-ci-template.yml' - - project: 'my-group/my-project' ref: 787123b47f14b552955ca2786bc9542ae66fee5b # Git SHA file: '/templates/.gitlab-ci-template.yml' ``` -You can include multiple files from the same project: - -```yaml -include: - - project: 'my-group/my-project' - ref: main - file: - - '/templates/.builds.yml' - - '/templates/.tests.yml' -``` - **Additional details**: - All [nested includes](includes.md#use-nested-includes) are executed in the scope of the target project. @@ -379,7 +373,7 @@ start. Jobs in the current stage are not stopped and continue to run. - If a job does not specify a [`stage`](#stage), the job is assigned the `test` stage. - If a stage is defined but no jobs use it, the stage is not visible in the pipeline, - which can help [compliance pipeline configurations](../../user/group/manage.md#configure-a-compliance-pipeline): + which can help [compliance pipeline configurations](../../user/group/compliance_frameworks.md#configure-a-compliance-pipeline): - Stages can be defined in the compliance configuration but remain hidden if not used. - The defined stages become visible when developers use them in job definitions. @@ -414,12 +408,33 @@ All pipelines are assigned the defined name. Any leading or trailing spaces in t **Possible inputs**: - A string. +- [CI/CD variables](../variables/where_variables_can_be_used.md#gitlab-ciyml-file). +- A combination of both. -**Example of `workflow:name`**: +**Examples of `workflow:name`**: + +A simple pipeline name with a predefined variable: ```yaml workflow: - name: 'Pipeline name' + name: 'Pipeline for branch: $CI_COMMIT_BRANCH' +``` + +A configuration with different pipeline names depending on the pipeline conditions: + +```yaml +variables: + PIPELINE_NAME: 'Default pipeline name' + +workflow: + name: '$PIPELINE_NAME' + rules: + - if: '$CI_PIPELINE_SOURCE == "merge_request_event"' + variables: + PIPELINE_NAME: 'MR pipeline: $CI_COMMIT_BRANCH' + - if: '$CI_MERGE_REQUEST_LABELS =~ /pipeline:run-in-ruby3/' + variables: + PIPELINE_NAME: 'Ruby 3 pipeline' ``` #### `workflow:rules` @@ -982,7 +997,7 @@ rspec: - Combining reports in parent pipelines using [artifacts from child pipelines](#needspipelinejob) is not supported. Track progress on adding support in [this issue](https://gitlab.com/gitlab-org/gitlab/-/issues/215725). -- To be able to browse the report output files, include the [`artifacts:paths`](#artifactspaths) keyword. Please note that this will upload and store the artifact twice. +- To be able to browse the report output files, include the [`artifacts:paths`](#artifactspaths) keyword. This will upload and store the artifact twice. - The test reports are collected regardless of the job results (success or failure). You can use [`artifacts:expire_in`](#artifactsexpire_in) to set up an expiration date for artifacts reports. @@ -1134,7 +1149,7 @@ that use the same cache key use the same cache, including in different pipelines If not set, the default key is `default`. All jobs with the `cache` keyword but no `cache:key` share the `default` cache. -Must be used with `cache: path`, or nothing is cached. +Must be used with `cache: paths`, or nothing is cached. **Keyword type**: Job keyword. You can use it only as part of a job or in the [`default` section](#default). @@ -1304,7 +1319,7 @@ rspec: Use `cache:when` to define when to save the cache, based on the status of the job. -Must be used with `cache: path`, or nothing is cached. +Must be used with `cache: paths`, or nothing is cached. **Keyword type**: Job keyword. You can use it only as part of a job or in the [`default` section](#default). @@ -1344,7 +1359,7 @@ Use the `pull` policy when you have many jobs executing in parallel that use the This policy speeds up job execution and reduces load on the cache server. You can use a job with the `push` policy to build the cache. -Must be used with `cache: path`, or nothing is cached. +Must be used with `cache: paths`, or nothing is cached. **Keyword type**: Job keyword. You can use it only as part of a job or in the [`default` section](#default). @@ -1464,8 +1479,8 @@ to select a specific site profile and scanner profile. **Related topics**: -- [Site profile](../../user/application_security/dast/index.md#site-profile). -- [Scanner profile](../../user/application_security/dast/index.md#scanner-profile). +- [Site profile](../../user/application_security/dast/proxy-based.md#site-profile). +- [Scanner profile](../../user/application_security/dast/proxy-based.md#scanner-profile). ### `dependencies` @@ -1749,6 +1764,11 @@ deploy: deployment_tier: production ``` +**Additional details**: + +- Enviroments created from this job definition are assigned a [tier](../environments/index.md#deployment-tier-of-environments) based on this value. +- Existing environments don't have their tier updated if this value is added later. Existing enviroments must have their tier updated via the [Environments API](../../api/environments.md#update-an-existing-environment). + **Related topics**: - [Deployment tier of environments](../environments/index.md#deployment-tier-of-environments). @@ -2809,6 +2829,13 @@ deploystacks: [vultr, data] deploystacks: [vultr, processing] ``` +**Additional details**: + +- `parallel:matrix` jobs add the variable values to the job names to differentiate + the jobs from each other, but [large values can cause names to exceed limits](https://gitlab.com/gitlab-org/gitlab/-/issues/362262): + - Job names must be [255 characters or fewer](../jobs/index.md#job-name-limitations). + - When using [`needs`](#needs), job names must be 128 characters or fewer. + **Related topics**: - [Run a one-dimensional matrix of parallel jobs](../jobs/job_control.md#run-a-one-dimensional-matrix-of-parallel-jobs). @@ -3229,7 +3256,8 @@ Use `rules:if` clauses to specify when to add a job to a pipeline: - If no `if` statements are true, do not add the job to the pipeline. `if` clauses are evaluated based on the values of [predefined CI/CD variables](../variables/predefined_variables.md) -or [custom CI/CD variables](../variables/index.md#custom-cicd-variables). +or [custom CI/CD variables](../variables/index.md#custom-cicd-variables), with +[some exceptions](../variables/where_variables_can_be_used.md#gitlab-ciyml-file). **Keyword type**: Job-specific and pipeline-specific. You can use it as part of a job to configure the job behavior, or with [`workflow`](#workflow) to configure the pipeline behavior. @@ -3403,7 +3431,8 @@ relative to `refs/heads/branch1` and the pipeline source is a merge request even #### `rules:exists` -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/24021) in GitLab 12.4. +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/24021) in GitLab 12.4. +> - CI/CD variable support [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/283881) in GitLab 15.6. Use `exists` to run a job when certain files exist in the repository. @@ -3411,8 +3440,7 @@ Use `exists` to run a job when certain files exist in the repository. **Possible inputs**: -- An array of file paths. Paths are relative to the project directory (`$CI_PROJECT_DIR`) - and can't directly link outside it. File paths can use glob patterns. +- An array of file paths. Paths are relative to the project directory (`$CI_PROJECT_DIR`) and can't directly link outside it. File paths can use glob patterns and [CI/CD variables](../variables/where_variables_can_be_used.md#gitlab-ciyml-file). **Example of `rules:exists`**: @@ -3573,7 +3601,7 @@ Use `secrets:vault` to specify secrets provided by a [HashiCorp Vault](https://w **Example of `secrets:vault`**: -To specify all details explicitly and use the [KV-V2](https://www.vaultproject.io/docs/secrets/kv/kv-v2) secrets engine: +To specify all details explicitly and use the [KV-V2](https://developer.hashicorp.com/vault/docs/secrets/kv/kv-v2) secrets engine: ```yaml job: @@ -3938,14 +3966,15 @@ Use `trigger` to declare that a job is a "trigger job" which starts a Trigger jobs can use only a limited set of GitLab CI/CD configuration keywords. The keywords available for use in trigger jobs are: -- [`trigger`](#trigger). -- [`stage`](#stage). - [`allow_failure`](#allow_failure). -- [`rules`](#rules). -- [`only` and `except`](#only--except). -- [`when`](#when) (only with a value of `on_success`, `on_failure`, or `always`). - [`extends`](#extends). - [`needs`](#needs), but not [`needs:project`](#needsproject). +- [`only` and `except`](#only--except). +- [`rules`](#rules). +- [`stage`](#stage). +- [`trigger`](#trigger). +- [`variables`](#variables). +- [`when`](#when) (only with a value of `on_success`, `on_failure`, or `always`). **Keyword type**: Job keyword. You can use it only as part of a job. @@ -3969,6 +3998,8 @@ trigger-multi-project-pipeline: - In [GitLab 13.5 and later](https://gitlab.com/gitlab-org/gitlab/-/issues/201938), you can use [`when:manual`](#when) in the same job as `trigger`. In GitLab 13.4 and earlier, using them together causes the error `jobs:#{job-name} when should be on_success, on_failure or always`. +- You cannot [manually specify CI/CD variables](../jobs/index.md#specifying-variables-when-running-manual-jobs) + before running a manual trigger job. - [Manual pipeline variables](../variables/index.md#override-a-defined-cicd-variable) and [scheduled pipeline variables](../pipelines/schedules.md#add-a-pipeline-schedule) are not passed to downstream pipelines by default. Use [trigger:forward](#triggerforward) @@ -4079,6 +4110,8 @@ successfully complete before starting. shows **pending** (**{status_pending}**) if the downstream pipeline status is **waiting for manual action** (**{status_manual}**) due to manual jobs. By default, jobs in later stages do not start until the trigger job completes. +- If the downstream pipeline has a failed job, but the job uses [`allow_failure: true`](#allow_failure), + the downstream pipeline is considered successful and the trigger job shows **success**. #### `trigger:forward` @@ -4222,6 +4255,38 @@ variables: - A global variable defined with `value` but no `description` behaves the same as [`variables`](#variables). +#### `variables:expand` + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/353991) in GitLab 15.6 [with a flag](../../administration/feature_flags.md) named `ci_raw_variables_in_yaml_config`. Disabled by default. + +Use the `expand` keyword to configure a variable to be expandable or not. + +**Keyword type**: Global and job keyword. You can use it at the global level, and also at the job level. + +**Possible inputs**: + +- `true` (default): The variable is expandable. +- `false`: The variable is not expandable. + +**Example of `variables:expand`**: + +```yaml +variables: + VAR1: value1 + VAR2: value2 $VAR1 + VAR3: + value: value3 $VAR1 + expand: false +``` + +- The result of `VAR2` is `value2 value1`. +- The result of `VAR3` is `value3 $VAR1`. + +**Additional details**: + +- The `expand` keyword can only be used with the global and job-level `variables` keywords. + You can't use it with [`rules:variables`](#rulesvariables) or [`workflow:rules:variables`](#workflowrulesvariables). + ### `when` Use `when` to configure the conditions for when jobs run. If not defined in a job, |
