diff options
| author | GitLab Bot <gitlab-bot@gitlab.com> | 2020-11-19 08:27:35 +0000 |
|---|---|---|
| committer | GitLab Bot <gitlab-bot@gitlab.com> | 2020-11-19 08:27:35 +0000 |
| commit | 7e9c479f7de77702622631cff2628a9c8dcbc627 (patch) | |
| tree | c8f718a08e110ad7e1894510980d2155a6549197 /doc/ci | |
| parent | e852b0ae16db4052c1c567d9efa4facc81146e88 (diff) | |
| download | gitlab-ce-7e9c479f7de77702622631cff2628a9c8dcbc627.tar.gz | |
Add latest changes from gitlab-org/gitlab@13-6-stable-eev13.6.0-rc42
Diffstat (limited to 'doc/ci')
78 files changed, 1675 insertions, 1272 deletions
diff --git a/doc/ci/README.md b/doc/ci/README.md index dca6d8baa79..45cf56df894 100644 --- a/doc/ci/README.md +++ b/doc/ci/README.md @@ -77,7 +77,7 @@ While building your `.gitlab-ci.yml`, you can use the [CI/CD configuration visua For a broader overview, see the [CI/CD getting started](quick_start/README.md) guide. -Once you're familiar with how GitLab CI/CD works, see the +After you're familiar with how GitLab CI/CD works, see the [`.gitlab-ci.yml` full reference](yaml/README.md) for all the attributes you can set and use. diff --git a/doc/ci/caching/index.md b/doc/ci/caching/index.md index df41f7da2d9..dfa92d469bc 100644 --- a/doc/ci/caching/index.md +++ b/doc/ci/caching/index.md @@ -411,6 +411,8 @@ job B: - cat vendor/hello.txt cache: key: build-cache + paths: + - vendor/ ``` Here's what happens behind the scenes: diff --git a/doc/ci/ci_cd_for_external_repos/github_integration.md b/doc/ci/ci_cd_for_external_repos/github_integration.md index 7ee9f98bd39..b6f885ff220 100644 --- a/doc/ci/ci_cd_for_external_repos/github_integration.md +++ b/doc/ci/ci_cd_for_external_repos/github_integration.md @@ -22,7 +22,7 @@ cannot be used to authenticate with GitHub as an external CI/CD repository. ## Connect with Personal Access Token Personal access tokens can only be used to connect GitHub.com -repositories to GitLab, and the GitHub user must have the [owner role](https://docs.github.com/en/github/getting-started-with-github/access-permissions-on-github). +repositories to GitLab, and the GitHub user must have the [owner role](https://docs.github.com/en/free-pro-team@latest/github/getting-started-with-github/access-permissions-on-github). To perform a one-off authorization with GitHub to grant GitLab access your repositories: diff --git a/doc/ci/ci_cd_for_external_repos/img/github_omniauth.png b/doc/ci/ci_cd_for_external_repos/img/github_omniauth.png Binary files differdeleted file mode 100644 index 71a3a057a41..00000000000 --- a/doc/ci/ci_cd_for_external_repos/img/github_omniauth.png +++ /dev/null diff --git a/doc/ci/ci_cd_for_external_repos/img/github_push_webhook.png b/doc/ci/ci_cd_for_external_repos/img/github_push_webhook.png Binary files differdeleted file mode 100644 index e8c17d664e1..00000000000 --- a/doc/ci/ci_cd_for_external_repos/img/github_push_webhook.png +++ /dev/null diff --git a/doc/ci/ci_cd_for_external_repos/index.md b/doc/ci/ci_cd_for_external_repos/index.md index c6590599849..ae6cb759d23 100644 --- a/doc/ci/ci_cd_for_external_repos/index.md +++ b/doc/ci/ci_cd_for_external_repos/index.md @@ -77,7 +77,7 @@ created. GitLab CI/CD will create 2 pipelines in this case. One for the branch push and one for the external pull request. -Once the Pull Request is closed, no pipelines are created for the external pull +After the Pull Request is closed, no pipelines are created for the external pull request, even if new changes are pushed to the same branch. ### Additional predefined variables diff --git a/doc/ci/cloud_deployment/index.md b/doc/ci/cloud_deployment/index.md index af7df0e1153..cbaebde0b00 100644 --- a/doc/ci/cloud_deployment/index.md +++ b/doc/ci/cloud_deployment/index.md @@ -239,7 +239,7 @@ pass three JSON input objects, based on existing templates: - [Template for the _Deploy to EC2_ step on AWS](https://docs.aws.amazon.com/codedeploy/latest/APIReference/API_CreateDeployment.html). -1. Once you have completed these three templates based on your requirements, you +1. After you have completed these three templates based on your requirements, you have two ways to pass in these JSON objects: - They can be three actual files located in your project. You must specify their path relative to @@ -281,3 +281,40 @@ When running your project pipeline at this point: - Your built application is pushed to your S3 bucket then and deployed to your EC2 instance, based on the related JSON object's content. The deployment job finishes whenever the deployment to EC2 is done or has failed. + +#### Custom build job for Auto DevOps + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/216008) in GitLab 13.6. + +To leverage [Auto DevOps](../../topics/autodevops/index.md) for your project when deploying to +AWS EC2, first you must define [your AWS credentials as environment variables](#run-aws-commands-from-gitlab-cicd). + +Next, define a job for the `build` stage. To do so, you must reference the +`Auto-DevOps.gitlab-ci.yml` template and include a job named `build_artifact` in your +`.gitlab-ci.yml` file. For example: + +```yaml +# .gitlab-ci.yml + +include: + - template: Auto-DevOps.gitlab-ci.yml + +variables: + - AUTO_DEVOPS_PLATFORM_TARGET: EC2 + +build_artifact: + stage: build + script: + - <your build script goes here> + artifacts: + paths: + - <built artifact> +``` + +### Deploy to Amazon EKS + +- [How to deploy your application to a GitLab-managed Amazon EKS cluster with Auto DevOps](https://about.gitlab.com/blog/2020/05/05/deploying-application-eks/) + +## Deploy to Google Cloud + +- [Deploying with GitLab on Google Cloud](https://about.gitlab.com/solutions/google-cloud-platform/) diff --git a/doc/ci/directed_acyclic_graph/index.md b/doc/ci/directed_acyclic_graph/index.md index 19da0496f8a..04f27c584b7 100644 --- a/doc/ci/directed_acyclic_graph/index.md +++ b/doc/ci/directed_acyclic_graph/index.md @@ -79,26 +79,28 @@ are certain use cases that you may need to work around. For more information: - [`needs` requirements and limitations](../yaml/README.md#requirements-and-limitations). - Related epic [tracking planned improvements](https://gitlab.com/groups/gitlab-org/-/epics/1716). -## DAG Visualization +## Needs Visualization > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/215517) in GitLab 13.1 as a [Beta feature](https://about.gitlab.com/handbook/product/#beta). > - It was deployed behind a feature flag, disabled by default. > - It became [enabled by default](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/36802) in 13.2. > - It became a [standard feature](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/38517) in 13.3. > - It's enabled on GitLab.com. -> - For GitLab self-managed instances, GitLab administrators can opt to [disable it](#enable-or-disable-dag-visualization). +> - For GitLab self-managed instances, GitLab administrators can opt to [disable it](#enable-or-disable-needs-visualization). -The DAG visualization makes it easier to visualize the relationships between dependent jobs in a DAG. This graph will display all the jobs in a pipeline that need or are needed by other jobs. Jobs with no relationships are not displayed in this view. +The needs visualization makes it easier to visualize the relationships between dependent jobs in a DAG. This graph will display all the jobs in a pipeline that need or are needed by other jobs. Jobs with no relationships are not displayed in this view. - +To see the needs visualization, click on the **Needs** tab when viewing a pipeline that uses the `needs:` keyword. + + Clicking a node will highlight all the job paths it depends on. - + -### Enable or disable DAG Visualization **(CORE ONLY)** +### Enable or disable Needs Visualization **(CORE ONLY)** -DAG Visualization is deployed behind a feature flag that is **enabled by default**. +The needs visualization is deployed behind a feature flag that is **enabled by default**. [GitLab administrators with access to the GitLab Rails console](../../administration/feature_flags.md) can opt to disable it for your instance: diff --git a/doc/ci/docker/using_docker_build.md b/doc/ci/docker/using_docker_build.md index e3123cde1cd..ebbfde09c67 100644 --- a/doc/ci/docker/using_docker_build.md +++ b/doc/ci/docker/using_docker_build.md @@ -90,7 +90,7 @@ GitLab Runner then executes job scripts as the `gitlab-runner` user. 1. You can now use `docker` command (and **install** `docker-compose` if needed). By adding `gitlab-runner` to the `docker` group you are effectively granting `gitlab-runner` full root permissions. -For more information please read [On Docker security: `docker` group considered harmful](https://www.andreas-jung.com/contents/on-docker-security-docker-group-considered-harmful). +For more information please read [On Docker security: `docker` group considered harmful](https://blog.zopyx.com/on-docker-security-docker-group-considered-harmful/). ### Use Docker-in-Docker workflow with Docker executor @@ -103,7 +103,7 @@ image in privileged mode. CI builds, follow the `docker-compose` [installation instructions](https://docs.docker.com/compose/install/). -DANGER: **Danger:** +DANGER: **Warning:** By enabling `--docker-privileged`, you are effectively disabling all of the security mechanisms of containers and exposing your host to privilege escalation which can lead to container breakout. For more information, check @@ -302,7 +302,149 @@ build: - docker run my-docker-image /script/to/run/tests ``` -### Use Docker socket binding +#### Enable registry mirror for `docker:dind` service + +When the Docker daemon starts inside of the service container, it uses +the default configuration. You may want to configure a [registry +mirror](https://docs.docker.com/registry/recipes/mirror/) for +performance improvements and ensuring you don't reach DockerHub rate limits. + +##### Inside `.gitlab-ci.yml` + +You can append extra CLI flags to the `dind` service to set the registry +mirror: + +```yaml +services: + - name: docker:19.03.13-dind + command: ["--registry-mirror", "https://registry-mirror.example.com"] # Specify the registry mirror to use. +``` + +##### DinD service defined inside of GitLab Runner configuration + +> [Introduced](https://gitlab.com/gitlab-org/gitlab-runner/-/issues/27173) in GitLab Runner 13.6. + +If you are an administrator of GitLab Runner and you have the `dind` +service defined for the [Docker +executor](https://docs.gitlab.com/runner/configuration/advanced-configuration.html#the-runnersdockerservices-section), +or the [Kubernetes +executor](https://docs.gitlab.com/runner/executors/kubernetes.html#using-services) +you can specify the `command` to configure the registry mirror for the +Docker daemon. + +Docker: + +```toml +[[runners]] + ... + executor = "docker" + [runners.docker] + ... + privileged = true + [[runners.docker.services]] + name = "docker:19.03.13-dind" + command = ["--registry-mirror", "https://registry-mirror.example.com"] +``` + +Kubernetes: + +```toml +[[runners]] + ... + name = "kubernetes" + [runners.kubernetes] + ... + privileged = true + [[runners.kubernetes.services]] + name = "docker:19.03.13-dind" + command = ["--registry-mirror", "https://registry-mirror.example.com"] +``` + +##### Docker executor inside GitLab Runner configuration + +If you are an administrator of GitLab Runner and you want to use +the mirror for every `dind` service, update the +[configuration](https://docs.gitlab.com/runner/configuration/advanced-configuration.html) +to specify a [volume +mount](https://docs.gitlab.com/runner/configuration/advanced-configuration.html#volumes-in-the-runnersdocker-section). + +For example, if you have a `/opt/docker/daemon.json` file with the following +content: + +```json +{ + "registry-mirrors": [ + "https://registry-mirror.example.com" + ] +} +``` + +Update the `config.toml` file to mount the file to +`/etc/docker/daemon.json`. This would mount the file for **every** +container that is created by GitLab Runner. The configuration will be +picked up by the `dind` service. + +```toml +[[runners]] + ... + executor = "docker" + [runners.docker] + image = "alpine:3.12" + privileged = true + volumes = ["/opt/docker/daemon.json:/etc/docker/daemon.json:ro"] +``` + +##### Kubernetes executor inside GitLab Runner configuration + +> [Introduced](https://gitlab.com/gitlab-org/gitlab-runner/-/issues/3223) in GitLab Runner 13.6. + +If you are an administrator of GitLab Runner and you want to use +the mirror for every `dind` service, update the +[configuration](https://docs.gitlab.com/runner/configuration/advanced-configuration.html) +to specify a [ConfigMap volume +mount](https://docs.gitlab.com/runner/executors/kubernetes.html#using-volumes). + +For example, if you have a `/tmp/daemon.json` file with the following +content: + +```json +{ + "registry-mirrors": [ + "https://registry-mirror.example.com" + ] +} +``` + +Create a [ConfigMap](https://kubernetes.io/docs/concepts/configuration/configmap/) with the content +of this file. You can do this with a command like: + +```shell +kubectl create configmap docker-daemon --namespace gitlab-runner --from-file /tmp/daemon.json +``` + +NOTE: **Note:** +Make sure to use the namespace that GitLab Runner Kubernetes executor uses +to create job pods in. + +After the ConfigMap is created, you can update the `config.toml` +file to mount the file to `/etc/docker/daemon.json`. This update +mounts the file for **every** container that is created by GitLab Runner. +The configuration is picked up by the `dind` service. + +```toml +[[runners]] + ... + executor = "kubernetes" + [runners.kubernetes] + image = "alpine:3.12" + privileged = true + [[runners.kubernetes.volumes.config_map]] + name = "docker-daemon" + mount_path = "/etc/docker/daemon.json" + sub_path = "daemon.json" +``` + +#### Use Docker socket binding The third approach is to bind-mount `/var/run/docker.sock` into the container so that Docker is available in the context of that image. @@ -502,7 +644,7 @@ and [using the OverlayFS storage driver](https://docs.docker.com/engine/userguid ## Using the GitLab Container Registry -Once you've built a Docker image, you can push it up to the built-in +After you've built a Docker image, you can push it up to the built-in [GitLab Container Registry](../../user/packages/container_registry/index.md#build-and-push-by-using-gitlab-cicd). ## Troubleshooting diff --git a/doc/ci/docker/using_docker_images.md b/doc/ci/docker/using_docker_images.md index f7d54aa7d78..b8563182bd9 100644 --- a/doc/ci/docker/using_docker_images.md +++ b/doc/ci/docker/using_docker_images.md @@ -452,9 +452,9 @@ CI jobs: from `Dockerfile` that may be overridden in `.gitlab-ci.yml`) 1. The runner attaches itself to a running container. 1. The runner prepares a script (the combination of - [`before_script`](../yaml/README.md#before_script-and-after_script), + [`before_script`](../yaml/README.md#before_script), [`script`](../yaml/README.md#script), - and [`after_script`](../yaml/README.md#before_script-and-after_script)). + and [`after_script`](../yaml/README.md#after_script)). 1. The runner sends the script to the container's shell STDIN and receives the output. @@ -561,10 +561,11 @@ See below for examples of each. #### Determining your `DOCKER_AUTH_CONFIG` data -As an example, let's assume that you want to use the `registry.example.com:5000/private/image:latest` -image which is private and requires you to login into a private container registry. +As an example, let's assume you want to use the `registry.example.com:5000/private/image:latest` +image, which is private and requires you to sign in to a private container +registry. -Let's also assume that these are the login credentials: +Let's also assume that these are the sign-in credentials: | Key | Value | |:---------|:----------------------------| @@ -572,9 +573,9 @@ Let's also assume that these are the login credentials: | username | `my_username` | | password | `my_password` | -There are two ways to determine the value of `DOCKER_AUTH_CONFIG`: +Use one of the following methods to determine the value of `DOCKER_AUTH_CONFIG`: -- **First way -** Do a `docker login` on your local machine: +- Do a `docker login` on your local machine: ```shell docker login registry.example.com:5000 --username my_username --password my_password @@ -589,12 +590,11 @@ There are two ways to determine the value of `DOCKER_AUTH_CONFIG`: docker logout registry.example.com:5000 ``` -- **Second way -** In some setups, it's possible that Docker client - uses the available system key store to store the result of `docker - login`. In that case, it's impossible to read `~/.docker/config.json`, - so you need to prepare the required base64-encoded version of - `${username}:${password}` and create the Docker configuration JSON manually. - Open a terminal and execute the following command: +- In some setups, it's possible that Docker client uses the available system key + store to store the result of `docker login`. In that case, it's impossible to + read `~/.docker/config.json`, so you need to prepare the required + base64-encoded version of `${username}:${password}` and create the Docker + configuration JSON manually. Open a terminal and execute the following command: ```shell # The use of "-n" - prevents encoding a newline in the password. diff --git a/doc/ci/environments/img/protected_access_group_v13_6.png b/doc/ci/environments/img/protected_access_group_v13_6.png Binary files differnew file mode 100644 index 00000000000..9c39e4362e8 --- /dev/null +++ b/doc/ci/environments/img/protected_access_group_v13_6.png diff --git a/doc/ci/environments/index.md b/doc/ci/environments/index.md index baf2156e64a..361b7217d17 100644 --- a/doc/ci/environments/index.md +++ b/doc/ci/environments/index.md @@ -34,8 +34,7 @@ currently being deployed or has been deployed on your servers. It's important to know that: - Environments are like tags for your CI jobs, describing where code gets deployed. -- Deployments are created when [jobs](../yaml/README.md#introduction) deploy versions of code to environments, - so every environment can have one or more deployments. +- Deployments are created when [GitLab CI/CD](../yaml/README.md) is used to deploy versions of code to environments. GitLab: @@ -219,10 +218,17 @@ You can also specify a static part of the URL at `environment:url:`, such as The assigned URL for the `review/your-branch-name` environment is [visible in the UI](#using-the-environment-url). -> **Notes:** -> -> - `stop_review` doesn't generate a dotenv report artifact, so it won't recognize the `DYNAMIC_ENVIRONMENT_URL` variable. Therefore you should not set `environment:url:` in the `stop_review` job. -> - If the environment URL is not valid (for example, the URL is malformed), the system doesn't update the environment URL. +Note the following: + +- `stop_review` doesn't generate a dotenv report artifact, so it won't recognize the + `DYNAMIC_ENVIRONMENT_URL` variable. Therefore you shouldn't set `environment:url:` in the + `stop_review` job. +- If the environment URL isn't valid (for example, the URL is malformed), the system doesn't update + the environment URL. +- If the script that runs in `stop_review` exists only in your repository and therefore can't use + `GIT_STRATEGY: none`, configure [pipelines for merge requests](../../ci/merge_request_pipelines/index.md) + for these jobs. This ensures that runners can fetch the repository even after a feature branch is + deleted. For more information, see [Ref Specs for Runners](../pipelines/index.md#ref-specs-for-runners). ### Configuring manual deployments @@ -304,7 +310,7 @@ Dynamic environments are a fundamental part of [Review apps](../review_apps/inde #### Allowed variables -The `name` and `url` parameters for dynamic environments can use most available CI/CD variables, +The `name` and `url` keywords for dynamic environments can use most available CI/CD variables, including: - [Predefined environment variables](../variables/README.md#predefined-environment-variables) @@ -436,7 +442,7 @@ The configuration in this section provides a full development workflow where you - Tested. - Built. - Deployed as a Review App. -- Deployed to a staging server once the merge request is merged. +- Deployed to a staging server after the merge request is merged. - Finally, able to be manually deployed to the production server. The following combines the previous configuration examples, including: @@ -675,24 +681,23 @@ deploy_review: name: review/$CI_COMMIT_REF_NAME url: https://$CI_ENVIRONMENT_SLUG.example.com on_stop: stop_review - only: - - branches - except: - - master + rules: + - if: $CI_MERGE_REQUEST_ID stop_review: stage: deploy - variables: - GIT_STRATEGY: none script: - echo "Remove review app" - when: manual environment: name: review/$CI_COMMIT_REF_NAME action: stop + rules: + - if: $CI_MERGE_REQUEST_ID + when: manual ``` -Setting the [`GIT_STRATEGY`](../yaml/README.md#git-strategy) to `none` is necessary in the +If you can't use [Pipelines for merge requests](../merge_request_pipelines/index.md), +setting the [`GIT_STRATEGY`](../runners/README.md#git-strategy) to `none` is necessary in the `stop_review` job so that the [runner](https://docs.gitlab.com/runner/) won't try to check out the code after the branch is deleted. @@ -748,13 +753,17 @@ review_app: name: review/$CI_COMMIT_REF_NAME on_stop: stop_review_app auto_stop_in: 1 week + rules: + - if: $CI_MERGE_REQUEST_ID stop_review_app: script: stop-review-app environment: name: review/$CI_COMMIT_REF_NAME action: stop - when: manual + rules: + - if: $CI_MERGE_REQUEST_ID + when: manual ``` As long as a merge request is active and keeps getting new commits, @@ -923,11 +932,10 @@ the [Kubernetes integration](../../user/project/clusters/index.md)), GitLab can a terminal session to your environment. This is a powerful feature that allows you to debug issues without leaving the comfort -of your web browser. To enable it, just follow the instructions given in the service integration +of your web browser. To enable it, follow the instructions given in the service integration documentation. -NOTE: **Note:** -Container-based deployments often lack basic tools (like an editor), and may +Note that container-based deployments often lack basic tools (like an editor), and may be stopped or restarted at any time. If this happens, you will lose all your changes. Treat this as a debugging tool, not a comprehensive online IDE. diff --git a/doc/ci/environments/protected_environments.md b/doc/ci/environments/protected_environments.md index 87bced29906..eeb95947ba1 100644 --- a/doc/ci/environments/protected_environments.md +++ b/doc/ci/environments/protected_environments.md @@ -45,6 +45,61 @@ To protect an environment: The protected environment will now appear in the list of protected environments. +### Use the API to protect an environment + +Alternatively, you can use the API to protect an environment: + +1. Use a project with a CI that creates an environment. For example: + + ```yaml + stages: + - test + - deploy + + test: + stage: test + script: + - 'echo "Testing Application: ${CI_PROJECT_NAME}"' + + production: + stage: deploy + when: manual + script: + - 'echo "Deploying to ${CI_ENVIRONMENT_NAME}"' + environment: + name: ${CI_JOB_NAME} + ``` + +1. Use the UI to [create a new group](../../user/group/index.md#create-a-new-group). + For example, this group is called `protected-access-group` and has the group ID `9899826`. Note + that the rest of the examples in these steps use this group. + +  + +1. Use the API to add a user to the group as a reporter: + + ```shell + $ curl --request POST --header "PRIVATE-TOKEN: xxxxxxxxxxxx" --data "user_id=3222377&access_level=20" "https://gitlab.com/api/v4/groups/9899826/members" + + {"id":3222377,"name":"Sean Carroll","username":"sfcarroll","state":"active","avatar_url":"https://assets.gitlab-static.net/uploads/-/system/user/avatar/3222377/avatar.png","web_url":"https://gitlab.com/sfcarroll","access_level":20,"created_at":"2020-10-26T17:37:50.309Z","expires_at":null} + ``` + +1. Use the API to add the group to the project as a reporter: + + ```shell + $ curl --request POST --header "PRIVATE-TOKEN: xxxxxxxxxxxx" --request POST "https://gitlab.com/api/v4/projects/22034114/share?group_id=9899826&group_access=20" + + {"id":1233335,"project_id":22034114,"group_id":9899826,"group_access":20,"expires_at":null} + ``` + +1. Use the API to add the group with protected environment access: + + ```shell + curl --header 'Content-Type: application/json' --request POST --data '{"name": "production", "deploy_access_levels": [{"group_id": 9899826}]}' --header "PRIVATE-TOKEN: xxxxxxxxxxx" "https://gitlab.com/api/v4/projects/22034114/protected_environments" + ``` + +The group now has access and can be seen in the UI. + ## Environment access by group membership A user may be granted access to protected environments as part of @@ -72,8 +127,7 @@ they have the following privileges: Users granted access to a protected environment, but not push or merge access to the branch deployed to it, are only granted access to deploy the environment. -NOTE: **Note:** -Deployment-only access is the only possible access level for users with +Note that deployment-only access is the only possible access level for users with [Reporter permissions](../../user/permissions.md). ## Modifying and unprotecting environments @@ -84,7 +138,6 @@ Maintainers can: **Allowed to Deploy** dropdown menu. - Unprotect a protected environment by clicking the **Unprotect** button for that environment. -NOTE: **Note:** After an environment is unprotected, all access entries are deleted and must be re-entered if the environment is re-protected. diff --git a/doc/ci/examples/README.md b/doc/ci/examples/README.md index 34e3b276d3e..1abb33005ca 100644 --- a/doc/ci/examples/README.md +++ b/doc/ci/examples/README.md @@ -15,7 +15,7 @@ Examples are available in several forms. As a collection of: - `.gitlab-ci.yml` [template files](#cicd-templates) maintained in GitLab, for many common frameworks and programming languages. -- Repositories with [example projects](https://gitlab.com/gitlab-examples) for various languages. You can fork and adjust them to your own needs. Projects include demonstrations of [multi-project pipelines](https://gitlab.com/gitlab-examples/multi-project-pipelines) and using [Review Apps with a static site served by NGINX](https://gitlab.com/gitlab-examples/review-apps-nginx/). +- Repositories with [example projects](https://gitlab.com/gitlab-examples) for various languages. You can fork and adjust them to your own needs. Projects include an example of using [Review Apps with a static site served by NGINX](https://gitlab.com/gitlab-examples/review-apps-nginx/). - Examples and [other resources](#other-resources) listed below. ## CI/CD examples @@ -42,6 +42,7 @@ The following table lists examples with step-by-step tutorials that are containe | Scala on Heroku | [Test and deploy a Scala application to Heroku](test-scala-application.md). | | Parallel testing Ruby & JS | [GitLab CI/CD parallel jobs testing for Ruby & JavaScript projects](https://docs.knapsackpro.com/2019/how-to-run-parallel-jobs-for-rspec-tests-on-gitlab-ci-pipeline-and-speed-up-ruby-javascript-testing). | | Secrets management with Vault | [Authenticating and Reading Secrets With Hashicorp Vault](authenticating-with-hashicorp-vault/index.md). | +| Multi project pipeline | [Build, test deploy using multi project pipeline](https://gitlab.com/gitlab-examples/upstream-project). | ### Contributing examples diff --git a/doc/ci/examples/authenticating-with-hashicorp-vault/index.md b/doc/ci/examples/authenticating-with-hashicorp-vault/index.md index 4a0ff2fa6ac..c0fb94acdf2 100644 --- a/doc/ci/examples/authenticating-with-hashicorp-vault/index.md +++ b/doc/ci/examples/authenticating-with-hashicorp-vault/index.md @@ -22,7 +22,8 @@ This tutorial assumes you are familiar with GitLab CI/CD and Vault. To follow along, you will need: - An account on GitLab. -- A running Vault server and the access required to configure authentication and create roles and policies. +- A running Vault server and access to it is required to configure authentication and create roles + and policies. For HashiCorp Vaults, this can be the Open Source or Enterprise version. NOTE: **Note:** You will need to replace the `vault.example.com` URL below with the URL of your Vault server and `gitlab.example.com` with the URL of your GitLab instance. @@ -55,7 +56,7 @@ The JWT's payload looks like this: } ``` -The JWT is encoded by using RS256 and signed with your GitLab instance's OpenID Connect private key. The expire time for the token will be set to job's timeout, if specified, or 5 minutes if it is not. The key used to sign this token may change without any notice. In such case retrying the job will generate new JWT using the current signing key. +The JWT is encoded by using RS256 and signed with a dedicated private key. The expire time for the token will be set to job's timeout, if specified, or 5 minutes if it is not. The key used to sign this token may change without any notice. In such case retrying the job will generate new JWT using the current signing key. You can use this JWT and your instance's JWKS endpoint (`https://gitlab.example.com/-/jwks`) to authenticate with a Vault server that is configured to allow the JWT Authentication method for authentication. diff --git a/doc/ci/examples/deploy_spring_boot_to_cloud_foundry/index.md b/doc/ci/examples/deploy_spring_boot_to_cloud_foundry/index.md index b113b10f2e3..25b866a08ed 100644 --- a/doc/ci/examples/deploy_spring_boot_to_cloud_foundry/index.md +++ b/doc/ci/examples/deploy_spring_boot_to_cloud_foundry/index.md @@ -103,27 +103,27 @@ Hub](https://hub.docker.com/). We've also added the [`only` clause](../../yaml/README.md#onlyexcept-basic) to ensure our deployments only happen when we push to the master branch. -Now, since the steps defined in `.gitlab-ci.yml` require credentials to login -to CF, you'll need to add your CF credentials as [environment -variables](../../variables/README.md#predefined-environment-variables) -on GitLab CI/CD. To set the environment variables, navigate to your project's -**Settings > CI/CD** and expand **Variables**. Name the variables +Because the steps defined in `.gitlab-ci.yml` require credentials to sign in to +CF, you'll need to add your CF credentials as +[environment variables](../../variables/README.md#predefined-environment-variables) +in GitLab CI/CD. To set the environment variables, navigate to your project's +**Settings > CI/CD**, and then expand **Variables**. Name the variables `CF_USERNAME` and `CF_PASSWORD` and set them to the correct values.  -Once set up, GitLab CI/CD will deploy your app to CF at every push to your -repository's default branch. To see the build logs or watch your builds running -live, navigate to **CI/CD > Pipelines**. +After set up, GitLab CI/CD deploys your app to CF at every push to your +repository's default branch. To review the build logs or watch your builds +running live, navigate to **CI/CD > Pipelines**. CAUTION: **Caution:** -It is considered best practice for security to create a separate deploy -user for your application and add its credentials to GitLab instead of using -a developer's credentials. +It's considered best practice for security to create a separate deploy user for +your application and add its credentials to GitLab instead of using a +developer's credentials. To start a manual deployment in GitLab go to **CI/CD > Pipelines** then click -on **Run Pipeline**. Once the app is finished deploying it will display the URL -of your application in the logs for the `production` job like: +on **Run Pipeline**. After the app is finished deploying, it will display the +URL of your application in the logs for the `production` job like: ```shell requested state: started diff --git a/doc/ci/examples/devops_and_game_dev_with_gitlab_ci_cd/index.md b/doc/ci/examples/devops_and_game_dev_with_gitlab_ci_cd/index.md index 836141af91e..39cad3a0c93 100644 --- a/doc/ci/examples/devops_and_game_dev_with_gitlab_ci_cd/index.md +++ b/doc/ci/examples/devops_and_game_dev_with_gitlab_ci_cd/index.md @@ -249,7 +249,7 @@ pipeline to include running the tests along with the existing build job. ## Continuous Integration -To ensure our changes don't break the build and all tests still pass, we utilize +To ensure our changes don't break the build and all tests still pass, we use Continuous Integration (CI) to run these checks automatically for every push. Read through this article to understand [Continuous Integration, Continuous Delivery, and Continuous Deployment](https://about.gitlab.com/blog/2016/08/05/continuous-integration-delivery-and-deployment-with-gitlab/), and how these methods are leveraged by GitLab. @@ -261,7 +261,7 @@ Please read through the [documentation on CI/CD configuration](../../../ci/yaml/ ### Build your game with GitLab CI/CD We need to update our build job to ensure tests get run as well. Add `gulp build-test` -to the end of the `script` array for the existing `build` job. Once these commands run, +to the end of the `script` array for the existing `build` job. After these commands run, we know we will need to access everything in the `built` folder, given by GitLab CI/CD's `artifacts`. We'll also cache `node_modules` to avoid having to do a full re-pull of those dependencies: just pack them up in the cache. Here is the full `build` job: diff --git a/doc/ci/examples/end_to_end_testing_webdriverio/index.md b/doc/ci/examples/end_to_end_testing_webdriverio/index.md index aa6e6f73a0d..42725e8aef9 100644 --- a/doc/ci/examples/end_to_end_testing_webdriverio/index.md +++ b/doc/ci/examples/end_to_end_testing_webdriverio/index.md @@ -14,7 +14,7 @@ environment, reducing the effort to assess the impact of changes. Thus, when we and it will immediately be clear that the application can still be properly built and deployed. After all, you can _see_ it running! -<img src="img/deployed_dependency_update.png" alt="dependencies.io" class="image-noshadow"> +<img src="img/deployed_dependency_update.png" alt="dependencies.io"> However, looking at the freshly deployed code to check whether it still looks and behaves as expected is repetitive manual work, which means it is a prime candidate for automation. This is @@ -140,7 +140,7 @@ new browser window interacting with your app as you specified. Which brings us to the exciting part: how do we run this in GitLab CI/CD? There are two things we need to do for this: -1. Set up [CI/CD jobs](../../yaml/README.md#introduction) that actually have a browser available. +1. Set up [CI/CD jobs](../../yaml/README.md) that actually have a browser available. 1. Update our WebdriverIO configuration to use those browsers to visit the review apps. For the scope of this article, we've defined an additional [CI/CD stage](../../yaml/README.md#stages) diff --git a/doc/ci/examples/laravel_with_gitlab_and_envoy/index.md b/doc/ci/examples/laravel_with_gitlab_and_envoy/index.md index aaa34afeddf..f692a8042f5 100644 --- a/doc/ci/examples/laravel_with_gitlab_and_envoy/index.md +++ b/doc/ci/examples/laravel_with_gitlab_and_envoy/index.md @@ -632,7 +632,7 @@ After our code passed through the pipeline successfully, we can deploy to our pr  -Once the deploy pipeline passed successfully, navigate to **Pipelines > Environments**. +After the deploy pipeline passed successfully, navigate to **Pipelines > Environments**.  diff --git a/doc/ci/examples/test_phoenix_app_with_gitlab_ci_cd/index.md b/doc/ci/examples/test_phoenix_app_with_gitlab_ci_cd/index.md index c62f0dec598..9bc9ac5423e 100644 --- a/doc/ci/examples/test_phoenix_app_with_gitlab_ci_cd/index.md +++ b/doc/ci/examples/test_phoenix_app_with_gitlab_ci_cd/index.md @@ -298,7 +298,7 @@ project. - mix ecto.migrate ``` - This ensures that [rebar3](https://www.rebar3.org) and [hex](https://hex.pm) are both installed + This ensures that [rebar3](https://rebar3.org) and [hex](https://hex.pm) are both installed before attempting to fetch the dependencies that are required to run the tests. Next, the `postgres` db is created and migrated with `ecto`, to ensure it's up-to-date. diff --git a/doc/ci/git_submodules.md b/doc/ci/git_submodules.md index 5bcfe8fa3f1..c28febd15d7 100644 --- a/doc/ci/git_submodules.md +++ b/doc/ci/git_submodules.md @@ -78,7 +78,7 @@ correctly with your CI jobs: GIT_SUBMODULE_STRATEGY: recursive ``` - See the [`.gitlab-ci.yml` reference](yaml/README.md#git-submodule-strategy) + See the [GitLab Runner documentation](runners/README.md#git-submodule-strategy) for more details about `GIT_SUBMODULE_STRATEGY`. 1. If you are using an older version of `gitlab-runner`, then use diff --git a/doc/ci/interactive_web_terminal/img/interactive_web_terminal_running_job.png b/doc/ci/interactive_web_terminal/img/interactive_web_terminal_running_job.png Binary files differindex 3ee5e39afc0..8082d17ae6a 100644 --- a/doc/ci/interactive_web_terminal/img/interactive_web_terminal_running_job.png +++ b/doc/ci/interactive_web_terminal/img/interactive_web_terminal_running_job.png diff --git a/doc/ci/interactive_web_terminal/index.md b/doc/ci/interactive_web_terminal/index.md index 125e7dabecf..a131d21039d 100644 --- a/doc/ci/interactive_web_terminal/index.md +++ b/doc/ci/interactive_web_terminal/index.md @@ -16,7 +16,7 @@ is deployed, some [security precautions](../../administration/integration/termin taken to protect the users. NOTE: **Note:** -[Shared runners on GitLab.com](../quick_start/README.md#shared-runners) do not +[Shared runners on GitLab.com](../runners/README.md#shared-runners) do not provide an interactive web terminal. Follow [this issue](https://gitlab.com/gitlab-org/gitlab/-/issues/24674) for progress on adding support. For groups and projects hosted on GitLab.com, interactive web diff --git a/doc/ci/introduction/img/gitlab_workflow_example_11_9.png b/doc/ci/introduction/img/gitlab_workflow_example_11_9.png Binary files differindex f3fb9444b55..1f11db55f81 100644 --- a/doc/ci/introduction/img/gitlab_workflow_example_11_9.png +++ b/doc/ci/introduction/img/gitlab_workflow_example_11_9.png diff --git a/doc/ci/introduction/img/job_running.png b/doc/ci/introduction/img/job_running.png Binary files differindex d5f922ceb8c..efd138fd4f8 100644 --- a/doc/ci/introduction/img/job_running.png +++ b/doc/ci/introduction/img/job_running.png diff --git a/doc/ci/introduction/index.md b/doc/ci/introduction/index.md index d1f3e449e5b..b24ee66fdba 100644 --- a/doc/ci/introduction/index.md +++ b/doc/ci/introduction/index.md @@ -17,6 +17,14 @@ Out-of-the-box management systems can decrease hours spent on maintaining toolch Watch our ["Mastering continuous software development"](https://about.gitlab.com/webcast/mastering-ci-cd/) webcast to learn about continuous methods and how GitLab’s built-in CI can help you simplify and scale software development. +> For some additional information about GitLab CI/CD: +> +> - <i class="fa fa-youtube-play youtube" aria-hidden="true"></i> Watch the [CI/CD Ease of configuration](https://www.youtube.com/embed/opdLqwz6tcE) video. +> - Watch the [Making the case for CI/CD in your organization](https://about.gitlab.com/compare/github-actions-alternative/) +> webcast to learn the benefits of CI/CD and how to measure the results of CI/CD automation. +> - <i class="fa fa-youtube-play youtube" aria-hidden="true"></i> Learn how [Verizon reduced rebuilds](https://about.gitlab.com/blog/2019/02/14/verizon-customer-story/) +> from 30 days to under 8 hours with GitLab. + ## Introduction to CI/CD methodologies The continuous methodologies of software development are based on @@ -93,7 +101,7 @@ In this file, you can define the scripts you want to run, define include and cache dependencies, choose commands you want to run in sequence and those you want to run in parallel, define where you want to deploy your app, and specify whether you will want to run the scripts automatically -or trigger any of them manually. Once you're familiar with +or trigger any of them manually. After you're familiar with GitLab CI/CD you can add more advanced steps into the configuration file. To add scripts to that file, you'll need to organize them in a @@ -102,7 +110,7 @@ the tests you wish to perform. To visualize the process, imagine that all the scripts you add to the configuration file are the same as the commands you run on a terminal on your computer. -Once you've added your `.gitlab-ci.yml` configuration file to your +After you've added your `.gitlab-ci.yml` configuration file to your repository, GitLab will detect it and run your scripts with the tool called [GitLab Runner](https://docs.gitlab.com/runner/), which works similarly to your terminal. @@ -149,7 +157,7 @@ Consider the following example for how GitLab CI/CD fits in a common development workflow. Assume that you have discussed a code implementation in an issue -and worked locally on your proposed changes. Once you push your +and worked locally on your proposed changes. After you push your commits to a feature branch in a remote repository in GitLab, the CI/CD pipeline set for your project is triggered. By doing so, GitLab CI/CD: @@ -159,7 +167,7 @@ so, GitLab CI/CD: - Preview the changes per merge request with Review Apps, as you would see in your `localhost`. -Once you're happy with your implementation: +After you're happy with your implementation: - Get your code reviewed and approved. - Merge the feature branch into the default branch. diff --git a/doc/ci/pipelines/img/collapsible_log_v12_6.png b/doc/ci/jobs/img/collapsible_log_v12_6.png Binary files differindex a1e9aeb244a..a1e9aeb244a 100644 --- a/doc/ci/pipelines/img/collapsible_log_v12_6.png +++ b/doc/ci/jobs/img/collapsible_log_v12_6.png diff --git a/doc/ci/pipelines/img/job_failure_reason.png b/doc/ci/jobs/img/job_failure_reason.png Binary files differindex d44b8e6d1be..d44b8e6d1be 100644 --- a/doc/ci/pipelines/img/job_failure_reason.png +++ b/doc/ci/jobs/img/job_failure_reason.png diff --git a/doc/ci/pipelines/img/job_group_v12_10.png b/doc/ci/jobs/img/job_group_v12_10.png Binary files differindex 27e6bfbfc0f..27e6bfbfc0f 100644 --- a/doc/ci/pipelines/img/job_group_v12_10.png +++ b/doc/ci/jobs/img/job_group_v12_10.png diff --git a/doc/ci/jobs/img/manual_job_variables.png b/doc/ci/jobs/img/manual_job_variables.png Binary files differnew file mode 100644 index 00000000000..63801ade21f --- /dev/null +++ b/doc/ci/jobs/img/manual_job_variables.png diff --git a/doc/ci/pipelines/img/pipeline_incremental_rollout.png b/doc/ci/jobs/img/pipeline_incremental_rollout.png Binary files differindex b3498e9a5a5..b3498e9a5a5 100644 --- a/doc/ci/pipelines/img/pipeline_incremental_rollout.png +++ b/doc/ci/jobs/img/pipeline_incremental_rollout.png diff --git a/doc/ci/pipelines/img/pipelines_grouped.png b/doc/ci/jobs/img/pipelines_grouped.png Binary files differindex 82814754747..82814754747 100644 --- a/doc/ci/pipelines/img/pipelines_grouped.png +++ b/doc/ci/jobs/img/pipelines_grouped.png diff --git a/doc/ci/pipelines/img/pipelines_mini_graph_sorting.png b/doc/ci/jobs/img/pipelines_mini_graph_sorting.png Binary files differindex 3a4e5453360..3a4e5453360 100644 --- a/doc/ci/pipelines/img/pipelines_mini_graph_sorting.png +++ b/doc/ci/jobs/img/pipelines_mini_graph_sorting.png diff --git a/doc/ci/jobs/index.md b/doc/ci/jobs/index.md new file mode 100644 index 00000000000..dc306ac7ecb --- /dev/null +++ b/doc/ci/jobs/index.md @@ -0,0 +1,251 @@ +--- +stage: Verify +group: Continuous Integration +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +--- + +# Jobs + +Pipeline configuration begins with jobs. Jobs are the most fundamental element of a `.gitlab-ci.yml` file. + +Jobs are: + +- Defined with constraints stating under what conditions they should be executed. +- Top-level elements with an arbitrary name and must contain at least the [`script`](../yaml/README.md#script) clause. +- Not limited in how many can be defined. + +For example: + +```yaml +job1: + script: "execute-script-for-job1" + +job2: + script: "execute-script-for-job2" +``` + +The above example is the simplest possible CI/CD configuration with two separate +jobs, where each of the jobs executes a different command. +Of course a command can execute code directly (`./configure;make;make install`) +or run a script (`test.sh`) in the repository. + +Jobs are picked up by [runners](../runners/README.md) and executed within the +environment of the runner. What is important is that each job is run +independently from each other. + +## View jobs in a pipeline + +When you access a pipeline, you can see the related jobs for that pipeline. + +Clicking an individual job shows you its job log, and allows you to: + +- Cancel the job. +- Retry the job. +- Erase the job log. + +## See why a job failed + +> [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/17782) in GitLab 10.7. + +When a pipeline fails or is allowed to fail, there are several places where you +can find the reason: + +- In the [pipeline graph](../pipelines/index.md#visualize-pipelines), on the pipeline detail view. +- In the pipeline widgets, in the merge requests and commit pages. +- In the job views, in the global and detailed views of a job. + +In each place, if you hover over the failed job you can see the reason it failed. + + + +In [GitLab 10.8](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/17814) and later, +you can also see the reason it failed on the Job detail page. + +## The order of jobs in a pipeline + +The order of jobs in a pipeline depends on the type of pipeline graph. + +- For [regular pipeline graphs](../pipelines/index.md#regular-pipeline-graphs), jobs are sorted by name. +- For [pipeline mini graphs](../pipelines/index.md#pipeline-mini-graphs), jobs are sorted by severity and then by name. + +The order of severity is: + +- failed +- warning +- pending +- running +- manual +- scheduled +- canceled +- success +- skipped +- created + +For example: + + + +## Group jobs in a pipeline + +> [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/6242) in GitLab 8.12. + +If you have many similar jobs, your [pipeline graph](../pipelines/index.md#visualize-pipelines) becomes long and hard +to read. + +You can automatically group similar jobs together. If the job names are formatted in a certain way, +they are collapsed into a single group in regular pipeline graphs (not the mini graphs). + +You can recognize when a pipeline has grouped jobs if you don't see the retry or +cancel button inside them. Hovering over them shows the number of grouped +jobs. Click to expand them. + + + +To create a group of jobs, in the [CI/CD pipeline configuration file](../yaml/README.md), +separate each job name with a number and one of the following: + +- A slash (`/`), for example, `test 1/3`, `test 2/3`, `test 3/3`. +- A colon (`:`), for example, `test 1:3`, `test 2:3`, `test 3:3`. +- A space, for example `test 0 3`, `test 1 3`, `test 2 3`. + +You can use these symbols interchangeably. + +In the example below, these three jobs are in a group named `build ruby`: + +```yaml +build ruby 1/3: + stage: build + script: + - echo "ruby1" + +build ruby 2/3: + stage: build + script: + - echo "ruby2" + +build ruby 3/3: + stage: build + script: + - echo "ruby3" +``` + +In the pipeline, the result is a group named `build ruby` with three jobs: + + + +The jobs are be ordered by comparing the numbers from left to right. You +usually want the first number to be the index and the second number to be the total. + +[This regular expression](https://gitlab.com/gitlab-org/gitlab/blob/2f3dc314f42dbd79813e6251792853bc231e69dd/app/models/commit_status.rb#L99) +evaluates the job names: `\d+[\s:\/\\]+\d+\s*`. + +## Specifying variables when running manual jobs + +> [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/30485) in GitLab 12.2. + +When running manual jobs you can supply additional job specific variables. + +You can do this from the job page of the manual job you want to run with +additional variables. To access this page, click on the **name** of the manual job in +the pipeline view, *not* the play (**{play}**) button. + +This is useful when you want to alter the execution of a job that uses +[custom environment variables](../variables/README.md#custom-environment-variables). +Add a variable name (key) and value here to override the value defined in +[the UI or `.gitlab-ci.yml`](../variables/README.md#custom-environment-variables), +for a single run of the manual job. + + + +## Delay a job + +> [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/21767) in GitLab 11.4. + +When you do not want to run a job immediately, you can use the [`when:delayed`](../yaml/README.md#whendelayed) keyword to +delay a job's execution for a certain period. + +This is especially useful for timed incremental rollout where new code is rolled out gradually. + +For example, if you start rolling out new code and: + +- Users do not experience trouble, GitLab can automatically complete the deployment from 0% to 100%. +- Users experience trouble with the new code, you can stop the timed incremental rollout by canceling the pipeline + and [rolling](../environments/index.md#retrying-and-rolling-back) back to the last stable version. + + + +## Expand and collapse job log sections + +> [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/14664) in GitLab 12.0. + +Job logs are divided into sections that can be collapsed or expanded. Each section displays +the duration. + +In the following example: + +- Two sections are collapsed and can be expanded. +- Three sections are expanded and can be collapsed. + + + +### Custom collapsible sections + +> [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/14664) in GitLab 12.0. + +You can create [collapsible sections in job logs](#expand-and-collapse-job-log-sections) +by manually outputting special codes +that GitLab uses to determine what sections to collapse: + +- Section start marker: `section_start:UNIX_TIMESTAMP:SECTION_NAME\r\e[0K` + `TEXT_OF_SECTION_HEADER` +- Section end marker: `section_end:UNIX_TIMESTAMP:SECTION_NAME\r\e[0K` + +You must add these codes to the script section of the CI configuration. For example, +using `echo`: + +```yaml +job1: + script: + - echo -e "section_start:`date +%s`:my_first_section\r\e[0KHeader of the 1st collapsible section" + - echo 'this line should be hidden when collapsed' + - echo -e "section_end:`date +%s`:my_first_section\r\e[0K" +``` + +In the example above: + +- `date +%s`: The Unix timestamp (for example `1560896352`). +- `my_first_section`: The name given to the section. +- `\r\e[0K`: Prevents the section markers from displaying in the rendered (colored) + job log, but they are displayed in the raw job log. To see them, in the top right + of the job log, click **{doc-text}** (**Show complete raw**). + - `\r`: carriage return. + - `\e[0K`: clear line ANSI escape code. + +Sample raw job log: + +```plaintext +section_start:1560896352:my_first_section\r\e[0KHeader of the 1st collapsible section +this line should be hidden when collapsed +section_end:1560896353:my_first_section\r\e[0K +``` + +### Pre-collapse sections + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/198413) in GitLab 13.5. + +You can make the job log automatically collapse collapsible sections by adding the `collapsed` option to the section start. +Add `[collapsed=true]` after the section name and before the `\r`. The section end marker +remains unchanged: + +- Section start marker with `[collapsed=true]`: `section_start:UNIX_TIMESTAMP:SECTION_NAME[collapsed=true]\r\e[0K` + `TEXT_OF_SECTION_HEADER` +- Section end marker: `section_end:UNIX_TIMESTAMP:SECTION_NAME\r\e[0K` + +Add the updated section start text to the CI configuration. For example, +using `echo`: + +```yaml +job1: + script: + - echo -e "section_start:`date +%s`:my_first_section[collapsed=true]\r\e[0KHeader of the 1st collapsible section" + - echo 'this line should be hidden automatically after loading the job log' + - echo -e "section_end:`date +%s`:my_first_section\r\e[0K" +``` diff --git a/doc/ci/large_repositories/index.md b/doc/ci/large_repositories/index.md index 0019eb5f40c..f25ef7c725a 100644 --- a/doc/ci/large_repositories/index.md +++ b/doc/ci/large_repositories/index.md @@ -56,29 +56,16 @@ test: > Introduced in GitLab Runner 8.9. -By default, GitLab is configured to always prefer the `GIT_STRATEGY: fetch` strategy. -The `GIT_STRATEGY: fetch` strategy will re-use existing worktrees if found -on disk. This is different to the `GIT_STRATEGY: clone` strategy -as in case of clones, if a worktree is found, it is removed before clone. - -Usage of `fetch` is preferred because it reduces the amount of data to transfer and +By default, GitLab is configured to use the [`fetch` Git strategy](../runners/README.md#git-strategy), +which is recommended for large repositories. +This strategy reduces the amount of data to transfer and does not really impact the operations that you might do on a repository from CI. -However, `fetch` does require access to the previous worktree. This works -well when using the `shell` or `docker` executor because these -try to preserve worktrees and try to re-use them by default. - -This does not work today for `kubernetes` executor and has limitations when using -`docker+machine`. `kubernetes` executor today always clones into ephemeral directory. - -GitLab also offers the `GIT_STRATEGY: none` strategy. This disables any `fetch` and `checkout` commands -done by GitLab, requiring you to do them. - ## Git clone path > Introduced in GitLab Runner 11.10. -[`GIT_CLONE_PATH`](../yaml/README.md#custom-build-directories) allows you to +[`GIT_CLONE_PATH`](../runners/README.md#custom-build-directories) allows you to control where you clone your sources. This can have implications if you heavily use big repositories with fork workflow. @@ -90,7 +77,7 @@ In such cases, ideally you want to make the GitLab Runner executor be used only for the given project and not shared across different projects to make this process more efficient. -The [`GIT_CLONE_PATH`](../yaml/README.md#custom-build-directories) has to be +The [`GIT_CLONE_PATH`](../runners/README.md#custom-build-directories) has to be within the `$CI_BUILDS_DIR`. Currently, it is impossible to pick any path from disk. @@ -98,12 +85,12 @@ from disk. > Introduced in GitLab Runner 11.10. -[`GIT_CLEAN_FLAGS`](../yaml/README.md#git-clean-flags) allows you to control +[`GIT_CLEAN_FLAGS`](../runners/README.md#git-clean-flags) allows you to control whether or not you require the `git clean` command to be executed for each CI job. By default, GitLab ensures that you have your worktree on the given SHA, and that your repository is clean. -[`GIT_CLEAN_FLAGS`](../yaml/README.md#git-clean-flags) is disabled when set +[`GIT_CLEAN_FLAGS`](../runners/README.md#git-clean-flags) is disabled when set to `none`. On very big repositories, this might be desired because `git clean` is disk I/O intensive. Controlling that with `GIT_CLEAN_FLAGS: -ffdx -e .build/` (for example) allows you to control and disable removal of some @@ -112,7 +99,7 @@ the incremental builds. This has the biggest effect if you re-use existing machines and have an existing worktree that you can re-use for builds. For exact parameters accepted by -[`GIT_CLEAN_FLAGS`](../yaml/README.md#git-clean-flags), see the documentation +[`GIT_CLEAN_FLAGS`](../runners/README.md#git-clean-flags), see the documentation for [`git clean`](https://git-scm.com/docs/git-clean). The available parameters are dependent on Git version. @@ -120,14 +107,14 @@ are dependent on Git version. > [Introduced](https://gitlab.com/gitlab-org/gitlab-runner/-/issues/4142) in GitLab Runner 13.1. -[`GIT_FETCH_EXTRA_FLAGS`](../yaml/README.md#git-fetch-extra-flags) allows you +[`GIT_FETCH_EXTRA_FLAGS`](../runners/README.md#git-fetch-extra-flags) allows you to modify `git fetch` behavior by passing extra flags. For example, if your project contains a large number of tags that your CI jobs don't rely on, you could add [`--no-tags`](https://git-scm.com/docs/git-fetch#Documentation/git-fetch.txt---no-tags) to the extra flags to make your fetches faster and more compact. -See the [`GIT_FETCH_EXTRA_FLAGS` documentation](../yaml/README.md#git-fetch-extra-flags) +See the [`GIT_FETCH_EXTRA_FLAGS` documentation](../runners/README.md#git-fetch-extra-flags) for more information. ## Fork-based workflow diff --git a/doc/ci/lint.md b/doc/ci/lint.md index 716a4218d97..cef71a68d72 100644 --- a/doc/ci/lint.md +++ b/doc/ci/lint.md @@ -1,3 +1,9 @@ +--- +stage: none +group: unassigned +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +--- + # CI Lint If you want to test the validity of your GitLab CI/CD configuration before committing @@ -11,7 +17,8 @@ in your project and click **CI lint**. ## Validate basic logic and syntax By default, the CI lint checks the syntax of your CI YAML configuration and also runs -some basic logical validations. +some basic logical validations. Configuration added with the [`includes` keyword](yaml/README.md#include), +is also validated. To use the CI lint, paste a complete CI configuration (`.gitlab-ci.yml` for example) into the text box and click **Validate**: diff --git a/doc/ci/merge_request_pipelines/img/merge_request_pipelines_doubled_MR_v12_09.png b/doc/ci/merge_request_pipelines/img/merge_request_pipelines_doubled_MR_v12_09.png Binary files differdeleted file mode 100644 index 3e4c72b9279..00000000000 --- a/doc/ci/merge_request_pipelines/img/merge_request_pipelines_doubled_MR_v12_09.png +++ /dev/null diff --git a/doc/ci/merge_request_pipelines/img/merge_request_pipelines_doubled_branch_v12_09.png b/doc/ci/merge_request_pipelines/img/merge_request_pipelines_doubled_branch_v12_09.png Binary files differdeleted file mode 100644 index dd70c3f8c20..00000000000 --- a/doc/ci/merge_request_pipelines/img/merge_request_pipelines_doubled_branch_v12_09.png +++ /dev/null diff --git a/doc/ci/merge_request_pipelines/index.md b/doc/ci/merge_request_pipelines/index.md index cc0b4ac1f86..ac9cda4e46c 100644 --- a/doc/ci/merge_request_pipelines/index.md +++ b/doc/ci/merge_request_pipelines/index.md @@ -56,7 +56,7 @@ below. When you use this method, you have to specify `only: - merge_requests` for each job. In this example, the pipeline contains a `test` job that is configured to run on merge requests. -The `build` and `deploy` jobs don't have the `only: - merge_requests` parameter, +The `build` and `deploy` jobs don't have the `only: - merge_requests` keyword, so they will not run on merge requests. ```yaml @@ -81,8 +81,8 @@ deploy: #### Excluding certain jobs -The behavior of the `only: [merge_requests]` parameter is such that _only_ jobs with -that parameter are run in the context of a merge request; no other jobs will be run. +The behavior of the `only: [merge_requests]` keyword is such that _only_ jobs with +that keyword are run in the context of a merge request; no other jobs will be run. However, you can invert this behavior and have all of your jobs run _except_ for one or two. diff --git a/doc/ci/merge_request_pipelines/pipelines_for_merged_results/img/merge_request_pipeline_config.png b/doc/ci/merge_request_pipelines/pipelines_for_merged_results/img/merge_request_pipeline_config.png Binary files differdeleted file mode 100644 index 3ee9d8ec93b..00000000000 --- a/doc/ci/merge_request_pipelines/pipelines_for_merged_results/img/merge_request_pipeline_config.png +++ /dev/null diff --git a/doc/ci/merge_request_pipelines/pipelines_for_merged_results/index.md b/doc/ci/merge_request_pipelines/pipelines_for_merged_results/index.md index 2330bdb4c7c..9c6fbba1337 100644 --- a/doc/ci/merge_request_pipelines/pipelines_for_merged_results/index.md +++ b/doc/ci/merge_request_pipelines/pipelines_for_merged_results/index.md @@ -56,7 +56,7 @@ To enable pipelines for merged results for your project: 1. [Configure your CI/CD configuration file](../index.md#configuring-pipelines-for-merge-requests) so that the pipeline or individual jobs run for merge requests. 1. Visit your project's **Settings > General** and expand **Merge requests**. -1. Check **Enable merge trains and pipelines for merged results**. +1. Check **Enable merged results pipelines.**. 1. Click **Save changes**. CAUTION: **Caution:** diff --git a/doc/ci/merge_request_pipelines/pipelines_for_merged_results/merge_trains/img/merge_train_config_v12_0.png b/doc/ci/merge_request_pipelines/pipelines_for_merged_results/merge_trains/img/merge_train_config_v12_0.png Binary files differdeleted file mode 100644 index 9da959ad440..00000000000 --- a/doc/ci/merge_request_pipelines/pipelines_for_merged_results/merge_trains/img/merge_train_config_v12_0.png +++ /dev/null diff --git a/doc/ci/merge_request_pipelines/pipelines_for_merged_results/merge_trains/img/merge_train_immediate_merge.png b/doc/ci/merge_request_pipelines/pipelines_for_merged_results/merge_trains/img/merge_train_immediate_merge.png Binary files differdeleted file mode 100644 index 03bc61129ba..00000000000 --- a/doc/ci/merge_request_pipelines/pipelines_for_merged_results/merge_trains/img/merge_train_immediate_merge.png +++ /dev/null diff --git a/doc/ci/merge_request_pipelines/pipelines_for_merged_results/merge_trains/index.md b/doc/ci/merge_request_pipelines/pipelines_for_merged_results/merge_trains/index.md index 45cae49377f..d4099cdeafd 100644 --- a/doc/ci/merge_request_pipelines/pipelines_for_merged_results/merge_trains/index.md +++ b/doc/ci/merge_request_pipelines/pipelines_for_merged_results/merge_trains/index.md @@ -82,9 +82,13 @@ To enable merge trains for your project: 1. If you are on a self-managed GitLab instance, ensure the [feature flag](#merge-trains-feature-flag) is set correctly. 1. [Configure your CI/CD configuration file](../../index.md#configuring-pipelines-for-merge-requests) so that the pipeline or individual jobs run for merge requests. -1. Visit your project's **Settings > General** and expand **Merge requests**. -1. Check **Enable merge trains and pipelines for merged results**. -1. Click **Save changes**. +1. Visit your project's **Settings > General** and expand **Merge requests** +1. Check **Enable merged results pipelines.** (if not enabled) +1. Check **Enable merge trains.** +1. Click **Save changes** + +In GitLab 13.5 and earlier, there is only one checkbox, named +**Enable merge trains and pipelines for merged results**. CAUTION: **Caution:** If you select the check box but don't configure your CI/CD to use @@ -200,17 +204,26 @@ for more information. ### Merge Trains feature flag **(PREMIUM ONLY)** -To enable and disable the Merge Trains feature, use the `:disable_merge_trains` feature flag. +In [GitLab 13.6 and later](https://gitlab.com/gitlab-org/gitlab/-/issues/244831), +you can [enable or disable merge trains in the project settings](#enable-merge-trains). + +In GitLab 13.5 and earlier, merge trains are automatically enabled when +[pipelines for merged results](../index.md#pipelines-for-merged-results) are enabled. +To use pipelines for merged results without using merge trains, you can enable a +[feature flag](../../../../user/feature_flags.md) that blocks the merge trains feature. -To check if the feature flag is enabled on your GitLab instance, -ask an administrator to execute the following commands: +[GitLab administrators with access to the GitLab Rails console](../../../../administration/feature_flags.md) +can enable the feature flag to disable merge trains: -```shell -> sudo gitlab-rails console # Login to Rails console of GitLab instance. -> Feature.enabled?(:disable_merge_trains) # Check if it's disabled or not. -> Feature.enable(:disable_merge_trains) # Disable Merge Trains. -> Feature.disable(:disable_merge_trains) # Enable Merge Trains. +```ruby +Feature.enable(:disable_merge_trains) ``` -When you disable this feature, all existing merge trains are cancelled and +After you enable this feature flag, all existing merge trains are cancelled and the **Start/Add to Merge Train** button no longer appears in merge requests. + +To disable the feature flag, and enable merge trains again: + +```ruby +Feature.disable(:disable_merge_trains) +``` diff --git a/doc/ci/metrics_reports.md b/doc/ci/metrics_reports.md index 53e097760e6..dbc0397bb0b 100644 --- a/doc/ci/metrics_reports.md +++ b/doc/ci/metrics_reports.md @@ -17,7 +17,7 @@ You can configure your job to use custom Metrics Reports, and GitLab will displa ## Use cases -Consider the following examples of data that can utilize Metrics Reports: +Consider the following examples of data that can use Metrics Reports: 1. Memory usage 1. Load testing results diff --git a/doc/ci/migration/circleci.md b/doc/ci/migration/circleci.md index 6de494bceaf..13190c15cca 100644 --- a/doc/ci/migration/circleci.md +++ b/doc/ci/migration/circleci.md @@ -10,7 +10,7 @@ type: index, howto If you are currently using CircleCI, you can migrate your CI/CD pipelines to [GitLab CI/CD](../introduction/index.md), and start making use of all its powerful features. Check out our -[CircleCI vs GitLab](https://about.gitlab.com/devops-tools/circle-ci-vs-gitlab.html) +[CircleCI vs GitLab](https://about.gitlab.com/devops-tools/circle-ci-vs-gitlab/) comparison to see what's different. We have collected several resources that you may find useful before starting to migrate. @@ -27,7 +27,7 @@ CircleCI's `config.yml` configuration file defines scripts, jobs, and workflows ### Jobs -In CircleCI, jobs are a collection of steps to perform a specific task. In GitLab, [jobs](../yaml/README.md#introduction) are also a fundamental element in the configuration file. The `checkout` parameter is not necessary in GitLab CI/CD as the repository is automatically fetched. +In CircleCI, jobs are a collection of steps to perform a specific task. In GitLab, [jobs](../jobs/index.md) are also a fundamental element in the configuration file. The `checkout` keyword is not necessary in GitLab CI/CD as the repository is automatically fetched. CircleCI example job definition: diff --git a/doc/ci/migration/jenkins.md b/doc/ci/migration/jenkins.md index 1130c11f472..afec94ca91c 100644 --- a/doc/ci/migration/jenkins.md +++ b/doc/ci/migration/jenkins.md @@ -258,7 +258,7 @@ stages: ``` Setting a step to be performed before and after any job can be done via the -[`before_script` and `after_script` keywords](../yaml/README.md#before_script-and-after_script): +[`before_script`](../yaml/README.md#before_script) and [`after_script`](../yaml/README.md#after_script) keywords: ```yaml default: diff --git a/doc/ci/multi_project_pipelines.md b/doc/ci/multi_project_pipelines.md index 378adcd35e9..5837bf6cf6b 100644 --- a/doc/ci/multi_project_pipelines.md +++ b/doc/ci/multi_project_pipelines.md @@ -305,7 +305,7 @@ Some features are not implemented yet. For example, support for environments. - `when` (only with `on_success`, `on_failure`, and `always` values) - `extends` -## Trigger a pipeline when an upstream project is rebuilt +## Trigger a pipeline when an upstream project is rebuilt **(PREMIUM)** > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/9045) in [GitLab Premium](https://about.gitlab.com/pricing/) 12.8. @@ -322,6 +322,9 @@ will now trigger a pipeline on the current project's default branch. The maximum number of upstream pipeline subscriptions is 2 by default, for both the upstream and downstream projects. This [application limit](../administration/instance_limits.md#number-of-cicd-subscriptions-to-a-project) can be changed on self-managed instances by a GitLab administrator. +The upstream project needs to be [public](../public_access/public_access.md) for +pipeline subscription to work. + ## Downstream private projects confidentiality concern If you trigger a pipeline in a downstream private project, the name of the project diff --git a/doc/ci/pipelines/img/manual_job_variables.png b/doc/ci/pipelines/img/manual_job_variables.png Binary files differdeleted file mode 100644 index a5ed351fdcd..00000000000 --- a/doc/ci/pipelines/img/manual_job_variables.png +++ /dev/null diff --git a/doc/ci/pipelines/index.md b/doc/ci/pipelines/index.md index f7e3698b6d4..f22d2373e5f 100644 --- a/doc/ci/pipelines/index.md +++ b/doc/ci/pipelines/index.md @@ -68,7 +68,7 @@ Pipelines can be configured in many different ways: Pipelines and their component jobs and stages are defined in the CI/CD pipeline configuration file for each project. -- Jobs are the [basic configuration](../yaml/README.md#introduction) component. +- [Jobs](../jobs/index.md) are the basic configuration component. - Stages are defined by using the [`stages`](../yaml/README.md#stages) keyword. For a list of configuration options in the CI pipeline file, see the [GitLab CI/CD Pipeline Configuration Reference](../yaml/README.md). @@ -79,6 +79,27 @@ You can also configure specific aspects of your pipelines through the GitLab UI. - [Pipeline schedules](schedules.md). - [Custom CI/CD variables](../variables/README.md#custom-environment-variables). +### Ref Specs for Runners + +When a runner picks a pipeline job, GitLab provides that job's metadata. This includes the [Git refspecs](https://git-scm.com/book/en/v2/Git-Internals-The-Refspec), +which indicate which ref (branch, tag, and so on) and commit (SHA1) are checked out from your +project repository. + +This table lists the refspecs injected for each pipeline type: + +| Pipeline type | Refspecs | +|--------------- |---------------------------------------- | +| Pipeline for Branches | `+refs/pipelines/<id>:refs/pipelines/<id>` and `+refs/heads/<name>:refs/remotes/origin/<name>` | +| pipeline for Tags | `+refs/pipelines/<id>:refs/pipelines/<id>` and `+refs/tags/<name>:refs/tags/<name>` | +| [Pipeline for Merge Requests](../merge_request_pipelines/index.md) | `+refs/pipelines/<id>:refs/pipelines/<id>` | + +The refs `refs/heads/<name>` and `refs/tags/<name>` exist in your +project repository. GitLab generates the special ref `refs/pipelines/<id>` during a +running pipeline job. This ref can be created even after the associated branch or tag has been +deleted. It's therefore useful in some features such as [automatically stopping an environment](../environments/index.md#automatically-stopping-an-environment), +and [merge trains](../merge_request_pipelines/pipelines_for_merged_results/merge_trains/index.md) +that might run pipelines after branch deletion. + ### View pipelines You can find the current and historical pipeline runs under your project's @@ -157,7 +178,7 @@ For each `var` or `file_var`, a key and value are required. > [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/7931) in GitLab 8.15. -Manual actions, configured using the [`when:manual`](../yaml/README.md#whenmanual) parameter, +Manual actions, configured using the [`when:manual`](../yaml/README.md#whenmanual) keyword, allow you to require manual interaction before moving forward in the pipeline. You can do this straight from the pipeline graph. Just click the play button @@ -174,7 +195,7 @@ stage has a job with a manual action. > [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/27188) in GitLab 11.11. Multiple manual actions in a single stage can be started at the same time using the "Play all manual" button. -Once you click this button, each individual manual action is triggered and refreshed +After you click this button, each individual manual action is triggered and refreshed to an updated status. This functionality is only available: @@ -266,223 +287,6 @@ preserving deployment keys and other credentials from being unintentionally accessed. In order to ensure that jobs intended to be executed on protected runners do not use regular runners, they must be tagged accordingly. -## View jobs in a pipeline - -When you access a pipeline, you can see the related jobs for that pipeline. - -Clicking an individual job shows you its job log, and allows you to: - -- Cancel the job. -- Retry the job. -- Erase the job log. - -### See why a job failed - -> [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/17782) in GitLab 10.7. - -When a pipeline fails or is allowed to fail, there are several places where you -can find the reason: - -- In the [pipeline graph](#visualize-pipelines), on the pipeline detail view. -- In the pipeline widgets, in the merge requests and commit pages. -- In the job views, in the global and detailed views of a job. - -In each place, if you hover over the failed job you can see the reason it failed. - - - -In [GitLab 10.8](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/17814) and later, -you can also see the reason it failed on the Job detail page. - -### The order of jobs in a pipeline - -The order of jobs in a pipeline depends on the type of pipeline graph. - -- For [regular pipeline graphs](#regular-pipeline-graphs), jobs are sorted by name. -- For [pipeline mini graphs](#pipeline-mini-graphs), jobs are sorted by severity and then by name. - -The order of severity is: - -- failed -- warning -- pending -- running -- manual -- scheduled -- canceled -- success -- skipped -- created - -For example: - - - -### Group jobs in a pipeline - -> [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/6242) in GitLab 8.12. - -If you have many similar jobs, your [pipeline graph](#visualize-pipelines) becomes long and hard -to read. - -You can automatically group similar jobs together. If the job names are formatted in a certain way, -they are collapsed into a single group in regular pipeline graphs (not the mini graphs). - -You can recognize when a pipeline has grouped jobs if you don't see the retry or -cancel button inside them. Hovering over them shows the number of grouped -jobs. Click to expand them. - - - -To create a group of jobs, in the [CI/CD pipeline configuration file](../yaml/README.md), -separate each job name with a number and one of the following: - -- A slash (`/`), for example, `test 1/3`, `test 2/3`, `test 3/3`. -- A colon (`:`), for example, `test 1:3`, `test 2:3`, `test 3:3`. -- A space, for example `test 0 3`, `test 1 3`, `test 2 3`. - -You can use these symbols interchangeably. - -In the example below, these three jobs are in a group named `build ruby`: - -```yaml -build ruby 1/3: - stage: build - script: - - echo "ruby1" - -build ruby 2/3: - stage: build - script: - - echo "ruby2" - -build ruby 3/3: - stage: build - script: - - echo "ruby3" -``` - -In the pipeline, the result is a group named `build ruby` with three jobs: - - - -The jobs are be ordered by comparing the numbers from left to right. You -usually want the first number to be the index and the second number to be the total. - -[This regular expression](https://gitlab.com/gitlab-org/gitlab/blob/2f3dc314f42dbd79813e6251792853bc231e69dd/app/models/commit_status.rb#L99) -evaluates the job names: `\d+[\s:\/\\]+\d+\s*`. - -### Specifying variables when running manual jobs - -> [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/30485) in GitLab 12.2. - -When running manual jobs you can supply additional job specific variables. - -You can do this from the job page of the manual job you want to run with -additional variables. To access this page, click on the **name** of the manual job in -the pipeline view, *not* the play (**{play}**) button. - -This is useful when you want to alter the execution of a job that uses -[custom environment variables](../variables/README.md#custom-environment-variables). -Add a variable name (key) and value here to override the value defined in -[the UI or `.gitlab-ci.yml`](../variables/README.md#custom-environment-variables), -for a single run of the manual job. - - - -### Delay a job - -> [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/21767) in GitLab 11.4. - -When you do not want to run a job immediately, you can use the [`when:delayed`](../yaml/README.md#whendelayed) parameter to -delay a job's execution for a certain period. - -This is especially useful for timed incremental rollout where new code is rolled out gradually. - -For example, if you start rolling out new code and: - -- Users do not experience trouble, GitLab can automatically complete the deployment from 0% to 100%. -- Users experience trouble with the new code, you can stop the timed incremental rollout by canceling the pipeline - and [rolling](../environments/index.md#retrying-and-rolling-back) back to the last stable version. - - - -### Expand and collapse job log sections - -> [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/14664) in GitLab 12.0. - -Job logs are divided into sections that can be collapsed or expanded. Each section displays -the duration. - -In the following example: - -- Two sections are collapsed and can be expanded. -- Three sections are expanded and can be collapsed. - - - -#### Custom collapsible sections - -> [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/14664) in GitLab 12.0. - -You can create [collapsible sections in job logs](../pipelines/index.md#expand-and-collapse-job-log-sections) -by manually outputting special codes -that GitLab uses to determine what sections to collapse: - -- Section start marker: `section_start:UNIX_TIMESTAMP:SECTION_NAME\r\e[0K` + `TEXT_OF_SECTION_HEADER` -- Section end marker: `section_end:UNIX_TIMESTAMP:SECTION_NAME\r\e[0K` - -You must add these codes to the script section of the CI configuration. For example, -using `echo`: - -```yaml -job1: - script: - - echo -e "section_start:`date +%s`:my_first_section\r\e[0KHeader of the 1st collapsible section" - - echo 'this line should be hidden when collapsed' - - echo -e "section_end:`date +%s`:my_first_section\r\e[0K" -``` - -In the example above: - -- `date +%s`: The Unix timestamp (for example `1560896352`). -- `my_first_section`: The name given to the section. -- `\r\e[0K`: Prevents the section markers from displaying in the rendered (colored) - job log, but they are displayed in the raw job log. To see them, in the top right - of the job log, click **{doc-text}** (**Show complete raw**). - - `\r`: carriage return. - - `\e[0K`: clear line ANSI escape code. - -Sample raw job log: - -```plaintext -section_start:1560896352:my_first_section\r\e[0KHeader of the 1st collapsible section -this line should be hidden when collapsed -section_end:1560896353:my_first_section\r\e[0K -``` - -#### Pre-collapse sections - -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/198413) in GitLab 13.5. - -You can make the job log automatically collapse collapsible sections by adding the `collapsed` option to the section start. -Add `[collapsed=true]` after the section name and before the `\r`. The section end marker -remains unchanged: - -- Section start marker with `[collapsed=true]`: `section_start:UNIX_TIMESTAMP:SECTION_NAME[collapsed=true]\r\e[0K` + `TEXT_OF_SECTION_HEADER` -- Section end marker: `section_end:UNIX_TIMESTAMP:SECTION_NAME\r\e[0K` - -Add the updated section start text to the CI configuration. For example, -using `echo`: - -```yaml -job1: - script: - - echo -e "section_start:`date +%s`:my_first_section[collapsed=true]\r\e[0KHeader of the 1st collapsible section" - - echo 'this line should be hidden automatically after loading the job log' - - echo -e "section_end:`date +%s`:my_first_section\r\e[0K" -``` - ## Visualize pipelines > [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/5742) in GitLab 8.11. diff --git a/doc/ci/pipelines/job_artifacts.md b/doc/ci/pipelines/job_artifacts.md index d904452a011..160399e4bc4 100644 --- a/doc/ci/pipelines/job_artifacts.md +++ b/doc/ci/pipelines/job_artifacts.md @@ -44,7 +44,7 @@ are relative to the repository that was cloned during the build. By default, the artifacts upload when the job succeeds. You can also set artifacts to upload when the job fails, or always, by using [`artifacts:when`](../yaml/README.md#artifactswhen) -parameter. GitLab keeps these uploaded artifacts for 1 week, as defined +keyword. GitLab keeps these uploaded artifacts for 1 week, as defined by the `expire_in` definition. You can keep the artifacts from expiring via the [web interface](#browsing-artifacts). If the expiry time is not defined, it defaults to the [instance wide setting](../../user/admin_area/settings/continuous_integration.md#default-artifacts-expiration). @@ -114,7 +114,9 @@ There are a couple of exceptions to the [original dotenv rules](https://github.c - The variable key can contain only letters, digits, and underscores (`_`). - The maximum size of the `.env` file is 5 KB. -- The maximum number of variables is 10. +- In GitLab 13.5 and older, the maximum number of inherited variables is 10. +- In [GitLab 13.6 and later](https://gitlab.com/gitlab-org/gitlab/-/issues/247913), + the maximum number of inherited variables is 20. - Variable substitution in the `.env` file is not supported. - The `.env` file can't have empty lines or comments (starting with `#`). - Key values in the `env` file cannot have spaces or newline characters (`\n`), including when using single or double quotes. @@ -137,10 +139,10 @@ third party ports for other languages like JavaScript, Python, Ruby, and so on. > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/207528) in GitLab 13.0. > - Requires [GitLab Runner](https://docs.gitlab.com/runner/) 11.5 and above. -The `terraform` report obtains a Terraform `tfplan.json` file. [JQ processing required to remove credentials](../../user/infrastructure/index.md#output-terraform-plan-information-into-a-merge-request). The collected Terraform +The `terraform` report obtains a Terraform `tfplan.json` file. [JQ processing required to remove credentials](../../user/infrastructure/mr_integration.md#setup). The collected Terraform plan report uploads to GitLab as an artifact and displays in merge requests. For more information, see -[Output `terraform plan` information into a merge request](../../user/infrastructure/index.md#output-terraform-plan-information-into-a-merge-request). +[Output `terraform plan` information into a merge request](../../user/infrastructure/mr_integration.md). #### `artifacts:reports:codequality` @@ -414,7 +416,7 @@ information in the UI. ## Erasing artifacts -DANGER: **Danger:** +DANGER: **Warning:** This is a destructive action that leads to data loss. Use with caution. You can erase a single job via the UI, which also removes the job's diff --git a/doc/ci/pipelines/pipeline_efficiency.md b/doc/ci/pipelines/pipeline_efficiency.md index c4febba8f44..149c596430c 100644 --- a/doc/ci/pipelines/pipeline_efficiency.md +++ b/doc/ci/pipelines/pipeline_efficiency.md @@ -222,6 +222,8 @@ Methods to reduce Docker image size: - Create a dedicated development image. - Disable man pages and docs installed by packages to save space. - Reduce the `RUN` layers and combine software installation steps. +- Use [multi-stage builds](https://blog.alexellis.io/mutli-stage-docker-builds/) + to merge multiple Dockerfiles that use the builder pattern into one Dockerfile, which can reduce image size. - If using `apt`, add `--no-install-recommends` to avoid unnecessary packages. - Clean up caches and files that are no longer needed at the end. For example `rm -rf /var/lib/apt/lists/*` for Debian and Ubuntu, or `yum clean all` for RHEL and CentOS. diff --git a/doc/ci/pipelines/schedules.md b/doc/ci/pipelines/schedules.md index bcdb7c4c8b6..9df73957293 100644 --- a/doc/ci/pipelines/schedules.md +++ b/doc/ci/pipelines/schedules.md @@ -85,10 +85,10 @@ job: ### Advanced configuration -The pipelines won't be executed exactly on schedule because schedules are handled by +The pipelines are not executed exactly on schedule because schedules are handled by Sidekiq, which runs according to its interval. -For example, only two pipelines will be created per day if: +For example, only two pipelines are created per day if: - You set a schedule to create a pipeline every minute (`* * * * *`). - The Sidekiq worker runs on 00:00 and 12:00 every day (`0 */12 * * *`). @@ -112,8 +112,8 @@ To trigger a pipeline schedule manually, click the "Play" button:  -This will schedule a background job to run the pipeline schedule. A flash -message will provide a link to the CI/CD Pipeline index page. +This schedules a background job to run the pipeline schedule. A flash +message provides a link to the CI/CD Pipeline index page. NOTE: **Note:** To help avoid abuse, users are rate limited to triggering a pipeline once per @@ -124,12 +124,12 @@ minute. Pipelines are executed as a user, who owns a schedule. This influences what projects and other resources the pipeline has access to. If a user does not own a pipeline, you can take ownership by clicking the **Take ownership** button. -The next time a pipeline is scheduled, your credentials will be used. +The next time a pipeline is scheduled, your credentials are used.  -If the owner of a pipeline schedule doesn't have the ability to create -pipelines on the target branch, the schedule will stop creating new +If the owner of a pipeline schedule does not have the ability to create +pipelines on the target branch, the schedule stops creating new pipelines. This can happen if, for example: diff --git a/doc/ci/pipelines/settings.md b/doc/ci/pipelines/settings.md index 143a5346e88..d2d2cb26256 100644 --- a/doc/ci/pipelines/settings.md +++ b/doc/ci/pipelines/settings.md @@ -26,10 +26,11 @@ There are two options. Using: - `git clone`, which is slower since it clones the repository from scratch for every job, ensuring that the local working copy is always pristine. -- `git fetch`, which is faster as it re-uses the local working copy (falling +- `git fetch`, which is GitLab's default and faster as it re-uses the local working copy (falling back to clone if it doesn't exist). + This is recommended, especially for [large repositories](../large_repositories/index.md#git-strategy). -The default Git strategy can be overridden by the [GIT_STRATEGY variable](../yaml/README.md#git-strategy) +The configured Git strategy can be overridden by the [`GIT_STRATEGY` variable](../runners/README.md#git-strategy) in `.gitlab-ci.yml`. ## Git shallow clone @@ -183,7 +184,7 @@ Job logs and artifacts are [not visible for guest users and non-project members] If **Public pipelines** is enabled (default): - For **public** projects, anyone can view the pipelines and related features. -- For **internal** projects, any logged in user can view the pipelines +- For **internal** projects, any logged in user except [external users](../../user/permissions.md#external-users) can view the pipelines and related features. - For **private** projects, any project member (guest or higher) can view the pipelines and related features. @@ -192,7 +193,7 @@ If **Public pipelines** is disabled: - For **public** projects, anyone can view the pipelines, but only members (reporter or higher) can access the related features. -- For **internal** projects, any logged in user can view the pipelines. +- For **internal** projects, any logged in user except [external users](../../user/permissions.md#external-users) can view the pipelines. However, only members (reporter or higher) can access the job related features. - For **private** projects, only project members (reporter or higher) can view the pipelines or access the related features. @@ -231,6 +232,16 @@ When enabled, any older deployments job are skipped when a new deployment starts For more information, see [Deployment safety](../environments/deployment_safety.md). +## Retry outdated jobs + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/211339) in GitLab 13.6. + +A deployment job can fail because a newer one has run. If you retry the failed deployment job, the +environment could be overwritten with older source code. If you click **Retry**, a modal warns you +about this and asks for confirmation. + +For more information, see [Deployment safety](../environments/deployment_safety.md). + ## Pipeline Badges In the pipelines settings page you can find pipeline status and test coverage diff --git a/doc/ci/quick_start/README.md b/doc/ci/quick_start/README.md index 246430a6458..f3e60fae13a 100644 --- a/doc/ci/quick_start/README.md +++ b/doc/ci/quick_start/README.md @@ -5,224 +5,153 @@ info: To determine the technical writer assigned to the Stage/Group associated w type: reference --- -# Getting started with GitLab CI/CD +# Get started with GitLab CI/CD -GitLab offers a [continuous integration](https://about.gitlab.com/stages-devops-lifecycle/continuous-integration/) service. For each commit or push to trigger your CI -[pipeline](../pipelines/index.md), you must: +Use this document to get started with +GitLab [continuous integration](https://about.gitlab.com/stages-devops-lifecycle/continuous-integration/). -- Add a [`.gitlab-ci.yml` file](#creating-a-gitlab-ciyml-file) to your repository's root directory. -- Ensure your project is configured to use a [runner](#configuring-a-runner). +Before you start, make sure you have: -The `.gitlab-ci.yml` file tells the runner what to do. A simple pipeline commonly has -three [stages](../yaml/README.md#stages): +- A project in GitLab that you would like to use CI/CD for. +- Maintainer or owner access for the project. -- `build` -- `test` -- `deploy` +If you are migrating from another CI/CD tool, view this documentation: -You do not need to use all three stages; stages with no jobs are ignored. +- [Migrate from CircleCI](../migration/circleci.md). +- [Migrate from Jenkins](../migration/jenkins.md). -The pipeline appears under the project's **CI/CD > Pipelines** page. If everything runs OK (no non-zero -return values), you get a green check mark associated with the commit. This makes it easy to see -whether a commit caused any of the tests to fail before you even look at the job (test) log. Many projects use -GitLab's CI service to run the test suite, so developers get immediate feedback if they broke -something. +## CI/CD process overview -It's also common to use pipelines to automatically deploy -tested code to staging and production environments. +To use GitLab CI/CD: -If you're already familiar with general CI/CD concepts, you can review which -[pipeline architectures](../pipelines/pipeline_architectures.md) can be used -in your projects. If you're coming over to GitLab from Jenkins, you can check out -our [reference](../migration/jenkins.md) for converting your pre-existing pipelines -over to our format. +1. [Ensure you have runners available](#ensure-you-have-runners-available) to run your jobs. + If you don't have a runner, [install GitLab Runner](https://docs.gitlab.com/runner/install/) + and [register a runner](https://docs.gitlab.com/runner/register/) for your instance, project, or group. +1. [Create a `.gitlab-ci.yml` file](#create-a-gitlab-ciyml-file) + at the root of your repository. This file is where you define your CI/CD jobs. -This guide assumes that you have: +When you commit the file to your repository, the runner runs your jobs. +The job results [are displayed in a pipeline](#view-the-status-of-your-pipeline-and-jobs). -- A working GitLab instance of version 8.0+ or are using - [GitLab.com](https://gitlab.com). -- A project in GitLab that you would like to use CI for. -- Maintainer or owner access to the project +### Ensure you have runners available -Let's break it down to pieces and work on solving the GitLab CI/CD puzzle. +In GitLab, runners are agents that run your CI/CD jobs. -## Creating a `.gitlab-ci.yml` file +You might already have runners available for your project, including +[shared runners](../runners/README.md#shared-runners), which are +available to all projects in your GitLab instance. -Before you create `.gitlab-ci.yml` let's first explain in brief what this is -all about. +To view available runners: -### What is `.gitlab-ci.yml` +- Go to **Settings > CI/CD** and expand **Runners**. -The `.gitlab-ci.yml` file is where you configure what CI does with your project. -It lives in the root of your repository. +As long as you have at least one runner that's active, with a green circle next to it, +you have a runner available to process your jobs. -On any push to your repository, GitLab will look for the `.gitlab-ci.yml` -file and start jobs on _runners_ according to the contents of the file, -for that commit. +If no runners are listed on the **Runners** page in the UI, you or an administrator +must [install GitLab Runner](https://docs.gitlab.com/runner/install/) and +[register](https://docs.gitlab.com/runner/register/) at least one runner. -Because `.gitlab-ci.yml` is in the repository and is version controlled, old -versions still build successfully, forks can easily make use of CI, branches can -have different pipelines and jobs, and you have a single source of truth for CI. -You can read more about the reasons why we are using `.gitlab-ci.yml` [in our -blog about it](https://about.gitlab.com/blog/2015/05/06/why-were-replacing-gitlab-ci-jobs-with-gitlab-ci-dot-yml/). +If you are testing CI/CD, you can install GitLab Runner and register runners on your local machine. +When your CI/CD jobs run, they will run on your local machine. -### Creating a simple `.gitlab-ci.yml` file +### Create a `.gitlab-ci.yml` file -You need to create a file named `.gitlab-ci.yml` in the root directory of your -repository. This is a [YAML](https://en.wikipedia.org/wiki/YAML) file -so you have to pay extra attention to indentation. Always use spaces, not tabs. +The `.gitlab-ci.yml` file is a [YAML](https://en.wikipedia.org/wiki/YAML) file where +you configure specific instructions for GitLab CI/CD. -Below is an example for a Ruby on Rails project: +In this file, you define: -```yaml -default: - image: ruby:2.5 - before_script: - - apt-get update - - apt-get install -y sqlite3 libsqlite3-dev nodejs - - ruby -v - - which ruby - - gem install bundler --no-document - - bundle install --jobs $(nproc) "${FLAGS[@]}" +- The structure and order of jobs that the runner should execute. +- The decisions the runner should make when specific conditions are encountered. -rspec: - script: - - bundle exec rspec +For example, you might want to run a suite of tests when you commit to +any branch except `master`. When you commit to `master`, you want +to run the same suite, but also publish your application. -rubocop: - script: - - bundle exec rubocop -``` +All of this is defined in the `.gitlab-ci.yml` file. -This is the simplest possible configuration that will work for most Ruby -applications: +To create a `.gitlab-ci.yml` file: -1. Define two jobs `rspec` and `rubocop` (the names are arbitrary) with - different commands to be executed. -1. Before every job, the commands defined by `before_script` are executed. +1. Go to **Project overview > Details**. +1. Above the file list, select the branch you want to commit to, + click the plus icon, then select **New file**: -The `.gitlab-ci.yml` file defines sets of jobs with constraints of how and when -they should be run. The jobs are defined as top-level elements with a name (in -our case `rspec` and `rubocop`) and always have to contain the `script` keyword. -Jobs are used to create jobs, which are then picked by -[runners](../runners/README.md) and executed within the environment of the runner. +  -What is important is that each job is run independently from each other. +1. For the **File name** type `.gitlab-ci.yml` and in the larger window, + paste this sample code: -If you want to check whether the `.gitlab-ci.yml` of your project is valid, there is a -[CI Lint tool](../lint.md) available in every project. + ```yaml + build-job: + stage: build + script: + - echo "Hello, $GITLAB_USER_LOGIN!" -You can use the [CI/CD configuration visualization](../yaml/visualization.md) to -see a graphical representation of your `.gitlab-ci.yml`. + test-job1: + stage: test + script: + - echo "This job tests something" -For more information and a complete `.gitlab-ci.yml` syntax, please read -[the reference documentation on `.gitlab-ci.yml`](../yaml/README.md). + test-job2: + stage: test + script: + - echo "This job tests something, but takes more time than test-job1." + - echo "After the echo commands complete, it runs the sleep command for 20 seconds" + - echo "which simulates a test that runs 20 seconds longer than test-job1" + - sleep 20 -TIP: **Tip:** -A GitLab team member has made an [unofficial visual pipeline editor](https://unofficial.gitlab.tools/visual-pipelines/). -There is a [plan to make it an official part of GitLab](https://gitlab.com/groups/gitlab-org/-/epics/4069) -in the future, but it's available for anyone who wants to try it at the above link. + deploy-prod: + stage: deploy + script: + - echo "This job deploys something from the $CI_COMMIT_BRANCH branch." + ``` -### Push `.gitlab-ci.yml` to GitLab + `$GITLAB_USER_LOGIN` and `$CI_COMMIT_BRANCH` are + [predefined variables](../variables/predefined_variables.md) + that populate when the job runs. -Once you've created `.gitlab-ci.yml`, you should add it to your Git repository -and push it to GitLab. +1. Click **Commit changes**. -```shell -git add .gitlab-ci.yml -git commit -m "Add .gitlab-ci.yml" -git push origin master -``` +The pipeline starts when the commit is committed. -Now if you go to the **Pipelines** page you will see that the pipeline is -pending. +#### `.gitlab-ci.yml` tips -NOTE: **Note:** -If you have a [mirrored repository where GitLab pulls from](../../user/project/repository/repository_mirroring.md#pulling-from-a-remote-repository), -you may need to enable pipeline triggering in your project's -**Settings > Repository > Pull from a remote repository > Trigger pipelines for mirror updates**. +- If you want the runner to use a Docker image to run the jobs, edit the `.gitlab-ci.yml` file + to include your image name: -You can also go to the **Commits** page and notice the little pause icon next -to the commit SHA. + ```yaml + default: + image: ruby:2.7.2 + ``` - + This command tells the runner to use a Ruby image from Docker Hub. -Clicking on it you will be directed to the jobs page for that specific commit. +- To validate your `.gitlab-ci.yml` file, use the + [CI Lint tool](../lint.md), which is available in every project. +- You can also use [CI/CD configuration visualization](../yaml/visualization.md) to + view a graphical representation of your `.gitlab-ci.yml` file. +- For the complete `.gitlab-ci.yml` syntax, see + [the `.gitlab-ci.yml` reference topic](../yaml/README.md). - +### View the status of your pipeline and jobs -Notice that there is a pending job which is named after what we wrote in -`.gitlab-ci.yml`. "stuck" indicates that there is no runner configured -yet for this job. +When you committed your changes, a pipeline started. -The next step is to configure a runner so that it picks the pending jobs. +To view your pipeline: -## Configuring a runner +- Go **CI/CD > Pipelines**. -In GitLab, runners run the jobs that you define in `.gitlab-ci.yml`. A runner -can be a virtual machine, a VPS, a bare-metal machine, a Docker container, or -even a cluster of containers. GitLab and the runner communicate through an API, -so the only requirement is that the runner's machine has network access to the -GitLab server. + A pipeline with three stages should be displayed: -A runner can be specific to a certain project or serve multiple projects in -GitLab. If it serves all projects, it's called a _shared runner_. +  -Find more information about runners in the -[runner](../runners/README.md) documentation. +- To view a visual representation of your pipeline, click the pipeline ID. -The official runner supported by GitLab is written in Go. -View [the documentation](https://docs.gitlab.com/runner/). +  -For a runner to be available in GitLab, you must: +- To view details of a job, click the job name, for example, `deploy-prod`. -1. [Install GitLab Runner](https://docs.gitlab.com/runner/install/). -1. [Register a runner for your group or project](https://docs.gitlab.com/runner/register/). +  -When a runner is available, you can view it by -clicking **Settings > CI/CD** and expanding **Runners**. - - - -### Shared runners - -If you use [GitLab.com](https://gitlab.com/), you can use the **shared runners** -provided by GitLab. - -These are special virtual machines that run on GitLab's infrastructure and can -build any project. - -To enable shared runners, go to your project's or group's -**Settings > CI/CD** and click **Enable shared runners**. - -[Read more about shared runners](../runners/README.md#shared-runners). - -## Viewing the status of your pipeline and jobs - -After configuring the runner successfully, you should see the status of your -last commit change from _pending_ to either _running_, _success_ or _failed_. - -You can view all pipelines by going to the **Pipelines** page in your project. - - - -Or you can view all jobs, by going to the **Pipelines ➔ Jobs** page. - - - -By clicking on a job's status, you will be able to see the log of that job. -This is important to diagnose why a job failed or acted differently than -you expected. - - - -You are also able to view the status of any commit in the various pages in -GitLab, such as **Commits** and **Merge requests**. - -## Additional resources - -Visit the [examples README](../examples/README.md) to see a list of examples using GitLab -CI with various languages. - -For help making your new pipelines faster and more efficient, see the -[pipeline efficiency documentation](../pipelines/pipeline_efficiency.md). +If the job status is `stuck`, check to ensure a runner is probably configured for the project. diff --git a/doc/ci/quick_start/img/build_log.png b/doc/ci/quick_start/img/build_log.png Binary files differdeleted file mode 100644 index 16698629edc..00000000000 --- a/doc/ci/quick_start/img/build_log.png +++ /dev/null diff --git a/doc/ci/quick_start/img/builds_status.png b/doc/ci/quick_start/img/builds_status.png Binary files differdeleted file mode 100644 index b4aeeb988d2..00000000000 --- a/doc/ci/quick_start/img/builds_status.png +++ /dev/null diff --git a/doc/ci/quick_start/img/job_details_v13_6.png b/doc/ci/quick_start/img/job_details_v13_6.png Binary files differnew file mode 100644 index 00000000000..e94287f90ba --- /dev/null +++ b/doc/ci/quick_start/img/job_details_v13_6.png diff --git a/doc/ci/quick_start/img/new_commit.png b/doc/ci/quick_start/img/new_commit.png Binary files differdeleted file mode 100644 index 507eb93ac0c..00000000000 --- a/doc/ci/quick_start/img/new_commit.png +++ /dev/null diff --git a/doc/ci/quick_start/img/new_file_v13_6.png b/doc/ci/quick_start/img/new_file_v13_6.png Binary files differnew file mode 100644 index 00000000000..c71923f9b90 --- /dev/null +++ b/doc/ci/quick_start/img/new_file_v13_6.png diff --git a/doc/ci/quick_start/img/pipeline_graph_v13_6.png b/doc/ci/quick_start/img/pipeline_graph_v13_6.png Binary files differnew file mode 100644 index 00000000000..fcf7e02d1f3 --- /dev/null +++ b/doc/ci/quick_start/img/pipeline_graph_v13_6.png diff --git a/doc/ci/quick_start/img/pipelines_status.png b/doc/ci/quick_start/img/pipelines_status.png Binary files differdeleted file mode 100644 index 39a77a26b25..00000000000 --- a/doc/ci/quick_start/img/pipelines_status.png +++ /dev/null diff --git a/doc/ci/quick_start/img/single_commit_status_pending.png b/doc/ci/quick_start/img/single_commit_status_pending.png Binary files differdeleted file mode 100644 index ffc7054d3b0..00000000000 --- a/doc/ci/quick_start/img/single_commit_status_pending.png +++ /dev/null diff --git a/doc/ci/quick_start/img/three_stages_v13_6.png b/doc/ci/quick_start/img/three_stages_v13_6.png Binary files differnew file mode 100644 index 00000000000..a6c049e3e6c --- /dev/null +++ b/doc/ci/quick_start/img/three_stages_v13_6.png diff --git a/doc/ci/review_apps/img/view_on_env_mr.png b/doc/ci/review_apps/img/view_on_env_mr.png Binary files differindex 2c0bd25a4f2..0e61814a65d 100644 --- a/doc/ci/review_apps/img/view_on_env_mr.png +++ b/doc/ci/review_apps/img/view_on_env_mr.png diff --git a/doc/ci/review_apps/index.md b/doc/ci/review_apps/index.md index 7110117709f..ba23eff0434 100644 --- a/doc/ci/review_apps/index.md +++ b/doc/ci/review_apps/index.md @@ -27,7 +27,7 @@ In the above example: - A Review App is built every time a commit is pushed to `topic branch`. - The reviewer fails two reviews before passing the third review. -- Once the review has passed, `topic branch` is merged into `master` where it is deployed to staging. +- After the review has passed, `topic branch` is merged into `master` where it is deployed to staging. - After having been approved in staging, the changes that were merged into `master` are deployed in to production. ## How Review Apps work @@ -169,7 +169,7 @@ will match `/source\/(.+?\.html).*/` instead of `/source\/(.*)/`, and will result in a public path of `index.html`, instead of `index.html.haml`. -Once you have the route mapping set up, it will take effect in the following locations: +After you have the route mapping set up, it will take effect in the following locations: - In the merge request widget. The: - **View app** button will take you to the environment URL set in `.gitlab-ci.yml`. diff --git a/doc/ci/runners/README.md b/doc/ci/runners/README.md index a3cc46f59bf..4392fa3c78b 100644 --- a/doc/ci/runners/README.md +++ b/doc/ci/runners/README.md @@ -6,8 +6,6 @@ type: reference --- # Configuring runners in GitLab -<!-- This topic contains several commented-out sections that were accidentally added in 13.2.--> -<!-- The commented-out sections will be added back in a future release.--> In GitLab CI/CD, runners run the code defined in [`.gitlab-ci.yml`](../yaml/README.md). A runner is a lightweight, highly-scalable agent that picks up a CI job through @@ -16,6 +14,10 @@ the coordinator API of GitLab CI/CD, runs the job, and sends the result back to Runners are created by an administrator and are visible in the GitLab UI. Runners can be specific to certain projects or available to all projects. +This documentation is focused on using runners in GitLab. +If you need to install and configure GitLab Runner, see +[the GitLab Runner documentation](https://docs.gitlab.com/runner/). + ## Types of runners In the GitLab UI there are three types of runners, based on who you want to have access: @@ -36,7 +38,7 @@ multiple projects. If you are using a self-managed instance of GitLab: - Your administrator can install and register shared runners by [following the documentation](https://docs.gitlab.com/runner/install/index.html). - <!-- going to your project's + <!-- going to your project's--> <!-- **Settings > CI / CD**, expanding the **Runners** section, and clicking **Show runner installation instructions**.--> <!-- These instructions are also available [in the documentation](https://docs.gitlab.com/runner/install/index.html).--> - The administrator can also configure a maximum number of shared runner [pipeline minutes for @@ -390,6 +392,9 @@ You must set up a runner to be able to run all the different types of jobs that it may encounter on the projects it's shared over. This would be problematic for large amounts of projects, if it weren't for tags. +GitLab CI tags are not the same as Git tags. GitLab CI tags are associated with runners. +Git tags are associated with commits. + By tagging a runner for the types of jobs it can handle, you can make sure shared runners will [only run the jobs they are equipped to run](../yaml/README.md#tags). @@ -450,6 +455,339 @@ Example 2: 1. A job that has no tags defined is executed and run. 1. A second job that has a `docker` tag defined is stuck. +## Configure runner behavior with variables + +You can use [CI/CD variables](../variables/README.md) to configure runner Git behavior +globally or for individual jobs: + +- [`GIT_STRATEGY`](#git-strategy) +- [`GIT_SUBMODULE_STRATEGY`](#git-submodule-strategy) +- [`GIT_CHECKOUT`](#git-checkout) +- [`GIT_CLEAN_FLAGS`](#git-clean-flags) +- [`GIT_FETCH_EXTRA_FLAGS`](#git-fetch-extra-flags) +- [`GIT_DEPTH`](#shallow-cloning) (shallow cloning) +- [`GIT_CLONE_PATH`](#custom-build-directories) (custom build directories) + +You can also use variables to configure how many times a runner +[attempts certain stages of job execution](#job-stages-attempts). + +### Git strategy + +> - Introduced in GitLab 8.9 as an experimental feature. +> - `GIT_STRATEGY=none` requires GitLab Runner v1.7+. + +You can set the `GIT_STRATEGY` used to fetch the repository content, either +globally or per-job in the [`variables`](../yaml/README.md#variables) section: + +```yaml +variables: + GIT_STRATEGY: clone +``` + +There are three possible values: `clone`, `fetch`, and `none`. If left unspecified, +jobs use the [project's pipeline setting](../pipelines/settings.md#git-strategy). + +`clone` is the slowest option. It clones the repository from scratch for every +job, ensuring that the local working copy is always pristine. +If an existing worktree is found, it is removed before cloning. + +`fetch` is faster as it re-uses the local working copy (falling back to `clone` +if it does not exist). `git clean` is used to undo any changes made by the last +job, and `git fetch` is used to retrieve commits made after the last job ran. + +However, `fetch` does require access to the previous worktree. This works +well when using the `shell` or `docker` executor because these +try to preserve worktrees and try to re-use them by default. + +This has limitations when using the [Docker Machine executor](https://docs.gitlab.com/runner/executors/docker_machine.html). + +It does not work for [the `kubernetes` executor](https://docs.gitlab.com/runner/executors/kubernetes.html), +but a [feature proposal](https://gitlab.com/gitlab-org/gitlab-runner/-/issues/3847) exists. +The `kubernetes` executor always clones into an temporary directory. + +A Git strategy of `none` also re-uses the local working copy, but skips all Git +operations normally done by GitLab. GitLab Runner pre-clone scripts are also skipped, +if present. This strategy could mean you need to add `fetch` and `checkout` commands +to [your `.gitlab-ci.yml` script](../yaml/README.md#script). + +It can be used for jobs that operate exclusively on artifacts, like a deployment job. +Git repository data may be present, but it's likely out of date. You should only +rely on files brought into the local working copy from cache or artifacts. + +### Git submodule strategy + +> Requires GitLab Runner v1.10+. + +The `GIT_SUBMODULE_STRATEGY` variable is used to control if / how Git +submodules are included when fetching the code before a build. You can set them +globally or per-job in the [`variables`](../yaml/README.md#variables) section. + +There are three possible values: `none`, `normal`, and `recursive`: + +- `none` means that submodules are not included when fetching the project + code. This is the default, which matches the pre-v1.10 behavior. + +- `normal` means that only the top-level submodules are included. It's + equivalent to: + + ```shell + git submodule sync + git submodule update --init + ``` + +- `recursive` means that all submodules (including submodules of submodules) + are included. This feature needs Git v1.8.1 and later. When using a + GitLab Runner with an executor not based on Docker, make sure the Git version + meets that requirement. It's equivalent to: + + ```shell + git submodule sync --recursive + git submodule update --init --recursive + ``` + +For this feature to work correctly, the submodules must be configured +(in `.gitmodules`) with either: + +- the HTTP(S) URL of a publicly-accessible repository, or +- a relative path to another repository on the same GitLab server. See the + [Git submodules](../git_submodules.md) documentation. + +### Git checkout + +> Introduced in GitLab Runner 9.3. + +The `GIT_CHECKOUT` variable can be used when the `GIT_STRATEGY` is set to either +`clone` or `fetch` to specify whether a `git checkout` should be run. If not +specified, it defaults to true. You can set them globally or per-job in the +[`variables`](../yaml/README.md#variables) section. + +If set to `false`, the runner: + +- when doing `fetch` - updates the repository and leaves the working copy on + the current revision, +- when doing `clone` - clones the repository and leaves the working copy on the + default branch. + +If `GIT_CHECKOUT` is set to `true`, both `clone` and `fetch` work the same way. +The runner checks out the working copy of a revision related +to the CI pipeline: + +```yaml +variables: + GIT_STRATEGY: clone + GIT_CHECKOUT: "false" +script: + - git checkout -B master origin/master + - git merge $CI_COMMIT_SHA +``` + +### Git clean flags + +> Introduced in GitLab Runner 11.10 + +The `GIT_CLEAN_FLAGS` variable is used to control the default behavior of +`git clean` after checking out the sources. You can set it globally or per-job in the +[`variables`](../yaml/README.md#variables) section. + +`GIT_CLEAN_FLAGS` accepts all possible options of the [`git clean`](https://git-scm.com/docs/git-clean) +command. + +`git clean` is disabled if `GIT_CHECKOUT: "false"` is specified. + +If `GIT_CLEAN_FLAGS` is: + +- Not specified, `git clean` flags default to `-ffdx`. +- Given the value `none`, `git clean` is not executed. + +For example: + +```yaml +variables: + GIT_CLEAN_FLAGS: -ffdx -e cache/ +script: + - ls -al cache/ +``` + +### Git fetch extra flags + +> [Introduced](https://gitlab.com/gitlab-org/gitlab-runner/-/issues/4142) in GitLab Runner 13.1. + +The `GIT_FETCH_EXTRA_FLAGS` variable is used to control the behavior of +`git fetch`. You can set it globally or per-job in the [`variables`](../yaml/README.md#variables) section. + +`GIT_FETCH_EXTRA_FLAGS` accepts all options of the [`git fetch`](https://git-scm.com/docs/git-fetch) command. However, `GIT_FETCH_EXTRA_FLAGS` flags are appended after the default flags that can't be modified. + +The default flags are: + +- [GIT_DEPTH](#shallow-cloning). +- The list of [refspecs](https://git-scm.com/book/en/v2/Git-Internals-The-Refspec). +- A remote called `origin`. + +If `GIT_FETCH_EXTRA_FLAGS` is: + +- Not specified, `git fetch` flags default to `--prune --quiet` along with the default flags. +- Given the value `none`, `git fetch` is executed only with the default flags. + +For example, the default flags are `--prune --quiet`, so you can make `git fetch` more verbose by overriding this with just `--prune`: + +```yaml +variables: + GIT_FETCH_EXTRA_FLAGS: --prune +script: + - ls -al cache/ +``` + +The configuration above results in `git fetch` being called this way: + +```shell +git fetch origin $REFSPECS --depth 50 --prune +``` + +Where `$REFSPECS` is a value provided to the runner internally by GitLab. + +### Shallow cloning + +> Introduced in GitLab 8.9 as an experimental feature. + +You can specify the depth of fetching and cloning using `GIT_DEPTH`. +`GIT_DEPTH` does a shallow clone of the repository and can significantly speed up cloning. +It can be helpful for repositories with a large number of commits or old, large binaries. The value is +passed to `git fetch` and `git clone`. + +In GitLab 12.0 and later, newly-created projects automatically have a +[default `git depth` value of `50`](../pipelines/settings.md#git-shallow-clone). + +If you use a depth of `1` and have a queue of jobs or retry +jobs, jobs may fail. + +Git fetching and cloning is based on a ref, such as a branch name, so runners +can't clone a specific commit SHA. If multiple jobs are in the queue, or +you're retrying an old job, the commit to be tested must be within the +Git history that is cloned. Setting too small a value for `GIT_DEPTH` can make +it impossible to run these old commits and `unresolved reference` is displayed in +job logs. You should then reconsider changing `GIT_DEPTH` to a higher value. + +Jobs that rely on `git describe` may not work correctly when `GIT_DEPTH` is +set since only part of the Git history is present. + +To fetch or clone only the last 3 commits: + +```yaml +variables: + GIT_DEPTH: "3" +``` + +You can set it globally or per-job in the [`variables`](../yaml/README.md#variables) section. + +### Custom build directories + +> [Introduced](https://gitlab.com/gitlab-org/gitlab-runner/-/issues/2211) in GitLab Runner 11.10. + +By default, GitLab Runner clones the repository in a unique subpath of the +`$CI_BUILDS_DIR` directory. However, your project might require the code in a +specific directory (Go projects, for example). In that case, you can specify +the `GIT_CLONE_PATH` variable to tell the runner the directory to clone the +repository in: + +```yaml +variables: + GIT_CLONE_PATH: $CI_BUILDS_DIR/project-name + +test: + script: + - pwd +``` + +The `GIT_CLONE_PATH` has to always be within `$CI_BUILDS_DIR`. The directory set in `$CI_BUILDS_DIR` +is dependent on executor and configuration of [runners.builds_dir](https://docs.gitlab.com/runner/configuration/advanced-configuration.html#the-runners-section) +setting. + +This can only be used when `custom_build_dir` is enabled in the +[runner's configuration](https://docs.gitlab.com/runner/configuration/advanced-configuration.html#the-runnerscustom_build_dir-section). +This is the default configuration for the `docker` and `kubernetes` executors. + +#### Handling concurrency + +An executor that uses a concurrency greater than `1` might lead +to failures. Multiple jobs might be working on the same directory if the `builds_dir` +is shared between jobs. + +The runner does not try to prevent this situation. It's up to the administrator +and developers to comply with the requirements of runner configuration. + +To avoid this scenario, you can use a unique path within `$CI_BUILDS_DIR`, because runner +exposes two additional variables that provide a unique `ID` of concurrency: + +- `$CI_CONCURRENT_ID`: Unique ID for all jobs running within the given executor. +- `$CI_CONCURRENT_PROJECT_ID`: Unique ID for all jobs running within the given executor and project. + +The most stable configuration that should work well in any scenario and on any executor +is to use `$CI_CONCURRENT_ID` in the `GIT_CLONE_PATH`. For example: + +```yaml +variables: + GIT_CLONE_PATH: $CI_BUILDS_DIR/$CI_CONCURRENT_ID/project-name + +test: + script: + - pwd +``` + +The `$CI_CONCURRENT_PROJECT_ID` should be used in conjunction with `$CI_PROJECT_PATH` +as the `$CI_PROJECT_PATH` provides a path of a repository. That is, `group/subgroup/project`. For example: + +```yaml +variables: + GIT_CLONE_PATH: $CI_BUILDS_DIR/$CI_CONCURRENT_ID/$CI_PROJECT_PATH + +test: + script: + - pwd +``` + +#### Nested paths + +The value of `GIT_CLONE_PATH` is expanded once and nesting variables +within is not supported. + +For example, you define both the variables below in your +`.gitlab-ci.yml` file: + +```yaml +variables: + GOPATH: $CI_BUILDS_DIR/go + GIT_CLONE_PATH: $GOPATH/src/namespace/project +``` + +The value of `GIT_CLONE_PATH` is expanded once into +`$CI_BUILDS_DIR/go/src/namespace/project`, and results in failure +because `$CI_BUILDS_DIR` is not expanded. + +### Job stages attempts + +> Introduced in GitLab, it requires GitLab Runner v1.9+. + +You can set the number of attempts that the running job tries to execute +the following stages: + +| Variable | Description | +|---------------------------------|--------------------------------------------------------| +| `ARTIFACT_DOWNLOAD_ATTEMPTS` | Number of attempts to download artifacts running a job | +| `EXECUTOR_JOB_SECTION_ATTEMPTS` | [In GitLab 12.10](https://gitlab.com/gitlab-org/gitlab-runner/-/issues/4450) and later, the number of attempts to run a section in a job after a [`No Such Container`](https://gitlab.com/gitlab-org/gitlab-runner/-/issues/4450) error ([Docker executor](https://docs.gitlab.com/runner/executors/docker.html) only). | +| `GET_SOURCES_ATTEMPTS` | Number of attempts to fetch sources running a job | +| `RESTORE_CACHE_ATTEMPTS` | Number of attempts to restore the cache running a job | + +The default is one single attempt. + +Example: + +```yaml +variables: + GET_SOURCES_ATTEMPTS: 3 +``` + +You can set them globally or per-job in the [`variables`](../yaml/README.md#variables) section. + ## System calls not available on GitLab.com shared runners GitLab.com shared runners run on CoreOS. This means that you cannot use some system calls, like `getlogin`, from the C standard library. diff --git a/doc/ci/runners/img/shared_runners_admin.png b/doc/ci/runners/img/shared_runners_admin.png Binary files differdeleted file mode 100644 index e049b339b36..00000000000 --- a/doc/ci/runners/img/shared_runners_admin.png +++ /dev/null diff --git a/doc/ci/ssh_keys/README.md b/doc/ci/ssh_keys/README.md index 12478000a0a..a329331df08 100644 --- a/doc/ci/ssh_keys/README.md +++ b/doc/ci/ssh_keys/README.md @@ -93,7 +93,7 @@ to access it. This is where an SSH key pair comes in handy. # - git config --global user.name "User name" ``` - The [`before_script`](../yaml/README.md#before_script-and-after_script) can be set globally + The [`before_script`](../yaml/README.md#before_script) can be set globally or per-job. 1. Make sure the private server's [SSH host keys are verified](#verifying-the-ssh-host-keys). diff --git a/doc/ci/triggers/README.md b/doc/ci/triggers/README.md index bcd19f0de6f..6fca482975c 100644 --- a/doc/ci/triggers/README.md +++ b/doc/ci/triggers/README.md @@ -32,7 +32,7 @@ This also applies when using the `pipelines` or `triggers` keywords with the leg A unique trigger token can be obtained when [adding a new trigger](#adding-a-new-trigger). -DANGER: **Danger:** +DANGER: **Warning:** Passing plain text tokens in public projects is a security issue. Potential attackers can impersonate the user that exposed their trigger token publicly in their `.gitlab-ci.yml` file. Use [variables](../variables/README.md#gitlab-cicd-environment-variables) @@ -50,7 +50,7 @@ with the [GitLab Container Registry](../../user/packages/container_registry/inde This way of triggering can only be used when invoked inside `.gitlab-ci.yml`, and it creates a dependent pipeline relation visible on the -[pipeline graph](../multi_project_pipelines.md#overview). For example: +[pipeline graph](../multi_project_pipelines.md). For example: ```yaml build_docs: diff --git a/doc/ci/troubleshooting.md b/doc/ci/troubleshooting.md index 992b51b6b3d..5dc1d3663c5 100644 --- a/doc/ci/troubleshooting.md +++ b/doc/ci/troubleshooting.md @@ -225,7 +225,7 @@ should disable **Pipelines must succeed** so you can accept merge requests. Pipeline configuration warnings are shown when you: -- [Validate configuration with the CI Lint tool](yaml/README.md#validate-the-gitlab-ciyml). +- [Validate configuration with the CI Lint tool](yaml/README.md). - [Manually run a pipeline](pipelines/index.md#run-a-pipeline-manually). ### "Job may allow multiple pipelines to run for a single action" warning diff --git a/doc/ci/unit_test_reports.md b/doc/ci/unit_test_reports.md index b9c1809bf0d..0a1f0000969 100644 --- a/doc/ci/unit_test_reports.md +++ b/doc/ci/unit_test_reports.md @@ -244,6 +244,44 @@ Test: - ./**/*test-result.xml ``` +### JavaScript example + +There are a few tools that can produce JUnit report format XML files in JavaScript. + +#### Jest + +The [jest-junit](https://github.com/jest-community/jest-junit) npm package can generate test reports for JavaScript applications. +In the following `.gitlab-ci.yml` example, the `javascript` job uses Jest to generate the test reports: + +```yaml +javascript: + stage: test + script: + - 'jest --ci --reporters=default --reporters=jest-junit' + artifacts: + when: always + reports: + junit: + - junit.xml +``` + +#### Karma + +The [Karma-junit-reporter](https://github.com/karma-runner/karma-junit-reporter) npm package can generate test reports for JavaScript applications. +In the following `.gitlab-ci.yml` example, the `javascript` job uses Karma to generate the test reports: + +```yaml +javascript: + stage: test + script: + - karma start --reporters junit + artifacts: + when: always + reports: + junit: + - junit.xml +``` + ## Viewing Unit test reports on GitLab > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/24792) in GitLab 12.5 behind a feature flag (`junit_pipeline_view`), disabled by default. diff --git a/doc/ci/variables/README.md b/doc/ci/variables/README.md index 9c8fb994bf7..c5eee2ed960 100644 --- a/doc/ci/variables/README.md +++ b/doc/ci/variables/README.md @@ -70,7 +70,7 @@ When you need a specific custom environment variable, you can or directly [in the `.gitlab-ci.yml` file](#create-a-custom-variable-in-gitlab-ciyml). The variables are used by the runner any time the pipeline runs. -You can also [override variable values manually for a specific pipeline](../pipelines/index.md#specifying-variables-when-running-manual-jobs). +You can also [override variable values manually for a specific pipeline](../jobs/index.md#specifying-variables-when-running-manual-jobs). There are two types of variables: **Variable** and **File**. You cannot set types in the `.gitlab-ci.yml` file, but you can set them in the UI and API. @@ -423,7 +423,7 @@ Group-level variables can be added by: 1. Inputting variable types, keys, and values in the **Variables** section. Any variables of [subgroups](../../user/group/subgroups/index.md) are inherited recursively. -Once you set them, they are available for all subsequent pipelines. Any group-level user defined variables can be viewed in projects by: +After you set them, they are available for all subsequent pipelines. Any group-level user defined variables can be viewed in projects by: 1. Navigating to the project's **Settings > CI/CD** page. 1. Expanding the **Variables** section. @@ -841,7 +841,7 @@ Available on GitLab Runner v1.7+, this feature enables the shell's execution log Before enabling this, you should ensure jobs are visible to [team members only](../../user/permissions.md#project-features). You should -also [erase](../pipelines/index.md#view-jobs-in-a-pipeline) all generated job logs +also [erase](../jobs/index.md#view-jobs-in-a-pipeline) all generated job logs before making them visible again. To enable debug logs (traces), set the `CI_DEBUG_TRACE` variable to `true`: diff --git a/doc/ci/variables/predefined_variables.md b/doc/ci/variables/predefined_variables.md index 08aaacd2620..055cb2f6a61 100644 --- a/doc/ci/variables/predefined_variables.md +++ b/doc/ci/variables/predefined_variables.md @@ -40,7 +40,7 @@ Kubernetes-specific environment variables are detailed in the | `CI_COMMIT_REF_SLUG` | 9.0 | all | `$CI_COMMIT_REF_NAME` lowercased, shortened to 63 bytes, and with everything except `0-9` and `a-z` replaced with `-`. No leading / trailing `-`. Use in URLs, host names and domain names. | | `CI_COMMIT_SHA` | 9.0 | all | The commit revision for which project is built | | `CI_COMMIT_SHORT_SHA` | 11.7 | all | The first eight characters of `CI_COMMIT_SHA` | -| `CI_COMMIT_BRANCH` | 12.6 | 0.5 | The commit branch name. Present only when building branches. | +| `CI_COMMIT_BRANCH` | 12.6 | 0.5 | The commit branch name. Present in branch pipelines, including pipelines for the default branch. Not present in merge request pipelines. | | `CI_COMMIT_TAG` | 9.0 | 0.5 | The commit tag name. Present only when building tags. | | `CI_COMMIT_TITLE` | 10.8 | all | The title of the commit - the full first line of the message | | `CI_COMMIT_TIMESTAMP` | 13.4 | all | The timestamp of the commit in the ISO 8601 format. | @@ -69,14 +69,14 @@ Kubernetes-specific environment variables are detailed in the | `CI_JOB_MANUAL` | 8.12 | all | The flag to indicate that job was manually started | | `CI_JOB_NAME` | 9.0 | 0.5 | The name of the job as defined in `.gitlab-ci.yml` | | `CI_JOB_STAGE` | 9.0 | 0.5 | The name of the stage as defined in `.gitlab-ci.yml` | -| `CI_JOB_STATUS` | all | 13.5 | The state of the job as each runner stage is executed. Use with [`after_script`](../yaml/README.md#before_script-and-after_script) where `CI_JOB_STATUS` can be either: `success`, `failed` or `canceled`. | +| `CI_JOB_STATUS` | all | 13.5 | The state of the job as each runner stage is executed. Use with [`after_script`](../yaml/README.md#after_script) where `CI_JOB_STATUS` can be either: `success`, `failed` or `canceled`. | | `CI_JOB_TOKEN` | 9.0 | 1.2 | Token used for authenticating with [a few API endpoints](../../api/README.md#gitlab-ci-job-token) and downloading [dependent repositories](../../user/project/new_ci_build_permissions_model.md#dependent-repositories). The token is valid as long as the job is running. | | `CI_JOB_JWT` | 12.10 | all | RS256 JSON web token that can be used for authenticating with third party systems that support JWT authentication, for example [HashiCorp's Vault](../secrets/index.md). | | `CI_JOB_URL` | 11.1 | 0.5 | Job details URL | | `CI_KUBERNETES_ACTIVE` | 13.0 | all | Included with the value `true` only if the pipeline has a Kubernetes cluster available for deployments. Not included if no cluster is available. Can be used as an alternative to [`only:kubernetes`/`except:kubernetes`](../yaml/README.md#onlykubernetesexceptkubernetes) with [`rules:if`](../yaml/README.md#rulesif) | | `CI_MERGE_REQUEST_ASSIGNEES` | 11.9 | all | Comma-separated list of username(s) of assignee(s) for the merge request if [the pipelines are for merge requests](../merge_request_pipelines/index.md). Available only if `only: [merge_requests]` or [`rules`](../yaml/README.md#rules) syntax is used and the merge request is created. | -| `CI_MERGE_REQUEST_ID` | 11.6 | all | The instance-level ID of the merge request. Only available if [the pipelines are for merge requests](../merge_request_pipelines/index.md) and the merge request is created. | -| `CI_MERGE_REQUEST_IID` | 11.6 | all | The project-level IID (internal ID) of the merge request. Only available If [the pipelines are for merge requests](../merge_request_pipelines/index.md) and the merge request is created. | +| `CI_MERGE_REQUEST_ID` | 11.6 | all | The instance-level ID of the merge request. Only available if [the pipelines are for merge requests](../merge_request_pipelines/index.md) and the merge request is created. This is a unique ID across all projects on GitLab. | +| `CI_MERGE_REQUEST_IID` | 11.6 | all | The project-level IID (internal ID) of the merge request. Only available If [the pipelines are for merge requests](../merge_request_pipelines/index.md) and the merge request is created. This ID is unique for the current project. | | `CI_MERGE_REQUEST_LABELS` | 11.9 | all | Comma-separated label names of the merge request if [the pipelines are for merge requests](../merge_request_pipelines/index.md). Available only if `only: [merge_requests]` or [`rules`](../yaml/README.md#rules) syntax is used and the merge request is created. | | `CI_MERGE_REQUEST_MILESTONE` | 11.9 | all | The milestone title of the merge request if [the pipelines are for merge requests](../merge_request_pipelines/index.md). Available only if `only: [merge_requests]` or [`rules`](../yaml/README.md#rules) syntax is used and the merge request is created. | | `CI_MERGE_REQUEST_PROJECT_ID` | 11.6 | all | The ID of the project of the merge request if [the pipelines are for merge requests](../merge_request_pipelines/index.md). Available only if `only: [merge_requests]` or [`rules`](../yaml/README.md#rules) syntax is used and the merge request is created. | @@ -96,8 +96,8 @@ Kubernetes-specific environment variables are detailed in the | `CI_NODE_TOTAL` | 11.5 | all | Total number of instances of this job running in parallel. If the job is not parallelized, this variable is set to `1`. | | `CI_PAGES_DOMAIN` | 11.8 | all | The configured domain that hosts GitLab Pages. | | `CI_PAGES_URL` | 11.8 | all | URL to GitLab Pages-built pages. Always belongs to a subdomain of `CI_PAGES_DOMAIN`. | -| `CI_PIPELINE_ID` | 8.10 | all | The instance-level ID of the current pipeline. | -| `CI_PIPELINE_IID` | 11.0 | all | The project-level IID (internal ID) of the current pipeline. | +| `CI_PIPELINE_ID` | 8.10 | all | The instance-level ID of the current pipeline. This is a unique ID across all projects on GitLab. | +| `CI_PIPELINE_IID` | 11.0 | all | The project-level IID (internal ID) of the current pipeline. This ID is unique for the current project. | | `CI_PIPELINE_SOURCE` | 10.0 | all | Indicates how the pipeline was triggered. Possible options are: `push`, `web`, `schedule`, `api`, `external`, `chat`, `webide`, `merge_request_event`, `external_pull_request_event`, `parent_pipeline`, [`trigger`, or `pipeline`](../triggers/README.md#authentication-tokens) (renamed to `cross_project_pipeline` since 13.0). For pipelines created before GitLab 9.5, this is displayed as `unknown`. | | `CI_PIPELINE_TRIGGERED` | all | all | The flag to indicate that job was [triggered](../triggers/README.md) | | `CI_PIPELINE_URL` | 11.1 | 0.5 | Pipeline details URL | diff --git a/doc/ci/variables/where_variables_can_be_used.md b/doc/ci/variables/where_variables_can_be_used.md index b4236ca34c2..b914f3db572 100644 --- a/doc/ci/variables/where_variables_can_be_used.md +++ b/doc/ci/variables/where_variables_can_be_used.md @@ -99,7 +99,7 @@ In the case of `after_script` scripts, they can: - Not use variables defined in `before_script` and `script`. These restrictions are because `after_script` scripts are executed in a -[separated shell context](../yaml/README.md#before_script-and-after_script). +[separated shell context](../yaml/README.md#after_script). ## Persisted variables diff --git a/doc/ci/yaml/README.md b/doc/ci/yaml/README.md index 2e4ab68a0e8..1057a1389de 100644 --- a/doc/ci/yaml/README.md +++ b/doc/ci/yaml/README.md @@ -7,105 +7,29 @@ type: reference # GitLab CI/CD pipeline configuration reference -GitLab CI/CD [pipelines](../pipelines/index.md) are configured using a YAML file called `.gitlab-ci.yml` within each project. +This document lists the configuration options for your GitLab `.gitlab-ci.yml` file. -The `.gitlab-ci.yml` file defines the structure and order of the pipelines and determines: - -- What to execute using [GitLab Runner](https://docs.gitlab.com/runner/). -- What decisions to make when specific conditions are encountered. For example, when a process succeeds or fails. - -This topic covers CI/CD pipeline configuration. For other CI/CD configuration information, see: - -- [GitLab CI/CD Variables](../variables/README.md), for configuring the environment the pipelines run in. -- [GitLab Runner advanced configuration](https://docs.gitlab.com/runner/configuration/advanced-configuration.html), for configuring GitLab Runner. - -We have complete examples of configuring pipelines: - -- For a quick introduction to GitLab CI/CD, follow our [quick start guide](../quick_start/README.md). +- For a quick introduction to GitLab CI/CD, follow the [quick start guide](../quick_start/README.md). - For a collection of examples, see [GitLab CI/CD Examples](../examples/README.md). -- To see a large `.gitlab-ci.yml` file used in an enterprise, see the [`.gitlab-ci.yml` file for `gitlab`](https://gitlab.com/gitlab-org/gitlab/blob/master/.gitlab-ci.yml). - -> For some additional information about GitLab CI/CD: -> -> - <i class="fa fa-youtube-play youtube" aria-hidden="true"></i> Watch the [CI/CD Ease of configuration](https://www.youtube.com/embed/opdLqwz6tcE) video. -> - Watch the [Making the case for CI/CD in your organization](https://about.gitlab.com/compare/github-actions-alternative/) -> webcast to learn the benefits of CI/CD and how to measure the results of CI/CD automation. -> - <i class="fa fa-youtube-play youtube" aria-hidden="true"></i> Learn how [Verizon reduced rebuilds](https://about.gitlab.com/blog/2019/02/14/verizon-customer-story/) -> from 30 days to under 8 hours with GitLab. - -If you have a [mirrored repository that GitLab pulls from](../../user/project/repository/repository_mirroring.md#pulling-from-a-remote-repository), -you may need to enable pipeline triggering. Go to your project's **Settings > Repository > Pull from a remote repository > Trigger pipelines for mirror updates**. +- To view a large `.gitlab-ci.yml` file used in an enterprise, see the [`.gitlab-ci.yml` file for `gitlab`](https://gitlab.com/gitlab-org/gitlab/blob/master/.gitlab-ci.yml). -## Introduction - -Pipeline configuration begins with jobs. Jobs are the most fundamental element of a `.gitlab-ci.yml` file. - -Jobs are: - -- Defined with constraints stating under what conditions they should be executed. -- Top-level elements with an arbitrary name and must contain at least the [`script`](#script) clause. -- Not limited in how many can be defined. - -For example: - -```yaml -job1: - script: "execute-script-for-job1" - -job2: - script: "execute-script-for-job2" -``` - -The above example is the simplest possible CI/CD configuration with two separate -jobs, where each of the jobs executes a different command. -Of course a command can execute code directly (`./configure;make;make install`) -or run a script (`test.sh`) in the repository. - -Jobs are picked up by [runners](../runners/README.md) and executed within the -environment of the runner. What is important is that each job is run -independently from each other. - -### Validate the `.gitlab-ci.yml` - -Each instance of GitLab CI/CD has an embedded debug tool called Lint, which validates the -content of your `.gitlab-ci.yml` files. You can find the Lint under the page `ci/lint` of your +While you are authoring your `.gitlab-ci.yml` file, you can validate it +by using the [CI Lint](../lint.md) tool. project namespace. For example, `https://gitlab.example.com/gitlab-org/project-123/-/ci/lint`. -### Unavailable names for jobs - -Each job must have a unique name, but there are a few **reserved `keywords` that -can't be used as job names**: - -- `image` -- `services` -- `stages` -- `types` -- `before_script` -- `after_script` -- `variables` -- `cache` -- `include` +## Job keywords -### Using reserved keywords +A job is defined as a list of keywords that define the job's behavior. -If you get validation error when using specific values (for example, `true` or `false`), try to: - -- Quote them. -- Change them to a different form. For example, `/bin/true`. - -## Configuration parameters - -A job is defined as a list of parameters that define the job's behavior. - -The following table lists available parameters for jobs: +The following table lists available keywords for jobs: | Keyword | Description | |:---------------------------------------------------|:------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | [`script`](#script) | Shell script that is executed by a runner. | -| [`after_script`](#before_script-and-after_script) | Override a set of commands that are executed after job. | +| [`after_script`](#after_script) | Override a set of commands that are executed after job. | | [`allow_failure`](#allow_failure) | Allow job to fail. Failed job does not contribute to commit status. | | [`artifacts`](#artifacts) | List of files and directories to attach to a job on success. Also available: `artifacts:paths`, `artifacts:exclude`, `artifacts:expose_as`, `artifacts:name`, `artifacts:untracked`, `artifacts:when`, `artifacts:expire_in`, and `artifacts:reports`. | -| [`before_script`](#before_script-and-after_script) | Override a set of commands that are executed before job. | +| [`before_script`](#before_script) | Override a set of commands that are executed before job. | | [`cache`](#cache) | List of files that should be cached between subsequent runs. Also available: `cache:paths`, `cache:key`, `cache:untracked`, `cache:when`, and `cache:policy`. | | [`coverage`](#coverage) | Code coverage settings for a given job. | | [`dependencies`](#dependencies) | Restrict which artifacts are passed to a specific job by providing a list of jobs to fetch artifacts from. | @@ -130,22 +54,44 @@ The following table lists available parameters for jobs: | [`variables`](#variables) | Define job variables on a job level. | | [`when`](#when) | When to run job. Also available: `when:manual` and `when:delayed`. | -## Global parameters +### Unavailable names for jobs -Some parameters must be defined at a global level, affecting all jobs in the pipeline. +Each job must have a unique name, but there are a few **reserved `keywords` that +can't be used as job names**: + +- `image` +- `services` +- `stages` +- `types` +- `before_script` +- `after_script` +- `variables` +- `cache` +- `include` + +## Global keywords + +Some keywords must be defined at a global level, affecting all jobs in the pipeline. + +### Using reserved keywords + +If you get validation error when using specific values (for example, `true` or `false`), try to: + +- Quote them. +- Change them to a different form. For example, `/bin/true`. ### Global defaults -Some parameters can be set globally as the default for all jobs using the -`default:` keyword. Default parameters can then be overridden by job-specific +Some keywords can be set globally as the default for all jobs using the +`default:` keyword. Default keywords can then be overridden by job-specific configuration. -The following job parameters can be defined inside a `default:` block: +The following job keywords can be defined inside a `default:` block: - [`image`](#image) - [`services`](#services) -- [`before_script`](#before_script-and-after_script) -- [`after_script`](#before_script-and-after_script) +- [`before_script`](#before_script) +- [`after_script`](#after_script) - [`tags`](#tags) - [`cache`](#cache) - [`artifacts`](#artifacts) @@ -173,20 +119,20 @@ rspec 2.6: > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/207484) in GitLab 12.9. You can disable inheritance of globally defined defaults -and variables with the `inherit:` parameter. +and variables with the `inherit:` keyword. -To enable or disable the inheritance of all `variables:` or `default:` parameters, use the following format: +To enable or disable the inheritance of all `variables:` or `default:` keywords, use the following format: - `default: true` or `default: false` - `variables: true` or `variables: false` -To inherit only a subset of `default:` parameters or `variables:`, specify what +To inherit only a subset of `default:` keywords or `variables:`, specify what you wish to inherit. Anything not listed is **not** inherited. Use one of the following formats: ```yaml inherit: - default: [parameter1, parameter2] + default: [keyword1, keyword2] variables: [VARIABLE1, VARIABLE2] ``` @@ -195,8 +141,8 @@ Or: ```yaml inherit: default: - - parameter1 - - parameter2 + - keyword1 + - keyword2 variables: - VARIABLE1 - VARIABLE2 @@ -288,23 +234,23 @@ There are also two edge cases worth mentioning: > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/29654) in GitLab 12.5 -The top-level `workflow:` key applies to the entirety of a pipeline, and -determines whether or not a pipeline is created. It accepts a single -`rules:` key that operates similarly to [`rules:` defined within jobs](#rules), -enabling dynamic configuration of the pipeline. +The top-level `workflow:` keyword determines whether or not a pipeline is created. +It accepts a single `rules:` keyword that is similar to [`rules:` defined within jobs](#rules). +Use it to define what can trigger a new pipeline. -If you are new to GitLab CI/CD and `workflow: rules`, you may find the [`workflow:rules` templates](#workflowrules-templates) useful. +You can use the [`workflow:rules` templates](#workflowrules-templates) to import +a preconfigured `workflow: rules` entry. -To define your own `workflow: rules`, the available configuration options are: +`workflow: rules` accepts these keywords: -- [`if`](#rulesif): Define a rule. -- [`when`](#when): May be set to `always` or `never` only. If not provided, the default value is `always`​. +- [`if`](#rulesif): Check this rule to determine when to run a pipeline. +- [`when`](#when): Specify what to do when the `if` rule evaluates to true. + - To run a pipeline, set to `always`. + - To prevent pipelines from running, set to `never`. -If a pipeline attempts to run but matches no rule, it's dropped and doesn't run. +When no rules evaluate to true, the pipeline does not run. -Use the example rules below exactly as written to allow pipelines that match the rule -to run. Add `when: never` to prevent pipelines that match the rule from running. See -the [common `if` clauses for `rules`](#common-if-clauses-for-rules) for more examples. +Some example `if` clauses for `workflow: rules`: | Example rules | Details | |------------------------------------------------------|-----------------------------------------------------------| @@ -313,9 +259,12 @@ the [common `if` clauses for `rules`](#common-if-clauses-for-rules) for more exa | `if: $CI_COMMIT_TAG` | Control when tag pipelines run. | | `if: $CI_COMMIT_BRANCH` | Control when branch pipelines run. | +See the [common `if` clauses for `rules`](#common-if-clauses-for-rules) for more examples. + For example, in the following configuration, pipelines run for all `push` events (changes to -branches and new tags). Only push events with `-wip` in the commit message are excluded. Scheduled -pipelines and merge request pipelines don't run, as there's no rule allowing them. +branches and new tags). Pipelines for push events with `-wip` in the commit message +don't run, because they are set to `when: never`. Pipelines for schedules or merge requests +don't run either, because no rules evaluate to true for them: ```yaml workflow: @@ -325,11 +274,11 @@ workflow: - if: '$CI_PIPELINE_SOURCE == "push"' ``` -This example has strict rules, and no other pipelines can run. +This example has strict rules, and pipelines do **not** run in any other case. -Alternatively, you can have loose rules by using only `when: never` rules, followed -by a final `when: always` rule. This allows all types of pipelines, except for any -that match the `when: never` rules: +Alternatively, all of the rules can be `when: never`, with a final +`when: always` rule. Pipelines that match the `when: never` rules do not run. +All other pipeline types run: ```yaml workflow: @@ -341,12 +290,13 @@ workflow: - when: always ``` -This example never allows pipelines for schedules or `push` (branches and tags) pipelines, -but does allow pipelines in **all** other cases, *including* merge request pipelines. +This example prevents pipelines for schedules or `push` (branches and tags) pipelines. +The final `when: always` rule lets all other pipeline types run, **including** merge +request pipelines. -Be careful not to use a configuration that might run -merge request pipelines and branch pipelines at the same time. As with `rules` defined in jobs, -it can cause [duplicate pipelines](#prevent-duplicate-pipelines). +Be careful not to have rules that match both branch pipelines +and merge request pipelines. Similar to `rules` defined in jobs, this can cause +[duplicate pipelines](#prevent-duplicate-pipelines). #### `workflow:rules` templates @@ -405,7 +355,7 @@ of using YAML anchors, you can use the [`extends` keyword](#extends). `include` supports the following inclusion methods: -| Method | Description | +| Keyword | Method | |:--------------------------------|:------------------------------------------------------------------| | [`local`](#includelocal) | Include a file from the local project repository. | | [`file`](#includefile) | Include a file from a different project repository. | @@ -471,7 +421,7 @@ include: file: '/templates/.gitlab-ci-template.yml' ``` -You can also specify `ref`, with the default being the `HEAD` of the project: +You can also specify a `ref`. If not specified, it defaults to the `HEAD` of the project: ```yaml include: @@ -492,11 +442,47 @@ All [nested includes](#nested-includes) are executed in the scope of the target This means you can use local (relative to target project), project, remote, or template includes. +##### Multiple files from a project + +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/26793) in GitLab 13.6. +> - It's [deployed behind a feature flag](../../user/feature_flags.md), enabled by default. +> - It's enabled on GitLab.com. +> - It's recommended for production use. +> - For GitLab self-managed instances, GitLab administrators can opt to disable it. **(CORE ONLY)** + +You can include multiple files from the same project: + +```yaml +include: + - project: 'my-group/my-project' + ref: master + file: + - '/templates/.builds.yml' + - '/templates/.tests.yml' +``` + +Including multiple files from the same project is under development but ready for production use. It is +deployed behind a feature flag that is **enabled by default**. +[GitLab administrators with access to the GitLab Rails console](../../administration/feature_flags.md) +can opt to disable it. + +To enable it: + +```ruby +Feature.enable(:ci_include_multiple_files_from_project) +``` + +To disable it: + +```ruby +Feature.disable(:ci_include_multiple_files_from_project) +``` + #### `include:remote` `include:remote` can be used to include a file from a different location, -using HTTP/HTTPS, referenced by using the full URL. The remote file must be -publicly accessible through a simple GET request as authentication schemas +using HTTP/HTTPS, referenced by the full URL. The remote file must be +publicly accessible by a GET request, because authentication schemas in the remote URL are not supported. For example: ```yaml @@ -542,15 +528,15 @@ Nested includes allow you to compose a set of includes. A total of 100 includes is allowed, but duplicate includes are considered a configuration error. In [GitLab 12.4](https://gitlab.com/gitlab-org/gitlab/-/issues/28212) and later, the time limit -for resolving all files is 30 seconds. +to resolve all files is 30 seconds. #### Additional `includes` examples There is a list of [additional `includes` examples](includes.md) available. -## Parameter details +## Keyword details -The following are detailed explanations for parameters used to configure CI/CD pipelines. +The following are detailed explanations for keywords used to configure CI/CD pipelines. ### `image` @@ -558,7 +544,7 @@ Used to specify [a Docker image](../docker/using_docker_images.md#what-is-an-ima For: -- Simple definition examples, see [Define `image` and `services` from `.gitlab-ci.yml`](../docker/using_docker_images.md#define-image-and-services-from-gitlab-ciyml). +- Usage examples, see [Define `image` and `services` from `.gitlab-ci.yml`](../docker/using_docker_images.md#define-image-and-services-from-gitlab-ciyml). - Detailed usage information, refer to [Docker integration](../docker/README.md) documentation. #### `image:name` @@ -579,7 +565,7 @@ Used to specify a [service Docker image](../docker/using_docker_images.md#what-i For: -- Simple definition examples, see [Define `image` and `services` from `.gitlab-ci.yml`](../docker/using_docker_images.md#define-image-and-services-from-gitlab-ciyml). +- Usage examples, see [Define `image` and `services` from `.gitlab-ci.yml`](../docker/using_docker_images.md#define-image-and-services-from-gitlab-ciyml). - Detailed usage information, refer to [Docker integration](../docker/README.md) documentation. - For example services, see [GitLab CI/CD Services](../services/README.md). @@ -617,9 +603,9 @@ job: script: "bundle exec rspec" ``` -[YAML anchors for scripts](#yaml-anchors-for-script) are available. +You can use [YAML anchors with `script`](#yaml-anchors-for-scripts). -This parameter can also contain several commands using an array: +This keyword can also contain several commands in an array: ```yaml job: @@ -635,8 +621,8 @@ a "key: value" pair. Be careful when using special characters: `:`, `{`, `}`, `[`, `]`, `,`, `&`, `*`, `#`, `?`, `|`, `-`, `<`, `>`, `=`, `!`, `%`, `@`, `` ` ``. If any of the script commands return an exit code other than zero, the job -fails and further commands are not executed. You can avoid this behavior by -storing the exit code in a variable: +fails and further commands are not executed. Store the exit code in a variable to +avoid this behavior: ```yaml job: @@ -645,191 +631,86 @@ job: - if [ $exit_code -ne 0 ]; then echo "Previous command failed"; fi; ``` -#### `before_script` and `after_script` +#### `before_script` > Introduced in GitLab 8.7 and requires GitLab Runner v1.2. -`before_script` is used to define commands that should be run before each -job, including deploy jobs, but after the restoration of any [artifacts](#artifacts). -This must be an array. +`before_script` is used to define commands that should run before each job, including +deploy jobs, but after the restoration of any [artifacts](#artifacts). This must be an array. Scripts specified in `before_script` are concatenated with any scripts specified in the main [`script`](#script), and executed together in a single shell. -`after_script` is used to define commands that run after each -job, including failed jobs. This must be an array. If a job times out or is cancelled, -the `after_script` commands are not executed. Support for executing `after_script` -commands for timed-out or cancelled jobs -[is planned](https://gitlab.com/gitlab-org/gitlab/-/issues/15603). - -Scripts specified in `after_script` are executed in a new shell, separate from any -`before_script` or `script` scripts. As a result, they: - -- Have a current working directory set back to the default. -- Have no access to changes done by scripts defined in `before_script` or `script`, including: - - Command aliases and variables exported in `script` scripts. - - Changes outside of the working tree (depending on the runner executor), like - software installed by a `before_script` or `script` script. -- Have a separate timeout, which is hard coded to 5 minutes. See - [related issue](https://gitlab.com/gitlab-org/gitlab-runner/-/issues/2716) for details. -- Don't affect the job's exit code. If the `script` section succeeds and the - `after_script` times out or fails, the job exits with code `0` (`Job Succeeded`). - -It's possible to overwrite a globally defined `before_script` or `after_script` -if you set it per-job: +It's possible to overwrite a globally defined `before_script` if you define it in a job: ```yaml default: before_script: - - global before script - -job: - before_script: - - execute this instead of global before script - script: - - my command - after_script: - - execute this after my script -``` - -[YAML anchors for `before_script` and `after_script`](#yaml-anchors-for-before_script-and-after_script) are available. + - echo "Execute this in all jobs that don't already have a before_script section." -#### Coloring script output - -Script output can be colored using [ANSI escape codes](https://en.wikipedia.org/wiki/ANSI_escape_code#Colors), -or by running commands or programs that output ANSI escape codes. - -For example, using [Bash with color codes](https://misc.flogisoft.com/bash/tip_colors_and_formatting): - -```yaml -job: +job1: script: - - echo -e "\e[31mThis text is red,\e[0m but this text isn't\e[31m however this text is red again." -``` + - echo "This executes after the global before_script." -You can define the color codes in Shell variables, or even [custom environment variables](../variables/README.md#custom-environment-variables), -which makes the commands easier to read and reusable. - -For example, using the same example as above and variables defined in a `before_script`: - -```yaml job: before_script: - - TXT_RED="\e[31m" && TXT_CLEAR="\e[0m" + - echo "Execute this instead of the global before_script." script: - - echo -e "${TXT_RED}This text is red,${TXT_CLEAR} but this part isn't${TXT_RED} however this part is again." - - echo "This text is not colored" + - echo "This executes after the job's `before_script`" ``` -Or with [PowerShell color codes](https://superuser.com/a/1259916): +You can use [YAML anchors with `before_script`](#yaml-anchors-for-scripts). -```yaml -job: - before_script: - - $esc="$([char]27)"; $TXT_RED="$esc[31m"; $TXT_CLEAR="$esc[0m" - script: - - Write-Host $TXT_RED"This text is red,"$TXT_CLEAR" but this text isn't"$TXT_RED" however this text is red again." - - Write-Host "This text is not colored" -``` - -#### Multi-line commands +#### `after_script` -You can split long commands into multi-line commands to improve readability -using [`|` (literal) and `>` (folded) YAML multi-line block scalar indicators](https://yaml-multiline.info/). - -CAUTION: **Warning:** -If multiple commands are combined into one command string, only the last command's -failure or success is reported. -[Failures from earlier commands are ignored due to a bug](https://gitlab.com/gitlab-org/gitlab-runner/-/issues/25394). -To work around this, -run each command as a separate `script:` item, or add an `exit 1` command -to each command string. +Introduced in GitLab 8.7 and requires GitLab Runner v1.2. -You can use the `|` (literal) YAML multiline block scalar indicator to write -commands over multiple lines in the `script` section of a job description. -Each line is treated as a separate command. -Only the first command is repeated in the job log, but additional -commands are still executed: +`after_script` is used to define commands that run after each job, including failed +jobs. This must be an array. -```yaml -job: - script: - - | - echo "First command line." - echo "Second command line." - echo "Third command line." -``` - -The example above renders in the job log as: +If a job times out or is cancelled, the `after_script` commands are not executed. +Support for executing `after_script` commands for timed-out or cancelled jobs +[is planned](https://gitlab.com/gitlab-org/gitlab/-/issues/15603). -```shell -$ echo First command line # collapsed multi-line command -First command line -Second command line. -Third command line. -``` +Scripts specified in `after_script` are executed in a new shell, separate from any +`before_script` or `script` scripts. As a result, they: -The `>` (folded) YAML multiline block scalar indicator treats empty lines between -sections as the start of a new command: +- Have a current working directory set back to the default. +- Have no access to changes done by scripts defined in `before_script` or `script`, including: + - Command aliases and variables exported in `script` scripts. + - Changes outside of the working tree (depending on the runner executor), like + software installed by a `before_script` or `script` script. +- Have a separate timeout, which is hard coded to 5 minutes. See the + [related issue](https://gitlab.com/gitlab-org/gitlab-runner/-/issues/2716) for details. +- Don't affect the job's exit code. If the `script` section succeeds and the + `after_script` times out or fails, the job exits with code `0` (`Job Succeeded`). ```yaml -job: - script: - - > - echo "First command line - is split over two lines." - - echo "Second command line." -``` - -This behaves similarly to writing multiline commands without the `>` or `|` block -scalar indicators: +default: + after_script: + - echo "Execute this in all jobs that don't already have an after_script section." -```yaml -job: +job1: script: - - echo "First command line - is split over two lines." - - echo "Second command line." -``` - -Both examples above render in the job log as: - -```shell -$ echo First command line is split over two lines. # collapsed multi-line command -First command line is split over two lines. -Second command line. -``` - -When you omit the `>` or `|` block scalar indicators, GitLab forms the command -by concatenating non-empty lines. Make sure the lines can run when concatenated. + - echo "This executes first. When it completes, the global after_script executes." -Shell [here documents](https://en.wikipedia.org/wiki/Here_document) work with the -`|` and `>` operators as well. The example below transliterates the lower case letters -to upper case: - -```yaml job: script: - - | - tr a-z A-Z << END_TEXT - one two three - four five six - END_TEXT + - echo "This executes first. When it completes, the job's `after_script` executes." + after_script: + - echo "Execute this instead of the global after_script." ``` -Results in: +You can use [YAML anchors with `after_script`](#yaml-anchors-for-scripts). -```shell -$ tr a-z A-Z << END_TEXT # collapsed multi-line command - ONE TWO THREE - FOUR FIVE SIX -``` +#### Script syntax -#### Custom collapsible sections +You can use special syntax in [`script`](README.md#script) sections to: -See [custom collapsible sections](../pipelines/index.md#custom-collapsible-sections). +- [Split long commands](script.md#split-long-commands) into multiline commands. +- [Use color codes](script.md#add-color-codes-to-script-output) to make job logs easier to review. +- [Create custom collapsible sections](../jobs/index.md#custom-collapsible-sections) + to simplify job log output. ### `stage` @@ -968,7 +849,7 @@ rspec: - $RSPEC ``` -If you do want to include the `rake test`, see [`before_script` and `after_script`](#before_script-and-after_script). +If you do want to include the `rake test`, see [`before_script`](#before_script) or [`after_script`](#after_script). `.tests` in this example is a [hidden job](#hide-jobs), but it's possible to inherit from regular jobs as well. @@ -1422,29 +1303,20 @@ Other commonly used variables for `if` clauses: #### `rules:changes` -To determine if jobs should be added to a pipeline, `rules: changes` clauses check -the files changed by Git push events. +`rules:changes` determines whether or not to add jobs to a pipeline by checking for +changes to specific files. `rules: changes` works exactly the same way as [`only: changes` and `except: changes`](#onlychangesexceptchanges), -accepting an array of paths. Similarly, it always returns true if there is no -Git push event, for example, when a new tag is created. It's recommended to use it -only with branch pipelines or merge request pipelines. For example, it's common to -use `rules: changes` with one of the following `if` clauses: - -- `if: $CI_COMMIT_BRANCH` -- `if: '$CI_PIPELINE_SOURCE == "merge_request_event"'` - -For example: +accepting an array of paths. It's recommended to only use `rules: changes` with branch +pipelines or merge request pipelines. For example, it's common to use `rules: changes` +with merge request pipelines: ```yaml -workflow: - rules: - - if: '$CI_PIPELINE_SOURCE == "merge_request_event"' - docker build: script: docker build -t my-image:$CI_COMMIT_REF_SLUG . rules: - - changes: + - if: '$CI_PIPELINE_SOURCE == "merge_request_event"' + changes: - Dockerfile when: manual allow_failure: true @@ -1452,14 +1324,77 @@ docker build: In this example: -- [`workflow: rules`](#workflowrules) allows only pipelines for merge requests for all jobs. +- If the pipeline is a merge request pipeline, check `Dockerfile` for changes. - If `Dockerfile` has changed, add the job to the pipeline as a manual job, and allow the pipeline to continue running even if the job is not triggered (`allow_failure: true`). - If `Dockerfile` has not changed, do not add job to any pipeline (same as `when: never`). -To implement a rule similar to [`except: changes`](#onlychangesexceptchanges), +To use `rules: changes` with branch pipelines instead of merge request pipelines, +change the `if:` clause in the example above to: + +```yaml +rules: + - if: $CI_PIPELINE_SOURCE == "push" && $CI_COMMIT_BRANCH +``` + +To implement a rule similar to [`except:changes`](#onlychangesexceptchanges), use `when: never`. +CAUTION: **Caution:** +You can use `rules: changes` with other pipeline types, but it is not recommended +because `rules: changes` always evaluates to true when there is no Git `push` event. +Tag pipelines, scheduled pipelines, and so on do **not** have a Git `push` event +associated with them. A `rules: changes` job is **always** added to those pipeline +if there is no `if:` statement that limits the job to branch or merge request pipelines. + +##### Variables in `rules:changes` + +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/34272) in GitLab 13.6. +> - It was [deployed behind a feature flag](../../user/feature_flags.md), disabled by default. +> - [Became enabled by default](https://gitlab.com/gitlab-org/gitlab/-/issues/267192) in GitLab 13.6. +> - It's enabled on GitLab.com. +> - It's recommended for production use. +> - For GitLab self-managed instances, GitLab administrators can opt to [disable it](#enable-or-disable-variables-support-in-ruleschanges). **(CORE ONLY)** + +CAUTION: **Warning:** +This feature might not be available to you. Check the **version history** note above for details. + +Environment variables can be used in `rules:changes` expressions to determine when +to add jobs to a pipeline: + +```yaml +docker build: + variables: + DOCKERFILES_DIR: 'path/to/files/' + script: docker build -t my-image:$CI_COMMIT_REF_SLUG . + rules: + - changes: + - $DOCKERFILES_DIR/* +``` + +The `$` character can be used for both variables and paths. For example, if the +`$DOCKERFILES_DIR` variable exists, its value is used. If it does not exist, the +`$` is interpreted as being part of a path. + +###### Enable or disable variables support in `rules:changes` **(CORE ONLY)** + +Variables support in `rules:changes` is under development, but is ready for production use. +It is deployed behind a feature flag that is **enabled by default**. +[GitLab administrators with access to the GitLab Rails console](../../administration/feature_flags.md) +can opt to disable it. + +To enable it: + +```ruby +Feature.enable(:ci_variable_expansion_in_rules_changes) +``` + +To disable it: + +```ruby +Feature.disable(:ci_variable_expansion_in_rules_changes) +``` + #### `rules:exists` > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/24021) in GitLab 12.4. @@ -1567,7 +1502,7 @@ The [`rules`](#rules) syntax is an improved, more powerful solution for defining when jobs should run or not. Consider using `rules` instead of `only/except` to get the most out of your pipelines. -`only` and `except` are two parameters that set a job policy to limit when +`only` and `except` are two keywords that set a job policy to limit when jobs are created: 1. `only` defines the names of branches and tags the job runs for. @@ -1695,7 +1630,7 @@ while just `/issue/` would also match a branch called `severe-issues`. #### Supported `only`/`except` regexp syntax In GitLab 11.9.4, GitLab began internally converting the regexp used -in `only` and `except` parameters to [RE2](https://github.com/google/re2/wiki/Syntax). +in `only` and `except` keywords to [RE2](https://github.com/google/re2/wiki/Syntax). [RE2](https://github.com/google/re2/wiki/Syntax) limits the set of available features due to computational complexity, and some features, like negative lookaheads, became unavailable. @@ -2489,7 +2424,7 @@ deployment to the `production` environment. > - Before GitLab 8.11, the name of an environment could be defined as a string like > `environment: production`. The recommended way now is to define it under the > `name` keyword. -> - The `name` parameter can use any of the defined CI variables, +> - The `name` keyword can use any of the defined CI variables, > including predefined, secure variables and `.gitlab-ci.yml` [`variables`](#variables). > You however can't use variables defined under `script`. @@ -2525,7 +2460,7 @@ deploy to production: > - Introduced in GitLab 8.11. > - Before GitLab 8.11, the URL could be added only in GitLab's UI. The > recommended way now is to define it in `.gitlab-ci.yml`. -> - The `url` parameter can use any of the defined CI variables, +> - The `url` keyword can use any of the defined CI variables, > including predefined, secure variables and `.gitlab-ci.yml` [`variables`](#variables). > You however can't use variables defined under `script`. @@ -2624,7 +2559,7 @@ In the example above, if the configuration is not identical: > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/20956) in GitLab 12.8. -The `auto_stop_in` keyword is for specifying life period of the environment, +The `auto_stop_in` keyword is for specifying the lifetime of the environment, that when expired, GitLab automatically stops them. For example, @@ -2637,8 +2572,8 @@ review_app: auto_stop_in: 1 day ``` -When `review_app` job is executed and a review app is created, a life period of -the environment is set to `1 day`. +When the environment for `review_app` is created, the environment's lifetime is set to `1 day`. +Every time the review app is deployed, that lifetime is also reset to `1 day`. For more information, see [the environments auto-stop documentation](../environments/index.md#environments-auto-stop) @@ -2679,7 +2614,7 @@ To follow progress on support for GitLab-managed clusters, see the > - [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/21971) in GitLab 8.12 and GitLab Runner 1.6. > - The `$CI_ENVIRONMENT_SLUG` was [introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/22864) in GitLab 8.15. -> - The `name` and `url` parameters can use any of the defined CI variables, +> - The `name` and `url` keywords can use any of the defined CI variables, > including predefined, secure variables and `.gitlab-ci.yml` [`variables`](#variables). > You however can't use variables defined under `script`. @@ -2774,11 +2709,7 @@ Otherwise cache content can be overwritten. > Introduced in GitLab Runner v1.0.0. -The cache is shared between jobs, so if you're using different -paths for different jobs, you should also set a different `cache:key`. -Otherwise cache content can be overwritten. - -The `key` parameter defines the affinity of caching between jobs. +The `key` keyword defines the affinity of caching between jobs. You can have a single cache for all jobs, cache per-job, cache per-branch, or any other way that fits your workflow. This way, you can fine tune caching, including caching data between different jobs or even different branches. @@ -2812,6 +2743,32 @@ URI-encoded `%2F`. A value made only of dots (`.`, `%2E`) is also forbidden. You can specify a [fallback cache key](#fallback-cache-key) to use if the specified `cache:key` is not found. +#### Fallback cache key + +> [Introduced](https://gitlab.com/gitlab-org/gitlab-runner/-/merge_requests/1534) in GitLab Runner 13.4. + +You can use the `$CI_COMMIT_REF_SLUG` [variable](#variables) to specify your [`cache:key`](#cachekey). +For example, if your `$CI_COMMIT_REF_SLUG` is `test` you can set a job +to download cache that's tagged with `test`. + +If a cache with this tag is not found, you can use `CACHE_FALLBACK_KEY` to +specify a cache to use when none exists. + +For example: + +```yaml +variables: + CACHE_FALLBACK_KEY: fallback-key + +cache: + key: "$CI_COMMIT_REF_SLUG" + paths: + - binaries/ +``` + +In this example, if the `$CI_COMMIT_REF_SLUG` is not found, the job uses the key defined +by the `CACHE_FALLBACK_KEY` variable. + ##### `cache:key:files` > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/18986) in GitLab v12.5. @@ -2847,7 +2804,7 @@ use the new cache, instead of rebuilding the dependencies. > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/18986) in GitLab v12.5. When you want to combine a prefix with the SHA computed for `cache:key:files`, -use the `prefix` parameter with `key:files`. +use the `prefix` keyword with `key:files`. For example, if you add a `prefix` of `test`, the resulting key is: `test-feef9576d21ee9b6a32e30c5c79d0a0ceb68d1e5`. If neither file was changed in any commits, the prefix is added to `default`, so the key in the example would be `test-default`. @@ -3307,7 +3264,7 @@ It also exposes these reports in GitLab's UI (merge requests, pipeline views, an These are the available report types: -| Parameter | Description | +| Keyword | Description | |--------------------------------------------------------------------------------------------------------------------------------------|-------------| | [`artifacts:reports:cobertura`](../pipelines/job_artifacts.md#artifactsreportscobertura) | The `cobertura` report collects Cobertura coverage XML files. | | [`artifacts:reports:codequality`](../pipelines/job_artifacts.md#artifactsreportscodequality) | The `codequality` report collects CodeQuality issues. | @@ -3329,7 +3286,7 @@ These are the available report types: > Introduced in GitLab 8.6 and GitLab Runner v1.1.1. By default, all [`artifacts`](#artifacts) from previous [stages](#stages) -are passed to each job. However, you can use the `dependencies` parameter to +are passed to each job. However, you can use the `dependencies` keyword to define a limited list of jobs to fetch artifacts from. You can also set a job to download no artifacts at all. To use this feature, define `dependencies` in context of the job and pass @@ -3495,7 +3452,7 @@ Possible values for `when` are: - `scheduler_failure`: Retry if the scheduler failed to assign the job to a runner. - `data_integrity_failure`: Retry if there was a structural integrity problem detected. -You can specify the number of [retry attempts for certain stages of job execution](#job-stages-attempts) using variables. +You can specify the number of [retry attempts for certain stages of job execution](../runners/README.md#job-stages-attempts) using variables. ### `timeout` @@ -3522,7 +3479,7 @@ exceed the runner-specific timeout. > [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/21480) in GitLab 11.5. Use `parallel` to configure how many instances of a job to run in -parallel. This value has to be greater than or equal to two (2) and less than or equal to 50. +parallel. This value can be from 2 to 50. This creates N instances of the same job that run in parallel. They are named sequentially from `job_name 1/N` to `job_name N/N`. @@ -3574,9 +3531,6 @@ There can be from 2 to 50 jobs. [In GitLab 13.5](https://gitlab.com/gitlab-org/gitlab/-/issues/26362) and later, you can have one-dimensional matrices with a single job. -The ability to have one-dimensional matrices is [deployed behind a feature flag](../../user/feature_flags.md), -enabled by default. It's enabled on GitLab.com. For self-managed GitLab instances, -administrators can opt to disable it by [disabling the `one_dimensional_matrix:` feature flag](../../administration/feature_flags.md). **(CORE ONLY)** Every job gets the same `CI_NODE_TOTAL` [environment variable](../variables/README.md#predefined-environment-variables) value, and a unique `CI_NODE_INDEX` value. @@ -3624,8 +3578,8 @@ Use `trigger` to define a downstream pipeline trigger. When GitLab starts a job with a `trigger` definition, a downstream pipeline is created. Jobs with `trigger` can only use a [limited set of keywords](../multi_project_pipelines.md#limitations). -For example, you can't run commands with [`script`](#script), [`before_script`](#before_script-and-after_script), -or [`after_script`](#before_script-and-after_script). +For example, you can't run commands with [`script`](#script), [`before_script`](#before_script), +or [`after_script`](#after_script). You can use this keyword to create two different types of downstream pipelines: @@ -3784,7 +3738,7 @@ starting, which reduces parallelization. To force a rebuild of a specific branch, tag, or commit, you can use an API call with a trigger token. -The trigger token is different than the [`trigger`](#trigger) parameter. +The trigger token is different than the [`trigger`](#trigger) keyword. [Read more in the triggers documentation.](../triggers/README.md) @@ -3884,8 +3838,8 @@ For more information, see [Deployments Safety](../environments/deployment_safety These methods are supported: - [`tag_name`](#releasetag_name) +- [`description`](#releasedescription) - [`name`](#releasename) (optional) -- [`description`](#releasedescription) (optional) - [`ref`](#releaseref) (optional) - [`milestones`](#releasemilestones) (optional) - [`released_at`](#releasereleased_at) (optional) @@ -3943,7 +3897,7 @@ For example, when creating a Release from a Git tag: job: release: tag_name: $CI_COMMIT_TAG - description: changelog.txt + description: 'Release description' ``` It is also possible to create any unique tag, in which case `only: tags` is not mandatory. @@ -3953,7 +3907,7 @@ A semantic versioning example: job: release: tag_name: ${MAJOR}_${MINOR}_${REVISION} - description: changelog.txt + description: 'Release description' ``` - The Release is created only if the job's main script succeeds. @@ -4160,8 +4114,8 @@ Read more on [GitLab Pages user documentation](../../user/project/pages/index.md > Introduced in GitLab Runner v0.5.0. -Variables are configurable values that are passed to jobs. They can be set -globally and per-job. +[CI/CD variables](../variables/README.md) are configurable values that are passed to jobs. +They can be set globally and per-job. There are two types of variables. @@ -4178,372 +4132,49 @@ Variables are meant for non-sensitive project configuration, for example: ```yaml variables: - DATABASE_URL: "postgres://postgres@postgres/my_database" + DEPLOY_SITE: "https://example.com/" + +deploy_job: + stage: deploy + script: + - deploy-script --url $DEPLOY_SITE --path "/" + +deploy_review_job: + stage: deploy + variables: + REVIEW_PATH: "/review" + script: + - deploy-review-script --url $DEPLOY_SITE --path $REVIEW_PATH ``` -You can use integers and strings for the variable's name and value. -You cannot use floats. +You can use only integers and strings for the variable's name and value. If you define a variable at the top level of the `gitlab-ci.yml` file, it is global, -meaning it applies to all jobs. - -If you define a variable within a job, it's available to that job only. +meaning it applies to all jobs. If you define a variable within a job, it's available +to that job only. If a variable of the same name is defined globally and for a specific job, the [job-specific variable is used](../variables/README.md#priority-of-environment-variables). All YAML-defined variables are also set to any linked -[service containers](../docker/using_docker_images.md#what-is-a-service). - -[YAML anchors for variables](#yaml-anchors-for-variables) are available. - -Learn more about [variables and their priority](../variables/README.md). - -### Git strategy - -> - Introduced in GitLab 8.9 as an experimental feature. -> - `GIT_STRATEGY=none` requires GitLab Runner v1.7+. - -You can set the `GIT_STRATEGY` used for getting recent application code, either -globally or per-job in the [`variables`](#variables) section. If left -unspecified, the default from the project settings is used. - -There are three possible values: `clone`, `fetch`, and `none`. - -`clone` is the slowest option. It clones the repository from scratch for every -job, ensuring that the local working copy is always pristine. - -```yaml -variables: - GIT_STRATEGY: clone -``` - -`fetch` is faster as it re-uses the local working copy (falling back to `clone` -if it does not exist). `git clean` is used to undo any changes made by the last -job, and `git fetch` is used to retrieve commits made since the last job ran. - -```yaml -variables: - GIT_STRATEGY: fetch -``` - -`none` also re-uses the local working copy. However, it skips all Git operations, -including GitLab Runner's pre-clone script, if present. - -It's useful for jobs that operate exclusively on artifacts, like a deployment job. -Git repository data may be present, but it's likely out-of-date. You should only -rely on files brought into the local working copy from cache or artifacts. - -```yaml -variables: - GIT_STRATEGY: none -``` - -NOTE: **Note:** -`GIT_STRATEGY` is not supported for -[Kubernetes executor](https://docs.gitlab.com/runner/executors/kubernetes.html), -but may be in the future. See the [support Git strategy with Kubernetes executor feature proposal](https://gitlab.com/gitlab-org/gitlab-runner/-/issues/3847) -for updates. - -### Git submodule strategy - -> Requires GitLab Runner v1.10+. - -The `GIT_SUBMODULE_STRATEGY` variable is used to control if / how Git -submodules are included when fetching the code before a build. You can set them -globally or per-job in the [`variables`](#variables) section. - -There are three possible values: `none`, `normal`, and `recursive`: - -- `none` means that submodules are not included when fetching the project - code. This is the default, which matches the pre-v1.10 behavior. - -- `normal` means that only the top-level submodules are included. It's - equivalent to: - - ```shell - git submodule sync - git submodule update --init - ``` - -- `recursive` means that all submodules (including submodules of submodules) - are included. This feature needs Git v1.8.1 and later. When using a - GitLab Runner with an executor not based on Docker, make sure the Git version - meets that requirement. It's equivalent to: - - ```shell - git submodule sync --recursive - git submodule update --init --recursive - ``` - -For this feature to work correctly, the submodules must be configured -(in `.gitmodules`) with either: - -- the HTTP(S) URL of a publicly-accessible repository, or -- a relative path to another repository on the same GitLab server. See the - [Git submodules](../git_submodules.md) documentation. - -### Git checkout - -> Introduced in GitLab Runner 9.3. - -The `GIT_CHECKOUT` variable can be used when the `GIT_STRATEGY` is set to either -`clone` or `fetch` to specify whether a `git checkout` should be run. If not -specified, it defaults to true. You can set them globally or per-job in the -[`variables`](#variables) section. - -If set to `false`, the runner: - -- when doing `fetch` - updates the repository and leaves the working copy on - the current revision, -- when doing `clone` - clones the repository and leaves the working copy on the - default branch. - -If `GIT_CHECKOUT` is set to `true`, both `clone` and `fetch` work the same way. -The runner checks out the working copy of a revision related -to the CI pipeline: - -```yaml -variables: - GIT_STRATEGY: clone - GIT_CHECKOUT: "false" -script: - - git checkout -B master origin/master - - git merge $CI_COMMIT_SHA -``` - -### Git clean flags - -> Introduced in GitLab Runner 11.10 - -The `GIT_CLEAN_FLAGS` variable is used to control the default behavior of -`git clean` after checking out the sources. You can set it globally or per-job in the -[`variables`](#variables) section. - -`GIT_CLEAN_FLAGS` accepts all possible options of the [`git clean`](https://git-scm.com/docs/git-clean) -command. - -`git clean` is disabled if `GIT_CHECKOUT: "false"` is specified. - -If `GIT_CLEAN_FLAGS` is: - -- Not specified, `git clean` flags default to `-ffdx`. -- Given the value `none`, `git clean` is not executed. - -For example: - -```yaml -variables: - GIT_CLEAN_FLAGS: -ffdx -e cache/ -script: - - ls -al cache/ -``` - -### Git fetch extra flags - -> [Introduced](https://gitlab.com/gitlab-org/gitlab-runner/-/issues/4142) in GitLab Runner 13.1. - -The `GIT_FETCH_EXTRA_FLAGS` variable is used to control the behavior of -`git fetch`. You can set it globally or per-job in the [`variables`](#variables) section. - -`GIT_FETCH_EXTRA_FLAGS` accepts all options of the [`git fetch`](https://git-scm.com/docs/git-fetch) command. However, `GIT_FETCH_EXTRA_FLAGS` flags are appended after the default flags that can't be modified. - -The default flags are: - -- [GIT_DEPTH](#shallow-cloning). -- The list of [refspecs](https://git-scm.com/book/en/v2/Git-Internals-The-Refspec). -- A remote called `origin`. - -If `GIT_FETCH_EXTRA_FLAGS` is: - -- Not specified, `git fetch` flags default to `--prune --quiet` along with the default flags. -- Given the value `none`, `git fetch` is executed only with the default flags. - -For example, the default flags are `--prune --quiet`, so you can make `git fetch` more verbose by overriding this with just `--prune`: - -```yaml -variables: - GIT_FETCH_EXTRA_FLAGS: --prune -script: - - ls -al cache/ -``` - -The configuration above results in `git fetch` being called this way: +[Docker service containers](../docker/using_docker_images.md#what-is-a-service). -```shell -git fetch origin $REFSPECS --depth 50 --prune -``` - -Where `$REFSPECS` is a value provided to the runner internally by GitLab. - -### Job stages attempts - -> Introduced in GitLab, it requires GitLab Runner v1.9+. - -You can set the number of attempts that the running job tries to execute -the following stages: - -| Variable | Description | -|-----------------------------------|--------------------------------------------------------| -| **ARTIFACT_DOWNLOAD_ATTEMPTS** | Number of attempts to download artifacts running a job | -| **EXECUTOR_JOB_SECTION_ATTEMPTS** | [In GitLab 12.10](https://gitlab.com/gitlab-org/gitlab-runner/-/issues/4450) and later, the number of attempts to run a section in a job after a [`No Such Container`](https://gitlab.com/gitlab-org/gitlab-runner/-/issues/4450) error ([Docker executor](https://docs.gitlab.com/runner/executors/docker.html) only). | -| **GET_SOURCES_ATTEMPTS** | Number of attempts to fetch sources running a job | -| **RESTORE_CACHE_ATTEMPTS** | Number of attempts to restore the cache running a job | - -The default is one single attempt. - -Example: - -```yaml -variables: - GET_SOURCES_ATTEMPTS: 3 -``` - -You can set them globally or per-job in the [`variables`](#variables) section. - -### Fallback cache key - -> [Introduced](https://gitlab.com/gitlab-org/gitlab-runner/-/merge_requests/1534) in GitLab Runner 13.4. - -You can use the `$CI_COMMIT_REF_SLUG` variable to specify your [`cache:key`](#cachekey). -For example, if your `$CI_COMMIT_REF_SLUG` is `test` you can set a job -to download cache that's tagged with `test`. - -If a cache with this tag is not found, you can use `CACHE_FALLBACK_KEY` to -specify a cache to use when none exists. - -For example: - -```yaml -variables: - CACHE_FALLBACK_KEY: fallback-key - -cache: - key: "$CI_COMMIT_REF_SLUG" - paths: - - binaries/ -``` - -In this example, if the `$CI_COMMIT_REF_SLUG` is not found, the job uses the key defined -by the `CACHE_FALLBACK_KEY` variable. - -### Shallow cloning - -> Introduced in GitLab 8.9 as an experimental feature. - -You can specify the depth of fetching and cloning using `GIT_DEPTH`. -`GIT_DEPTH` does a shallow clone of the repository and can significantly speed up cloning. -It can be helpful for repositories with a large number of commits or old, large binaries. The value is -passed to `git fetch` and `git clone`. - -In GitLab 12.0 and later, newly-created projects automatically have a -[default `git depth` value of `50`](../pipelines/settings.md#git-shallow-clone). - -If you use a depth of `1` and have a queue of jobs or retry -jobs, jobs may fail. - -Git fetching and cloning is based on a ref, such as a branch name, so runners -can't clone a specific commit SHA. If multiple jobs are in the queue, or -you're retrying an old job, the commit to be tested must be within the -Git history that is cloned. Setting too small a value for `GIT_DEPTH` can make -it impossible to run these old commits and `unresolved reference` is displayed in -job logs. You should then reconsider changing `GIT_DEPTH` to a higher value. - -Jobs that rely on `git describe` may not work correctly when `GIT_DEPTH` is -set since only part of the Git history is present. - -To fetch or clone only the last 3 commits: - -```yaml -variables: - GIT_DEPTH: "3" -``` - -You can set it globally or per-job in the [`variables`](#variables) section. - -### Custom build directories +You can use [YAML anchors for variables](#yaml-anchors-for-variables). -> [Introduced](https://gitlab.com/gitlab-org/gitlab-runner/-/issues/2211) in GitLab Runner 11.10. +### Configure runner behavior with variables -By default, GitLab Runner clones the repository in a unique subpath of the -`$CI_BUILDS_DIR` directory. However, your project might require the code in a -specific directory (Go projects, for example). In that case, you can specify -the `GIT_CLONE_PATH` variable to tell the runner the directory to clone the -repository in: +You can use [CI/CD variables](../variables/README.md) to configure runner Git behavior: -```yaml -variables: - GIT_CLONE_PATH: $CI_BUILDS_DIR/project-name - -test: - script: - - pwd -``` - -The `GIT_CLONE_PATH` has to always be within `$CI_BUILDS_DIR`. The directory set in `$CI_BUILDS_DIR` -is dependent on executor and configuration of [runners.builds_dir](https://docs.gitlab.com/runner/configuration/advanced-configuration.html#the-runners-section) -setting. +- [`GIT_STRATEGY`](../runners/README.md#git-strategy) +- [`GIT_SUBMODULE_STRATEGY`](../runners/README.md#git-submodule-strategy) +- [`GIT_CHECKOUT`](../runners/README.md#git-checkout) +- [`GIT_CLEAN_FLAGS`](../runners/README.md#git-clean-flags) +- [`GIT_FETCH_EXTRA_FLAGS`](../runners/README.md#git-fetch-extra-flags) +- [`GIT_DEPTH`](../runners/README.md#shallow-cloning) (shallow cloning) +- [`GIT_CLONE_PATH`](../runners/README.md#custom-build-directories) (custom build directories) -This can only be used when `custom_build_dir` is enabled in the -[runner's configuration](https://docs.gitlab.com/runner/configuration/advanced-configuration.html#the-runnerscustom_build_dir-section). -This is the default configuration for the `docker` and `kubernetes` executors. - -#### Handling concurrency - -An executor that uses a concurrency greater than `1` might lead -to failures. Multiple jobs might be working on the same directory if the `builds_dir` -is shared between jobs. - -The runner does not try to prevent this situation. It's up to the administrator -and developers to comply with the requirements of runner configuration. - -To avoid this scenario, you can use a unique path within `$CI_BUILDS_DIR`, because runner -exposes two additional variables that provide a unique `ID` of concurrency: - -- `$CI_CONCURRENT_ID`: Unique ID for all jobs running within the given executor. -- `$CI_CONCURRENT_PROJECT_ID`: Unique ID for all jobs running within the given executor and project. - -The most stable configuration that should work well in any scenario and on any executor -is to use `$CI_CONCURRENT_ID` in the `GIT_CLONE_PATH`. For example: - -```yaml -variables: - GIT_CLONE_PATH: $CI_BUILDS_DIR/$CI_CONCURRENT_ID/project-name - -test: - script: - - pwd -``` - -The `$CI_CONCURRENT_PROJECT_ID` should be used in conjunction with `$CI_PROJECT_PATH` -as the `$CI_PROJECT_PATH` provides a path of a repository. That is, `group/subgroup/project`. For example: - -```yaml -variables: - GIT_CLONE_PATH: $CI_BUILDS_DIR/$CI_CONCURRENT_ID/$CI_PROJECT_PATH - -test: - script: - - pwd -``` - -#### Nested paths - -The value of `GIT_CLONE_PATH` is expanded once and nesting variables -within is not supported. - -For example, you define both the variables below in your -`.gitlab-ci.yml` file: - -```yaml -variables: - GOPATH: $CI_BUILDS_DIR/go - GIT_CLONE_PATH: $GOPATH/src/namespace/project -``` - -The value of `GIT_CLONE_PATH` is expanded once into -`$CI_BUILDS_DIR/go/src/namespace/project`, and results in failure -because `$CI_BUILDS_DIR` is not expanded. +You can also use variables to configure how many times a runner +[attempts certain stages of job execution](../runners/README.md#job-stages-attempts). ## Special YAML features @@ -4573,7 +4204,7 @@ feature. Anchors are only valid within the file they were defined in. Instead of using YAML anchors, you can use the [`extends` keyword](#extends). The following example uses anchors and map merging. It creates two jobs, -`test1` and `test2`, that inherit the parameters of `.job_template`, each +`test1` and `test2`, that inherit the `.job_template` configuration, each with their own custom `script` defined: ```yaml @@ -4696,50 +4327,30 @@ test:mysql: You can see that the hidden jobs are conveniently used as templates, and `tags: [dev]` has been overwritten by `tags: [postgres]`. -#### YAML anchors for `before_script` and `after_script` +#### YAML anchors for scripts > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/23005) in GitLab 12.5. -You can use [YAML anchors](#anchors) with `before_script` and `after_script`, -which makes it possible to include a predefined list of commands in multiple -jobs. - -Example: +You can use [YAML anchors](#anchors) with [script](#script), [`before_script`](#before_script), +and [`after_script`](#after_script) to use predefined commands in multiple jobs: ```yaml -.something_before: &something_before - - echo 'something before' - -.something_after: &something_after - - echo 'something after' - - echo 'another thing after' - -job_name: - before_script: - - *something_before - script: - - echo 'this is the script' - after_script: - - *something_after -``` - -#### YAML anchors for `script` +.some-script: &some-script + - echo "Execute this in `before_script` sections" -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/23005) in GitLab 12.5. +.some-script-before: &some-script-before + - echo "Execute this in `script` sections" -You can use [YAML anchors](#anchors) with scripts, which makes it possible to -include a predefined list of commands in multiple jobs. - -For example: - -```yaml -.something: &something - - echo 'something' +.some-script-after: &some-script-after + - echo "Execute this in `after_script` sections" job_name: + before_script: + - *some-script-before script: - - *something - - echo 'this is the script' + - *some-script + before_script: + - *some-script-after ``` #### YAML anchors for variables @@ -4809,19 +4420,19 @@ This limitation does not affect any of the updated merge request pipelines. All updated merge requests have a pipeline created when using [pipelines for merge requests](../merge_request_pipelines/index.md). -## Deprecated parameters +## Deprecated keywords -The following parameters are deprecated. +The following keywords are deprecated. ### Globally-defined `types` -CAUTION: **Deprecated:** +DANGER: **Deprecated:** `types` is deprecated, and could be removed in a future release. Use [`stages`](#stages) instead. ### Job-defined `type` -CAUTION: **Deprecated:** +DANGER: **Deprecated:** `type` is deprecated, and could be removed in one of the future releases. Use [`stage`](#stage) instead. diff --git a/doc/ci/yaml/script.md b/doc/ci/yaml/script.md new file mode 100644 index 00000000000..87885c8e548 --- /dev/null +++ b/doc/ci/yaml/script.md @@ -0,0 +1,147 @@ +--- +stage: Verify +group: Continuous Integration +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +type: reference +--- + +# GitLab CI/CD script syntax + +You can use special syntax in [`script`](README.md#script) sections to: + +- [Split long commands](#split-long-commands) into multiline commands. +- [Use color codes](#add-color-codes-to-script-output) to make job logs easier to review. +- [Create custom collapsible sections](../jobs/index.md#custom-collapsible-sections) + to simplify job log output. + +## Split long commands + +You can split long commands into multiline commands to improve readability with +`|` (literal) and `>` (folded) [YAML multiline block scalar indicators](https://yaml-multiline.info/). + +CAUTION: **Warning:** +If multiple commands are combined into one command string, only the last command's +failure or success is reported. +[Failures from earlier commands are ignored due to a bug](https://gitlab.com/gitlab-org/gitlab-runner/-/issues/25394). +To work around this, run each command as a separate `script:` item, or add an `exit 1` +command to each command string. + +You can use the `|` (literal) YAML multiline block scalar indicator to write +commands over multiple lines in the `script` section of a job description. +Each line is treated as a separate command. +Only the first command is repeated in the job log, but additional +commands are still executed: + +```yaml +job: + script: + - | + echo "First command line." + echo "Second command line." + echo "Third command line." +``` + +The example above renders in the job log as: + +```shell +$ echo First command line # collapsed multiline command +First command line +Second command line. +Third command line. +``` + +The `>` (folded) YAML multiline block scalar indicator treats empty lines between +sections as the start of a new command: + +```yaml +job: + script: + - > + echo "First command line + is split over two lines." + + echo "Second command line." +``` + +This behaves similarly to multiline commands without the `>` or `|` block +scalar indicators: + +```yaml +job: + script: + - echo "First command line + is split over two lines." + + echo "Second command line." +``` + +Both examples above render in the job log as: + +```shell +$ echo First command line is split over two lines. # collapsed multiline command +First command line is split over two lines. +Second command line. +``` + +When you omit the `>` or `|` block scalar indicators, GitLab concatenates non-empty +lines to form the command. Make sure the lines can run when concatenated. + +[Shell here documents](https://en.wikipedia.org/wiki/Here_document) work with the +`|` and `>` operators as well. The example below transliterates lower case letters +to upper case: + +```yaml +job: + script: + - | + tr a-z A-Z << END_TEXT + one two three + four five six + END_TEXT +``` + +Results in: + +```shell +$ tr a-z A-Z << END_TEXT # collapsed multiline command + ONE TWO THREE + FOUR FIVE SIX +``` + +## Add color codes to script output + +Script output can be colored using [ANSI escape codes](https://en.wikipedia.org/wiki/ANSI_escape_code#Colors), +or by running commands or programs that output ANSI escape codes. + +For example, using [Bash with color codes](https://misc.flogisoft.com/bash/tip_colors_and_formatting): + +```yaml +job: + script: + - echo -e "\e[31mThis text is red,\e[0m but this text isn't\e[31m however this text is red again." +``` + +You can define the color codes in Shell variables, or even [custom environment variables](../variables/README.md#custom-environment-variables), +which makes the commands easier to read and reusable. + +For example, using the same example as above and variables defined in a `before_script`: + +```yaml +job: + before_script: + - TXT_RED="\e[31m" && TXT_CLEAR="\e[0m" + script: + - echo -e "${TXT_RED}This text is red,${TXT_CLEAR} but this part isn't${TXT_RED} however this part is again." + - echo "This text is not colored" +``` + +Or with [PowerShell color codes](https://superuser.com/a/1259916): + +```yaml +job: + before_script: + - $esc="$([char]27)"; $TXT_RED="$esc[31m"; $TXT_CLEAR="$esc[0m" + script: + - Write-Host $TXT_RED"This text is red,"$TXT_CLEAR" but this text isn't"$TXT_RED" however this text is red again." + - Write-Host "This text is not colored" +``` diff --git a/doc/ci/yaml/visualization.md b/doc/ci/yaml/visualization.md index ac5f5c8fd14..391ff9f852a 100644 --- a/doc/ci/yaml/visualization.md +++ b/doc/ci/yaml/visualization.md @@ -1,3 +1,9 @@ +--- +stage: Verify +group: Pipeline Authoring +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +--- + # Visualize your CI/CD configuration > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/241722) in GitLab 13.5. |
