diff options
Diffstat (limited to 'doc/ci')
39 files changed, 1320 insertions, 987 deletions
diff --git a/doc/ci/README.md b/doc/ci/README.md index 43e66adbd35..6f15fa52550 100644 --- a/doc/ci/README.md +++ b/doc/ci/README.md @@ -63,7 +63,7 @@ the following documents: If you're migrating from another CI/CD tool, check out our handy references: - [Migrating from CircleCI](migration/circleci.md) -- [Migrating from Jenkins](jenkins/index.md) +- [Migrating from Jenkins](migration/jenkins.md) You can also get started by using one of the [`.gitlab-ci.yml` templates](https://gitlab.com/gitlab-org/gitlab-foss/tree/master/lib/gitlab/ci/templates) diff --git a/doc/ci/ci_cd_for_external_repos/github_integration.md b/doc/ci/ci_cd_for_external_repos/github_integration.md index 5365ef49944..dc1135742ea 100644 --- a/doc/ci/ci_cd_for_external_repos/github_integration.md +++ b/doc/ci/ci_cd_for_external_repos/github_integration.md @@ -23,7 +23,7 @@ cannot be used to authenticate with GitHub as an external CI/CD repository. NOTE: **Note:** Personal access tokens can only be used to connect GitHub.com -repositories to GitLab, and the GitHub user must have the [owner role](https://help.github.com/en/github/getting-started-with-github/access-permissions-on-github). +repositories to GitLab, and the GitHub user must have the [owner role](https://docs.github.com/en/github/getting-started-with-github/access-permissions-on-github). To perform a one-off authorization with GitHub to grant GitLab access your repositories: diff --git a/doc/ci/cloud_deployment/index.md b/doc/ci/cloud_deployment/index.md index 29ce8bdf625..355bc7813d9 100644 --- a/doc/ci/cloud_deployment/index.md +++ b/doc/ci/cloud_deployment/index.md @@ -81,11 +81,12 @@ path to point to your ECR image. ### Deploy your application to the AWS Elastic Container Service (ECS) -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/207962) in GitLab 12.9. +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/207962) in GitLab 12.9. +> - The `Deploy-ECS.gitlab-ci.yml` template was [moved](https://gitlab.com/gitlab-org/gitlab/-/issues/220821) to `AWS/Deploy-ECS.gitlab-ci.yml` in GitLab 13.2. GitLab provides a series of [CI templates that you can include in your project](../yaml/README.md#include). To automate deployments of your application to your [Amazon Elastic Container Service](https://aws.amazon.com/ecs/) (AWS ECS) -cluster, you can `include` the `Deploy-ECS.gitlab-ci.yml` template in your `.gitlab-ci.yml` file. +cluster, you can `include` the `AWS/Deploy-ECS.gitlab-ci.yml` template in your `.gitlab-ci.yml` file. GitLab also provides [Docker images](https://gitlab.com/gitlab-org/cloud-deploy/-/tree/master/aws) that can be used in your `gitlab-ci.yml` file to simplify working with AWS: @@ -96,11 +97,25 @@ Before getting started with this process, you need a cluster on AWS ECS, as well components, like an ECS service, ECS task definition, a database on AWS RDS, etc. [Read more about AWS ECS](https://docs.aws.amazon.com/AmazonECS/latest/developerguide/Welcome.html). -After you're all set up on AWS ECS, follow these steps: +The ECS task definition can be: + +- An existing task definition in AWS ECS +- A JSON file containing a task definition. Create the JSON file by using the template provided in + the [AWS documentation](https://docs.aws.amazon.com/AmazonECS/latest/developerguide/create-task-definition.html#task-definition-template). + Copy the task definition into a new file in your project, for example `<project-root>/ci/aws/task-definition.json`. + [Available](https://gitlab.com/gitlab-org/gitlab/-/issues/222618) in GitLab 13.3 and later. + +After you have these prerequisites ready, follow these steps: 1. Make sure your AWS credentials are set up as environment variables for your project. You can follow [the steps above](#run-aws-commands-from-gitlab-cicd) to complete this setup. -1. Add these variables to your project's `.gitlab-ci.yml` file: +1. Add these variables to your project's `.gitlab-ci.yml` file, or in the project's + [CI/CD settings](../variables/README.md#create-a-custom-variable-in-the-ui): + + - `CI_AWS_ECS_CLUSTER`: The name of the AWS ECS cluster that you're targeting for your deployments. + - `CI_AWS_ECS_SERVICE`: The name of the targeted service tied to your AWS ECS cluster. + - `CI_AWS_ECS_TASK_DEFINITION`: The name of an existing task definition in ECS tied + to the service mentioned above. ```yaml variables: @@ -109,19 +124,37 @@ After you're all set up on AWS ECS, follow these steps: CI_AWS_ECS_TASK_DEFINITION: my-task-definition ``` - Three variables are defined in this snippet: - - - `CI_AWS_ECS_CLUSTER`: The name of your AWS ECS cluster that you're - targeting for your deployments. - - `CI_AWS_ECS_SERVICE`: The name of the targeted service tied to - your AWS ECS cluster. - - `CI_AWS_ECS_TASK_DEFINITION`: The name of the task definition tied - to the service mentioned above. - You can find these names after selecting the targeted cluster on your [AWS ECS dashboard](https://console.aws.amazon.com/ecs/home):  + Alternatively, if you want to use a task definition defined in a JSON file, use + `CI_AWS_ECS_TASK_DEFINITION_FILE` instead: + + ```yaml + variables: + CI_AWS_ECS_CLUSTER: my-cluster + CI_AWS_ECS_SERVICE: my-service + CI_AWS_ECS_TASK_DEFINITION_FILE: ci/aws/my_task_definition.json + ``` + + You can create your `CI_AWS_ECS_TASK_DEFINITION_FILE` variable as a + [file-typed environment variable](../variables/README.md#custom-environment-variables-of-type-file) instead of a + regular environment variable. If you choose to do so, set the variable value to be the full contents of + the JSON task definition. You can then remove the JSON file from your project. + + In both cases, make sure that the value for the `containerDefinitions[].name` attribute is + the same as the `Container name` defined in your targeted ECS service. + + CAUTION: **Warning:** + `CI_AWS_ECS_TASK_DEFINITION_FILE` takes precedence over `CI_AWS_ECS_TASK_DEFINITION` if both these environment + variables are defined within your project. + + NOTE: **Note:** + If the name of the task definition you wrote in your JSON file is the same name + as an existing task definition on AWS, then a new revision is created for it. + Otherwise, a brand new task definition is created, starting at revision 1. + 1. Include this template in `.gitlab-ci.yml`: ```yaml @@ -129,12 +162,15 @@ After you're all set up on AWS ECS, follow these steps: - template: AWS/Deploy-ECS.gitlab-ci.yml ``` - The `Deploy-ECS` template ships with GitLab and is available [on + The `AWS/Deploy-ECS` template ships with GitLab and is available [on GitLab.com](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/ci/templates/AWS/Deploy-ECS.gitlab-ci.yml). 1. Commit and push your updated `.gitlab-ci.yml` to your project's repository, and you're done! Your application Docker image will be rebuilt and pushed to the GitLab registry. + If your image is located in a private registry, make sure your task definition is + [configured with a `repositoryCredentials` attribute](https://docs.aws.amazon.com/AmazonECS/latest/developerguide/private-auth.html). + Then the targeted task definition will be updated with the location of the new Docker image, and a new revision will be created in ECS as result. @@ -143,17 +179,17 @@ After you're all set up on AWS ECS, follow these steps: application. CAUTION: **Warning:** -The [`Deploy-ECS.gitlab-ci.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/ci/templates/AWS/Deploy-ECS.gitlab-ci.yml) +The [`AWS/Deploy-ECS.gitlab-ci.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/ci/templates/AWS/Deploy-ECS.gitlab-ci.yml) template includes both the [`Jobs/Build.gitlab-ci.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/ci/templates/Jobs/Build.gitlab-ci.yml) and [`Jobs/Deploy/ECS.gitlab-ci.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/ci/templates/Jobs/Deploy/ECS.gitlab-ci.yml) "sub-templates". Do not include these "sub-templates" on their own, and only include the main -`Deploy-ECS.gitlab-ci.yml` template. The "sub-templates" are designed to only be +`AWS/Deploy-ECS.gitlab-ci.yml` template. The "sub-templates" are designed to only be used along with the main template. They may move or change unexpectedly causing your pipeline to fail if you didn't include the main template. Also, the job names within these templates may change. Do not override these jobs names in your own pipeline, as the override will stop working when the name changes. -Alternatively, if you don't wish to use the `Deploy-ECS.gitlab-ci.yml` template +Alternatively, if you don't wish to use the `AWS/Deploy-ECS.gitlab-ci.yml` template to deploy to AWS ECS, you can always use our `aws-base` Docker image to run your own [AWS CLI commands for ECS](https://docs.aws.amazon.com/cli/latest/reference/ecs/index.html#cli-aws-ecs). diff --git a/doc/ci/directed_acyclic_graph/index.md b/doc/ci/directed_acyclic_graph/index.md index b7dd74a0230..8fc58df51fe 100644 --- a/doc/ci/directed_acyclic_graph/index.md +++ b/doc/ci/directed_acyclic_graph/index.md @@ -84,6 +84,7 @@ are certain use cases that you may need to work around. For more information: > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/215517) in GitLab 13.1 as a [Beta feature](https://about.gitlab.com/handbook/product/#beta). > - It was deployed behind a feature flag, disabled by default. > - It became [enabled by default](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/36802) in 13.2. +> - It became a [standard feature](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/38517) in 13.3. > - It's enabled on GitLab.com. > - For GitLab self-managed instances, GitLab administrators can opt to [disable it](#enable-or-disable-dag-visualization-core-only). @@ -97,9 +98,7 @@ Clicking a node will highlight all the job paths it depends on. ### Enable or disable DAG Visualization **(CORE ONLY)** -DAG Visualization is under development, but is being made available as a beta feature so users can check its limitations and uses. - -It is deployed behind a feature flag that is **enabled by default**. +DAG Visualization is deployed behind a feature flag that is **enabled by default**. [GitLab administrators with access to the GitLab Rails console](../../administration/feature_flags.md) can opt to disable it for your instance: diff --git a/doc/ci/docker/using_docker_build.md b/doc/ci/docker/using_docker_build.md index 4bed6d9e323..88d6dc3aae4 100644 --- a/doc/ci/docker/using_docker_build.md +++ b/doc/ci/docker/using_docker_build.md @@ -32,7 +32,7 @@ during jobs. ## Runner Configuration There are three methods to enable the use of `docker build` and `docker run` -during jobs; each with their own tradeoffs. +during jobs, each with their own tradeoffs. An alternative to using `docker build` is to [use kaniko](using_kaniko.md). This avoids having to execute Runner in privileged mode. @@ -61,8 +61,8 @@ GitLab Runner then executes job scripts as the `gitlab-runner` user. 1. Install Docker Engine on server. - For more information how to install Docker Engine on different systems - checkout the [Supported installations](https://docs.docker.com/engine/installation/). + For more information how to install Docker Engine on different systems, + check out the [Supported installations](https://docs.docker.com/engine/installation/). 1. Add `gitlab-runner` user to `docker` group: @@ -118,13 +118,13 @@ not without its own challenges: - When using Docker-in-Docker, each job is in a clean environment without the past history. Concurrent jobs work fine because every build gets its own - instance of Docker engine so they won't conflict with each other. But this + instance of Docker engine so they don't conflict with each other. But this also means that jobs can be slower because there's no caching of layers. - By default, Docker 17.09 and higher uses `--storage-driver overlay2` which is the recommended storage driver. See [Using the overlayfs driver](#use-the-overlayfs-driver) for details. - Since the `docker:19.03.12-dind` container and the Runner container don't share their - root filesystem, the job's working directory can be used as a mount point for + root file system, the job's working directory can be used as a mount point for child containers. For example, if you have files you want to share with a child container, you may create a subdirectory under `/builds/$CI_PROJECT_PATH` and use it as your mount point (for a more thorough explanation, check [issue @@ -143,8 +143,8 @@ An example project using this approach can be found here: <https://gitlab.com/gi In the examples below, we are using Docker images tags to specify a specific version, such as `docker:19.03.12`. If tags like `docker:stable` -are used, you have no control over what version is going to be used and this -can lead to unpredictable behavior, especially when new versions are +are used, you have no control over what version is used. This can lead to +unpredictable behavior, especially when new versions are released. #### TLS enabled @@ -179,18 +179,18 @@ support this. --docker-volumes "/certs/client" ``` - The above command will register a new Runner to use the special + The above command registers a new Runner to use the special `docker:19.03.12` image, which is provided by Docker. **Notice that it's using the `privileged` mode to start the build and service containers.** If you want to use [Docker-in-Docker](https://www.docker.com/blog/docker-can-now-run-within-docker/) mode, you always have to use `privileged = true` in your Docker containers. - This will also mount `/certs/client` for the service and build + This also mounts `/certs/client` for the service and build container, which is needed for the Docker client to use the - certificates inside of that directory. For more information how - Docker with TLS works check <https://hub.docker.com/_/docker/#tls>. + certificates inside of that directory. For more information on how + Docker with TLS works, check <https://hub.docker.com/_/docker/#tls>. - The above command will create a `config.toml` entry similar to this: + The above command creates a `config.toml` entry similar to this: ```toml [[runners]] @@ -215,7 +215,7 @@ support this. image: docker:19.03.12 variables: - # When using dind service, we need to instruct docker, to talk with + # When using dind service, we need to instruct docker to talk with # the daemon started inside of the service. The daemon is available # with a network connection instead of the default # /var/run/docker.sock socket. Docker 19.03 does this automatically @@ -337,13 +337,13 @@ In order to do that, follow the steps: --docker-volumes /var/run/docker.sock:/var/run/docker.sock ``` - The above command will register a new Runner to use the special + The above command registers a new Runner to use the special `docker:19.03.12` image which is provided by Docker. **Notice that it's using the Docker daemon of the Runner itself, and any containers spawned by Docker - commands will be siblings of the Runner rather than children of the Runner.** + commands are siblings of the Runner rather than children of the Runner.** This may have complications and limitations that are unsuitable for your workflow. - The above command will create a `config.toml` entry similar to this: + The above command creates a `config.toml` entry similar to this: ```toml [[runners]] @@ -387,7 +387,7 @@ aware of the following implications: containers. - Concurrent jobs may not work; if your tests create containers with specific names, they may conflict with each other. -- Sharing files and directories from the source repo into containers may not +- Sharing files and directories from the source repository into containers may not work as expected since volume mounting is done in the context of the host machine, not the build container. For example: @@ -397,7 +397,7 @@ aware of the following implications: ## Making Docker-in-Docker builds faster with Docker layer caching -When using Docker-in-Docker, Docker will download all layers of your image every +When using Docker-in-Docker, Docker downloads all layers of your image every time you create a build. Recent versions of Docker (Docker 1.13 and above) can use a pre-existing image as a cache during the `docker build` step, considerably speeding up the build process. @@ -504,7 +504,7 @@ environment variable in the environment = ["DOCKER_DRIVER=overlay2"] ``` -If you're running multiple Runners you will have to modify all configuration files. +If you're running multiple Runners, you have to modify all configuration files. NOTE: **Note:** Read more about the [Runner configuration](https://docs.gitlab.com/runner/configuration/) @@ -523,7 +523,7 @@ This is a common error when you are using [Docker in Docker](#use-docker-in-docker-workflow-with-docker-executor) v19.03 or higher. -This occurs because Docker starts on TLS automatically, so you need to do some set up. +This occurs because Docker starts on TLS automatically, so you need to do some setup. If: - This is the first time setting it up, carefully read diff --git a/doc/ci/docker/using_docker_images.md b/doc/ci/docker/using_docker_images.md index 735cf35584f..db39532bbf2 100644 --- a/doc/ci/docker/using_docker_images.md +++ b/doc/ci/docker/using_docker_images.md @@ -16,12 +16,12 @@ instance. [Docker Hub](https://hub.docker.com/) has a rich database of pre-built used to test and build your applications. When used with GitLab CI/CD, Docker runs each job in a separate and isolated -container using the predefined image that is set up in +container using the predefined image that's set up in [`.gitlab-ci.yml`](../yaml/README.md). This makes it easier to have a simple and reproducible build environment that can also run on your workstation. The added benefit is that you can test all -the commands that we will explore later from your shell, rather than having to +the commands that we explore later from your shell, rather than having to test them on a dedicated CI server. ## Register Docker Runner @@ -54,16 +54,16 @@ sudo gitlab-runner register \ --docker-image ruby:2.6 ``` -The registered runner will use the `ruby:2.6` Docker image and will run two -services, `postgres:latest` and `mysql:latest`, both of which will be +The registered runner uses the `ruby:2.6` Docker image and runs two +services, `postgres:latest` and `mysql:latest`, both of which are accessible during the build process. ## What is an image The `image` keyword is the name of the Docker image the Docker executor -will run to perform the CI tasks. +runs to perform the CI tasks. -By default, the executor will only pull images from [Docker Hub](https://hub.docker.com/), +By default, the executor only pulls images from [Docker Hub](https://hub.docker.com/), however this can be configured in the `gitlab-runner/config.toml` by setting the [Docker pull policy](https://docs.gitlab.com/runner/executors/docker.html#how-pull-policies-work) to allow using local images. @@ -72,16 +72,16 @@ the [Docker Fundamentals](https://docs.docker.com/engine/understanding-docker/) ## What is a service -The `services` keyword defines just another Docker image that is run during +The `services` keyword defines just another Docker image that's run during your job and is linked to the Docker image that the `image` keyword defines. This allows you to access the service image during build time. The service image can run any application, but the most common use case is to -run a database container, e.g., `mysql`. It's easier and faster to use an +run a database container, for example, `mysql`. It's easier and faster to use an existing image and run it as an additional container than install `mysql` every time the project is built. -You are not limited to have only database services. You can add as many +You're not limited to have only database services. You can add as many services you need to `.gitlab-ci.yml` or manually modify `config.toml`. Any image found at [Docker Hub](https://hub.docker.com/) or your private Container Registry can be used as a service. @@ -97,10 +97,10 @@ You can see some widely used services examples in the relevant documentation of To better understand how the container linking works, read [Linking containers together](https://docs.docker.com/engine/userguide/networking/default_network/dockerlinks/). -To summarize, if you add `mysql` as service to your application, the image will -then be used to create a container that is linked to the job container. +To summarize, if you add `mysql` as service to your application, the image is +then used to create a container that's linked to the job container. -The service container for MySQL will be accessible under the hostname `mysql`. +The service container for MySQL is accessible under the hostname `mysql`. So, in order to access your database service you have to connect to the host named `mysql` instead of a socket or `localhost`. Read more in [accessing the services](#accessing-the-services). @@ -109,7 +109,7 @@ services](#accessing-the-services). Services are designed to provide additional functionality which is **network accessible**. It may be a database like MySQL, or Redis, and even `docker:stable-dind` which -allows you to use Docker in Docker. It can be practically anything that is +allows you to use Docker in Docker. It can be practically anything that's required for the CI/CD job to proceed and is accessed by network. To make sure this works, the Runner: @@ -121,8 +121,8 @@ When the second stage of the check fails, either because there is no opened port service, or the service was not started properly before the timeout and the port is not responding, it prints the warning: `*** WARNING: Service XYZ probably didn't start properly`. -In most cases it will affect the job, but there may be situations when the job -will still succeed even if that warning was printed. For example: +In most cases it affects the job, but there may be situations when the job +still succeeds even if that warning was printed. For example: - The service was started a little after the warning was raised, and the job is not using the linked service from the beginning. In that case, when the @@ -130,8 +130,8 @@ will still succeed even if that warning was printed. For example: connections. - The service container is not providing any networking service, but it's doing something with the job's directory (all services have the job directory mounted - as a volume under `/builds`). In that case, the service will do its job, and - since the job is not trying to connect to it, it won't fail. + as a volume under `/builds`). In that case, the service does its job, and + since the job is not trying to connect to it, it does not fail. ### What services are not for @@ -139,12 +139,12 @@ As it was mentioned before, this feature is designed to provide **network access services. A database is the simplest example of such a service. NOTE: **Note:** -The services feature is not designed to, and will not add any software from the +The services feature is not designed to, and does not add any software from the defined `services` image(s) to the job's container. For example, if you have the following `services` defined in your job, the `php`, -`node` or `go` commands will **not** be available for your script, and thus -the job will fail: +`node` or `go` commands are **not** available for your script, and thus +the job fails: ```yaml job: @@ -163,7 +163,7 @@ If you need to have `php`, `node` and `go` available for your script, you should either: - Choose an existing Docker image that contains all required tools. -- Create your own Docker image, which will have all the required tools included +- Create your own Docker image, with all the required tools included and use that in your job. ### Accessing the services @@ -180,7 +180,7 @@ services: ``` If you don't [specify a service alias](#available-settings-for-services), -when the job is run, `tutum/wordpress` will be started and you will have +when the job is run, `tutum/wordpress` is started and you have access to it from your build container under two hostnames to choose from: - `tutum-wordpress` @@ -204,7 +204,7 @@ To override the default behavior, you can ## Define `image` and `services` from `.gitlab-ci.yml` -You can simply define an image that will be used for all jobs and a list of +You can simply define an image that's used for all jobs and a list of services that you want to use during build time: ```yaml @@ -228,7 +228,7 @@ The image name must be in one of the following formats: - `image: <image-name>:<tag>` - `image: <image-name>@<digest>` -It is also possible to define different images and services per job: +It's also possible to define different images and services per job: ```yaml default: @@ -280,7 +280,7 @@ to fine tune your Docker `images` and `services` directly in the `.gitlab-ci.yml For more information, see [custom environment variables](../variables/README.md#gitlab-ciyml-defined-variables) ```yaml -# The following variables will automatically be passed down to the Postgres container +# The following variables are automatically passed down to the Postgres container # as well as the Ruby container and available within each. variables: HTTPS_PROXY: "https://10.1.1.1:8090" @@ -353,7 +353,7 @@ For example, the following two definitions are equal: | Setting | Required | GitLab version | Description | |------------|----------|----------------| ----------- | | `name` | yes, when used with any other option | 9.4 |Full name of the image that should be used. It should contain the Registry part if needed. | -| `entrypoint` | no | 9.4 |Command or script that should be executed as the container's entrypoint. It will be translated to Docker's `--entrypoint` option while creating the container. The syntax is similar to [`Dockerfile`'s `ENTRYPOINT`](https://docs.docker.com/engine/reference/builder/#entrypoint) directive, where each shell token is a separate string in the array. | +| `entrypoint` | no | 9.4 |Command or script that should be executed as the container's entrypoint. It's translated to Docker's `--entrypoint` option while creating the container. The syntax is similar to [`Dockerfile`'s `ENTRYPOINT`](https://docs.docker.com/engine/reference/builder/#entrypoint) directive, where each shell token is a separate string in the array. | ### Available settings for `services` @@ -362,8 +362,8 @@ For example, the following two definitions are equal: | Setting | Required | GitLab version | Description | |------------|----------|----------------| ----------- | | `name` | yes, when used with any other option | 9.4 | Full name of the image that should be used. It should contain the Registry part if needed. | -| `entrypoint` | no | 9.4 |Command or script that should be executed as the container's entrypoint. It will be translated to Docker's `--entrypoint` option while creating the container. The syntax is similar to [`Dockerfile`'s `ENTRYPOINT`](https://docs.docker.com/engine/reference/builder/#entrypoint) directive, where each shell token is a separate string in the array. | -| `command` | no | 9.4 |Command or script that should be used as the container's command. It will be translated to arguments passed to Docker after the image's name. The syntax is similar to [`Dockerfile`'s `CMD`](https://docs.docker.com/engine/reference/builder/#cmd) directive, where each shell token is a separate string in the array. | +| `entrypoint` | no | 9.4 |Command or script that should be executed as the container's entrypoint. It's translated to Docker's `--entrypoint` option while creating the container. The syntax is similar to [`Dockerfile`'s `ENTRYPOINT`](https://docs.docker.com/engine/reference/builder/#entrypoint) directive, where each shell token is a separate string in the array. | +| `command` | no | 9.4 |Command or script that should be used as the container's command. It's translated to arguments passed to Docker after the image's name. The syntax is similar to [`Dockerfile`'s `CMD`](https://docs.docker.com/engine/reference/builder/#cmd) directive, where each shell token is a separate string in the array. | | `alias` | no | 9.4 |Additional alias that can be used to access the service from the job's container. Read [Accessing the services](#accessing-the-services) for more information. | NOTE: **Note:** @@ -398,8 +398,8 @@ services: alias: mysql-2 ``` -The Runner will still start two containers using the `mysql:latest` image, -however now each of them will also be accessible with the alias configured +The Runner still starts two containers using the `mysql:latest` image, +however now each of them are also accessible with the alias configured in `.gitlab-ci.yml` file. ### Setting a command for the service @@ -408,7 +408,7 @@ in `.gitlab-ci.yml` file. Let's assume you have a `super/sql:latest` image with some SQL database inside it and you would like to use it as a service for your job. Let's also -assume that this image doesn't start the database process while starting +assume that this image does not start the database process while starting the container and the user needs to manually use `/usr/bin/super-sql run` as a command to start the database. @@ -462,8 +462,8 @@ CI jobs: output. To override the entrypoint of a Docker image, the recommended solution is to -define an empty `entrypoint` in `.gitlab-ci.yml`, so the Runner doesn't start -a useless shell layer. However, that will not work for all Docker versions, and +define an empty `entrypoint` in `.gitlab-ci.yml`, so the Runner does not start +a useless shell layer. However, that does not work for all Docker versions, and you should check which one your Runner is using. Specifically: - If Docker 17.06 or later is used, the `entrypoint` can be set to an empty value. @@ -476,8 +476,8 @@ Let's assume you have a `super/sql:experimental` image with some SQL database inside it and you would like to use it as a base image for your job because you want to execute some tests with this database binary. Let's also assume that this image is configured with `/usr/bin/super-sql run` as an entrypoint. That -means that when starting the container without additional options, it will run -the database's process, while Runner expects that the image will have no +means that when starting the container without additional options, it runs +the database's process, while Runner expects that the image has no entrypoint or that the entrypoint is prepared to start a shell command. With the extended Docker configuration options, instead of creating your @@ -511,7 +511,7 @@ Look for the `[runners.docker]` section: services = ["mysql:latest", "postgres:latest"] ``` -The image and services defined this way will be added to all job run by +The image and services defined this way are added to all job run by that runner. ## Define an image from a private Container Registry @@ -530,12 +530,12 @@ To define which should be used, the GitLab Runner process reads the configuratio - `DOCKER_AUTH_CONFIG` variable provided as environment variable in `config.toml` of the Runner. - `config.json` file placed in `$HOME/.docker` directory of the user running GitLab Runner process. If the `--user` flag is provided to run the GitLab Runner child processes as unprivileged user, - the home directory of the main GitLab Runner process user will be used. + the home directory of the main GitLab Runner process user is used. NOTE: **Note:** GitLab Runner reads this configuration **only** from `config.toml` and ignores it if it's provided as an environment variable. This is because GitLab Runner uses **only** -`config.toml` configuration and doesn't interpolate **ANY** environment variables at +`config.toml` configuration and does not interpolate **ANY** environment variables at runtime. ### Requirements and limitations @@ -593,9 +593,9 @@ There are two ways to determine the value of `DOCKER_AUTH_CONFIG`: ``` - **Second way -** In some setups, it's possible that Docker client - will use the available system key store to store the result of `docker + uses the available system key store to store the result of `docker login`. In that case, it's impossible to read `~/.docker/config.json`, - so you will need to prepare the required base64-encoded version of + so you need to prepare the required base64-encoded version of `${username}:${password}` and create the Docker configuration JSON manually. Open a terminal and execute the following command: @@ -644,7 +644,7 @@ follow these steps: image: registry.example.com:5000/namespace/image:tag ``` - In the example above, GitLab Runner will look at `registry.example.com:5000` for the + In the example above, GitLab Runner looks at `registry.example.com:5000` for the image `namespace/image:tag`. You can add configuration for as many registries as you want, adding more @@ -655,19 +655,19 @@ The full `hostname:port` combination is required everywhere for the Runner to match the `DOCKER_AUTH_CONFIG`. For example, if `registry.example.com:5000/namespace/image:tag` is specified in `.gitlab-ci.yml`, then the `DOCKER_AUTH_CONFIG` must also specify `registry.example.com:5000`. -Specifying only `registry.example.com` will not work. +Specifying only `registry.example.com` does not work. ### Configuring a Runner -If you have many pipelines that access the same registry, it'll -probably be better to setup registry access at the runner level. This +If you have many pipelines that access the same registry, it is +probably better to set up registry access at the runner level. This allows pipeline authors to have access to a private registry just by running a job on the appropriate runner. It also makes registry changes and credential rotations much simpler. Of course this means that any job on that runner can access the registry with the same privilege, even across projects. If you need to -control access to the registry, you'll need to be sure to control +control access to the registry, you need to be sure to control access to the runner. To add `DOCKER_AUTH_CONFIG` to a Runner: @@ -713,14 +713,14 @@ To configure credentials store, follow these steps: } ``` - - Or, if you are running self-managed Runners, add the above JSON to - `${GITLAB_RUNNER_HOME}/.docker/config.json`. GitLab Runner will read this configuration file - and will use the needed helper for this specific repository. + - Or, if you're running self-managed Runners, add the above JSON to + `${GITLAB_RUNNER_HOME}/.docker/config.json`. GitLab Runner reads this configuration file + and uses the needed helper for this specific repository. NOTE: **Note:** `credsStore` is used to access ALL the registries. -If you will want to use both images from private registry and public images from DockerHub, -pulling from DockerHub will fail, because Docker daemon will try to use the same credentials for **ALL** the registries. +If you want to use both images from private registry and public images from DockerHub, +pulling from DockerHub would fail, because Docker daemon tries to use the same credentials for **ALL** the registries. ### Using Credential Helpers @@ -762,9 +762,9 @@ To configure access for `aws_account_id.dkr.ecr.region.amazonaws.com`, follow th This configures Docker to use the credential helper for all Amazon ECR registries. - - Or, if you are running self-managed Runners, + - Or, if you're running self-managed Runners, add the above JSON to `${GITLAB_RUNNER_HOME}/.docker/config.json`. - GitLab Runner will read this configuration file and will use the needed helper for this + GitLab Runner reads this configuration file and uses the needed helper for this specific repository. 1. You can now use any private image from `aws_account_id.dkr.ecr.region.amazonaws.com` defined in @@ -774,7 +774,7 @@ To configure access for `aws_account_id.dkr.ecr.region.amazonaws.com`, follow th image: aws_account_id.dkr.ecr.region.amazonaws.com/private/image:latest ``` - In the example above, GitLab Runner will look at `aws_account_id.dkr.ecr.region.amazonaws.com` for the + In the example above, GitLab Runner looks at `aws_account_id.dkr.ecr.region.amazonaws.com` for the image `private/image:latest`. You can add configuration for as many registries as you want, adding more @@ -792,7 +792,7 @@ For all possible configuration variables check the documentation of each image provided in their corresponding Docker hub page. NOTE: **Note:** -All variables will be passed to all services containers. It's not +All variables are passed to all services containers. It's not designed to distinguish which variable should go where. ### PostgreSQL service example @@ -838,7 +838,7 @@ EOF ``` Here we use as an example the GitLab Runner repository which contains a -Makefile, so running `make` will execute the commands defined in the Makefile. +Makefile, so running `make` executes the commands defined in the Makefile. Your mileage may vary, so instead of `make` you could run the command which is specific to your project. @@ -849,9 +849,9 @@ docker run -d --name service-mysql mysql:latest docker run -d --name service-postgres postgres:latest ``` -This will create two service containers, named `service-mysql` and +This creates two service containers, named `service-mysql` and `service-postgres` which use the latest MySQL and PostgreSQL images -respectively. They will both run in the background (`-d`). +respectively. They both run in the background (`-d`). Finally, create a build container by executing the `build_script` file we created earlier: @@ -860,7 +860,7 @@ created earlier: docker run --name build -i --link=service-mysql:mysql --link=service-postgres:postgres ruby:2.6 /bin/bash < build_script ``` -The above command will create a container named `build` that is spawned from +The above command creates a container named `build` that's spawned from the `ruby:2.6` image and has two services linked to it. The `build_script` is piped using STDIN to the bash interpreter which in turn executes the `build_script` in the `build` container. @@ -872,6 +872,6 @@ with: docker rm -f -v build service-mysql service-postgres ``` -This will forcefully (`-f`) remove the `build` container, the two service +This forcefully (`-f`) removes the `build` container, the two service containers as well as all volumes (`-v`) that were created with the container creation. diff --git a/doc/ci/docker/using_kaniko.md b/doc/ci/docker/using_kaniko.md index 1580080ac6e..41c8f04f66e 100644 --- a/doc/ci/docker/using_kaniko.md +++ b/doc/ci/docker/using_kaniko.md @@ -63,6 +63,7 @@ build: name: gcr.io/kaniko-project/executor:debug entrypoint: [""] script: + - mkdir -p /kaniko/.docker - echo "{\"auths\":{\"$CI_REGISTRY\":{\"username\":\"$CI_REGISTRY_USER\",\"password\":\"$CI_REGISTRY_PASSWORD\"}}}" > /kaniko/.docker/config.json - /kaniko/executor --context $CI_PROJECT_DIR --dockerfile $CI_PROJECT_DIR/Dockerfile --destination $CI_REGISTRY_IMAGE:$CI_COMMIT_TAG only: diff --git a/doc/ci/environments/deployment_safety.md b/doc/ci/environments/deployment_safety.md index a4ff164a5ba..a0b7adb5279 100644 --- a/doc/ci/environments/deployment_safety.md +++ b/doc/ci/environments/deployment_safety.md @@ -1,3 +1,9 @@ +--- +stage: Release +group: Release Management +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +--- + # Deployment safety Deployment jobs can be more sensitive than other jobs in a pipeline, diff --git a/doc/ci/examples/authenticating-with-hashicorp-vault/img/vault-read-secrets-production.png b/doc/ci/examples/authenticating-with-hashicorp-vault/img/vault-read-secrets-production.png Binary files differindex 65c8546deb2..16c15071dd7 100644 --- a/doc/ci/examples/authenticating-with-hashicorp-vault/img/vault-read-secrets-production.png +++ b/doc/ci/examples/authenticating-with-hashicorp-vault/img/vault-read-secrets-production.png diff --git a/doc/ci/examples/authenticating-with-hashicorp-vault/img/vault-read-secrets-staging.png b/doc/ci/examples/authenticating-with-hashicorp-vault/img/vault-read-secrets-staging.png Binary files differindex 2399d8f8879..7a92a3c2d50 100644 --- a/doc/ci/examples/authenticating-with-hashicorp-vault/img/vault-read-secrets-staging.png +++ b/doc/ci/examples/authenticating-with-hashicorp-vault/img/vault-read-secrets-staging.png diff --git a/doc/ci/examples/laravel_with_gitlab_and_envoy/index.md b/doc/ci/examples/laravel_with_gitlab_and_envoy/index.md index ee0938b4d4c..c27566d38cf 100644 --- a/doc/ci/examples/laravel_with_gitlab_and_envoy/index.md +++ b/doc/ci/examples/laravel_with_gitlab_and_envoy/index.md @@ -150,7 +150,7 @@ Now, let's clone our repository on the server just to make sure the `deployer` u git clone git@gitlab.example.com:<USERNAME>/laravel-sample.git ``` ->**Note:** +NOTE: **Note:** Answer **yes** if asked `Are you sure you want to continue connecting (yes/no)?`. It adds GitLab.com to the known hosts. @@ -174,7 +174,7 @@ server { } ``` ->**Note:** +NOTE: **Note:** You may replace the app's name in `/var/www/app/current/public` with the folder name of your application. ## Setting up Envoy @@ -398,7 +398,7 @@ But let's take a step forward to do it automatically with [Continuous Delivery]( We need to check every commit with a set of automated tests to become aware of issues at the earliest, and then, we can deploy to the target environment if we are happy with the result of the tests. [GitLab CI/CD](../../README.md) allows us to use [Docker](https://www.docker.com) engine to handle the process of testing and deploying our app. -In the case you're not familiar with Docker, refer to [How to Automate Docker Deployments](http://paislee.io/how-to-automate-docker-deployments/). +In case you're not familiar with Docker, refer to [Set up automated builds](https://docs.docker.com/get-started/). To be able to build, test, and deploy our app with GitLab CI/CD, we need to prepare our work environment. To do that, we'll use a Docker image which has the minimum requirements that a Laravel app needs to run. @@ -464,14 +464,14 @@ docker build -t registry.gitlab.com/<USERNAME>/laravel-sample . docker push registry.gitlab.com/<USERNAME>/laravel-sample ``` ->**Note:** +NOTE: **Note:** To run the above commands, we first need to have [Docker](https://docs.docker.com/engine/installation/) installed on our machine. Congratulations! You just pushed the first Docker image to the GitLab Registry, and if you refresh the page you should be able to see it:  ->**Note:** +NOTE: **Note:** You can also [use GitLab CI/CD](https://about.gitlab.com/blog/2016/05/23/gitlab-container-registry/#use-with-gitlab-ci) to build and push your Docker images, rather than doing that on your machine. We'll use this image further down in the `.gitlab-ci.yml` configuration file to handle the process of testing and deploying our app. @@ -551,7 +551,7 @@ services: ... ``` ->**Note:** +NOTE: **Note:** If you wish to test your app with different PHP versions and [database management systems](../../services/README.md), you can define different `image` and `services` keywords for each test job. #### Variables diff --git a/doc/ci/examples/test_phoenix_app_with_gitlab_ci_cd/index.md b/doc/ci/examples/test_phoenix_app_with_gitlab_ci_cd/index.md index c901f6c9c66..f2620461c09 100644 --- a/doc/ci/examples/test_phoenix_app_with_gitlab_ci_cd/index.md +++ b/doc/ci/examples/test_phoenix_app_with_gitlab_ci_cd/index.md @@ -123,7 +123,7 @@ Generated hello_gitlab_ci app The database for HelloGitlabCi.Repo has been created ``` -> **Note:** +NOTE: **Note:** Phoenix assumes that our PostgreSQL database will have a `postgres` user account with the correct permissions and a password of `postgres`. If it's not your case, check [Ecto's instructions](https://hexdocs.pm/ecto/Ecto.html#module-repositories). @@ -211,7 +211,8 @@ when running our Phoenix in our `localhost`. Without `.gitkeep`, Git will not upload this empty directory and we'll got an error when running our test on GitLab. - > **Note:** If we add a folder via the GitLab UI, GitLab itself will add the `.gitkeep` to that new dir. + NOTE: **Note:** + If we add a folder via the GitLab UI, GitLab itself will add the `.gitkeep` to that new dir. Now, let's run a local test and see if everything we did didn't break anything. diff --git a/doc/ci/img/ci_lint.png b/doc/ci/img/ci_lint.png Binary files differnew file mode 100644 index 00000000000..e62de011293 --- /dev/null +++ b/doc/ci/img/ci_lint.png diff --git a/doc/ci/img/ci_lint_dry_run.png b/doc/ci/img/ci_lint_dry_run.png Binary files differnew file mode 100644 index 00000000000..4092b66d534 --- /dev/null +++ b/doc/ci/img/ci_lint_dry_run.png diff --git a/doc/ci/img/metrics_reports_advanced_v13_0.png b/doc/ci/img/metrics_reports_advanced_v13_0.png Binary files differindex c092c6a84e4..e96fdcf620a 100644 --- a/doc/ci/img/metrics_reports_advanced_v13_0.png +++ b/doc/ci/img/metrics_reports_advanced_v13_0.png diff --git a/doc/ci/introduction/index.md b/doc/ci/introduction/index.md index db18624d4e9..c97f4e51d30 100644 --- a/doc/ci/introduction/index.md +++ b/doc/ci/introduction/index.md @@ -230,7 +230,7 @@ syntax and with its attributes. This document [introduces the concepts of GitLab CI/CD in the scope of GitLab Pages](../../user/project/pages/getting_started/pages_from_scratch.md), for deploying static websites. Although it's meant for users who want to write their own Pages script from scratch, it also serves as an introduction to the setup process for GitLab CI/CD. -It covers the very first general steps of writing a CI/CD configuration +It covers the first general steps of writing a CI/CD configuration file, so we recommend you read through it to understand GitLab's CI/CD logic, and learn how to write your own script (or tweak an existing one) for any application. diff --git a/doc/ci/jenkins/index.md b/doc/ci/jenkins/index.md index 1d029dcdd14..34600dd6540 100644 --- a/doc/ci/jenkins/index.md +++ b/doc/ci/jenkins/index.md @@ -1,359 +1,5 @@ --- -stage: Verify -group: Continuous Integration -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers -comments: false -type: index, howto +redirect_to: '../migration/jenkins.md' --- -# Migrating from Jenkins - -A lot of GitLab users have successfully migrated to GitLab CI/CD from Jenkins. To make this -easier if you're just getting started, we've collected several resources here that you might find useful -before diving in. Think of this page as a "GitLab CI/CD for Jenkins Users" guide. - -The following list of recommended steps was created after observing organizations -that were able to quickly complete this migration: - -1. Start by reading the GitLab CI/CD [Quick Start Guide](../quick_start/README.md) and [important product differences](#important-product-differences). -1. Learn the importance of [managing the organizational transition](#managing-the-organizational-transition). -1. [Add Runners](../runners/README.md) to your GitLab instance. -1. Educate and enable your developers to independently perform the following steps in their projects: - 1. Review the [Quick Start Guide](../quick_start/README.md) and [Pipeline Configuration Reference](../yaml/README.md). - 1. Use the [Jenkins Wrapper](#jenkinsfile-wrapper) to temporarily maintain fragile Jenkins jobs. - 1. Migrate the build and CI jobs and configure them to show results directly in your merge requests. They can use [Auto DevOps](../../topics/autodevops/index.md) as a starting point, and [customize](../../topics/autodevops/customize.md) or [decompose](../../topics/autodevops/customize.md#using-components-of-auto-devops) the configuration as needed. - 1. Add [Review Apps](../review_apps/index.md). - 1. Migrate the deployment jobs using [cloud deployment templates](../cloud_deployment/index.md), adding [environments](../environments/index.md), and [deploy boards](../..//user/project/deploy_boards.md). - 1. Work to unwrap any jobs still running with the use of the Jenkins wrapper. -1. Take stock of any common CI/CD job definitions then create and share [templates](#templates) for them. - -For an example of how to convert a Jenkins pipeline into a GitLab CI/CD pipeline, -or how to use Auto DevOps to test your code automatically, watch the -[Migrating from Jenkins to GitLab](https://www.youtube.com/watch?v=RlEVGOpYF5Y) video. - -Otherwise, read on for important information that will help you get the ball rolling. Welcome -to GitLab! - -If you have questions that are not answered here, the [GitLab community forum](https://forum.gitlab.com/) -can be a great resource. - -## Managing the organizational transition - -An important part of transitioning from Jenkins to GitLab is the cultural and organizational -changes that comes with the move, and successfully managing them. There are a few -things we have found that helps this: - -- Setting and communicating a clear vision of what your migration goals are helps - your users understand why the effort is worth it. The value will be clear when - the work is done, but people need to be aware while it's in progress too. -- Sponsorship and alignment from the relevant leadership team helps with the point above. -- Spending time educating your users on what's different, sharing this document with them, - and so on will help ensure you are successful. -- Finding ways to sequence or delay parts of the migration can help a lot, but you - don't want to leave things in a non-migrated (or partially-migrated) state for too - long. To gain all the benefits of GitLab, moving your existing Jenkins setup over - as-is, including any current problems, will not be enough. You need to take advantage - of the improvements that GitLab offers, and this requires (eventually) updating - your implementation as part of the transition. - -## JenkinsFile Wrapper - -We are building a [JenkinsFile Wrapper](https://gitlab.com/gitlab-org/jfr-container-builder/) which will allow -you to run a complete Jenkins instance inside of a GitLab job, including plugins. This can help ease the process -of transition, by letting you delay the migration of less urgent pipelines for a period of time. - -If you are interested in helping GitLab test the wrapper, join our [public testing issue](https://gitlab.com/gitlab-org/gitlab/-/issues/215675) for instructions and to provide your feedback. - -## Important product differences - -There are some high level differences between the products worth mentioning: - -- With GitLab you don't need a root `pipeline` keyword to wrap everything. -- The way pipelines are triggered and [trigger other pipelines](../yaml/README.md#trigger) - is different than Jenkins. GitLab pipelines can be triggered: - - - on push - - on [schedule](../pipelines/schedules.md) - - from the [GitLab UI](../pipelines/index.md#run-a-pipeline-manually) - - by [API call](../triggers/README.md) - - by [webhook](../triggers/README.md#triggering-a-pipeline-from-a-webhook) - - by [ChatOps](../chatops/README.md) - -- You can control which jobs run in which cases, depending on how they are triggered, - with the [`rules` syntax](../yaml/README.md#rules). -- GitLab [pipeline scheduling concepts](../pipelines/schedules.md) are also different than with Jenkins. -- You can reuse pipeline configurations using the [`include` keyword](../yaml/README.md#include) - and [templates](#templates). Your templates can be kept in a central repository (with different - permissions), and then any project can use them. This central project could also - contain scripts or other reusable code. -- You can also use the [`extends` keyword](../yaml/README.md#extends) to reuse configuration - within a single pipeline configuration. -- All jobs within a single stage always run in parallel, and all stages run in sequence. We are planning - to allow certain jobs to break this sequencing as needed with our [directed acyclic graph](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/47063) - feature. -- The [`parallel`](../yaml/README.md#parallel) keyword can automatically parallelize tasks, - like tests that support parallelization. -- Normally all jobs within a single stage run in parallel, and all stages run in sequence. - There are different [pipeline architectures](../pipelines/pipeline_architectures.md) - that allow you to change this behavior. -- The new [`rules` syntax](../yaml/README.md#rules) is the recommended method of - controlling when different jobs run. It is more powerful than the `only/except` syntax. -- One important difference is that jobs run independently of each other and have a - fresh environment in each job. Passing artifacts between jobs is controlled using the - [`artifacts`](../yaml/README.md#artifacts) and [`dependencies`](../yaml/README.md#dependencies) - keywords. When finished, the planned [Workspaces](https://gitlab.com/gitlab-org/gitlab/-/issues/29265) - feature will allow you to more easily persist a common workspace between serial jobs. -- The `.gitlab-ci.yml` file is checked in to the root of your repository, much like a Jenkinsfile, but - is in the YAML format (see [complete reference](../yaml/README.md)) instead of a Groovy DSL. It's most - analogous to the declarative Jenkinsfile format. -- Manual approvals or gates can be set up as [`when:manual` jobs](../yaml/README.md#whenmanual). These can - also leverage [`protected environments`](../yaml/README.md#protecting-manual-jobs-premium) - to control who is able to approve them. -- GitLab comes with a [container registry](../../user/packages/container_registry/index.md), and we recommend using - container images to set up your build environment. For example, set up one pipeline that builds your build environment - itself and publish that to the container registry. Then, have your pipelines use this instead of each building their - own environment, which will be slower and may be less consistent. We have extensive docs on [how to use the Container Registry](../../user/packages/container_registry/index.md). -- A central utilities repository can be a great place to put assorted scheduled jobs - or other manual jobs that function like utilities. Jenkins installations tend to - have a few of these. - -## Agents vs. Runners - -Both Jenkins agents and GitLab Runners are the hosts that run jobs. To convert the -Jenkins agent, simply uninstall it and then [install and register the runner](../runners/README.md). -Runners do not require much overhead, so you can size them similarly to the Jenkins -agents you were using. - -There are some important differences in the way Runners work in comparison to agents: - -- Runners can be set up as [shared across an instance, be added at the group level, or set up at the project level](../runners/README.md#types-of-runners). - They will self-select jobs from the scopes you've defined automatically. -- You can also [use tags](../runners/README.md#use-tags-to-limit-the-number-of-jobs-using-the-runner) for finer control, and - associate runners with specific jobs. For example, you can use a tag for jobs that - require dedicated, more powerful, or specific hardware. -- GitLab has [autoscaling for Runners](https://docs.gitlab.com/runner/configuration/autoscale.html) - which will let you configure them to be provisioned as needed, and scaled down when not. - This is similar to ephemeral agents in Jenkins. - -If you are using `gitlab.com`, you can take advantage of our [shared Runner fleet](../../user/gitlab_com/index.md#shared-runners) -to run jobs without provisioning your own Runners. We are investigating making them -[available for self-managed instances](https://gitlab.com/groups/gitlab-org/-/epics/835) -as well. - -## Groovy vs. YAML - -Jenkins Pipelines are based on [Groovy](https://groovy-lang.org/), so the pipeline specification is written as code. -GitLab works a bit differently, we use the more highly structured [YAML](https://yaml.org/) format, which -places scripting elements inside of `script:` blocks separate from the pipeline specification itself. - -This is a strength of GitLab, in that it helps keep the learning curve much simpler to get up and running -and avoids some of the problem of unconstrained complexity which can make your Jenkinsfile hard to understand -and manage. - -That said, we do of course still value DRY (don't repeat yourself) principles and want to ensure that -behaviors of your jobs can be codified once and applied as needed. You can use the `extends:` syntax to -[reuse configuration in your jobs](../yaml/README.md#extends), and `include:` can -be used to [reuse pipeline configurations](../yaml/README.md#include) in pipelines -in different projects: - -```yaml -.in-docker: - tags: - - docker - image: alpine - -rspec: - extends: - - .in-docker - script: - - rake rspec -``` - -## Artifact publishing - -Artifacts may work a bit differently than you've used them with Jenkins. In GitLab, any job can define -a set of artifacts to be saved by using the `artifacts:` keyword. This can be configured to point to a file -or set of files that can then be persisted from job to job. Read more on our detailed -[artifacts documentation](../pipelines/job_artifacts.md): - -```yaml -pdf: - script: xelatex mycv.tex - artifacts: - paths: - - ./mycv.pdf - - ./output/ - expire_in: 1 week -``` - -Additionally, we have package management features like a built-in container, NPM, and Maven registry that you -can leverage. You can see the complete list of packaging features (which includes links to documentation) -in the [Packaging section of our documentation](../../README.md#package). - -## Integrated features - -Where you may have used plugins to get things like code quality, unit tests, security scanning, and so on working in Jenkins, -GitLab takes advantage of our connected ecosystem to automatically pull these kinds of results into -your Merge Requests, pipeline details pages, and other locations. You may find that you actually don't -need to configure anything to have these appear. - -If they aren't working as expected, or if you'd like to see what's available, our [CI feature index](../README.md#feature-set) has the full list -of bundled features and links to the documentation for each. - -### Templates - -For advanced CI/CD teams, project templates can enable the reuse of pipeline configurations, -as well as encourage inner sourcing. - -In self-managed GitLab instances, you can build an [Instance Template Repository](../../user/admin_area/settings/instance_template_repository.md). -Development teams across the whole organization can select templates from a dropdown menu. -A group administrator is able to set a group to use as the source for the -[custom project templates](../../user/admin_area/custom_project_templates.md), which can -be used by all projects in the group. An instance administrator can set a group as -the source for [instance project templates](../../user/group/custom_project_templates.md), -which can be used by projects in that instance. - -## Converting a declarative Jenkinsfile - -A declarative Jenkinsfile contains "Sections" and "Directives" which are used to control the behavior of your -pipelines. There are equivalents for all of these in GitLab, which we've documented below. - -This section is based on the [Jenkinsfile syntax documentation](https://www.jenkins.io/doc/book/pipeline/syntax/) -and is meant to be a mapping of concepts there to concepts in GitLab. - -### Sections - -#### `agent` - -The agent section is used to define how a pipeline will be executed. For GitLab, we use the [GitLab Runner](../runners/README.md) -to provide this capability. You can configure your own runners in Kubernetes or on any host, or take advantage -of our shared runner fleet (note that the shared runner fleet is only available for GitLab.com users.) The link above will bring you to the documentation which will describe how to get -up and running quickly. We also support using [tags](../runners/README.md#use-tags-to-limit-the-number-of-jobs-using-the-runner) to direct different jobs -to different Runners (execution agents). - -The `agent` section also allows you to define which Docker images should be used for execution, for which we use -the [`image`](../yaml/README.md#image) keyword. The `image` can be set on a single job or at the top level, in which -case it will apply to all jobs in the pipeline: - -```yaml -my_job: - image: alpine - ... -``` - -#### `post` - -The `post` section defines the actions that should be performed at the end of the pipeline. GitLab also supports -this through the use of stages. You can define your stages as follows, and any jobs assigned to the `before_pipeline` -or `after_pipeline` stages will run as expected. You can call these stages anything you like: - -```yaml -stages: - - before_pipeline - - build - - test - - deploy - - after_pipeline -``` - -Setting a step to be performed before and after any job can be done via the -[`before_script` and `after_script` keywords](../yaml/README.md#before_script-and-after_script): - -```yaml -default: - before_script: - - echo "I run before any jobs starts in the entire pipeline, and can be responsible for setting up the environment." -``` - -#### `stages` - -GitLab CI/CD also lets you define stages, but is a little bit more free-form to configure. The GitLab [`stages` keyword](../yaml/README.md#stages) -is a top level setting that enumerates the list of stages, but you are not required to nest individual jobs underneath -the `stages` section. Any job defined in the `.gitlab-ci.yml` can be made a part of any stage through use of the -[`stage:` keyword](../yaml/README.md#stage). - -Note that, unless otherwise specified, every pipeline is instantiated with a `build`, `test`, and `deploy` stage -which are run in that order. Jobs that have no `stage` defined are placed by default in the `test` stage. -Of course, each job that refers to a stage must refer to a stage that exists in the pipeline configuration. - -```yaml -stages: - - build - - test - - deploy - -my_job: - stage: build - ... -``` - -#### `steps` - -The `steps` section is equivalent to the [`script` section](../yaml/README.md#script) of an individual job. This is -a simple YAML array with each line representing an individual command to be run: - -```yaml -my_job: - script: - - echo "hello! the current time is:" - - time - ... -``` - -### Directives - -#### `environment` - -In GitLab, we use the [`variables` keyword](../yaml/README.md#variables) to define different variables at runtime. -These can also be set up through the GitLab UI, under CI/CD settings. See also our [general documentation on variables](../variables/README.md), -including the section on [protected variables](../variables/README.md#protect-a-custom-variable) which can be used -to limit access to certain variables to certain environments or runners: - -```yaml -variables: - POSTGRES_USER: user - POSTGRES_PASSWORD: testing_password -``` - -#### `options` - -Here, options for different things exist associated with the object in question itself. For example, options related -to jobs are defined in relation to the job itself. If you're looking for a certain option, you should be able to find -where it's located by searching our [complete configuration reference](../yaml/README.md) page. - -#### `parameters` - -GitLab does not require you to define which variables you want to be available when starting a manual job. A user -can provide any variables they like. - -#### `triggers` / `cron` - -Because GitLab is integrated tightly with Git, SCM polling options for triggers are not needed. We support an easy to use -[syntax for scheduling pipelines](../pipelines/schedules.md). - -#### `tools` - -GitLab does not support a separate `tools` directive. Our best-practice recommendation is to use pre-built -container images, which can be cached, and can be built to already contain the tools you need for your pipelines. Pipelines can -be set up to automatically build these images as needed and deploy them to the [container registry](../../user/packages/container_registry/index.md). - -If you're not using container images with Docker/Kubernetes, for example on Mac or FreeBSD, then the `shell` executor does require you to -set up your environment either in advance or as part of the jobs. You could create a `before_script` -action that handles this for you. - -#### `input` - -Similar to the `parameters` keyword, this is not needed because a manual job can always be provided runtime -variable entry. - -#### `when` - -GitLab does support a [`when` keyword](../yaml/README.md#when) which is used to indicate when a job should be -run in case of (or despite) failure, but most of the logic for controlling pipelines can be found in -our very powerful [`only/except` rules system](../yaml/README.md#onlyexcept-basic) -(see also our [advanced syntax](../yaml/README.md#onlyexcept-basic)): - -```yaml -my_job: - only: [branches] -``` +This document was moved to [another location](../migration/jenkins.md). diff --git a/doc/ci/junit_test_reports.md b/doc/ci/junit_test_reports.md index aedda3c8341..8bc55a6e4f3 100644 --- a/doc/ci/junit_test_reports.md +++ b/doc/ci/junit_test_reports.md @@ -66,10 +66,6 @@ execution time and the error output. ## How to set it up -NOTE: **Note:** -For a list of supported languages on JUnit tests, check the -[Wikipedia article](https://en.wikipedia.org/wiki/JUnit#Ports). - To enable the JUnit reports in merge requests, you need to add [`artifacts:reports:junit`](pipelines/job_artifacts.md#artifactsreportsjunit) in `.gitlab-ci.yml`, and specify the path(s) of the generated test reports. @@ -80,9 +76,11 @@ collects the JUnit test report from each job. After each job is executed, the XML reports are stored in GitLab as artifacts and their results are shown in the merge request widget. +To make the JUnit output files browsable, include them with the +[`artifacts:paths`](yaml/README.md#artifactspaths) keyword as well, as shown in the [Ruby example](#ruby-example). + NOTE: **Note:** -If you also want the ability to browse JUnit output files, include the -[`artifacts:paths`](yaml/README.md#artifactspaths) keyword. An example of this is shown in the Ruby example below. +You cannot have multiple tests with the same name and class in your JUnit report. ### Ruby example @@ -140,6 +138,9 @@ java: junit: build/test-results/test/**/TEST-*.xml ``` +NOTE: **Note:** +Support for `**` was added in [GitLab Runner 13.0](https://gitlab.com/gitlab-org/gitlab-runner/-/issues/2620). + #### Maven For parsing [Surefire](https://maven.apache.org/surefire/maven-surefire-plugin/) @@ -236,9 +237,8 @@ Test: ## Viewing JUnit test reports on GitLab -> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/24792) in GitLab 12.5. -> - It's deployed behind a feature flag, disabled by default. -> - To use it in GitLab self-managed instances, ask a GitLab administrator to [enable it](#enabling-the-junit-test-reports-feature-core-only). **(CORE ONLY)** +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/24792) in GitLab 12.5 behind a feature flag (`junit_pipeline_view`), disabled by default. +> - The feature flag was removed and the feature was [made generally available](https://gitlab.com/gitlab-org/gitlab/-/issues/216478) in GitLab 13.3. If JUnit XML files are generated and uploaded as part of a pipeline, these reports can be viewed inside the pipelines details page. The **Tests** tab on this page will @@ -247,27 +247,10 @@ display a list of test suites and cases reported from the XML file.  You can view all the known test suites and click on each of these to see further -details, including the cases that makeup the suite. Cases are ordered by status, -with failed showing at the top, skipped next and successful cases last. +details, including the cases that make up the suite. You can also retrieve the reports via the [GitLab API](../api/pipelines.md#get-a-pipelines-test-report). -### Enabling the JUnit test reports feature **(CORE ONLY)** - -This feature comes with the `:junit_pipeline_view` feature flag disabled by default. This -feature is disabled due to some performance issues with very large data sets. -When [the performance is improved](https://gitlab.com/groups/gitlab-org/-/epics/2854), the feature will be enabled by default. - -To enable this feature, ask a GitLab administrator with [Rails console access](../administration/feature_flags.md#how-to-enable-and-disable-features-behind-flags) to run the -following command: - -```ruby -Feature.enable(:junit_pipeline_view) - -# Enable the feature for a specific project, GitLab 13.0 and above only. -Feature.enable(:junit_pipeline_view, Project.find(<your-project-id-here>)) -``` - ## Viewing JUnit screenshots on GitLab > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/202114) in GitLab 13.0. diff --git a/doc/ci/large_repositories/index.md b/doc/ci/large_repositories/index.md index b4652050654..00e912d47dc 100644 --- a/doc/ci/large_repositories/index.md +++ b/doc/ci/large_repositories/index.md @@ -124,6 +124,10 @@ are dependent on Git version. [`GIT_FETCH_EXTRA_FLAGS`](../yaml/README.md#git-fetch-extra-flags) allows you to modify `git fetch` behavior by passing extra flags. +For example, if your project contains a large number of tags that your CI jobs don't rely on, +you could add [`--no-tags`](https://git-scm.com/docs/git-fetch#Documentation/git-fetch.txt---no-tags) +to the extra flags to make your fetches faster and more compact. + See the [`GIT_FETCH_EXTRA_FLAGS` documentation](../yaml/README.md#git-fetch-extra-flags) for more information. @@ -253,3 +257,11 @@ concurrent = 4 This makes the cloning configuration to be part of given Runner and does not require us to update each `.gitlab-ci.yml`. + +## Pre-clone step + +For very active repositories with a large number of references and files, you can also +optimize your CI jobs by seeding repository data with GitLab Runner's [`pre_clone_script`](https://docs.gitlab.com/runner/configuration/advanced-configuration.html#the-runners-section). + +See [our development documentation](../../development/pipelines.md#pre-clone-step) for +an overview of how we implemented this approach on GitLab.com for the main GitLab repository. diff --git a/doc/ci/lint.md b/doc/ci/lint.md new file mode 100644 index 00000000000..716a4218d97 --- /dev/null +++ b/doc/ci/lint.md @@ -0,0 +1,41 @@ +# CI Lint + +If you want to test the validity of your GitLab CI/CD configuration before committing +the changes, you can use the CI Lint tool. This tool checks for syntax and logical +errors by default, and can simulate pipeline creation to try to find more complicated +issues as well. + +To access the CI Lint tool, navigate to **CI/CD > Pipelines** or **CI/CD > Jobs** +in your project and click **CI lint**. + +## Validate basic logic and syntax + +By default, the CI lint checks the syntax of your CI YAML configuration and also runs +some basic logical validations. + +To use the CI lint, paste a complete CI configuration (`.gitlab-ci.yml` for example) +into the text box and click **Validate**: + + + +## Pipeline simulation + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/229794) in GitLab 13.3. + +Not all pipeline configuration issues can be found by the [basic CI lint validation](#validate-basic-logic-and-syntax). +You can simulate the creation of a pipeline for deeper validation that can discover +more complicated issues. + +To validate the configuration by running a pipeline simulation: + +1. Paste the GitLab CI configuration to verify into the text box. +1. Click the **Simulate pipeline creation for the default branch** checkbox. +1. Click **Validate**. + + + +### Pipeline simulation limitations + +Simulations run as `git push` events against the default branch. You must have +[permissions](../user/permissions.md#project-members-permissions) to create pipelines +on this branch to validate with a simulation. diff --git a/doc/ci/merge_request_pipelines/index.md b/doc/ci/merge_request_pipelines/index.md index 2a6008e6307..73b971c2d41 100644 --- a/doc/ci/merge_request_pipelines/index.md +++ b/doc/ci/merge_request_pipelines/index.md @@ -166,31 +166,33 @@ Read the [documentation on Pipelines for Merged Results](pipelines_for_merged_re Read the [documentation on Merge Trains](pipelines_for_merged_results/merge_trains/index.md). -## Important notes about merge requests from forked projects +## Run pipelines in the parent project for merge requests from a forked project **(STARTER)** -Note that the current behavior is subject to change. In the usual contribution -flow, external contributors follow the following steps: +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/217451) in GitLab 13.3. -1. Fork a parent project. -1. Create a merge request from the forked project that targets the `master` branch - in the parent project. -1. A pipeline runs on the merge request. -1. A maintainer from the parent project checks the pipeline result, and merge - into a target branch if the latest pipeline has passed. +By default, external contributors working from forks can't create pipelines in the +parent project. When a pipeline for merge requests is triggered by a merge request +coming from a fork: -Currently, those pipelines are created in a **forked** project, not in the -parent project. This means you cannot completely trust the pipeline result, -because, technically, external contributors can disguise their pipeline results -by tweaking their GitLab Runner in the forked project. +- It's created and runs in the fork (source) project, not the parent (target) project. +- It uses the fork project's CI/CD configuration and resources. -There are multiple reasons why GitLab doesn't allow those pipelines to be -created in the parent project, but one of the biggest reasons is security concern. -External users could steal secret variables from the parent project by modifying -`.gitlab-ci.yml`, which could be some sort of credentials. This should not happen. +Sometimes parent project members want the pipeline to run in the parent +project. This could be to ensure that the post-merge pipeline passes in the parent project. +For example, a fork project could try to use a corrupted Runner that doesn't execute +test scripts properly, but reports a passed pipeline. Reviewers in the parent project +could mistakenly trust the merge request because it passed a faked pipeline. -We're discussing a secure solution of running pipelines for merge requests -that are submitted from forked projects, -see [the issue about the permission extension](https://gitlab.com/gitlab-org/gitlab/-/issues/11934). +Parent project members with at least [Developer permissions](../../user/permissions.md) +can create pipelines in the parent project for merge requests +from a forked project. In the merge request, go to the **Pipelines** and click +**Run Pipeline** button. + +CAUTION: **Caution:** +Fork merge requests could contain malicious code that tries to steal secrets in the +parent project when the pipeline runs, even before merge. Reviewers must carefully +check the changes in the merge request before triggering the pipeline. GitLab shows +a warning that must be accepted before the pipeline can be triggered. ## Additional predefined variables @@ -206,19 +208,19 @@ The variable names begin with the `CI_MERGE_REQUEST_` prefix. ### Two pipelines created when pushing to a merge request If you are experiencing duplicated pipelines when using `rules`, take a look at -the [important differences between `rules` and `only`/`except`](../yaml/README.md#differences-between-rules-and-onlyexcept), +the [important differences between `rules` and `only`/`except`](../yaml/README.md#prevent-duplicate-pipelines), which will help you get your starting configuration correct. If you are seeing two pipelines when using `only/except`, please see the caveats related to using `only/except` above (or, consider moving to `rules`). +It is not possible to run a job for branch pipelines first, then only for merge request +pipelines after the merge request is created (skipping the duplicate branch pipeline). See +the [related issue](https://gitlab.com/gitlab-org/gitlab/-/issues/201845) for more details. + ### Two pipelines created when pushing an invalid CI configuration file Pushing to a branch with an invalid CI configuration file can trigger the creation of two types of failed pipelines. One pipeline is a failed merge request pipeline, and the other is a failed branch pipeline, but both are caused by the same invalid configuration. - -In rare cases, duplicate pipelines are created. - -See [this issue](https://gitlab.com/gitlab-org/gitlab/-/issues/201845) for details. diff --git a/doc/ci/merge_request_pipelines/pipelines_for_merged_results/index.md b/doc/ci/merge_request_pipelines/pipelines_for_merged_results/index.md index 84fbefb080f..685c93b3be4 100644 --- a/doc/ci/merge_request_pipelines/pipelines_for_merged_results/index.md +++ b/doc/ci/merge_request_pipelines/pipelines_for_merged_results/index.md @@ -45,8 +45,6 @@ To enable pipelines for merge results: - You must have maintainer [permissions](../../../user/permissions.md). - You must be using [GitLab Runner](https://gitlab.com/gitlab-org/gitlab-runner) 11.9 or later. -- You must not be forking or using cross-repo workflows. To follow progress, - see [#11934](https://gitlab.com/gitlab-org/gitlab/-/issues/11934). - You must not be using [fast forward merges](../../../user/project/merge_requests/fast_forward_merge.md) yet. To follow progress, see [#58226](https://gitlab.com/gitlab-org/gitlab/-/issues/26996). diff --git a/doc/ci/merge_request_pipelines/pipelines_for_merged_results/merge_trains/index.md b/doc/ci/merge_request_pipelines/pipelines_for_merged_results/merge_trains/index.md index e12c9d109cc..d91d88c8e12 100644 --- a/doc/ci/merge_request_pipelines/pipelines_for_merged_results/merge_trains/index.md +++ b/doc/ci/merge_request_pipelines/pipelines_for_merged_results/merge_trains/index.md @@ -11,6 +11,8 @@ last_update: 2019-07-03 > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/9186) in [GitLab Premium](https://about.gitlab.com/pricing/) 12.0. > - [Squash and merge](../../../../user/project/merge_requests/squash_and_merge.md) support [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/13001) in [GitLab Premium](https://about.gitlab.com/pricing/) 12.6. +For more information about why you might want to use Merge Trains, read [How merge trains keep your master green](https://about.gitlab.com/blog/2020/01/30/all-aboard-merge-trains/). + When [pipelines for merged results](../index.md#pipelines-for-merged-results-premium) are enabled, the pipeline jobs run as if the changes from your source branch have already been merged into the target branch. diff --git a/doc/ci/migration/circleci.md b/doc/ci/migration/circleci.md index 625b15ca4fb..78705815c24 100644 --- a/doc/ci/migration/circleci.md +++ b/doc/ci/migration/circleci.md @@ -1,4 +1,7 @@ --- +stage: Verify +group: Continuous Integration +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers comments: false type: index, howto --- diff --git a/doc/ci/migration/jenkins.md b/doc/ci/migration/jenkins.md new file mode 100644 index 00000000000..1d029dcdd14 --- /dev/null +++ b/doc/ci/migration/jenkins.md @@ -0,0 +1,359 @@ +--- +stage: Verify +group: Continuous Integration +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +comments: false +type: index, howto +--- + +# Migrating from Jenkins + +A lot of GitLab users have successfully migrated to GitLab CI/CD from Jenkins. To make this +easier if you're just getting started, we've collected several resources here that you might find useful +before diving in. Think of this page as a "GitLab CI/CD for Jenkins Users" guide. + +The following list of recommended steps was created after observing organizations +that were able to quickly complete this migration: + +1. Start by reading the GitLab CI/CD [Quick Start Guide](../quick_start/README.md) and [important product differences](#important-product-differences). +1. Learn the importance of [managing the organizational transition](#managing-the-organizational-transition). +1. [Add Runners](../runners/README.md) to your GitLab instance. +1. Educate and enable your developers to independently perform the following steps in their projects: + 1. Review the [Quick Start Guide](../quick_start/README.md) and [Pipeline Configuration Reference](../yaml/README.md). + 1. Use the [Jenkins Wrapper](#jenkinsfile-wrapper) to temporarily maintain fragile Jenkins jobs. + 1. Migrate the build and CI jobs and configure them to show results directly in your merge requests. They can use [Auto DevOps](../../topics/autodevops/index.md) as a starting point, and [customize](../../topics/autodevops/customize.md) or [decompose](../../topics/autodevops/customize.md#using-components-of-auto-devops) the configuration as needed. + 1. Add [Review Apps](../review_apps/index.md). + 1. Migrate the deployment jobs using [cloud deployment templates](../cloud_deployment/index.md), adding [environments](../environments/index.md), and [deploy boards](../..//user/project/deploy_boards.md). + 1. Work to unwrap any jobs still running with the use of the Jenkins wrapper. +1. Take stock of any common CI/CD job definitions then create and share [templates](#templates) for them. + +For an example of how to convert a Jenkins pipeline into a GitLab CI/CD pipeline, +or how to use Auto DevOps to test your code automatically, watch the +[Migrating from Jenkins to GitLab](https://www.youtube.com/watch?v=RlEVGOpYF5Y) video. + +Otherwise, read on for important information that will help you get the ball rolling. Welcome +to GitLab! + +If you have questions that are not answered here, the [GitLab community forum](https://forum.gitlab.com/) +can be a great resource. + +## Managing the organizational transition + +An important part of transitioning from Jenkins to GitLab is the cultural and organizational +changes that comes with the move, and successfully managing them. There are a few +things we have found that helps this: + +- Setting and communicating a clear vision of what your migration goals are helps + your users understand why the effort is worth it. The value will be clear when + the work is done, but people need to be aware while it's in progress too. +- Sponsorship and alignment from the relevant leadership team helps with the point above. +- Spending time educating your users on what's different, sharing this document with them, + and so on will help ensure you are successful. +- Finding ways to sequence or delay parts of the migration can help a lot, but you + don't want to leave things in a non-migrated (or partially-migrated) state for too + long. To gain all the benefits of GitLab, moving your existing Jenkins setup over + as-is, including any current problems, will not be enough. You need to take advantage + of the improvements that GitLab offers, and this requires (eventually) updating + your implementation as part of the transition. + +## JenkinsFile Wrapper + +We are building a [JenkinsFile Wrapper](https://gitlab.com/gitlab-org/jfr-container-builder/) which will allow +you to run a complete Jenkins instance inside of a GitLab job, including plugins. This can help ease the process +of transition, by letting you delay the migration of less urgent pipelines for a period of time. + +If you are interested in helping GitLab test the wrapper, join our [public testing issue](https://gitlab.com/gitlab-org/gitlab/-/issues/215675) for instructions and to provide your feedback. + +## Important product differences + +There are some high level differences between the products worth mentioning: + +- With GitLab you don't need a root `pipeline` keyword to wrap everything. +- The way pipelines are triggered and [trigger other pipelines](../yaml/README.md#trigger) + is different than Jenkins. GitLab pipelines can be triggered: + + - on push + - on [schedule](../pipelines/schedules.md) + - from the [GitLab UI](../pipelines/index.md#run-a-pipeline-manually) + - by [API call](../triggers/README.md) + - by [webhook](../triggers/README.md#triggering-a-pipeline-from-a-webhook) + - by [ChatOps](../chatops/README.md) + +- You can control which jobs run in which cases, depending on how they are triggered, + with the [`rules` syntax](../yaml/README.md#rules). +- GitLab [pipeline scheduling concepts](../pipelines/schedules.md) are also different than with Jenkins. +- You can reuse pipeline configurations using the [`include` keyword](../yaml/README.md#include) + and [templates](#templates). Your templates can be kept in a central repository (with different + permissions), and then any project can use them. This central project could also + contain scripts or other reusable code. +- You can also use the [`extends` keyword](../yaml/README.md#extends) to reuse configuration + within a single pipeline configuration. +- All jobs within a single stage always run in parallel, and all stages run in sequence. We are planning + to allow certain jobs to break this sequencing as needed with our [directed acyclic graph](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/47063) + feature. +- The [`parallel`](../yaml/README.md#parallel) keyword can automatically parallelize tasks, + like tests that support parallelization. +- Normally all jobs within a single stage run in parallel, and all stages run in sequence. + There are different [pipeline architectures](../pipelines/pipeline_architectures.md) + that allow you to change this behavior. +- The new [`rules` syntax](../yaml/README.md#rules) is the recommended method of + controlling when different jobs run. It is more powerful than the `only/except` syntax. +- One important difference is that jobs run independently of each other and have a + fresh environment in each job. Passing artifacts between jobs is controlled using the + [`artifacts`](../yaml/README.md#artifacts) and [`dependencies`](../yaml/README.md#dependencies) + keywords. When finished, the planned [Workspaces](https://gitlab.com/gitlab-org/gitlab/-/issues/29265) + feature will allow you to more easily persist a common workspace between serial jobs. +- The `.gitlab-ci.yml` file is checked in to the root of your repository, much like a Jenkinsfile, but + is in the YAML format (see [complete reference](../yaml/README.md)) instead of a Groovy DSL. It's most + analogous to the declarative Jenkinsfile format. +- Manual approvals or gates can be set up as [`when:manual` jobs](../yaml/README.md#whenmanual). These can + also leverage [`protected environments`](../yaml/README.md#protecting-manual-jobs-premium) + to control who is able to approve them. +- GitLab comes with a [container registry](../../user/packages/container_registry/index.md), and we recommend using + container images to set up your build environment. For example, set up one pipeline that builds your build environment + itself and publish that to the container registry. Then, have your pipelines use this instead of each building their + own environment, which will be slower and may be less consistent. We have extensive docs on [how to use the Container Registry](../../user/packages/container_registry/index.md). +- A central utilities repository can be a great place to put assorted scheduled jobs + or other manual jobs that function like utilities. Jenkins installations tend to + have a few of these. + +## Agents vs. Runners + +Both Jenkins agents and GitLab Runners are the hosts that run jobs. To convert the +Jenkins agent, simply uninstall it and then [install and register the runner](../runners/README.md). +Runners do not require much overhead, so you can size them similarly to the Jenkins +agents you were using. + +There are some important differences in the way Runners work in comparison to agents: + +- Runners can be set up as [shared across an instance, be added at the group level, or set up at the project level](../runners/README.md#types-of-runners). + They will self-select jobs from the scopes you've defined automatically. +- You can also [use tags](../runners/README.md#use-tags-to-limit-the-number-of-jobs-using-the-runner) for finer control, and + associate runners with specific jobs. For example, you can use a tag for jobs that + require dedicated, more powerful, or specific hardware. +- GitLab has [autoscaling for Runners](https://docs.gitlab.com/runner/configuration/autoscale.html) + which will let you configure them to be provisioned as needed, and scaled down when not. + This is similar to ephemeral agents in Jenkins. + +If you are using `gitlab.com`, you can take advantage of our [shared Runner fleet](../../user/gitlab_com/index.md#shared-runners) +to run jobs without provisioning your own Runners. We are investigating making them +[available for self-managed instances](https://gitlab.com/groups/gitlab-org/-/epics/835) +as well. + +## Groovy vs. YAML + +Jenkins Pipelines are based on [Groovy](https://groovy-lang.org/), so the pipeline specification is written as code. +GitLab works a bit differently, we use the more highly structured [YAML](https://yaml.org/) format, which +places scripting elements inside of `script:` blocks separate from the pipeline specification itself. + +This is a strength of GitLab, in that it helps keep the learning curve much simpler to get up and running +and avoids some of the problem of unconstrained complexity which can make your Jenkinsfile hard to understand +and manage. + +That said, we do of course still value DRY (don't repeat yourself) principles and want to ensure that +behaviors of your jobs can be codified once and applied as needed. You can use the `extends:` syntax to +[reuse configuration in your jobs](../yaml/README.md#extends), and `include:` can +be used to [reuse pipeline configurations](../yaml/README.md#include) in pipelines +in different projects: + +```yaml +.in-docker: + tags: + - docker + image: alpine + +rspec: + extends: + - .in-docker + script: + - rake rspec +``` + +## Artifact publishing + +Artifacts may work a bit differently than you've used them with Jenkins. In GitLab, any job can define +a set of artifacts to be saved by using the `artifacts:` keyword. This can be configured to point to a file +or set of files that can then be persisted from job to job. Read more on our detailed +[artifacts documentation](../pipelines/job_artifacts.md): + +```yaml +pdf: + script: xelatex mycv.tex + artifacts: + paths: + - ./mycv.pdf + - ./output/ + expire_in: 1 week +``` + +Additionally, we have package management features like a built-in container, NPM, and Maven registry that you +can leverage. You can see the complete list of packaging features (which includes links to documentation) +in the [Packaging section of our documentation](../../README.md#package). + +## Integrated features + +Where you may have used plugins to get things like code quality, unit tests, security scanning, and so on working in Jenkins, +GitLab takes advantage of our connected ecosystem to automatically pull these kinds of results into +your Merge Requests, pipeline details pages, and other locations. You may find that you actually don't +need to configure anything to have these appear. + +If they aren't working as expected, or if you'd like to see what's available, our [CI feature index](../README.md#feature-set) has the full list +of bundled features and links to the documentation for each. + +### Templates + +For advanced CI/CD teams, project templates can enable the reuse of pipeline configurations, +as well as encourage inner sourcing. + +In self-managed GitLab instances, you can build an [Instance Template Repository](../../user/admin_area/settings/instance_template_repository.md). +Development teams across the whole organization can select templates from a dropdown menu. +A group administrator is able to set a group to use as the source for the +[custom project templates](../../user/admin_area/custom_project_templates.md), which can +be used by all projects in the group. An instance administrator can set a group as +the source for [instance project templates](../../user/group/custom_project_templates.md), +which can be used by projects in that instance. + +## Converting a declarative Jenkinsfile + +A declarative Jenkinsfile contains "Sections" and "Directives" which are used to control the behavior of your +pipelines. There are equivalents for all of these in GitLab, which we've documented below. + +This section is based on the [Jenkinsfile syntax documentation](https://www.jenkins.io/doc/book/pipeline/syntax/) +and is meant to be a mapping of concepts there to concepts in GitLab. + +### Sections + +#### `agent` + +The agent section is used to define how a pipeline will be executed. For GitLab, we use the [GitLab Runner](../runners/README.md) +to provide this capability. You can configure your own runners in Kubernetes or on any host, or take advantage +of our shared runner fleet (note that the shared runner fleet is only available for GitLab.com users.) The link above will bring you to the documentation which will describe how to get +up and running quickly. We also support using [tags](../runners/README.md#use-tags-to-limit-the-number-of-jobs-using-the-runner) to direct different jobs +to different Runners (execution agents). + +The `agent` section also allows you to define which Docker images should be used for execution, for which we use +the [`image`](../yaml/README.md#image) keyword. The `image` can be set on a single job or at the top level, in which +case it will apply to all jobs in the pipeline: + +```yaml +my_job: + image: alpine + ... +``` + +#### `post` + +The `post` section defines the actions that should be performed at the end of the pipeline. GitLab also supports +this through the use of stages. You can define your stages as follows, and any jobs assigned to the `before_pipeline` +or `after_pipeline` stages will run as expected. You can call these stages anything you like: + +```yaml +stages: + - before_pipeline + - build + - test + - deploy + - after_pipeline +``` + +Setting a step to be performed before and after any job can be done via the +[`before_script` and `after_script` keywords](../yaml/README.md#before_script-and-after_script): + +```yaml +default: + before_script: + - echo "I run before any jobs starts in the entire pipeline, and can be responsible for setting up the environment." +``` + +#### `stages` + +GitLab CI/CD also lets you define stages, but is a little bit more free-form to configure. The GitLab [`stages` keyword](../yaml/README.md#stages) +is a top level setting that enumerates the list of stages, but you are not required to nest individual jobs underneath +the `stages` section. Any job defined in the `.gitlab-ci.yml` can be made a part of any stage through use of the +[`stage:` keyword](../yaml/README.md#stage). + +Note that, unless otherwise specified, every pipeline is instantiated with a `build`, `test`, and `deploy` stage +which are run in that order. Jobs that have no `stage` defined are placed by default in the `test` stage. +Of course, each job that refers to a stage must refer to a stage that exists in the pipeline configuration. + +```yaml +stages: + - build + - test + - deploy + +my_job: + stage: build + ... +``` + +#### `steps` + +The `steps` section is equivalent to the [`script` section](../yaml/README.md#script) of an individual job. This is +a simple YAML array with each line representing an individual command to be run: + +```yaml +my_job: + script: + - echo "hello! the current time is:" + - time + ... +``` + +### Directives + +#### `environment` + +In GitLab, we use the [`variables` keyword](../yaml/README.md#variables) to define different variables at runtime. +These can also be set up through the GitLab UI, under CI/CD settings. See also our [general documentation on variables](../variables/README.md), +including the section on [protected variables](../variables/README.md#protect-a-custom-variable) which can be used +to limit access to certain variables to certain environments or runners: + +```yaml +variables: + POSTGRES_USER: user + POSTGRES_PASSWORD: testing_password +``` + +#### `options` + +Here, options for different things exist associated with the object in question itself. For example, options related +to jobs are defined in relation to the job itself. If you're looking for a certain option, you should be able to find +where it's located by searching our [complete configuration reference](../yaml/README.md) page. + +#### `parameters` + +GitLab does not require you to define which variables you want to be available when starting a manual job. A user +can provide any variables they like. + +#### `triggers` / `cron` + +Because GitLab is integrated tightly with Git, SCM polling options for triggers are not needed. We support an easy to use +[syntax for scheduling pipelines](../pipelines/schedules.md). + +#### `tools` + +GitLab does not support a separate `tools` directive. Our best-practice recommendation is to use pre-built +container images, which can be cached, and can be built to already contain the tools you need for your pipelines. Pipelines can +be set up to automatically build these images as needed and deploy them to the [container registry](../../user/packages/container_registry/index.md). + +If you're not using container images with Docker/Kubernetes, for example on Mac or FreeBSD, then the `shell` executor does require you to +set up your environment either in advance or as part of the jobs. You could create a `before_script` +action that handles this for you. + +#### `input` + +Similar to the `parameters` keyword, this is not needed because a manual job can always be provided runtime +variable entry. + +#### `when` + +GitLab does support a [`when` keyword](../yaml/README.md#when) which is used to indicate when a job should be +run in case of (or despite) failure, but most of the logic for controlling pipelines can be found in +our very powerful [`only/except` rules system](../yaml/README.md#onlyexcept-basic) +(see also our [advanced syntax](../yaml/README.md#onlyexcept-basic)): + +```yaml +my_job: + only: [branches] +``` diff --git a/doc/ci/multi_project_pipelines.md b/doc/ci/multi_project_pipelines.md index 40d495a69f1..3df2db8539e 100644 --- a/doc/ci/multi_project_pipelines.md +++ b/doc/ci/multi_project_pipelines.md @@ -62,6 +62,14 @@ together, allowing you to visualize their relationships on pipeline graphs. These relationships are displayed in the pipeline graph by showing inbound and outbound connections for upstream and downstream pipeline dependencies. +When using: + +- Variables or [`rules`](yaml/README.md#rulesif) to control job behavior, the value of + the [`$CI_PIPELINE_SOURCE` predefined variable](variables/predefined_variables.md) is + `pipeline` for multi-project pipeline triggered through the API with `CI_JOB_TOKEN`. +- [`only/except`](yaml/README.md#onlyexcept-basic) to control job behavior, use the + `pipelines` keyword. + ## Creating multi-project pipelines from `.gitlab-ci.yml` > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/8997) in [GitLab Premium](https://about.gitlab.com/pricing/) 11.8. @@ -100,6 +108,15 @@ downstream project (`my/deployment` in this case). If a downstream project can not be found, or a user does not have access rights to create pipeline there, the `staging` job is going to be marked as _failed_. +When using: + +- Variables or [`rules`](yaml/README.md#rulesif) to control job behavior, the value of + the [`$CI_PIPELINE_SOURCE` predefined variable](variables/predefined_variables.md) is + `pipeline` for multi-project pipelines triggered with a bridge job (using the + [`trigger:`](yaml/README.md#trigger) keyword). +- [`only/except`](yaml/README.md#onlyexcept-basic) to control job behavior, use the + `pipelines` keyword. + CAUTION: **Caution:** In the example, `staging` will be marked as succeeded as soon as a downstream pipeline gets created. If you want to display the downstream pipeline's status instead, see @@ -138,6 +155,12 @@ Use: GitLab will use a commit that is currently on the HEAD of the branch when creating a downstream pipeline. +NOTE: **Note:** +Pipelines triggered on a protected branch in a downstream project use the [permissions](../user/permissions.md) +of the user that ran the trigger job in the upstream project. If the user does not +have permission to run CI/CD pipelines against the protected branch, the pipeline fails. See +[pipeline security for protected branches](pipelines/index.md#pipeline-security-on-protected-branches). + ### Passing variables to a downstream pipeline Sometimes you might want to pass variables to a downstream pipeline. @@ -254,5 +277,5 @@ tag in a different project: Any pipelines that complete successfully for new tags in the subscribed project will now trigger a pipeline on the current project's default branch. The maximum -number of upstream pipeline subscriptions is 2, for both the upstream and -downstream projects. +number of upstream pipeline subscriptions is 2 by default, for both the upstream and +downstream projects. This [application limit](../administration/instance_limits.md#number-of-cicd-subscriptions-to-a-project) can be changed on self-managed instances by a GitLab administrator. diff --git a/doc/ci/parent_child_pipelines.md b/doc/ci/parent_child_pipelines.md index c936bd8f5fe..1cfa698bfa5 100644 --- a/doc/ci/parent_child_pipelines.md +++ b/doc/ci/parent_child_pipelines.md @@ -43,8 +43,8 @@ Child pipelines work well with other GitLab CI/CD features: - Since the parent pipeline in `.gitlab-ci.yml` and the child pipeline run as normal pipelines, they can have their own behaviors and sequencing in relation to triggers. -All of this will work with the [`include:`](yaml/README.md#include) feature so you can compose -the child pipeline configuration. +See the [`trigger:`](yaml/README.md#trigger) keyword documentation for full details on how to +include the child pipeline configuration. <i class="fa fa-youtube-play youtube" aria-hidden="true"></i> For an overview, see [Parent-Child Pipelines feature demo](https://youtu.be/n8KpBSqZNbk). @@ -145,6 +145,8 @@ build a matrix of targets and architectures. <i class="fa fa-youtube-play youtube" aria-hidden="true"></i> For an overview, see [Create child pipelines using dynamically generated configurations](https://youtu.be/nMdfus2JWHM). +We also have an [example project using Dynamic Child Pipelines with Jsonnet](https://gitlab.com/gitlab-org/project-templates/jsonnet) which shows how to use a data templating language to generate your `.gitlab-ci.yml` at runtime. You could use a similar process for other templating languages like [Dhall](https://dhall-lang.org/) or [`ytt`](https://get-ytt.io/). + In GitLab 12.9, the child pipeline could fail to be created in certain cases, causing the parent pipeline to fail. This is [resolved in GitLab 12.10](https://gitlab.com/gitlab-org/gitlab/-/issues/209070). diff --git a/doc/ci/pipelines/index.md b/doc/ci/pipelines/index.md index 18b3fe10bec..8419b474d54 100644 --- a/doc/ci/pipelines/index.md +++ b/doc/ci/pipelines/index.md @@ -87,13 +87,13 @@ to its **Pipelines** tab.  -Clicking a pipeline will bring you to the **Pipeline Details** page and show +Click a pipeline to open the **Pipeline Details** page and show the jobs that were run for that pipeline. From here you can cancel a running pipeline, retry jobs on a failed pipeline, or [delete a pipeline](#delete-a-pipeline). [Starting in GitLab 12.3](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/50499), a link to the latest pipeline for the last commit of a given branch is available at `/project/pipelines/[branch]/latest`. -Also, `/project/pipelines/latest` will redirect you to the latest pipeline for the last commit +Also, `/project/pipelines/latest` redirects you to the latest pipeline for the last commit on the project's default branch. [Starting in GitLab 13.0](https://gitlab.com/gitlab-org/gitlab/-/issues/215367), @@ -120,14 +120,14 @@ To execute a pipeline manually: 1. Enter any [environment variables](../variables/README.md) required for the pipeline run. 1. Click the **Create pipeline** button. -The pipeline will execute the jobs as configured. +The pipeline now executes the jobs as configured. ### Run a pipeline by using a URL query string > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/24146) in GitLab 12.5. You can use a query string to pre-populate the **Run Pipeline** page. For example, the query string -`.../pipelines/new?ref=my_branch&var[foo]=bar&file_var[file_foo]=file_bar` will pre-populate the +`.../pipelines/new?ref=my_branch&var[foo]=bar&file_var[file_foo]=file_bar` pre-populates the **Run Pipeline** page with: - **Run for** field: `my_branch`. @@ -174,7 +174,7 @@ stage has a job with a manual action. > [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/27188) in GitLab 11.11. Multiple manual actions in a single stage can be started at the same time using the "Play all manual" button. -Once the user clicks this button, each individual manual action will be triggered and refreshed +Once you click this button, each individual manual action is triggered and refreshed to an updated status. This functionality is only available: @@ -193,7 +193,7 @@ page, then using the **Delete** button.  CAUTION: **Warning:** -Deleting a pipeline will expire all pipeline caches, and delete all related objects, +Deleting a pipeline expires all pipeline caches, and deletes all related objects, such as builds, logs, artifacts, and triggers. **This action cannot be undone.** ### Pipeline quotas @@ -252,6 +252,7 @@ on that specific branch: - Run manual pipelines (using the [Web UI](#run-a-pipeline-manually) or [pipelines API](#pipelines-api)). - Run scheduled pipelines. - Run pipelines using triggers. +- Run on-demand DAST scan. - Trigger manual actions on existing pipelines. - Retry or cancel existing jobs (using the Web UI or pipelines API). @@ -263,13 +264,13 @@ sensitive information like deployment credentials and tokens. branches, preventing untrusted code from executing on the protected runner and preserving deployment keys and other credentials from being unintentionally accessed. In order to ensure that jobs intended to be executed on protected -runners will not use regular runners, they must be tagged accordingly. +runners do not use regular runners, they must be tagged accordingly. ## View jobs in a pipeline When you access a pipeline, you can see the related jobs for that pipeline. -Clicking an individual job will show you its job log, and allow you to: +Clicking an individual job shows you its job log, and allows you to: - Cancel the job. - Retry the job. @@ -325,10 +326,10 @@ If you have many similar jobs, your [pipeline graph](#visualize-pipelines) becom to read. You can automatically group similar jobs together. If the job names are formatted in a certain way, -they will be collapsed into a single group in regular pipeline graphs (not the mini graphs). +they are collapsed into a single group in regular pipeline graphs (not the mini graphs). -You'll know when a pipeline has grouped jobs if you don't see the retry or -cancel button inside them. Hovering over them will show the number of grouped +You can recognize when a pipeline has grouped jobs if you don't see the retry or +cancel button inside them. Hovering over them shows the number of grouped jobs. Click to expand them.  @@ -342,7 +343,7 @@ separate each job name with a number and one of the following: You can use these symbols interchangeably. -For example, these three jobs will be in a group named `build ruby`: +In the example below, these three jobs are in a group named `build ruby`: ```yaml build ruby 1/3: @@ -365,7 +366,7 @@ In the pipeline, the result is a group named `build ruby` with three jobs:  -The jobs will be ordered by comparing the numbers from left to right. You +The jobs are be ordered by comparing the numbers from left to right. You usually want the first number to be the index and the second number to be the total. [This regular expression](https://gitlab.com/gitlab-org/gitlab/blob/2f3dc314f42dbd79813e6251792853bc231e69dd/app/models/commit_status.rb#L99) @@ -383,7 +384,7 @@ the pipeline view, *not* the play (**{play}**) button. This is useful when you want to alter the execution of a job that uses [custom environment variables](../variables/README.md#custom-environment-variables). -Adding a variable name (key) and value here will override the value defined in +Add a variable name (key) and value here to override the value defined in [the UI or `.gitlab-ci.yml`](../variables/README.md#custom-environment-variables), for a single run of the manual job. @@ -410,7 +411,7 @@ For example, if you start rolling out new code and: > [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/14664) in GitLab 12.0. -Job logs are divided into sections that can be collapsed or expanded. Each section will display +Job logs are divided into sections that can be collapsed or expanded. Each section displays the duration. In the following example: @@ -422,8 +423,11 @@ In the following example: #### Custom collapsible sections -You can create collapsible sections in job logs by manually outputting special codes -that GitLab will use to determine what sections to collapse: +> [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/14664) in GitLab 12.0. + +You can create [collapsible sections in job logs](../pipelines/index.md#expand-and-collapse-job-log-sections) +by manually outputting special codes +that GitLab uses to determine what sections to collapse: - Section start marker: `section_start:UNIX_TIMESTAMP:SECTION_NAME\r\e[0K` + `TEXT_OF_SECTION_HEADER` - Section end marker: `section_end:UNIX_TIMESTAMP:SECTION_NAME\r\e[0K` diff --git a/doc/ci/pipelines/job_artifacts.md b/doc/ci/pipelines/job_artifacts.md index c4457d17dc2..be6886fe6b2 100644 --- a/doc/ci/pipelines/job_artifacts.md +++ b/doc/ci/pipelines/job_artifacts.md @@ -11,13 +11,13 @@ type: reference, howto > - Introduced in GitLab 8.2 and GitLab Runner 0.7.0. > - Starting with GitLab 8.4 and GitLab Runner 1.0, the artifacts archive format changed to `ZIP`, and it's now possible to browse its contents, with the added ability of downloading the files separately. > - In GitLab 8.17, builds were renamed to jobs. -> - The artifacts browser will be available only for new artifacts that are sent to GitLab using GitLab Runner version 1.0 and up. It won't be possible to browse old artifacts already uploaded to GitLab. +> - The artifacts browser is available only for new artifacts that are sent to GitLab using GitLab Runner version 1.0 and up. You cannot browse old artifacts already uploaded to GitLab. Job artifacts are a list of files and directories created by a job once it finishes. This feature is [enabled by default](../../administration/job_artifacts.md) in all GitLab installations. -Job artifacts created by GitLab Runner are uploaded to GitLab and are downloadable as a single archive using the GitLab UI or the [GitLab API](../../api/jobs.md#get-job-artifacts). +Job artifacts created by GitLab Runner are uploaded to GitLab and are downloadable as a single archive using the GitLab UI or the [GitLab API](../../api/job_artifacts.md#get-job-artifacts). <i class="fa fa-youtube-play youtube" aria-hidden="true"></i> For an overview, watch the video [GitLab CI Pipeline, Artifacts, and Environments](https://www.youtube.com/watch?v=PCKDICEe10s). @@ -42,9 +42,9 @@ the latex source file `mycv.tex`. We then define the `artifacts` paths which in turn are defined with the `paths` keyword. All paths to files and directories are relative to the repository that was cloned during the build. -The artifacts will be uploaded when the job succeeds by default, but can be set to upload -when the job fails, or always, if the [`artifacts:when`](../yaml/README.md#artifactswhen) -parameter is used. These uploaded artifacts will be kept in GitLab for 1 week as defined +By default, the artifacts upload when the job succeeds. You can also set artifacts to upload +when the job fails, or always, by using [`artifacts:when`](../yaml/README.md#artifactswhen) +parameter. GitLab keeps these uploaded artifacts for 1 week, as defined by the `expire_in` definition. You can keep the artifacts from expiring via the [web interface](#browsing-artifacts). If the expiry time is not defined, it defaults to the [instance wide setting](../../user/admin_area/settings/continuous_integration.md#default-artifacts-expiration-core-only). @@ -94,13 +94,12 @@ rspec: junit: rspec.xml ``` -The collected JUnit reports will be uploaded to GitLab as an artifact and will -be automatically shown in merge requests. +The collected JUnit reports upload to GitLab as an artifact and display in merge requests. NOTE: **Note:** -In case the JUnit tool you use exports to multiple XML files, you can specify -multiple test report paths within a single job and they will be automatically -concatenated into a single file. Use a filename pattern (`junit: rspec-*.xml`), +If the JUnit tool you use exports to multiple XML files, specify +multiple test report paths within a single job to +concatenate them into a single file. Use a filename pattern (`junit: rspec-*.xml`), an array of filenames (`junit: [rspec-1.xml, rspec-2.xml, rspec-3.xml]`), or a combination thereof (`junit: [rspec.xml, test-results/TEST-*.xml]`). @@ -130,8 +129,8 @@ There are a couple of exceptions to the [original dotenv rules](https://github.c > - Requires [GitLab Runner](https://docs.gitlab.com/runner/) 11.5 and above. The `cobertura` report collects [Cobertura coverage XML files](../../user/project/merge_requests/test_coverage_visualization.md). -The collected Cobertura coverage reports will be uploaded to GitLab as an artifact -and will be automatically shown in merge requests. +The collected Cobertura coverage reports upload to GitLab as an artifact +and display in merge requests. Cobertura was originally developed for Java, but there are many third party ports for other languages like JavaScript, Python, Ruby, and so on. @@ -142,7 +141,7 @@ third party ports for other languages like JavaScript, Python, Ruby, and so on. > - Requires [GitLab Runner](https://docs.gitlab.com/runner/) 11.5 and above. The `terraform` report obtains a Terraform `tfplan.json` file. [JQ processing required to remove credentials](../../user/infrastructure/index.md#output-terraform-plan-information-into-a-merge-request). The collected Terraform -plan report will be uploaded to GitLab as an artifact and will be automatically shown +plan report uploads to GitLab as an artifact and displays in merge requests. For more information, see [Output `terraform plan` information into a merge request](../../user/infrastructure/index.md#output-terraform-plan-information-into-a-merge-request). @@ -155,8 +154,7 @@ in merge requests. For more information, see The `codequality` report collects [CodeQuality issues](../../user/project/merge_requests/code_quality.md) as artifacts. -The collected Code Quality report will be uploaded to GitLab as an artifact and will -be summarized in merge requests. +The collected Code Quality report uploads to GitLab as an artifact and is summarized in merge requests. #### `artifacts:reports:sast` **(ULTIMATE)** @@ -166,8 +164,8 @@ be summarized in merge requests. The `sast` report collects [SAST vulnerabilities](../../user/application_security/sast/index.md) as artifacts. -The collected SAST report will be uploaded to GitLab as an artifact and will be summarized -in the merge requests and pipeline view. It's also used to provide data for security +The collected SAST report uploads to GitLab as an artifact and is summarized +in merge requests and the pipeline view. It's also used to provide data for security dashboards. #### `artifacts:reports:secret_detection` **(ULTIMATE)** @@ -190,8 +188,7 @@ dashboards. The `dependency_scanning` report collects [Dependency Scanning vulnerabilities](../../user/application_security/dependency_scanning/index.md) as artifacts. -The collected Dependency Scanning report will be uploaded to GitLab as an artifact and will -be summarized in the merge requests and pipeline view. It's also used to provide data for security +The collected Dependency Scanning report uploads to GitLab as an artifact and is summarized in merge requests and the pipeline view. It's also used to provide data for security dashboards. #### `artifacts:reports:container_scanning` **(ULTIMATE)** @@ -202,8 +199,8 @@ dashboards. The `container_scanning` report collects [Container Scanning vulnerabilities](../../user/application_security/container_scanning/index.md) as artifacts. -The collected Container Scanning report will be uploaded to GitLab as an artifact and will -be summarized in the merge requests and pipeline view. It's also used to provide data for security +The collected Container Scanning report uploads to GitLab as an artifact and +is summarized in merge requests and the pipeline view. It's also used to provide data for security dashboards. #### `artifacts:reports:dast` **(ULTIMATE)** @@ -214,8 +211,7 @@ dashboards. The `dast` report collects [DAST vulnerabilities](../../user/application_security/dast/index.md) as artifacts. -The collected DAST report will be uploaded to GitLab as an artifact and will -be summarized in the merge requests and pipeline view. It's also used to provide data for security +The collected DAST report uploads to GitLab as an artifact and is summarized in merge requests and the pipeline view. It's also used to provide data for security dashboards. #### `artifacts:reports:license_management` **(ULTIMATE)** @@ -231,8 +227,7 @@ introduced in GitLab 12.8. The `license_management` report collects [Licenses](../../user/compliance/license_compliance/index.md) as artifacts. -The collected License Compliance report will be uploaded to GitLab as an artifact and will -be summarized in the merge requests and pipeline view. It's also used to provide data for security +The collected License Compliance report uploads to GitLab as an artifact and is summarized in merge requests and the pipeline view. It's also used to provide data for security dashboards. #### `artifacts:reports:license_scanning` **(ULTIMATE)** @@ -243,8 +238,7 @@ dashboards. The `license_scanning` report collects [Licenses](../../user/compliance/license_compliance/index.md) as artifacts. -The License Compliance report will be uploaded to GitLab as an artifact and will -be automatically shown in merge requests, pipeline view and provide data for security +The License Compliance report uploads to GitLab as an artifact and displays automatically in merge requests and the pipeline view, and provide data for security dashboards. #### `artifacts:reports:performance` **(PREMIUM)** @@ -255,8 +249,7 @@ dashboards. The `performance` report collects [Browser Performance Testing metrics](../../user/project/merge_requests/browser_performance_testing.md) as artifacts. -The collected Browser Performance report will be uploaded to GitLab as an artifact and will -be automatically shown in merge requests. +The collected Browser Performance report uploads to GitLab as an artifact and displays in merge requests. #### `artifacts:reports:load_performance` **(PREMIUM)** @@ -276,8 +269,7 @@ shown in merge requests automatically. The `metrics` report collects [Metrics](../metrics_reports.md) as artifacts. -The collected Metrics report will be uploaded to GitLab as an artifact and will -be automatically shown in merge requests. +The collected Metrics report uploads to GitLab as an artifact and displays in merge requests. #### `artifacts:reports:requirements` **(ULTIMATE)** @@ -286,15 +278,15 @@ be automatically shown in merge requests. The `requirements` report collects `requirements.json` files as artifacts. -The collected Requirements report will be uploaded to GitLab as an artifact and -existing [requirements](../../user/project/requirements/index.md) will be +The collected Requirements report uploads to GitLab as an artifact and +existing [requirements](../../user/project/requirements/index.md) are marked as Satisfied. ## Browsing artifacts > - From GitLab 9.2, PDFs, images, videos, and other formats can be previewed directly in the job artifacts browser without the need to download them. > - Introduced in [GitLab 10.1](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/14399), HTML files in a public project can be previewed directly in a new tab without the need to download them when [GitLab Pages](../../administration/pages/index.md) is enabled. The same applies for textual formats (currently supported extensions: `.txt`, `.json`, and `.log`). -> - Introduced in [GitLab 12.4](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/16675), artifacts in private projects can be previewed when [GitLab Pages access control](../../administration/pages/index.md#access-control) is enabled. +> - Introduced in [GitLab 12.4](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/16675), artifacts in internal and private projects can be previewed when [GitLab Pages access control](../../administration/pages/index.md#access-control) is enabled. After a job finishes, if you visit the job's specific page, there are three buttons. You can download the artifacts archive or browse its contents, whereas @@ -311,6 +303,8 @@ Below you can see what browsing looks like. In this case we have browsed inside the archive and at this point there is one directory, a couple files, and one HTML file that you can view directly online when [GitLab Pages](../../administration/pages/index.md) is enabled (opens in a new tab). +Select artifacts in internal and private projects can only be previewed when +[GitLab Pages access control](../../administration/pages/index.md#access-control) is enabled.  @@ -346,10 +340,8 @@ so you can use it for scripting purposes. NOTE: **Note:** The latest artifacts are created by jobs in the **most recent** successful pipeline -for the specific ref. If you run two types of pipelines for the same ref, the latest -artifact will be determined by timing. For example, if a branch pipeline created -by merging a merge request runs at the same time as a scheduled pipeline, the -latest artifact will be from the pipeline that completed most recently. +for the specific ref. If you run two types of pipelines for the same ref, timing determines the latest +artifact. For example, if a merge request creates a branch pipeline at the same time as a scheduled pipeline, the pipeline that completed most recently creates the latest artifact. Artifacts for other pipelines can be accessed with direct access to them. @@ -423,7 +415,7 @@ information in the UI. DANGER: **Danger:** This is a destructive action that leads to data loss. Use with caution. -You can erase a single job via the UI, which will also remove the job's +You can erase a single job via the UI, which also removes the job's artifacts and trace, if you are: - The owner of the job. @@ -437,7 +429,7 @@ To erase a job: ## Retrieve artifacts of private projects when using GitLab CI -In order to retrieve a job artifact of a different project, you might need to use a private token in order to [authenticate and download](../../api/jobs.md#get-job-artifacts) the artifacts. +In order to retrieve a job artifact of a different project, you might need to use a private token in order to [authenticate and download](../../api/job_artifacts.md#get-job-artifacts) the artifacts. <!-- ## Troubleshooting diff --git a/doc/ci/pipelines/settings.md b/doc/ci/pipelines/settings.md index 39acef14443..40093167213 100644 --- a/doc/ci/pipelines/settings.md +++ b/doc/ci/pipelines/settings.md @@ -125,7 +125,8 @@ can use <https://rubular.com> to test your regex. The regex returns the **last** match found in the output. If the pipeline succeeds, the coverage is shown in the merge request widget and -in the jobs table. +in the jobs table. If multiple jobs in the pipeline have coverage reports, they will +be averaged.  @@ -207,7 +208,7 @@ If you want all pending non-HEAD pipelines on branches to auto-cancel each time a new pipeline is created, such as after a Git push or manually from the UI, you can enable this in the project settings: -1. Go to **{settings}** **Settings > CI / CD**. +1. Go to **Settings > CI / CD**. 1. Expand **General Pipelines**. 1. Check the **Auto-cancel redundant, pending pipelines** checkbox. 1. Click **Save changes**. @@ -226,7 +227,7 @@ newer one, which may not be what you want. To avoid this scenario: -1. Go to **{settings}** **Settings > CI / CD**. +1. Go to **Settings > CI / CD**. 1. Expand **General pipelines**. 1. Check the **Skip outdated deployment jobs** checkbox. 1. Click **Save changes**. diff --git a/doc/ci/quick_start/README.md b/doc/ci/quick_start/README.md index 25469570603..050df243af4 100644 --- a/doc/ci/quick_start/README.md +++ b/doc/ci/quick_start/README.md @@ -7,24 +7,6 @@ type: reference # Getting started with GitLab CI/CD -NOTE: **Note:** -Starting from version 8.0, GitLab [Continuous Integration](https://about.gitlab.com/stages-devops-lifecycle/continuous-integration/) (CI) -is fully integrated into GitLab itself and is [enabled](../enable_or_disable_ci.md) by default on all -projects. - -NOTE: **Note:** -Please keep in mind that only project Maintainers and Admin users have -the permissions to access a project's settings. - -NOTE: **Note:** -Coming over to GitLab from Jenkins? Check out our [reference](../jenkins/index.md) -for converting your pre-existing pipelines over to our format. - -NOTE: **Note:** -There are a few different [basic pipeline architectures](../pipelines/pipeline_architectures.md) -that you can consider for use in your project. You may want to familiarize -yourself with these prior to getting started. - GitLab offers a [continuous integration](https://about.gitlab.com/stages-devops-lifecycle/continuous-integration/) service. For each commit or push to trigger your CI [pipeline](../pipelines/index.md), you must: @@ -49,7 +31,11 @@ something. It's also common to use pipelines to automatically deploy tested code to staging and production environments. ---- +If you're already familiar with general CI/CD concepts, you can review which +[pipeline architectures](../pipelines/pipeline_architectures.md) can be used +in your projects. If you're coming over to GitLab from Jenkins, you can check out +our [reference](../migration/jenkins.md) for converting your pre-existing pipelines +over to our format. This guide assumes that you have: @@ -82,18 +68,22 @@ blog about it](https://about.gitlab.com/blog/2015/05/06/why-were-replacing-gitla ### Creating a simple `.gitlab-ci.yml` file ->**Note:** -`.gitlab-ci.yml` is a [YAML](https://en.wikipedia.org/wiki/YAML) file -so you have to pay extra attention to indentation. Always use spaces, not tabs. +NOTE: **Note:** +A GitLab team member has made an [unofficial visual pipeline editor](https://unofficial.gitlab.tools/visual-pipelines/). +There is a [plan to make it an official part of GitLab](https://gitlab.com/groups/gitlab-org/-/epics/4069) +in the future, but it's available for anyone who wants to try it at the above link. You need to create a file named `.gitlab-ci.yml` in the root directory of your -repository. Below is an example for a Ruby on Rails project. +repository. This is a [YAML](https://en.wikipedia.org/wiki/YAML) file +so you have to pay extra attention to indentation. Always use spaces, not tabs. + +Below is an example for a Ruby on Rails project: ```yaml image: "ruby:2.5" before_script: - - apt-get update -qq && apt-get install -y -qq sqlite3 libsqlite3-dev nodejs + - sudo apt-get update -qq && sudo apt-get install -y -qq sqlite3 libsqlite3-dev nodejs - ruby -v - which ruby - gem install bundler --no-document @@ -124,9 +114,7 @@ Jobs are used to create jobs, which are then picked by What is important is that each job is run independently from each other. If you want to check whether the `.gitlab-ci.yml` of your project is valid, there is a -Lint tool under the page `/-/ci/lint` of your project namespace. You can also find -a "CI Lint" button to go to this page under **CI/CD ➔ Pipelines** and -**Pipelines ➔ Jobs** in your project. +[CI Lint tool](../lint.md) available in every project. For more information and a complete `.gitlab-ci.yml` syntax, please read [the reference documentation on `.gitlab-ci.yml`](../yaml/README.md). diff --git a/doc/ci/runners/README.md b/doc/ci/runners/README.md index 21c99f928d8..6d248156004 100644 --- a/doc/ci/runners/README.md +++ b/doc/ci/runners/README.md @@ -7,7 +7,7 @@ type: reference # Configuring GitLab Runners <!-- This topic contains several commented-out sections that were accidentally added in 13.2.--> -<!-- The commented-out sections will be added back in 13.3.--> +<!-- The commented-out sections are added back in 13.3.--> In GitLab CI/CD, Runners run the code defined in [`.gitlab-ci.yml`](../yaml/README.md). A GitLab Runner is a lightweight, highly-scalable agent that picks up a CI job through @@ -118,7 +118,7 @@ You can also enable shared Runners for individual projects. To enable shared Runners: -1. Go to the project's **{settings}** **Settings > CI/CD** and expand the **Runners** section. +1. Go to the project's **Settings > CI/CD** and expand the **Runners** section. 1. Click **Allow shared Runners**. #### Disable shared Runners @@ -128,12 +128,12 @@ You must have Owner permissions for the project<!-- or group-->. To disable shared Runners for a project: -1. Go to the project's **{settings}** **Settings > CI/CD** and expand the **Runners** section. +1. Go to the project's **Settings > CI/CD** and expand the **Runners** section. 1. In the **Shared Runners** area, click **Disable shared Runners**. <!--To disable shared Runners for a group: -1. Go to the group's **{settings}** **Settings > CI/CD** and expand the **Runners** section. +1. Go to the group's **Settings > CI/CD** and expand the **Runners** section. 1. In the **Shared Runners** area, click **Disable shared Runners globally**. 1. Optionally, to allow shared Runners to be enabled for individual projects or subgroups, click **Allow projects/subgroups to override the global setting**. @@ -155,20 +155,20 @@ To create a group Runner: 1. [Install Runner](https://docs.gitlab.com/runner/install/). 1. Go to the group you want to make the Runner work for. -1. Go to **{settings}** **Settings > CI/CD** and expand the **Runners** section. +1. Go to **Settings > CI/CD** and expand the **Runners** section. 1. Note the URL and token. 1. [Register the Runner](https://docs.gitlab.com/runner/register/). -<!-- #### View and manage group Runners +#### View and manage group Runners -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/37366/) in GitLab 13.3. +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/37366/) in GitLab 13.2. You can view and manage all Runners for a group, its subgroups, and projects. You can do this for your self-managed GitLab instance or for GitLab.com. You must have [Owner permissions](../../user/permissions.md#group-members-permissions) for the group. 1. Go to the group where you want to view the Runners. -1. Go to **{settings}** **Settings > CI/CD** and expand the **Runners** section. +1. Go to **Settings > CI/CD** and expand the **Runners** section. 1. The following fields are displayed. | Attribute | Description | @@ -183,7 +183,7 @@ You must have [Owner permissions](../../user/permissions.md#group-members-permis | Tags | Tags associated with the Runner | | Last contact | Timestamp indicating when the GitLab instance last contacted the Runner | -From this page, you can edit, pause, and remove Runners from the group, its subgroups, and projects. --> +From this page, you can edit, pause, and remove Runners from the group, its subgroups, and projects. #### Pause or remove a group Runner @@ -191,11 +191,11 @@ You can pause or remove a group Runner for your self-managed GitLab instance or You must have [Owner permissions](../../user/permissions.md#group-members-permissions) for the group. 1. Go to the group you want to remove or pause the Runner for. -1. Go to **{settings}** **Settings > CI/CD** and expand the **Runners** section. +1. Go to **Settings > CI/CD** and expand the **Runners** section. 1. Click **Pause** or **Remove Runner**. -<!-- - If you pause a group Runner that is used by multiple projects, the Runner pauses for all projects. --> -<!-- - From the group view, you cannot remove a Runner that is assigned to more than one project. --> -<!-- You must remove it from each project first. --> + - If you pause a group Runner that is used by multiple projects, the Runner pauses for all projects. + - From the group view, you cannot remove a Runner that is assigned to more than one project. + You must remove it from each project first. 1. On the confirmation dialog, click **OK**. ### Specific Runners @@ -223,7 +223,7 @@ You must have [Owner permissions](../../user/permissions.md#project-members-perm To create a specific Runner: 1. [Install Runner](https://docs.gitlab.com/runner/install/). -1. Go to the project's **{settings}** **Settings > CI/CD** and expand the **Runners** section. +1. Go to the project's **Settings > CI/CD** and expand the **Runners** section. 1. Note the URL and token. 1. [Register the Runner](https://docs.gitlab.com/runner/register/). @@ -237,7 +237,7 @@ enable a specific Runner to apply to additional projects. To enable or disable a specific Runner for a project: -1. Go to the project's **{settings}** **Settings > CI/CD** and expand the **Runners** section. +1. Go to the project's **Settings > CI/CD** and expand the **Runners** section. 1. Click **Enable for this project** or **Disable for this project**. #### Prevent a specific Runner from being enabled for other projects @@ -248,7 +248,7 @@ but can also be changed later. To lock or unlock a Runner: -1. Go to the project's **{settings}** **Settings > CI/CD** and expand the **Runners** section. +1. Go to the project's **Settings > CI/CD** and expand the **Runners** section. 1. Find the Runner you want to lock or unlock. Make sure it's enabled. 1. Click the pencil button. 1. Check the **Lock to current projects** option. @@ -266,7 +266,7 @@ if smaller than the [project defined timeout](../pipelines/settings.md#timeout), This feature can be used to prevent your shared Runner from being overwhelmed by a project that has jobs with a long timeout (for example, one week). -When not configured, Runners will not override the project timeout. +When not configured, Runners do not override the project timeout. How this feature works: @@ -317,7 +317,7 @@ and ignores other jobs. To protect or unprotect a Runner: -1. Go to the project's **{settings}** **Settings > CI/CD** and expand the **Runners** section. +1. Go to the project's **Settings > CI/CD** and expand the **Runners** section. 1. Find the Runner you want to protect or unprotect. Make sure it's enabled. 1. Click the pencil button. 1. Check the **Protected** option. @@ -329,8 +329,7 @@ To protect or unprotect a Runner: Whenever a project is forked, it copies the settings of the jobs that relate to it. This means that if you have shared Runners set up for a project and -someone forks that project, the shared Runners will also serve jobs of this -project. +someone forks that project, the shared Runners serve jobs of this project. ### Attack vectors in Runners @@ -346,14 +345,14 @@ may then be used to obtain the values of secret variables or to clone project co To reset the token: -1. Go to the project's **{settings}** **Settings > CI/CD**. +1. Go to the project's **Settings > CI/CD**. 1. Expand the **General pipelines settings** section. 1. Find the **Runner token** form field and click the **Reveal value** button. 1. Delete the value and save the form. 1. After the page is refreshed, expand the **Runners settings** section and check the registration token - it should be changed. -From now on the old token is no longer valid and will not register +From now on the old token is no longer valid and does not register any new Runners to the project. If you are using any tools to provision and register new Runners, the tokens used in those tools should be updated to reflect the value of the new token. @@ -376,7 +375,7 @@ different places. To view the IP address of a shared Runner you must have admin access to the GitLab instance. To determine this: -1. Visit **{admin}** **Admin Area > Overview > Runners**. +1. Visit **Admin Area > Overview > Runners**. 1. Look for the Runner in the table and you should see a column for **IP Address**.  @@ -386,7 +385,7 @@ the GitLab instance. To determine this: To can find the IP address of a Runner for a specific project, you must have Owner [permissions](../../user/permissions.md#project-members-permissions) for the project. -1. Go to the project's **{settings}** **Settings > CI/CD** and expand the **Runners** section. +1. Go to the project's **Settings > CI/CD** and expand the **Runners** section. 1. On the details page you should see a row for **IP Address**.  @@ -409,7 +408,7 @@ To change this, you must have Owner [permissions](../../user/permissions.md#proj To make a Runner pick untagged jobs: -1. Go to the project's **{settings}** **Settings > CI/CD** and expand the **Runners** section. +1. Go to the project's **Settings > CI/CD** and expand the **Runners** section. 1. Find the Runner you want to pick untagged jobs and make sure it's enabled. 1. Click the pencil button. 1. Check the **Run untagged jobs** option. diff --git a/doc/ci/services/redis.md b/doc/ci/services/redis.md index 193f78f240b..77668d9104b 100644 --- a/doc/ci/services/redis.md +++ b/doc/ci/services/redis.md @@ -34,7 +34,7 @@ And that's it. Redis will now be available to be used within your testing framework. You can also use any other Docker image available on [Docker Hub](https://hub.docker.com/_/redis). -For example, to use Redis 2.8 the service becomes `redis:2.8`. +For example, to use Redis 6.0 the service becomes `redis:6.0`. ## Use Redis with the Shell executor diff --git a/doc/ci/ssh_keys/README.md b/doc/ci/ssh_keys/README.md index 4ad4758b281..b1847ffbc60 100644 --- a/doc/ci/ssh_keys/README.md +++ b/doc/ci/ssh_keys/README.md @@ -66,7 +66,7 @@ to access it. This is where an SSH key pair comes in handy. ## Install ssh-agent if not already installed, it is required by Docker. ## (change apt-get to yum if you use an RPM-based image) ## - - 'which ssh-agent || ( apt-get update -y && apt-get install openssh-client -y )' + - 'command -v ssh-agent >/dev/null || ( apt-get update -y && apt-get install openssh-client -y )' ## ## Run ssh-agent (inside the build environment) diff --git a/doc/ci/triggers/README.md b/doc/ci/triggers/README.md index 47f11a6228c..2b006b8779b 100644 --- a/doc/ci/triggers/README.md +++ b/doc/ci/triggers/README.md @@ -7,12 +7,6 @@ type: tutorial # Triggering pipelines through the API -> **Notes**: -> -> - [Introduced](https://about.gitlab.com/releases/2015/08/22/gitlab-7-14-released/) in GitLab 7.14. -> - GitLab 8.12 has a completely redesigned job permissions system. Read all -> about the [new model and its implications](../../user/project/new_ci_build_permissions_model.md#pipeline-triggers). - Triggers can be used to force a pipeline rerun of a specific `ref` (branch or tag) with an API call. @@ -97,7 +91,7 @@ This allows you to use that for multi-project pipelines and download artifacts from any project to which you have access as this follows the same principles with the [permission model](../../user/permissions.md#job-permissions). -Read more about the [jobs API](../../api/jobs.md#download-the-artifacts-archive). +Read more about the [jobs API](../../api/job_artifacts.md#download-the-artifacts-archive). ## Adding a new trigger @@ -120,11 +114,6 @@ The action is irreversible. ## Triggering a pipeline -> **Notes**: -> -> - Valid refs are only the branches and tags. If you pass a commit SHA as a ref, -> it will not trigger a job. - To trigger a job you need to send a `POST` request to GitLab's API endpoint: ```plaintext @@ -132,8 +121,8 @@ POST /projects/:id/trigger/pipeline ``` The required parameters are the [trigger's `token`](#authentication-tokens) -and the Git `ref` on which the trigger will be performed. Valid refs are the -branch and the tag. The `:id` of a project can be found by +and the Git `ref` on which the trigger will be performed. Valid refs are +branches or tags. The `:id` of a project can be found by [querying the API](../../api/projects.md) or by visiting the **CI/CD** settings page which provides self-explanatory examples. @@ -142,16 +131,12 @@ UI under the **Jobs** page and the jobs are marked as triggered 'by API'.  ---- - You can see which trigger caused the rebuild by visiting the single job page. A part of the trigger's token is exposed in the UI as you can see from the image below.  ---- - By using cURL you can trigger a pipeline rerun with minimal effort, for example: ```shell @@ -191,14 +176,6 @@ This means that whenever a new tag is pushed on project A, the job will run and ## Triggering a pipeline from a webhook -> **Notes**: -> -> - Introduced in GitLab 8.14. -> - `ref` should be passed as part of the URL in order to take precedence over -> `ref` from the webhook body that designates the branch ref that fired the -> trigger in the source repository. -> - `ref` should be URL-encoded if it contains slashes. - To trigger a job from a webhook of another project you need to add the following webhook URL for Push and Tag events (change the project ID, ref and token): @@ -206,6 +183,10 @@ webhook URL for Push and Tag events (change the project ID, ref and token): https://gitlab.example.com/api/v4/projects/9/ref/master/trigger/pipeline?token=TOKEN ``` +`ref` should be passed as part of the URL in order to take precedence over +`ref` from the webhook body that designates the branch ref that fired the +trigger in the source repository. `ref` should be URL-encoded if it contains slashes. + ## Making use of trigger variables You can pass any number of arbitrary variables in the trigger API call and they @@ -271,7 +252,7 @@ of all types of variables. ## Using cron to trigger nightly pipelines ->**Note:** +NOTE: **Note:** The following behavior can also be achieved through GitLab's UI with [pipeline schedules](../pipelines/schedules.md). diff --git a/doc/ci/troubleshooting.md b/doc/ci/troubleshooting.md index a019f8232a9..96d94a6c165 100644 --- a/doc/ci/troubleshooting.md +++ b/doc/ci/troubleshooting.md @@ -7,6 +7,24 @@ type: reference # Troubleshooting CI/CD +## Pipeline warnings + +Pipeline configuration warnings are shown when you: + +- [View pipeline details](pipelines/index.md#view-pipelines). +- [Validate configuration with the CI Lint tool](yaml/README.md#validate-the-gitlab-ciyml). +- [Manually run a pipeline](pipelines/index.md#run-a-pipeline-manually). + +### "Job may allow multiple pipelines to run for a single action" + +When you use [`rules`](yaml/README.md#rules) with a `when:` clause without +an `if:` clause, multiple pipelines may run. Usually +this occurs when you push a commit to a branch that has an open merge request associated with it. + +To [prevent duplicate pipelines](yaml/README.md#prevent-duplicate-pipelines), use +[`workflow: rules`](yaml/README.md#workflowrules) or rewrite your rules +to control which pipelines can run. + ## Merge request pipeline widget The merge request pipeline widget shows information about the pipeline status in a Merge Request. It's displayed above the [merge request ability to merge widget](#merge-request-ability-to-merge-widget). @@ -15,16 +33,15 @@ There are several messages that can be displayed depending on the status of the ### "Checking pipeline status" -This message is shown when the merge request has no pipeline associated with the latest commit yet and [Pipelines must succeed](../user/project/merge_requests/merge_when_pipeline_succeeds.md#only-allow-merge-requests-to-be-merged-if-the-pipeline-succeeds) is turned on. This might be because: +This message is shown when the merge request has no pipeline associated with the latest commit yet. This might be because: - GitLab hasn't finished creating the pipeline yet. - You are using an external CI service and GitLab hasn't heard back from the service yet. - You are not using CI/CD pipelines in your project. +- The latest pipeline was deleted (this is a [known issue](https://gitlab.com/gitlab-org/gitlab/-/issues/214323)). After the pipeline is created, the message will update with the pipeline status. -Note: Currently if you delete the latest pipeline of a Merge Request, this message will be shown instead of a meaningful error message. This is a known issue and should be resolved soon. - ## Merge request ability to merge widget The merge request status widget shows the **Merge** button and whether or not a merge request is ready to merge. If the merge request can't be merged, the reason for this is displayed. diff --git a/doc/ci/variables/README.md b/doc/ci/variables/README.md index 4f9a1d8dd27..61bc466692e 100644 --- a/doc/ci/variables/README.md +++ b/doc/ci/variables/README.md @@ -74,8 +74,8 @@ pages: - echo $CI_PAGES_DOMAIN ``` -For GitLab.com users, the output will be `gitlab.io`. For your -private instance, the output will be whatever your sysadmin has +For GitLab.com users, the output is `gitlab.io`. For your +private instance, the output is whatever your sysadmin has defined. ## Custom environment variables @@ -120,8 +120,8 @@ From within the UI, you can add or update custom environment variables: - **Value**: No limitations. - **Type**: `File` or `Variable`. - **Environment scope**: `All`, or specific environments. - - **Protect variable** (Optional): If selected, the variable will only be available in pipelines that run on protected branches or tags. - - **Mask variable** (Optional): If selected, the variable's **Value** will be masked in job logs. The variable fails to save if the value does not meet the [masking requirements](#masked-variable-requirements). + - **Protect variable** (Optional): If selected, the variable is only available in pipelines that run on protected branches or tags. + - **Mask variable** (Optional): If selected, the variable's **Value** is masked in job logs. The variable fails to save if the value does not meet the [masking requirements](#masked-variable-requirements). After a variable is created, you can update any of the details by clicking the **{pencil}** **Edit** button. @@ -137,7 +137,7 @@ test_variable: - cat $GREETING # the temp file itself contains the variable value ``` -The output will be: +The output is:  @@ -187,7 +187,7 @@ kubectl config set-cluster e2e --server="$KUBE_URL" --certificate-authority="$KU > [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/13784) in GitLab 11.10 -Variables can be masked so that the value of the variable will be hidden in job logs. +Variables can be masked so that the value of the variable is hidden in job logs. To mask a variable: @@ -308,7 +308,7 @@ job_name: You can also list all environment variables with the `export` command in Bash or `dir env:` command in PowerShell. -Be aware that this will also expose the values of all the variables +Be aware that this also exposes the values of all the variables you set, in the job log: ```yaml @@ -376,8 +376,8 @@ These variables are saved in the repository, and they are meant to store non-sensitive project configuration, like `RAILS_ENV` or `DATABASE_URL`. -For example, if you set the variable below globally (not inside a job), it will -be used in all executed commands and scripts: +For example, if you set the variable below globally (not inside a job), it is +used in all executed commands and scripts: ```yaml variables: @@ -419,9 +419,9 @@ Group-level variables can be added by: 1. Navigating to your group's **Settings > CI/CD** page. 1. Inputting variable types, keys, and values in the **Variables** section. - Any variables of [subgroups](../../user/group/subgroups/index.md) will be inherited recursively. + Any variables of [subgroups](../../user/group/subgroups/index.md) are inherited recursively. -Once you set them, they will be available for all subsequent pipelines. Any group-level user defined variables can be viewed in projects by: +Once you set them, they are available for all subsequent pipelines. Any group-level user defined variables can be viewed in projects by: 1. Navigating to the project's **Settings > CI/CD** page. 1. Expanding the **Variables** section. @@ -444,11 +444,11 @@ To add an instance-level variable: 1. Navigate to your admin area's **Settings > CI/CD** and expand the **Variables** section. 1. Click the **Add variable** button, and fill in the details: - - **Key**: Must be one line, using only letters, numbers, or `_` (underscore), with no spaces. - - **Value**: 700 characters allowed. - - **Type**: `File` or `Variable`. - - **Protect variable** (Optional): If selected, the variable will only be available in pipelines that run on protected branches or tags. - - **Mask variable** (Optional): If selected, the variable's **Value** will not be shown in job logs. The variable will not be saved if the value does not meet the [masking requirements](#masked-variable-requirements). + - **Key**: Must be one line, using only letters, numbers, or `_` (underscore), with no spaces. + - **Value**: [Since GitLab 13.3](https://gitlab.com/gitlab-org/gitlab/-/issues/220028), 10,000 characters allowed. This is also bounded by the limits of the selected Runner operating system. In GitLab 13.0 to 13.2, 700 characters allowed. + - **Type**: `File` or `Variable`. + - **Protect variable** (Optional): If selected, the variable is only available in pipelines that run on protected branches or tags. + - **Mask variable** (Optional): If selected, the variable's **Value** is not shown in job logs. The variable is not saved if the value does not meet the [masking requirements](#masked-variable-requirements). After a variable is created, you can update any of the details by clicking the **{pencil}** **Edit** button. @@ -540,14 +540,14 @@ For example, if you define: - `API_TOKEN=secure` as a project variable. - `API_TOKEN=yaml` in your `.gitlab-ci.yml`. -`API_TOKEN` will take the value `secure` as the project +`API_TOKEN` takes the value `secure` as the project variables take precedence over those defined in `.gitlab-ci.yml`. ## Unsupported variables Variable names are limited by the underlying shell used to execute scripts (see [available shells](https://docs.gitlab.com/runner/shells/index.html). Each shell has its own unique set of reserved variable names. -You will also want to keep in mind the [scope of environment variables](where_variables_can_be_used.md) to ensure a variable is defined in the scope +You also want to keep in mind the [scope of environment variables](where_variables_can_be_used.md) to ensure a variable is defined in the scope in which you wish to use it. ## Where variables can be used @@ -585,8 +585,8 @@ pass CI variables to the running application by prefixing the key of the variable with `K8S_SECRET_`. These [prefixed -variables](../../topics/autodevops/customize.md#application-secret-variables) will -then be available as environment variables on the running application +variables](../../topics/autodevops/customize.md#application-secret-variables) are +then available as environment variables on the running application container. CAUTION: **Caution:** @@ -649,94 +649,132 @@ This follows the usual rules for [`only` / `except` policies](../yaml/README.md# ### Syntax of environment variable expressions -Below you can find supported syntax reference: +Below you can find supported syntax reference. + +#### Equality matching using a string + +Examples: + +- `$VARIABLE == "some value"` +- `$VARIABLE != "some value"` (introduced in GitLab 11.11) + +You can use equality operator `==` or `!=` to compare a variable content to a +string. We support both, double quotes and single quotes to define a string +value, so both `$VARIABLE == "some value"` and `$VARIABLE == 'some value'` +are supported. `"some value" == $VARIABLE` is correct too. + +#### Checking for an undefined value + +Examples: + +- `$VARIABLE == null` +- `$VARIABLE != null` (introduced in GitLab 11.11) + +It sometimes happens that you want to check whether a variable is defined +or not. To do that, you can compare a variable to `null` keyword, like +`$VARIABLE == null`. This expression evaluates to true if +variable is not defined when `==` is used, or to false if `!=` is used. + +#### Checking for an empty variable -1. Equality matching using a string +Examples: - Examples: +- `$VARIABLE == ""` +- `$VARIABLE != ""` (introduced in GitLab 11.11) - - `$VARIABLE == "some value"` - - `$VARIABLE != "some value"` (introduced in GitLab 11.11) +If you want to check whether a variable is defined, but is empty, you can +simply compare it against an empty string, like `$VAR == ''` or non-empty +string `$VARIABLE != ""`. - You can use equality operator `==` or `!=` to compare a variable content to a - string. We support both, double quotes and single quotes to define a string - value, so both `$VARIABLE == "some value"` and `$VARIABLE == 'some value'` - are supported. `"some value" == $VARIABLE` is correct too. +#### Comparing two variables -1. Checking for an undefined value +Examples: - Examples: +- `$VARIABLE_1 == $VARIABLE_2` +- `$VARIABLE_1 != $VARIABLE_2` (introduced in GitLab 11.11) - - `$VARIABLE == null` - - `$VARIABLE != null` (introduced in GitLab 11.11) +It is possible to compare two variables. This compares values +of these variables. - It sometimes happens that you want to check whether a variable is defined - or not. To do that, you can compare a variable to `null` keyword, like - `$VARIABLE == null`. This expression evaluates to true if - variable is not defined when `==` is used, or to false if `!=` is used. +#### Variable presence check -1. Checking for an empty variable +Example: `$STAGING` - Examples: +If you only want to create a job when there is some variable present, +which means that it is defined and non-empty, you can simply use +variable name as an expression, like `$STAGING`. If `$STAGING` variable +is defined, and is non empty, expression evaluates to `true`. +`$STAGING` value needs to be a string, with length higher than zero. +Variable that contains only whitespace characters is not an empty variable. - - `$VARIABLE == ""` - - `$VARIABLE != ""` (introduced in GitLab 11.11) +#### Regex pattern matching - If you want to check whether a variable is defined, but is empty, you can - simply compare it against an empty string, like `$VAR == ''` or non-empty - string `$VARIABLE != ""`. +> [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/43601) in GitLab 11.0 -1. Comparing two variables +Examples: - Examples: +- `=~`: True if pattern is matched. Ex: `$VARIABLE =~ /^content.*/` +- `!~`: True if pattern is not matched. Ex: `$VARIABLE_1 !~ /^content.*/` ([Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/61900) in GitLab 11.11) - - `$VARIABLE_1 == $VARIABLE_2` - - `$VARIABLE_1 != $VARIABLE_2` (introduced in GitLab 11.11) +Variable pattern matching with regular expressions uses the +[RE2 regular expression syntax](https://github.com/google/re2/wiki/Syntax). +Expressions evaluate as `true` if: - It is possible to compare two variables. This is going to compare values - of these variables. +- Matches are found when using `=~`. +- Matches are *not* found when using `!~`. -1. Variable presence check +Pattern matching is case-sensitive by default. Use `i` flag modifier, like +`/pattern/i` to make a pattern case-insensitive. - Example: `$STAGING` +#### Conjunction / Disjunction - If you only want to create a job when there is some variable present, - which means that it is defined and non-empty, you can simply use - variable name as an expression, like `$STAGING`. If `$STAGING` variable - is defined, and is non empty, expression will evaluate to truth. - `$STAGING` value needs to be a string, with length higher than zero. - Variable that contains only whitespace characters is not an empty variable. +> [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/62867) in GitLab 12.0 -1. Pattern matching (introduced in GitLab 11.0) +Examples: - Examples: +- `$VARIABLE1 =~ /^content.*/ && $VARIABLE2 == "something"` +- `$VARIABLE1 =~ /^content.*/ && $VARIABLE2 =~ /thing$/ && $VARIABLE3` +- `$VARIABLE1 =~ /^content.*/ || $VARIABLE2 =~ /thing$/ && $VARIABLE3` - - `=~`: True if pattern is matched. Ex: `$VARIABLE =~ /^content.*/` - - `!~`: True if pattern is not matched. Ex: `$VARIABLE_1 !~ /^content.*/` ([Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/61900) in GitLab 11.11) +It is possible to join multiple conditions using `&&` or `||`. Any of the otherwise +supported syntax may be used in a conjunctive or disjunctive statement. +Precedence of operators follows the +[Ruby 2.5 standard](https://ruby-doc.org/core-2.5.0/doc/syntax/precedence_rdoc.html), +so `&&` is evaluated before `||`. - Variable pattern matching with regular expressions uses the - [RE2 regular expression syntax](https://github.com/google/re2/wiki/Syntax). - Expressions evaluate as `true` if: +#### Parentheses - - Matches are found when using `=~`. - - Matches are *not* found when using `!~`. +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/230938) in GitLab 13.3 - Pattern matching is case-sensitive by default. Use `i` flag modifier, like - `/pattern/i` to make a pattern case-insensitive. +It is possible to use parentheses to group conditions. Parentheses have the highest +precedence of all operators. Expressions enclosed in parentheses are evaluated first, +and the result is used for the rest of the expression. -1. Conjunction / Disjunction ([introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/27925) in GitLab 12.0) +Many nested parentheses can be used to create complex conditions, and the inner-most +expressions in parentheses are evaluated first. For an expression to be valid an equal +number of `(` and `)` need to be used. - Examples: +Examples: - - `$VARIABLE1 =~ /^content.*/ && $VARIABLE2 == "something"` - - `$VARIABLE1 =~ /^content.*/ && $VARIABLE2 =~ /thing$/ && $VARIABLE3` - - `$VARIABLE1 =~ /^content.*/ || $VARIABLE2 =~ /thing$/ && $VARIABLE3` +- `($VARIABLE1 =~ /^content.*/ || $VARIABLE2) && ($VARIABLE3 =~ /thing$/ || $VARIABLE4)` +- `($VARIABLE1 =~ /^content.*/ || $VARIABLE2 =~ /thing$/) && $VARIABLE3` +- `$CI_COMMIT_BRANCH == "my-branch" || (($VARIABLE1 == "thing" || $VARIABLE2 == "thing") && $VARIABLE3)` - It is possible to join multiple conditions using `&&` or `||`. Any of the otherwise - supported syntax may be used in a conjunctive or disjunctive statement. - Precedence of operators follows the - [Ruby 2.5 standard](https://ruby-doc.org/core-2.5.0/doc/syntax/precedence_rdoc.html), - so `&&` is evaluated before `||`. +The feature is currently deployed behind a feature flag that is **enabled by default**. +[GitLab administrators with access to the GitLab Rails console](../../administration/feature_flags.md) +can opt to disable it for your instance. + +To enable it: + +```ruby +Feature.enable(:ci_if_parenthesis_enabled) +``` + +To disable it: + +```ruby +Feature.disable(:ci_if_parenthesis_enabled) +``` ### Storing regular expressions in variables diff --git a/doc/ci/variables/predefined_variables.md b/doc/ci/variables/predefined_variables.md index 98a2e7ae22f..c79ea4b0d05 100644 --- a/doc/ci/variables/predefined_variables.md +++ b/doc/ci/variables/predefined_variables.md @@ -16,9 +16,8 @@ version of Runner required. NOTE: **Note:** Starting with GitLab 9.0, we have deprecated some variables. Read the -[9.0 Renaming](deprecated_variables.md#gitlab-90-renamed-variables) section to find out their replacements. **You are -strongly advised to use the new variables as we will remove the old ones in -future GitLab releases.** +[9.0 Renaming](deprecated_variables.md#gitlab-90-renamed-variables) section to find out their replacements. +**To avoid problems with deprecated and removed variables in future releases, you are strongly advised to use the new variables.** You can add a command to your `.gitlab-ci.yml` file to [output the values of all variables available for a job](README.md#list-all-environment-variables). @@ -49,6 +48,7 @@ Kubernetes-specific environment variables are detailed in the | `CI_CONFIG_PATH` | 9.4 | 0.5 | The path to CI configuration file. Defaults to `.gitlab-ci.yml` | | `CI_DEBUG_TRACE` | all | 1.7 | Whether [debug logging (tracing)](README.md#debug-logging) is enabled | | `CI_DEFAULT_BRANCH` | 12.4 | all | The name of the default branch for the project. | +| `CI_DEPLOY_FREEZE` | 13.2 | all | Included with the value `true` if the pipeline runs during a [deploy freeze window](../../user/project/releases/index.md#prevent-unintentional-releases-by-setting-a-deploy-freeze). | | `CI_DEPLOY_PASSWORD` | 10.8 | all | Authentication password of the [GitLab Deploy Token](../../user/project/deploy_tokens/index.md#gitlab-deploy-token), only present if the Project has one related. | | `CI_DEPLOY_USER` | 10.8 | all | Authentication username of the [GitLab Deploy Token](../../user/project/deploy_tokens/index.md#gitlab-deploy-token), only present if the Project has one related. | | `CI_DISPOSABLE_ENVIRONMENT` | all | 10.1 | Marks that the job is executed in a disposable environment (something that is created only for this job and disposed of/destroyed after the execution - all executors except `shell` and `ssh`). If the environment is disposable, it is set to true, otherwise it is not defined at all. | @@ -56,6 +56,8 @@ Kubernetes-specific environment variables are detailed in the | `CI_ENVIRONMENT_SLUG` | 8.15 | all | A simplified version of the environment name, suitable for inclusion in DNS, URLs, Kubernetes labels, etc. Only present if [`environment:name`](../yaml/README.md#environmentname) is set. | | `CI_ENVIRONMENT_URL` | 9.3 | all | The URL of the environment for this job. Only present if [`environment:url`](../yaml/README.md#environmenturl) is set. | | `CI_EXTERNAL_PULL_REQUEST_IID` | 12.3 | all | Pull Request ID from GitHub if the [pipelines are for external pull requests](../ci_cd_for_external_repos/index.md#pipelines-for-external-pull-requests). Available only if `only: [external_pull_requests]` or [`rules`](../yaml/README.md#rules) syntax is used and the pull request is open. | +| `CI_EXTERNAL_PULL_REQUEST_SOURCE_REPOSITORY` | 13.3 | all | The source repository name of the pull request if [the pipelines are for external pull requests](../ci_cd_for_external_repos/index.md#pipelines-for-external-pull-requests). Available only if `only: [external_pull_requests]` or [`rules`](../yaml/README.md#rules) syntax is used and the pull request is open. | +| `CI_EXTERNAL_PULL_REQUEST_TARGET_REPOSITORY` | 13.3 | all | The target repository name of the pull request if [the pipelines are for external pull requests](../ci_cd_for_external_repos/index.md#pipelines-for-external-pull-requests). Available only if `only: [external_pull_requests]` or [`rules`](../yaml/README.md#rules) syntax is used and the pull request is open. | | `CI_EXTERNAL_PULL_REQUEST_SOURCE_BRANCH_NAME` | 12.3 | all | The source branch name of the pull request if [the pipelines are for external pull requests](../ci_cd_for_external_repos/index.md#pipelines-for-external-pull-requests). Available only if `only: [external_pull_requests]` or [`rules`](../yaml/README.md#rules) syntax is used and the pull request is open. | | `CI_EXTERNAL_PULL_REQUEST_SOURCE_BRANCH_SHA` | 12.3 | all | The HEAD SHA of the source branch of the pull request if [the pipelines are for external pull requests](../ci_cd_for_external_repos/index.md#pipelines-for-external-pull-requests). Available only if `only: [external_pull_requests]` or [`rules`](../yaml/README.md#rules) syntax is used and the pull request is open. | | `CI_EXTERNAL_PULL_REQUEST_TARGET_BRANCH_NAME` | 12.3 | all | The target branch name of the pull request if [the pipelines are for external pull requests](../ci_cd_for_external_repos/index.md#pipelines-for-external-pull-requests). Available only if `only: [external_pull_requests]` or [`rules`](../yaml/README.md#rules) syntax is used and the pull request is open. | @@ -71,8 +73,8 @@ Kubernetes-specific environment variables are detailed in the | `CI_JOB_URL` | 11.1 | 0.5 | Job details URL | | `CI_KUBERNETES_ACTIVE` | 13.0 | all | Included with the value `true` only if the pipeline has a Kubernetes cluster available for deployments. Not included if no cluster is available. Can be used as an alternative to [`only:kubernetes`/`except:kubernetes`](../yaml/README.md#onlykubernetesexceptkubernetes) with [`rules:if`](../yaml/README.md#rulesif) | | `CI_MERGE_REQUEST_ASSIGNEES` | 11.9 | all | Comma-separated list of username(s) of assignee(s) for the merge request if [the pipelines are for merge requests](../merge_request_pipelines/index.md). Available only if `only: [merge_requests]` or [`rules`](../yaml/README.md#rules) syntax is used and the merge request is created. | -| `CI_MERGE_REQUEST_ID` | 11.6 | all | The project-level ID of the merge request. Only available if [the pipelines are for merge requests](../merge_request_pipelines/index.md) and the merge request is created. | -| `CI_MERGE_REQUEST_IID` | 11.6 | all | The instance-level IID of the merge request. Only available If [the pipelines are for merge requests](../merge_request_pipelines/index.md) and the merge request is created. | +| `CI_MERGE_REQUEST_ID` | 11.6 | all | The instance-level ID of the merge request. Only available if [the pipelines are for merge requests](../merge_request_pipelines/index.md) and the merge request is created. | +| `CI_MERGE_REQUEST_IID` | 11.6 | all | The project-level IID (internal ID) of the merge request. Only available If [the pipelines are for merge requests](../merge_request_pipelines/index.md) and the merge request is created. | | `CI_MERGE_REQUEST_LABELS` | 11.9 | all | Comma-separated label names of the merge request if [the pipelines are for merge requests](../merge_request_pipelines/index.md). Available only if `only: [merge_requests]` or [`rules`](../yaml/README.md#rules) syntax is used and the merge request is created. | | `CI_MERGE_REQUEST_MILESTONE` | 11.9 | all | The milestone title of the merge request if [the pipelines are for merge requests](../merge_request_pipelines/index.md). Available only if `only: [merge_requests]` or [`rules`](../yaml/README.md#rules) syntax is used and the merge request is created. | | `CI_MERGE_REQUEST_PROJECT_ID` | 11.6 | all | The ID of the project of the merge request if [the pipelines are for merge requests](../merge_request_pipelines/index.md). Available only if `only: [merge_requests]` or [`rules`](../yaml/README.md#rules) syntax is used and the merge request is created. | @@ -92,9 +94,9 @@ Kubernetes-specific environment variables are detailed in the | `CI_NODE_TOTAL` | 11.5 | all | Total number of instances of this job running in parallel. If the job is not parallelized, this variable is set to `1`. | | `CI_PAGES_DOMAIN` | 11.8 | all | The configured domain that hosts GitLab Pages. | | `CI_PAGES_URL` | 11.8 | all | URL to GitLab Pages-built pages. Always belongs to a subdomain of `CI_PAGES_DOMAIN`. | -| `CI_PIPELINE_ID` | 8.10 | all | The unique ID of the current pipeline that GitLab CI/CD uses internally | -| `CI_PIPELINE_IID` | 11.0 | all | The unique ID of the current pipeline scoped to project | -| `CI_PIPELINE_SOURCE` | 10.0 | all | Indicates how the pipeline was triggered. Possible options are: `push`, `web`, `schedule`, `api`, `external`, `chat`, `webide`, `merge_request_event`, `external_pull_request_event`, `parent_pipeline`, [`trigger`, or `pipeline`](../triggers/README.md#authentication-tokens) (renamed to `cross_project_pipeline` since 13.0). For pipelines created before GitLab 9.5, this will show as `unknown`. | +| `CI_PIPELINE_ID` | 8.10 | all | The instance-level ID of the current pipeline. | +| `CI_PIPELINE_IID` | 11.0 | all | The project-level IID (internal ID) of the current pipeline. | +| `CI_PIPELINE_SOURCE` | 10.0 | all | Indicates how the pipeline was triggered. Possible options are: `push`, `web`, `schedule`, `api`, `external`, `chat`, `webide`, `merge_request_event`, `external_pull_request_event`, `parent_pipeline`, [`trigger`, or `pipeline`](../triggers/README.md#authentication-tokens) (renamed to `cross_project_pipeline` since 13.0). For pipelines created before GitLab 9.5, this is displayed as `unknown`. | | `CI_PIPELINE_TRIGGERED` | all | all | The flag to indicate that job was [triggered](../triggers/README.md) | | `CI_PIPELINE_URL` | 11.1 | 0.5 | Pipeline details URL | | `CI_PROJECT_DIR` | all | all | The full path where the repository is cloned and where the job is run. If the GitLab Runner `builds_dir` parameter is set, this variable is set relative to the value of `builds_dir`. For more information, see [Advanced configuration](https://docs.gitlab.com/runner/configuration/advanced-configuration.html#the-runners-section) for GitLab Runner. | @@ -108,7 +110,7 @@ Kubernetes-specific environment variables are detailed in the | `CI_PROJECT_TITLE` | 12.4 | all | The human-readable project name as displayed in the GitLab web interface. | | `CI_PROJECT_URL` | 8.10 | 0.5 | The HTTP(S) address to access project | | `CI_PROJECT_VISIBILITY` | 10.3 | all | The project visibility (internal, private, public) | -| `CI_REGISTRY` | 8.10 | 0.5 | If the Container Registry is enabled it returns the address of GitLab's Container Registry. This variable will include a `:port` value if one has been specified in the registry configuration. | +| `CI_REGISTRY` | 8.10 | 0.5 | If the Container Registry is enabled it returns the address of GitLab's Container Registry. This variable includes a `:port` value if one has been specified in the registry configuration. | | `CI_REGISTRY_IMAGE` | 8.10 | 0.5 | If the Container Registry is enabled for the project it returns the address of the registry tied to the specific project | | `CI_REGISTRY_PASSWORD` | 9.0 | all | The password to use to push containers to the GitLab Container Registry, for the current project. | | `CI_REGISTRY_USER` | 9.0 | all | The username to use to push containers to the GitLab Container Registry, for the current project. | diff --git a/doc/ci/yaml/README.md b/doc/ci/yaml/README.md index e1d1d27efed..694754a33d1 100644 --- a/doc/ci/yaml/README.md +++ b/doc/ci/yaml/README.md @@ -34,8 +34,8 @@ We have complete examples of configuring pipelines: > from 30 days to under 8 hours with GitLab. NOTE: **Note:** -If you have a [mirrored repository where GitLab pulls from](../../user/project/repository/repository_mirroring.md#pulling-from-a-remote-repository-starter), -you may need to enable pipeline triggering in your project's +If you have a [mirrored repository that GitLab pulls from](../../user/project/repository/repository_mirroring.md#pulling-from-a-remote-repository-starter), +you may need to enable pipeline triggering. Go to your project's **Settings > Repository > Pull from a remote repository > Trigger pipelines for mirror updates**. ## Introduction @@ -63,8 +63,8 @@ jobs, where each of the jobs executes a different command. Of course a command can execute code directly (`./configure;make;make install`) or run a script (`test.sh`) in the repository. -Jobs are picked up by [Runners](../runners/README.md) and executed within the -environment of the Runner. What is important, is that each job is run +Jobs are picked up by [runners](../runners/README.md) and executed within the +environment of the runner. What is important is that each job is run independently from each other. ### Validate the `.gitlab-ci.yml` @@ -103,37 +103,34 @@ The following table lists available parameters for jobs: | Keyword | Description | |:---------------------------------------------------|:------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| -| [`script`](#script) | Shell script which is executed by Runner. | -| [`image`](#image) | Use Docker images. Also available: `image:name` and `image:entrypoint`. | -| [`services`](#services) | Use Docker services images. Also available: `services:name`, `services:alias`, `services:entrypoint`, and `services:command`. | -| [`before_script`](#before_script-and-after_script) | Override a set of commands that are executed before job. | +| [`script`](#script) | Shell script that is executed by a runner. | | [`after_script`](#before_script-and-after_script) | Override a set of commands that are executed after job. | -| [`stage`](#stage) | Defines a job stage (default: `test`). | -| [`only`](#onlyexcept-basic) | Limit when jobs are created. Also available: [`only:refs`, `only:kubernetes`, `only:variables`, and `only:changes`](#onlyexcept-advanced). | -| [`except`](#onlyexcept-basic) | Limit when jobs are not created. Also available: [`except:refs`, `except:kubernetes`, `except:variables`, and `except:changes`](#onlyexcept-advanced). | -| [`rules`](#rules) | List of conditions to evaluate and determine selected attributes of a job, and whether or not it's created. May not be used alongside `only`/`except`. | -| [`tags`](#tags) | List of tags which are used to select Runner. | -| [`allow_failure`](#allow_failure) | Allow job to fail. Failed job does not contribute to commit status. | -| [`when`](#when) | When to run job. Also available: `when:manual` and `when:delayed`. | -| [`environment`](#environment) | Name of an environment to which the job deploys. Also available: `environment:name`, `environment:url`, `environment:on_stop`, `environment:auto_stop_in` and `environment:action`. | +| [`allow_failure`](#allow_failure) | Allow job to fail. Failed job does not contribute to commit status. | +| [`artifacts`](#artifacts) | List of files and directories to attach to a job on success. Also available: `artifacts:paths`, `artifacts:exclude`, `artifacts:expose_as`, `artifacts:name`, `artifacts:untracked`, `artifacts:when`, `artifacts:expire_in`, `artifacts:reports`. | +| [`before_script`](#before_script-and-after_script) | Override a set of commands that are executed before job. | | [`cache`](#cache) | List of files that should be cached between subsequent runs. Also available: `cache:paths`, `cache:key`, `cache:untracked`, and `cache:policy`. | -| [`artifacts`](#artifacts) | List of files and directories to attach to a job on success. Also available: `artifacts:paths`, `artifacts:exclude`, `artifacts:expose_as`, `artifacts:name`, `artifacts:untracked`, `artifacts:when`, `artifacts:expire_in`, `artifacts:reports`, `artifacts:reports:codequality`, `artifacts:reports:junit`, `artifacts:reports:cobertura`, and `artifacts:reports:terraform`.<br><br>In GitLab [Enterprise Edition](https://about.gitlab.com/pricing/), these are available: `artifacts:reports:sast`, `artifacts:reports:dependency_scanning`, `artifacts:reports:container_scanning`, `artifacts:reports:dast`, `artifacts:reports:license_scanning`, `artifacts:reports:license_management` (removed in GitLab 13.0), `artifacts:reports:performance`, `artifacts:reports:load_performance`, and `artifacts:reports:metrics`. | -| [`dependencies`](#dependencies) | Restrict which artifacts are passed to a specific job by providing a list of jobs to fetch artifacts from. | | [`coverage`](#coverage) | Code coverage settings for a given job. | +| [`dependencies`](#dependencies) | Restrict which artifacts are passed to a specific job by providing a list of jobs to fetch artifacts from. | +| [`environment`](#environment) | Name of an environment to which the job deploys. Also available: `environment:name`, `environment:url`, `environment:on_stop`, `environment:auto_stop_in` and `environment:action`. | +| [`except`](#onlyexcept-basic) | Limit when jobs are not created. Also available: [`except:refs`, `except:kubernetes`, `except:variables`, and `except:changes`](#onlyexcept-advanced). | +| [`extends`](#extends) | Configuration entries that this job inherits from. | +| [`image`](#image) | Use Docker images. Also available: `image:name` and `image:entrypoint`. | +| [`include`](#include) | Allows this job to include external YAML files. Also available: `include:local`, `include:file`, `include:template`, and `include:remote`. | +| [`interruptible`](#interruptible) | Defines if a job can be canceled when made redundant by a newer run. | +| [`only`](#onlyexcept-basic) | Limit when jobs are created. Also available: [`only:refs`, `only:kubernetes`, `only:variables`, and `only:changes`](#onlyexcept-advanced). | +| [`pages`](#pages) | Upload the result of a job to use with GitLab Pages. | +| [`parallel`](#parallel) | How many instances of a job should be run in parallel. | +| [`release`](#release) | Instructs the runner to generate a [Release](../../user/project/releases/index.md) object. | +| [`resource_group`](#resource_group) | Limit job concurrency. | | [`retry`](#retry) | When and how many times a job can be auto-retried in case of a failure. | +| [`rules`](#rules) | List of conditions to evaluate and determine selected attributes of a job, and whether or not it's created. May not be used alongside `only`/`except`. | +| [`services`](#services) | Use Docker services images. Also available: `services:name`, `services:alias`, `services:entrypoint`, and `services:command`. | +| [`stage`](#stage) | Defines a job stage (default: `test`). | +| [`tags`](#tags) | List of tags that are used to select a runner. | | [`timeout`](#timeout) | Define a custom job-level timeout that takes precedence over the project-wide setting. | -| [`parallel`](#parallel) | How many instances of a job should be run in parallel. | | [`trigger`](#trigger) | Defines a downstream pipeline trigger. | -| [`include`](#include) | Allows this job to include external YAML files. Also available: `include:local`, `include:file`, `include:template`, and `include:remote`. | -| [`extends`](#extends) | Configuration entries that this job is going to inherit from. | -| [`pages`](#pages) | Upload the result of a job to use with GitLab Pages. | | [`variables`](#variables) | Define job variables on a job level. | -| [`interruptible`](#interruptible) | Defines if a job can be canceled when made redundant by a newer run. | -| [`resource_group`](#resource_group) | Limit job concurrency. | -| [`release`](#release) | Instructs the Runner to generate a [Release](../../user/project/releases/index.md) object. | - -NOTE: **Note:** -Parameters `types` and `type` are [deprecated](#deprecated-parameters). +| [`when`](#when) | When to run job. Also available: `when:manual` and `when:delayed`. | ## Global parameters @@ -293,31 +290,66 @@ There are also two edge cases worth mentioning: > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/29654) in GitLab 12.5 -The top-level `workflow:` key applies to the entirety of a pipeline, and will -determine whether or not a pipeline is created. It currently accepts a single +The top-level `workflow:` key applies to the entirety of a pipeline, and +determines whether or not a pipeline is created. It accepts a single `rules:` key that operates similarly to [`rules:` defined within jobs](#rules), enabling dynamic configuration of the pipeline. If you are new to GitLab CI/CD and `workflow: rules`, you may find the [`workflow:rules` templates](#workflowrules-templates) useful. -To define your own `workflow: rules`, the configuration options currently available are: +To define your own `workflow: rules`, the available configuration options are: - [`if`](#rulesif): Define a rule. - [`when`](#when): May be set to `always` or `never` only. If not provided, the default value is `always`​. -The list of `if` rules is evaluated until a single one is matched. If none -match, the last `when` will be used: +If a pipeline attempts to run but matches no rule, it's dropped and doesn't run. + +Use the example rules below exactly as written to allow pipelines that match the rule +to run. Add `when: never` to prevent pipelines that match the rule from running. See +the [common `if` clauses for `rules`](#common-if-clauses-for-rules) for more examples. + +| Example rules | Details | +|------------------------------------------------------|-----------------------------------------------------------| +| `if: '$CI_PIPELINE_SOURCE == "merge_request_event"'` | Control when merge request pipelines run. | +| `if: '$CI_PIPELINE_SOURCE == "push"'` | Control when both branch pipelines and tag pipelines run. | +| `if: $CI_COMMIT_TAG` | Control when tag pipelines run. | +| `if: $CI_COMMIT_BRANCH` | Control when branch pipelines run. | + +For example, in the following configuration, pipelines run for all `push` events (changes to +branches and new tags). Only push events with `-wip` in the commit message are excluded. Scheduled +pipelines and merge request pipelines don't run, as there's no rule allowing them. ```yaml workflow: rules: - if: $CI_COMMIT_REF_NAME =~ /-wip$/ when: never - - if: $CI_COMMIT_TAG + - if: '$CI_PIPELINE_SOURCE == "push"' +``` + +This example has strict rules, and no other pipelines can run. + +Alternatively, you can have loose rules by using only `when: never` rules, followed +by a final `when: always` rule. This allows all types of pipelines, except for any +that match the `when: never` rules: + +```yaml +workflow: + rules: + - if: '$CI_PIPELINE_SOURCE == "schedule"' + when: never + - if: '$CI_PIPELINE_SOURCE == "push"' when: never - when: always ``` +This example never allows pipelines for schedules or `push` (branches and tags) pipelines, +but does allow pipelines in **all** other cases, *including* merge request pipelines. + +As with `rules` defined in jobs, be careful not to use a configuration that allows +merge request pipelines and branch pipelines to run at the same time, or you could +have [duplicate pipelines](#prevent-duplicate-pipelines). + #### `workflow:rules` templates > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/217732) in GitLab 13.0. @@ -367,7 +399,7 @@ It's also possible to have template files stored in a central repository and pro configuration files. This helps avoid duplicated configuration, for example, global default variables for all projects. `include` requires the external YAML file to have the extensions `.yml` or `.yaml`, -otherwise the external file won't be included. +otherwise the external file is not included. `include` supports the following inclusion methods: @@ -376,14 +408,14 @@ otherwise the external file won't be included. | [`local`](#includelocal) | Include a file from the local project repository. | | [`file`](#includefile) | Include a file from a different project repository. | | [`remote`](#includeremote) | Include a file from a remote URL. Must be publicly accessible. | -| [`template`](#includetemplate) | Include templates which are provided by GitLab. | +| [`template`](#includetemplate) | Include templates that are provided by GitLab. | The `include` methods do not support [variable expansion](../variables/where_variables_can_be_used.md#variables-usage). NOTE: **Note:** `.gitlab-ci.yml` configuration included by all methods is evaluated at pipeline creation. The configuration is a snapshot in time and persisted in the database. Any changes to -referenced `.gitlab-ci.yml` configuration won't be reflected in GitLab until the next pipeline is created. +referenced `.gitlab-ci.yml` configuration is not reflected in GitLab until the next pipeline is created. The files defined by `include` are: @@ -393,7 +425,7 @@ The files defined by `include` are: TIP: **Tip:** Use merging to customize and override included CI/CD configurations with local -definitions. Local definitions in `.gitlab-ci.yml` will override included definitions. +definitions. Local definitions in `.gitlab-ci.yml` override included definitions. NOTE: **Note:** Using [YAML anchors](#anchors) across different YAML files sourced by `include` is not @@ -405,11 +437,11 @@ of using YAML anchors, you can use the [`extends` keyword](#extends). `include:local` includes a file from the same repository as `.gitlab-ci.yml`. It's referenced using full paths relative to the root directory (`/`). -You can only use files that are currently tracked by Git on the same branch +You can only use files that are tracked by Git on the same branch your configuration file is on. In other words, when using a `include:local`, make sure that both `.gitlab-ci.yml` and the local file are on the same branch. -All [nested includes](#nested-includes) will be executed in the scope of the same project, +All [nested includes](#nested-includes) are executed in the scope of the same project, so it's possible to use local, project, remote, or template includes. NOTE: **Note:** @@ -423,7 +455,7 @@ include: ``` TIP: **Tip:** -Local includes can be used as a replacement for symbolic links which are not followed. +Local includes can be used as a replacement for symbolic links that are not followed. This can be defined as a short local include: @@ -462,7 +494,7 @@ include: file: '/templates/.gitlab-ci-template.yml' ``` -All [nested includes](#nested-includes) will be executed in the scope of the target project, +All [nested includes](#nested-includes) are executed in the scope of the target project, so it's possible to use local (relative to target project), project, remote or template includes. @@ -478,7 +510,7 @@ include: - remote: 'https://gitlab.com/awesome-project/raw/master/.gitlab-ci-template.yml' ``` -All [nested includes](#nested-includes) will be executed without context as public user, so only another remote +All [nested includes](#nested-includes) are executed without context as public user, so only another remote or public project, or template, is allowed. #### `include:template` @@ -504,7 +536,7 @@ include: - template: Auto-DevOps.gitlab-ci.yml ``` -All [nested includes](#nested-includes) will be executed only with the permission of the user, +All [nested includes](#nested-includes) are executed only with the permission of the user, so it's possible to use project, remote or template includes. #### Nested includes @@ -584,7 +616,7 @@ For more information, see [Available settings for `services`](../docker/using_do ### `script` `script` is the only required keyword that a job needs. It's a shell script -which is executed by the Runner. For example: +that is executed by the runner. For example: ```yaml job: @@ -603,14 +635,14 @@ job: ``` NOTE: **Note:** -Sometimes, `script` commands will need to be wrapped in single or double quotes. -For example, commands that contain a colon (`:`) need to be wrapped in quotes so +Sometimes, `script` commands must be wrapped in single or double quotes. +For example, commands that contain a colon (`:`) must be wrapped in quotes so that the YAML parser knows to interpret the whole thing as a string rather than a "key: value" pair. Be careful when using special characters: `:`, `{`, `}`, `[`, `]`, `,`, `&`, `*`, `#`, `?`, `|`, `-`, `<`, `>`, `=`, `!`, `%`, `@`, `` ` ``. -If any of the script commands return an exit code different from zero, the job -will fail and further commands won't be executed. This behavior can be avoided by +If any of the script commands return an exit code other than zero, the job +fails and further commands are not executed. You can avoid this behavior by storing the exit code in a variable: ```yaml @@ -631,7 +663,7 @@ This must be an array. Scripts specified in `before_script` are concatenated with any scripts specified in the main [`script`](#script), and executed together in a single shell. -`after_script` is used to define the command that will be run after each +`after_script` is used to define the command that runs after each job, including failed ones. This must be an array. Scripts specified in `after_script` are executed in a new shell, separate from any @@ -640,12 +672,12 @@ Scripts specified in `after_script` are executed in a new shell, separate from a - Have a current working directory set back to the default. - Have no access to changes done by scripts defined in `before_script` or `script`, including: - Command aliases and variables exported in `script` scripts. - - Changes outside of the working tree (depending on the Runner executor), like + - Changes outside of the working tree (depending on the runner executor), like software installed by a `before_script` or `script` script. - Have a separate timeout, which is hard coded to 5 minutes. See [related issue](https://gitlab.com/gitlab-org/gitlab-runner/-/issues/2716) for details. - Don't affect the job's exit code. If the `script` section succeeds and the - `after_script` times out or fails, the job will exit with code `0` (`Job Succeeded`). + `after_script` times out or fails, the job exits with code `0` (`Job Succeeded`). It's possible to overwrite a globally defined `before_script` or `after_script` if you set it per-job: @@ -704,14 +736,14 @@ job: - Write-Host "This text is not colored" ``` -#### Multiline commands +#### Multi-line commands You can split long commands into multi-line commands to improve readability -using [`|` (literal) and `>` (folded) YAML multiline block scalar indicators](https://yaml-multiline.info/). +using [`|` (literal) and `>` (folded) YAML multi-line block scalar indicators](https://yaml-multiline.info/). CAUTION: **Warning:** If multiple commands are combined into one command string, only the last command's -failure or success will be reported, +failure or success is reported, [incorrectly ignoring failures from earlier commands due to a bug](https://gitlab.com/gitlab-org/gitlab-runner/-/issues/25394). If the success of the job depends on the success or failure of these commands, you can run the commands as separate `script:` items, or add `exit 1` commands @@ -774,7 +806,7 @@ First command line is split over two lines. Second command line. ``` -When the `>` or `|` block scalar indicators are omitted, GitLab will form the command +When you omit the `>` or `|` block scalar indicators, GitLab forms the command by concatenating non-empty lines, so make sure the lines can run when concatenated. Shell [here documents](https://en.wikipedia.org/wiki/Here_document) work with the @@ -799,9 +831,13 @@ $ tr a-z A-Z << END_TEXT # collapsed multi-line command FOUR FIVE SIX ``` +#### Custom collapsible sections + +See [custom collapsible sections](../pipelines/index.md#custom-collapsible-sections). + ### `stage` -`stage` is defined per-job and relies on [`stages`](#stages) which is defined +`stage` is defined per-job and relies on [`stages`](#stages), which is defined globally. It allows to group jobs into different stages, and jobs of the same `stage` are executed in parallel (subject to [certain conditions](#using-your-own-runners)). For example: @@ -836,16 +872,16 @@ job 5: script: make something useful at the end of pipeline ``` -#### Using your own Runners +#### Using your own runners -When using your own Runners, GitLab Runner runs only one job at a time by default (see the -`concurrent` flag in [Runner global settings](https://docs.gitlab.com/runner/configuration/advanced-configuration.html#the-global-section) -for more information). +When you use your own runners, GitLab Runner runs only one job at a time by default. See the +`concurrent` flag in [runner global settings](https://docs.gitlab.com/runner/configuration/advanced-configuration.html#the-global-section) +for more information. -Jobs will run on your own Runners in parallel only if: +Jobs run on your own runners in parallel only if: -- Run on different Runners. -- The Runner's `concurrent` setting has been changed. +- Run on different runners. +- The runner's `concurrent` setting has been changed. #### `.pre` and `.post` @@ -890,14 +926,14 @@ For example, the following are equivalent configuration: ``` NOTE: **Note:** -A pipeline won't be created if it only contains jobs in `.pre` or `.post` stages. +A pipeline is not created if all jobs are in `.pre` or `.post` stages. ### `extends` > Introduced in GitLab 11.3. -`extends` defines entry names that a job that uses `extends` is going to -inherit from. +`extends` defines entry names that a job that uses `extends` +inherits from. It's an alternative to using [YAML anchors](#anchors) and is a little more flexible and readable: @@ -919,12 +955,12 @@ rspec: ``` In the example above, the `rspec` job inherits from the `.tests` template job. -GitLab will perform a reverse deep merge based on the keys. GitLab will: +GitLab performs a reverse deep merge based on the keys. GitLab: -- Merge the `rspec` contents into `.tests` recursively. -- Not merge the values of the keys. +- Merges the `rspec` contents into `.tests` recursively. +- Doesn't merge the values of the keys. -This results in the following `rspec` job: +The result is this `rspec` job: ```yaml rspec: @@ -945,8 +981,8 @@ If you do want to include the `rake test`, see [`before_script` and `after_scrip `.tests` in this example is a [hidden job](#hide-jobs), but it's possible to inherit from regular jobs as well. -`extends` supports multi-level inheritance, however it's not recommended to -use more than three levels. The maximum nesting level that is supported is 10. +`extends` supports multi-level inheritance. You should avoid using more than 3 levels, +but you can use as many as ten. The following example has two levels of inheritance: ```yaml @@ -980,7 +1016,7 @@ In GitLab 12.0 and later, it's also possible to use multiple parents for `extends` is able to merge hashes but not arrays. The algorithm used for merge is "closest scope wins", so -keys from the last member will always override anything defined on other +keys from the last member always override anything defined on other levels. For example: ```yaml @@ -1062,7 +1098,7 @@ useTemplate: extends: .template ``` -This will run a job called `useTemplate` that runs `echo Hello!` as defined in +This example runs a job called `useTemplate` that runs `echo Hello!` as defined in the `.template` job, and uses the `alpine` Docker image as defined in the local job. ### `rules` @@ -1077,8 +1113,8 @@ If included, the job also has [certain attributes](#rules-attributes) added to it. CAUTION: **Caution:** -`rules` can't be used in combination with [`only/except`](#onlyexcept-basic) because it is a replacement for -that functionality. If you attempt to do this, the linter returns a +`rules` replaces [`only/except`](#onlyexcept-basic) and can't be used in conjunction with it. +If you attempt to use both keywords in the same job, the linter returns a `key may not be used with rules` error. #### Rules attributes @@ -1182,19 +1218,22 @@ job: - In **all other cases**, the job is added to the pipeline, with `when: on_success`. CAUTION: **Caution:** -If you use `when: on_success`, `always`, or `delayed` as the final rule, two +If you use a `when:` clause as the final rule (not including `when: never`), two simultaneous pipelines may start. Both push pipelines and merge request pipelines can be triggered by the same event (a push to the source branch for an open merge request). -See the [important differences between `rules` and `only`/`except`](#differences-between-rules-and-onlyexcept) +See how to [prevent duplicate pipelines](#prevent-duplicate-pipelines) for more details. -#### Differences between `rules` and `only`/`except` +#### Prevent duplicate pipelines -Jobs defined with `only/except` do not trigger merge request pipelines by default. -You must explicitly add `only: merge_requests`. +Jobs defined with `rules` can trigger multiple pipelines with the same action. You +don't have to explicitly configure rules for each type of pipeline to trigger them +accidentally. Rules that are too loose (allowing too many types of pipelines) could +cause a second pipeline to run unexpectedly. -Jobs defined with `rules` can trigger all types of pipelines. -You do not have to explicitly configure each type. +Some configurations that have the potential to cause duplicate pipelines cause a +[pipeline warning](../troubleshooting.md#pipeline-warnings) to be displayed. +[Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/219431) in GitLab 13.3. For example: @@ -1210,19 +1249,79 @@ job: This job does not run when `$CUSTOM_VARIABLE` is false, but it *does* run in **all** other pipelines, including **both** push (branch) and merge request pipelines. With this configuration, every push to an open merge request's source branch -causes duplicated pipelines. Explicitly allowing both push and merge request pipelines -in the same job could have the same effect. +causes duplicated pipelines. + +There are multiple ways to avoid this: + +- Use [`workflow: rules`](#workflowrules) to specify which types of pipelines + can run. To eliminate duplicate pipelines, allow only merge request pipelines + or push (branch) pipelines. + +- Rewrite the rules to run the job only in very specific cases, + and avoid using a final `when:` rule: -We recommend using [`workflow: rules`](#workflowrules) to limit which types of pipelines -are permitted. Allowing only merge request pipelines, or only branch pipelines, -eliminates duplicated pipelines. Alternatively, you can rewrite the rules to be -stricter, or avoid using a final `when` (`always`, `on_success` or `delayed`). + ```yaml + job: + script: "echo This does NOT create double pipelines!" + rules: + - if: '$CUSTOM_VARIABLE == "true" && $CI_PIPELINE_SOURCE == "merge_request_event"' + ``` -Also, we don't recommend mixing `only/except` jobs with `rules` jobs in the same pipeline. -It may not cause YAML errors, but debugging the exact execution behavior can be complex -due to the different default behaviors of `only/except` and `rules`. +You can prevent duplicate pipelines by changing the job rules to avoid either push (branch) +pipelines or merge request pipelines. However, if you use a `- when: always` rule without +`workflow: rules`, GitLab still displays a [pipeline warning](../troubleshooting.md#pipeline-warnings). -##### `rules:if` +For example, the following does not trigger double pipelines, but is not recommended +without `workflow: rules`: + +```yaml +job: + script: "echo This does NOT create double pipelines!" + rules: + - if: '$CI_PIPELINE_SOURCE == "push"' + when: never + - when: always +``` + +Do not include both push and merge request pipelines in the same job: + +```yaml +job: + script: "echo This creates double pipelines!" + rules: + - if: '$CI_PIPELINE_SOURCE == "push"' + - if: '$CI_PIPELINE_SOURCE == "merge_request_event"' +``` + +Also, do not mix `only/except` jobs with `rules` jobs in the same pipeline. +It may not cause YAML errors, but the different default behaviors of `only/except` +and `rules` can cause issues that are difficult to troubleshoot: + +```yaml +job-with-no-rules: + script: "echo This job runs in branch pipelines." + +job-with-rules: + script: "echo This job runs in merge request pipelines." + rules: + - if: '$CI_PIPELINE_SOURCE == "merge_request_event"' +``` + +For every change pushed to the branch, duplicate pipelines run. One +branch pipeline runs a single job (`job-with-no-rules`), and one merge request pipeline +runs the other job (`job-with-rules`). Jobs with no rules default +to [`except: merge_requests`](#onlyexcept-basic), so `job-with-no-rules` +runs in all cases except merge requests. + +It is not possible to define rules based on whether or not a branch has an open +merge request associated with it. You can't configure a job to be included in: + +- Only branch pipelines when the branch doesn't have a merge request associated with it. +- Only merge request pipelines when the branch has a merge request associated with it. + +See the [related issue](https://gitlab.com/gitlab-org/gitlab/-/issues/201845) for more details. + +#### `rules:if` `rules:if` clauses determine whether or not jobs are added to a pipeline by evaluating a simple `if` statement. If the `if` statement is true, the job is either included @@ -1233,7 +1332,8 @@ or excluded from a pipeline. In plain English, `if` rules can be interpreted as `rules:if` differs slightly from `only:variables` by accepting only a single expression string per rule, rather than an array of them. Any set of expressions to be -evaluated can be conjoined into a single expression by using `&&` or `||`, and use +evaluated can be [conjoined into a single expression](../variables/README.md#conjunction--disjunction) +by using `&&` or `||`, and use the [variable matching syntax](../variables/README.md#syntax-of-environment-variable-expressions). `if:` clauses are evaluated based on the values of [predefined environment variables](../variables/predefined_variables.md) @@ -1264,23 +1364,25 @@ Some details regarding the logic that determines the `when` for the job: - You can define `when` once per rule, or once at the job-level, which applies to all rules. You can't mix `when` at the job-level with `when` in rules. +##### Common `if` clauses for `rules` + For behavior similar to the [`only`/`except` keywords](#onlyexcept-basic), you can -check the value of the `$CI_PIPELINE_SOURCE` variable. - -| Value | Description | -|-------------------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| -| `push` | For pipelines triggered by a `git push` event, including for branches and tags. | -| `web` | For pipelines created by using **Run pipeline** button in the GitLab UI, from the project's **CI/CD > Pipelines** section. | -| `trigger` | For pipelines created by using a trigger token. | -| `schedule` | For [scheduled pipelines](../pipelines/schedules.md). | -| `api` | For pipelines triggered by the [pipelines API](../../api/pipelines.md#create-a-new-pipeline). | -| `external` | When using CI services other than GitLab. | -| `pipelines` | For multi-project pipelines created by [using the API with `CI_JOB_TOKEN`](../triggers/README.md#when-used-with-multi-project-pipelines). | -| `chat` | For pipelines created by using a [GitLab ChatOps](../chatops/README.md) command. | -| `webide` | For pipelines created by using the [WebIDE](../../user/project/web_ide/index.md). | +check the value of the `$CI_PIPELINE_SOURCE` variable: + +| Value | Description | +|-------------------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| +| `api` | For pipelines triggered by the [pipelines API](../../api/pipelines.md#create-a-new-pipeline). | +| `chat` | For pipelines created by using a [GitLab ChatOps](../chatops/README.md) command. | +| `external` | When using CI services other than GitLab. | +| `external_pull_request_event` | When an external pull request on GitHub is created or updated. See [Pipelines for external pull requests](../ci_cd_for_external_repos/index.md#pipelines-for-external-pull-requests). | | `merge_request_event` | For pipelines created when a merge request is created or updated. Required to enable [merge request pipelines](../merge_request_pipelines/index.md), [merged results pipelines](../merge_request_pipelines/pipelines_for_merged_results/index.md), and [merge trains](../merge_request_pipelines/pipelines_for_merged_results/merge_trains/index.md). | -| `external_pull_request_event` | When an external pull request on GitHub is created or updated. See [Pipelines for external pull requests](../ci_cd_for_external_repos/index.md#pipelines-for-external-pull-requests). | -| `parent_pipeline` | For pipelines triggered by a [parent/child pipeline](../parent_child_pipelines.md) with `rules`, use this in the child pipeline configuration so that it can be triggered by the parent pipeline. | +| `parent_pipeline` | For pipelines triggered by a [parent/child pipeline](../parent_child_pipelines.md) with `rules`, use this in the child pipeline configuration so that it can be triggered by the parent pipeline. | +| `pipeline` | For [multi-project pipelines](../multi_project_pipelines.md) created by [using the API with `CI_JOB_TOKEN`](../multi_project_pipelines.md#triggering-multi-project-pipelines-through-api), or the [`trigger`](#trigger) keyword. | +| `push` | For pipelines triggered by a `git push` event, including for branches and tags. | +| `schedule` | For [scheduled pipelines](../pipelines/schedules.md). | +| `trigger` | For pipelines created by using a [trigger token](../triggers/README.md#trigger-token). | +| `web` | For pipelines created by using **Run pipeline** button in the GitLab UI, from the project's **CI/CD > Pipelines** section. | +| `webide` | For pipelines created by using the [WebIDE](../../user/project/web_ide/index.md). | For example: @@ -1325,14 +1427,14 @@ Other commonly used variables for `if` clauses: - `if: '$CUSTOM_VARIABLE == "value1"'`: If the custom variable `CUSTOM_VARIABLE` is exactly `value1`. -##### `rules:changes` +#### `rules:changes` To determine if jobs should be added to a pipeline, `rules: changes` clauses check the files changed by Git push events. `rules: changes` works exactly the same way as [`only: changes` and `except: changes`](#onlychangesexceptchanges), accepting an array of paths. Similarly, it always returns true if there is no -Git push event. It should only be used for branch pipelines or merge request pipelines. +Git push event, for example, when a new tag is created. It should only be used for branch pipelines or merge request pipelines. For example: @@ -1357,11 +1459,14 @@ In this example: to continue running even if the job is not triggered (`allow_failure: true`). - If `Dockerfile` has not changed, do not add job to any pipeline (same as `when: never`). -##### `rules:exists` +To implement a rule similar to [`except: changes`](#onlychangesexceptchanges), +use `when: never`. + +#### `rules:exists` > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/24021) in GitLab 12.4. -`exists` accepts an array of paths and will match if any of these paths exist +`exists` accepts an array of paths and matches if any of these paths exist as files in the repository. For example: @@ -1389,9 +1494,9 @@ job: NOTE: **Note:** For performance reasons, using `exists` with patterns is limited to 10000 -checks. After the 10000th check, rules with patterned globs will always match. +checks. After the 10000th check, rules with patterned globs always match. -##### `rules:allow_failure` +#### `rules:allow_failure` > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/30235) in GitLab 12.8. @@ -1412,18 +1517,18 @@ job: allow_failure: true ``` -In this example, if the first rule matches, then the job will have `when: manual` and `allow_failure: true`. +In this example, if the first rule matches, then the job has `when: manual` and `allow_failure: true`. #### Complex rule clauses -To conjoin `if`, `changes`, and `exists` clauses with an AND, use them in the +To conjoin `if`, `changes`, and `exists` clauses with an `AND`, use them in the same rule. In the following example: -- We run the job manually if `Dockerfile` or any file in `docker/scripts/` - has changed AND `$VAR == "string value"`. -- Otherwise, the job won't be included in the pipeline. +- If the dockerfile or any file in `/docker/scripts` has changed, and var=blah, + then the job runs manually +- Otherwise, the job isn't included in the pipeline. ```yaml docker build: @@ -1453,9 +1558,9 @@ the most out of your pipelines. `only` and `except` are two parameters that set a job policy to limit when jobs are created: -1. `only` defines the names of branches and tags for which the job will run. -1. `except` defines the names of branches and tags for which the job will - **not** run. +1. `only` defines the names of branches and tags the job runs for. +1. `except` defines the names of branches and tags the job does + **not** run for. There are a few rules that apply to the usage of job policy: @@ -1467,20 +1572,20 @@ There are a few rules that apply to the usage of job policy: In addition, `only` and `except` allow the use of special keywords: -| **Value** | **Description** | -|--------------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| -| `branches` | When the Git reference for a pipeline is a branch. | -| `tags` | When the Git reference for a pipeline is a tag. | -| `api` | For pipelines triggered by the [pipelines API](../../api/pipelines.md#create-a-new-pipeline). | -| `external` | When using CI services other than GitLab. | -| `pipelines` | For multi-project pipelines created by using the API with `CI_JOB_TOKEN`. | -| `pushes` | For pipelines triggered by a `git push` event, including for branches and tags. | -| `schedules` | For [scheduled pipelines](../pipelines/schedules.md). | -| `triggers` | For pipelines created by using a trigger token. | -| `web` | For pipelines created by using **Run pipeline** button in the GitLab UI, from the project's **CI/CD > Pipelines** section. | -| `merge_requests` | For pipelines created when a merge request is created or updated. Enables [merge request pipelines](../merge_request_pipelines/index.md), [merged results pipelines](../merge_request_pipelines/pipelines_for_merged_results/index.md), and [merge trains](../merge_request_pipelines/pipelines_for_merged_results/merge_trains/index.md). | -| `external_pull_requests` | When an external pull request on GitHub is created or updated (See [Pipelines for external pull requests](../ci_cd_for_external_repos/index.md#pipelines-for-external-pull-requests)). | -| `chat` | For pipelines created by using a [GitLab ChatOps](../chatops/README.md) command. | +| **Value** | **Description** | +|--------------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| +| `api` | For pipelines triggered by the [pipelines API](../../api/pipelines.md#create-a-new-pipeline). | +| `branches` | When the Git reference for a pipeline is a branch. | +| `chat` | For pipelines created by using a [GitLab ChatOps](../chatops/README.md) command. | +| `external` | When using CI services other than GitLab. | +| `external_pull_requests` | When an external pull request on GitHub is created or updated (See [Pipelines for external pull requests](../ci_cd_for_external_repos/index.md#pipelines-for-external-pull-requests)). | +| `merge_requests` | For pipelines created when a merge request is created or updated. Enables [merge request pipelines](../merge_request_pipelines/index.md), [merged results pipelines](../merge_request_pipelines/pipelines_for_merged_results/index.md), and [merge trains](../merge_request_pipelines/pipelines_for_merged_results/merge_trains/index.md). | +| `pipelines` | For [multi-project pipelines](../multi_project_pipelines.md) created by [using the API with `CI_JOB_TOKEN`](../multi_project_pipelines.md#triggering-multi-project-pipelines-through-api), or the [`trigger`](#trigger) keyword. | +| `pushes` | For pipelines triggered by a `git push` event, including for branches and tags. | +| `schedules` | For [scheduled pipelines](../pipelines/schedules.md). | +| `tags` | When the Git reference for a pipeline is a tag. | +| `triggers` | For pipelines created by using a [trigger token](../triggers/README.md#trigger-token). | +| `web` | For pipelines created by using **Run pipeline** button in the GitLab UI, from the project's **CI/CD > Pipelines** section. | In the example below, `job` will run only for refs that start with `issue-`, whereas all branches will be skipped: @@ -1863,8 +1968,8 @@ properly corrected any failures from previous pipelines. Without [pipelines for merge requests](../merge_request_pipelines/index.md), pipelines run on branches or tags that don't have an explicit association with a merge request. -In this case, a previous SHA is used to calculate the diff, which equivalent to `git diff HEAD~`. -This could result in some unexpected behavior, including: +In this case, a previous SHA is used to calculate the diff, which is equivalent to `git diff HEAD~`. +This can result in some unexpected behavior, including: - When pushing a new branch or a new tag to GitLab, the policy always evaluates to true. - When pushing a new commit, the changed files are calculated using the previous commit @@ -1943,9 +2048,7 @@ This example creates four paths of execution: - The maximum number of jobs that a single job can need in the `needs:` array is limited: - For GitLab.com, the limit is ten. For more information, see our [infrastructure issue](https://gitlab.com/gitlab-com/gl-infra/infrastructure/-/issues/7541). - - For self-managed instances, the limit is: - - 10, if the `ci_dag_limit_needs` feature flag is enabled (default). - - 50, if the `ci_dag_limit_needs` feature flag is disabled. + - For self-managed instances, the limit is: 50. This limit [can be changed](#changing-the-needs-job-limit-core-only). - If `needs:` refers to a job that is marked as `parallel:`. the current job will depend on all parallel jobs created. - `needs:` is similar to `dependencies:` in that it needs to use jobs from prior stages, @@ -1954,23 +2057,20 @@ This example creates four paths of execution: - Related to the above, stages must be explicitly defined for all jobs that have the keyword `needs:` or are referred to by one. -##### Changing the `needs:` job limit +##### Changing the `needs:` job limit **(CORE ONLY)** -The maximum number of jobs that can be defined within `needs:` defaults to 10, but -can be changed to 50 via a feature flag. To change the limit to 50, -[start a Rails console session](../../administration/troubleshooting/debug.md#starting-a-rails-console-session) -and run: +The maximum number of jobs that can be defined within `needs:` defaults to 50. -```ruby -Feature::disable(:ci_dag_limit_needs) -``` - -To set it back to 10, run the opposite command: +A GitLab administrator with [access to the GitLab Rails console](../../administration/feature_flags.md) +can choose a custom limit. For example, to set the limit to 100: ```ruby -Feature::enable(:ci_dag_limit_needs) +Plan.default.actual_limits.update!(ci_needs_size_limit: 100) ``` +NOTE: **Note:** +To disable the ability to use DAG, set the limit to `0`. + #### Artifact downloads with `needs` > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/14311) in GitLab v12.6. @@ -2023,7 +2123,7 @@ rspec: `needs` can be used to download artifacts from up to five jobs in pipelines on [other refs in the same project](#artifact-downloads-between-pipelines-in-the-same-project), -or pipelines in different projects: +or pipelines in different projects, groups and namespaces: ```yaml build_job: @@ -2031,14 +2131,18 @@ build_job: script: - ls -lhR needs: - - project: group/project-name + - project: namespace/group/project-name job: build-1 ref: master artifacts: true ``` `build_job` will download the artifacts from the latest successful `build-1` job -on the `master` branch in the `group/project-name` project. +on the `master` branch in the `group/project-name` project. If the project is in the +same group or namespace, you can omit them from the `project:` key. For example, +`project: group/project-name` or `project: project-name`. + +The user running the pipeline must have at least `reporter` access to the group or project, or the group/project must have public visibility. ##### Artifact downloads between pipelines in the same project @@ -2059,18 +2163,38 @@ build_job: artifacts: true ``` +Environment variables support for `project:`, `job:`, and `ref` was [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/202093) +in GitLab 13.3. This is under development, but it is ready for production use. It is deployed +behind the `ci_expand_names_for_cross_pipeline_artifacts` feature flag, which is **disabled by default**. +[GitLab administrators with access to the GitLab Rails console](../../administration/feature_flags.md) +can enable it for your instance. + +For example: + +```yaml +build_job: + stage: build + script: + - ls -lhR + needs: + - project: $CI_PROJECT_PATH + job: $DEPENDENCY_JOB_NAME + ref: $CI_COMMIT_BRANCH + artifacts: true +``` + NOTE: **Note:** Downloading artifacts from jobs that are run in [`parallel:`](#parallel) is not supported. ### `tags` -`tags` is used to select specific Runners from the list of all Runners that are +`tags` is used to select specific runners from the list of all runners that are allowed to run this project. -During the registration of a Runner, you can specify the Runner's tags, for +During the registration of a runner, you can specify the runner's tags, for example `ruby`, `postgres`, `development`. -`tags` allow you to run jobs with Runners that have the specified tags +`tags` allow you to run jobs with runners that have the specified tags assigned to them: ```yaml @@ -2080,11 +2204,11 @@ job: - postgres ``` -The specification above, will make sure that `job` is built by a Runner that +The specification above, will make sure that `job` is built by a runner that has both `ruby` AND `postgres` tags defined. Tags are also a great way to run different jobs on different platforms, for -example, given an OS X Runner with tag `osx` and Windows Runner with tag +example, given an OS X runner with tag `osx` and Windows runner with tag `windows`, the following jobs run on respective platforms: ```yaml @@ -2160,6 +2284,9 @@ failure. [manual actions](#whenmanual) below. 1. `delayed` - execute job after a certain period (added in GitLab 11.14). Read about [delayed actions](#whendelayed) below. +1. `never`: + - With [`rules`](#rules), don't execute job. + - With [`workflow:rules`](#workflowrules), don't run pipeline. For example: @@ -2323,8 +2450,8 @@ timed rollout 10%: You can stop the active timer of a delayed job by clicking the **{time-out}** (**Unschedule**) button. This job will never be executed in the future unless you execute the job manually. -You can start a delayed job immediately by clicking the **Play** button. -GitLab Runner will pick your job soon and start the job. +To start a delayed job immediately, click the **Play** button. +Soon GitLab Runner picks up and starts the job. ### `environment` @@ -2394,11 +2521,10 @@ deploy to production: > including predefined, secure variables and `.gitlab-ci.yml` [`variables`](#variables). > You however can't use variables defined under `script`. -This is an optional value that when set, it exposes buttons in various places -in GitLab which when clicked take you to the defined URL. +This optional value exposes buttons that take you to the defined URL -In the example below, if the job finishes successfully, it will create buttons -in the merge requests and in the environments/deployments pages which will point +In this example, if the job finishes successfully, it creates buttons +in the merge requests and in the environments/deployments pages that point to `https://prod.example.com`. ```yaml @@ -2427,8 +2553,13 @@ Read the `environment:action` section for an example. > [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/22191) in GitLab 8.13. -The `action` keyword is to be used in conjunction with `on_stop` and is defined -in the job that is called to close the environment. +The `action` keyword can be used to specify jobs that prepare, start, or stop environments. + +| **Value** | **Description** | +|-----------|---------------------------------------------------------------------------------------------------------------------------------------------------------------| +| start | Default value. Indicates that job starts the environment. Deployment will be created after job starts. | +| prepare | Indicates that job is only preparing the environment. Does not affect deployments. [Read more about environments](../environments/index.md#prepare-an-environment) | +| stop | Indicates that job stops deployment. See the example below. | Take for instance: @@ -2556,13 +2687,13 @@ deploy as review app: The `deploy as review app` job will be marked as deployment to dynamically create the `review/$CI_COMMIT_REF_NAME` environment, where `$CI_COMMIT_REF_NAME` -is an [environment variable](../variables/README.md) set by the Runner. The +is an [environment variable](../variables/README.md) set by the runner. The `$CI_ENVIRONMENT_SLUG` variable is based on the environment name, but suitable for inclusion in URLs. In this case, if the `deploy as review app` job was run in a branch named `pow`, this environment would be accessible with an URL like `https://review-pow.example.com/`. -This of course implies that the underlying server which hosts the application +This implies that the underlying server that hosts the application is properly configured. The common use case is to create dynamic environments for branches and use them @@ -2581,7 +2712,7 @@ TIP: **Learn more:** Read how caching works and find out some good practices in the [caching dependencies documentation](../caching/index.md). -`cache` is used to specify a list of files and directories which should be +`cache` is used to specify a list of files and directories that should be cached between jobs. You can only use paths that are within the local working copy. @@ -2590,7 +2721,7 @@ globally and all jobs will use that definition. #### `cache:paths` -Use the `paths` directive to choose which files or directories will be cached. Paths +Use the `paths` directive to choose which files or directories to cache. Paths are relative to the project directory (`$CI_PROJECT_DIR`) and can't directly link outside it. Wildcards can be used that follow the [glob](https://en.wikipedia.org/wiki/Glob_(programming)) patterns and: @@ -2645,8 +2776,8 @@ or any other way that fits your workflow. This way, you can fine tune caching, allowing you to cache data between different jobs or even different branches. The `cache:key` variable can use any of the -[predefined variables](../variables/README.md), and the default key, if not -set, is just literal `default` which means everything is shared between +[predefined variables](../variables/README.md). The default key, if not +set, is just literal `default`, which means everything is shared between pipelines and jobs by default, starting from GitLab 9.0. NOTE: **Note:** @@ -2818,7 +2949,7 @@ skip the download step. > - Not all executors are [supported](https://docs.gitlab.com/runner/executors/#compatibility-chart). > - Job artifacts are only collected for successful jobs by default. -`artifacts` is used to specify a list of files and directories which should be +`artifacts` is used to specify a list of files and directories that are attached to the job when it [succeeds, fails, or always](#artifactswhen). The artifacts will be sent to GitLab after the job finishes and will @@ -2961,11 +3092,11 @@ Note the following: > Introduced in GitLab 8.6 and GitLab Runner v1.1.0. -The `name` directive allows you to define the name of the created artifacts -archive. That way, you can have a unique name for every archive which could be -useful when you'd like to download the archive from GitLab. The `artifacts:name` +Use the `name` directive to define the name of the created artifacts +archive. You can specify a unique name for every archive, which can be +useful when you want to download the archive from GitLab. The `artifacts:name` variable can make use of any of the [predefined variables](../variables/README.md). -The default name is `artifacts`, which becomes `artifacts.zip` when downloaded. +The default name is `artifacts`, which becomes `artifacts.zip` when you download it. NOTE: **Note:** If your branch-name contains forward slashes @@ -3102,8 +3233,11 @@ stored on GitLab. If the expiry time is not defined, it defaults to the [instance wide setting](../../user/admin_area/settings/continuous_integration.md#default-artifacts-expiration-core-only) (30 days by default). -You can use the **Keep** button on the job page to override expiration and -keep artifacts forever. +To override the expiration date and protect artifacts from being automatically deleted: + +- Use the **Keep** button on the job page. +- Set the value of `expire_in` to `never`. [Available](https://gitlab.com/gitlab-org/gitlab/-/issues/22761) + in GitLab 13.3 and later. After their expiry, artifacts are deleted hourly by default (via a cron job), and are not accessible anymore. @@ -3118,6 +3252,7 @@ provided. Examples of valid values: - `6 mos 1 day` - `47 yrs 6 mos and 4d` - `3 weeks and 2 days` +- `never` To expire artifacts 1 week after being uploaded: @@ -3143,20 +3278,20 @@ These are the available report types: | Parameter | Description | |--------------------------------------------------------------------------------------------------------------------------------------|-------------| -| [`artifacts:reports:junit`](../pipelines/job_artifacts.md#artifactsreportsjunit) | The `junit` report collects JUnit XML files. | -| [`artifacts:reports:dotenv`](../pipelines/job_artifacts.md#artifactsreportsdotenv) | The `dotenv` report collects a set of environment variables. | | [`artifacts:reports:cobertura`](../pipelines/job_artifacts.md#artifactsreportscobertura) | The `cobertura` report collects Cobertura coverage XML files. | -| [`artifacts:reports:terraform`](../pipelines/job_artifacts.md#artifactsreportsterraform) | The `terraform` report collects Terraform `tfplan.json` files. | | [`artifacts:reports:codequality`](../pipelines/job_artifacts.md#artifactsreportscodequality) | The `codequality` report collects CodeQuality issues. | -| [`artifacts:reports:sast`](../pipelines/job_artifacts.md#artifactsreportssast-ultimate) **(ULTIMATE)** | The `sast` report collects Static Application Security Testing vulnerabilities. | -| [`artifacts:reports:dependency_scanning`](../pipelines/job_artifacts.md#artifactsreportsdependency_scanning-ultimate) **(ULTIMATE)** | The `dependency_scanning` report collects Dependency Scanning vulnerabilities. | | [`artifacts:reports:container_scanning`](../pipelines/job_artifacts.md#artifactsreportscontainer_scanning-ultimate) **(ULTIMATE)** | The `container_scanning` report collects Container Scanning vulnerabilities. | | [`artifacts:reports:dast`](../pipelines/job_artifacts.md#artifactsreportsdast-ultimate) **(ULTIMATE)** | The `dast` report collects Dynamic Application Security Testing vulnerabilities. | +| [`artifacts:reports:dependency_scanning`](../pipelines/job_artifacts.md#artifactsreportsdependency_scanning-ultimate) **(ULTIMATE)** | The `dependency_scanning` report collects Dependency Scanning vulnerabilities. | +| [`artifacts:reports:dotenv`](../pipelines/job_artifacts.md#artifactsreportsdotenv) | The `dotenv` report collects a set of environment variables. | +| [`artifacts:reports:junit`](../pipelines/job_artifacts.md#artifactsreportsjunit) | The `junit` report collects JUnit XML files. | | [`artifacts:reports:license_management`](../pipelines/job_artifacts.md#artifactsreportslicense_management-ultimate) **(ULTIMATE)** | The `license_management` report collects Licenses (*removed from GitLab 13.0*). | | [`artifacts:reports:license_scanning`](../pipelines/job_artifacts.md#artifactsreportslicense_scanning-ultimate) **(ULTIMATE)** | The `license_scanning` report collects Licenses. | -| [`artifacts:reports:performance`](../pipelines/job_artifacts.md#artifactsreportsperformance-premium) **(PREMIUM)** | The `performance` report collects Browser Performance metrics. | | [`artifacts:reports:load_performance`](../pipelines/job_artifacts.md#artifactsreportsload_performance-premium) **(PREMIUM)** | The `load_performance` report collects load performance metrics. | | [`artifacts:reports:metrics`](../pipelines/job_artifacts.md#artifactsreportsmetrics-premium) **(PREMIUM)** | The `metrics` report collects Metrics. | +| [`artifacts:reports:performance`](../pipelines/job_artifacts.md#artifactsreportsperformance-premium) **(PREMIUM)** | The `performance` report collects Browser Performance metrics. | +| [`artifacts:reports:sast`](../pipelines/job_artifacts.md#artifactsreportssast-ultimate) **(ULTIMATE)** | The `sast` report collects Static Application Security Testing vulnerabilities. | +| [`artifacts:reports:terraform`](../pipelines/job_artifacts.md#artifactsreportsterraform) | The `terraform` report collects Terraform `tfplan.json` files. | #### `dependencies` @@ -3167,7 +3302,7 @@ are passed, but you can use the `dependencies` parameter to define a limited list of jobs (or no jobs) to fetch artifacts from. To use this feature, define `dependencies` in context of the job and pass -a list of all previous jobs from which the artifacts should be downloaded. +a list of all previous jobs the artifacts should be downloaded from. You can only define jobs from stages that are executed before the current one. An error will be shown if you define jobs from the current stage or next ones. Defining an empty array will skip downloading any artifacts for that job. @@ -3251,7 +3386,7 @@ job1: ### `retry` > - [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/3442) in GitLab 9.5. -> - [Behavior expanded](https://gitlab.com/gitlab-org/gitlab-runner/-/issues/3515) in GitLab 11.5 to control on which failures to retry. +> - [Behavior expanded](https://gitlab.com/gitlab-org/gitlab-runner/-/issues/3515) in GitLab 11.5 to control which failures to retry on. `retry` allows you to configure how many times a job is going to be retried in case of a failure. @@ -3272,7 +3407,7 @@ test: ``` By default, a job will be retried on all failure cases. To have a better control -on which failures to retry, `retry` can be a hash with the following keys: +over which failures to retry, `retry` can be a hash with the following keys: - `max`: The maximum number of retries. - `when`: The failure cases to retry. @@ -3347,7 +3482,7 @@ test: The job-level timeout can exceed the [project-level timeout](../pipelines/settings.md#timeout) but can't -exceed the Runner-specific timeout. +exceed the runner-specific timeout. ### `parallel` @@ -3398,6 +3533,48 @@ Please be aware that semaphore_test_boosters reports usages statistics to the au You can then navigate to the **Jobs** tab of a new pipeline build and see your RSpec job split into three separate jobs. +#### Parallel `matrix` jobs + +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/15356) in GitLab 13.3. + +`matrix:` allows you to configure different variables for jobs that are running in parallel. +There can be from 2 to 50 jobs. + +Every job gets the same `CI_NODE_TOTAL` [environment variable](../variables/README.md#predefined-environment-variables) value, and a unique `CI_NODE_INDEX` value. + +```yaml +deploystacks: + stage: deploy + script: + - bin/deploy + parallel: + matrix: + - PROVIDER: aws + STACK: + - monitoring + - app1 + - app2 + - PROVIDER: ovh + STACK: [monitoring, backup, app] + - PROVIDER: [gcp, vultr] + STACK: [data, processing] +``` + +This generates 10 parallel `deploystacks` jobs, each with different values for `PROVIDER` and `STACK`: + +```plaintext +deploystacks 1/10 with PROVIDER=aws and STACK=monitoring +deploystacks 2/10 with PROVIDER=aws and STACK=app1 +deploystacks 3/10 with PROVIDER=aws and STACK=app2 +deploystacks 4/10 with PROVIDER=ovh and STACK=monitoring +deploystacks 5/10 with PROVIDER=ovh and STACK=backup +deploystacks 6/10 with PROVIDER=ovh and STACK=app +deploystacks 7/10 with PROVIDER=gcp and STACK=data +deploystacks 8/10 with PROVIDER=gcp and STACK=processing +deploystacks 9/10 with PROVIDER=vultr and STACK=data +deploystacks 10/10 with PROVIDER=vultr and STACK=processing +``` + ### `trigger` > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/8997) in [GitLab Premium](https://about.gitlab.com/pricing/) 11.8. @@ -3606,16 +3783,16 @@ Once an uninterruptible job is running, the pipeline will never be canceled, reg > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/15536) in GitLab 12.7. -Sometimes running multiples jobs or pipelines at the same time in an environment +Sometimes running multiple jobs or pipelines at the same time in an environment can lead to errors during the deployment. To avoid these errors, the `resource_group` attribute can be used to ensure that -the Runner won't run certain jobs simultaneously. +the runner doesn't run certain jobs simultaneously. When the `resource_group` key is defined for a job in `.gitlab-ci.yml`, job executions are mutually exclusive across different pipelines for the same project. If multiple jobs belonging to the same resource group are enqueued simultaneously, -only one of the jobs will be picked by the Runner, and the other jobs will wait until the +only one of the jobs is picked by the runner, and the other jobs wait until the `resource_group` is free. Here is a simple example: @@ -3645,8 +3822,7 @@ For more information, see [Deployments Safety](../environments/deployment_safety > [Introduced](https://gitlab.com/gitlab-org/gitlab/merge_requests/19298) in GitLab 13.2. -`release` indicates that the job creates a [Release](../../user/project/releases/index.md), -and optionally includes URLs for Release assets. +`release` indicates that the job creates a [Release](../../user/project/releases/index.md). These methods are supported: @@ -3779,25 +3955,46 @@ tags. These options cannot be used together, so choose one: # or can use a variable. ``` -- To create a release automatically when changes are pushed to the default branch, +- To create a release automatically when commits are pushed or merged to the default branch, using a new Git tag that is defined with variables: +NOTE: **Note:** +Environment variables set in `before_script` or `script` are not available for expanding +in the same job. Read more about +[potentially making variables available for expanding](https://gitlab.com/gitlab-org/gitlab-runner/-/issues/6400). + ```yaml + prepare_job: + stage: prepare # This stage must run before the release stage + rules: + - if: $CI_COMMIT_TAG + when: never # Do not run this job when a tag is created manually + - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH # Run this job when commits are pushed or merged to the default branch + script: + - echo "EXTRA_DESCRIPTION=some message" >> variables.env # Generate the EXTRA_DESCRIPTION and TAG environment variables + - echo "TAG=v$(cat VERSION)" >> variables.env # and append to the variables.env file + artifacts: + reports: + dotenv: variables.env # Use artifacts:reports:dotenv to expose the variables to other jobs + release_job: stage: release image: registry.gitlab.com/gitlab-org/release-cli:latest + needs: + - job: prepare_job + artifacts: true rules: - if: $CI_COMMIT_TAG when: never # Do not run this job when a tag is created manually - - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH # Run this job when the default branch changes + - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH # Run this job when commits are pushed or merged to the default branch script: - - echo 'running release_job' + - echo 'running release_job for $TAG' release: - name: 'Release $CI_COMMIT_SHA' - description: 'Created using the release-cli $EXTRA_DESCRIPTION' # $EXTRA_DESCRIPTION and the tag_name - tag_name: 'v${MAJOR}.${MINOR}.${REVISION}' # variables must be defined elsewhere - ref: '$CI_COMMIT_SHA' # in the pipeline. - milestones: + name: 'Release $TAG' + description: 'Created using the release-cli $EXTRA_DESCRIPTION' # $EXTRA_DESCRIPTION and the $TAG + tag_name: '$TAG' # variables must be defined elsewhere + ref: '$CI_COMMIT_SHA' # in the pipeline. For example, in the + milestones: # prepare_job - 'm1' - 'm2' - 'm3' @@ -3871,13 +4068,13 @@ These variables can be later used in all executed commands and scripts. The YAML-defined variables are also set to all created service containers, thus allowing to fine tune them. -Except for the user defined variables, there are also the ones [set up by the -Runner itself](../variables/README.md#predefined-environment-variables). -One example would be `CI_COMMIT_REF_NAME` which has the value of -the branch or tag name for which project is built. Apart from the variables -you can set in `.gitlab-ci.yml`, there are also the so called -[Variables](../variables/README.md#gitlab-cicd-environment-variables) -which can be set in GitLab's UI. +Except for the user-defined variables, there are also variables [set up by the +runner itself](../variables/README.md#predefined-environment-variables). +One example would be `CI_COMMIT_REF_NAME`, which has the value of +the branch or tag name the project is built for. Apart from the variables +you can set in `.gitlab-ci.yml`, there are also environment +[variables](../variables/README.md#gitlab-cicd-environment-variables), +which can be set in the GitLab UI. [YAML anchors for variables](#yaml-anchors-for-variables) are available. @@ -3978,7 +4175,7 @@ The `GIT_CHECKOUT` variable can be used when the `GIT_STRATEGY` is set to either specified, it defaults to true. You can set them globally or per-job in the [`variables`](#variables) section. -If set to `false`, the Runner will: +If set to `false`, the runner will: - when doing `fetch` - update the repository and leave working copy on the current revision, @@ -3986,7 +4183,7 @@ If set to `false`, the Runner will: default branch. Having this setting set to `true` will mean that for both `clone` and `fetch` -strategies the Runner will checkout the working copy to a revision related +strategies the runner will checkout the working copy to a revision related to the CI pipeline: ```yaml @@ -4060,7 +4257,7 @@ The configurtion above will result in `git fetch` being called this way: git fetch origin $REFSPECS --depth 50 --prune ``` -Where `$REFSPECS` is a value provided to the Runner internally by GitLab. +Where `$REFSPECS` is a value provided to the runner internally by GitLab. ### Job stages attempts @@ -4069,12 +4266,12 @@ Where `$REFSPECS` is a value provided to the Runner internally by GitLab. You can set the number for attempts the running job will try to execute each of the following stages: -| Variable | Description | -|-----------------------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| -| **GET_SOURCES_ATTEMPTS** | Number of attempts to fetch sources running a job | -| **ARTIFACT_DOWNLOAD_ATTEMPTS** | Number of attempts to download artifacts running a job | -| **RESTORE_CACHE_ATTEMPTS** | Number of attempts to restore the cache running a job | +| Variable | Description | +|-----------------------------------|--------------------------------------------------------| +| **ARTIFACT_DOWNLOAD_ATTEMPTS** | Number of attempts to download artifacts running a job | | **EXECUTOR_JOB_SECTION_ATTEMPTS** | [Since GitLab 12.10](https://gitlab.com/gitlab-org/gitlab-runner/-/issues/4450), the number of attempts to run a section in a job after a [`No Such Container`](https://gitlab.com/gitlab-org/gitlab-runner/-/issues/4450) error ([Docker executor](https://docs.gitlab.com/runner/executors/docker.html) only). | +| **GET_SOURCES_ATTEMPTS** | Number of attempts to fetch sources running a job | +| **RESTORE_CACHE_ATTEMPTS** | Number of attempts to restore the cache running a job | The default is one single attempt. @@ -4094,8 +4291,8 @@ You can set them globally or per-job in the [`variables`](#variables) section. NOTE: **Note:** As of GitLab 12.0, newly created projects will automatically have a [default `git depth` value of `50`](../pipelines/settings.md#git-shallow-clone). -You can specify the depth of fetching and cloning using `GIT_DEPTH`. This allows -shallow cloning of the repository which can significantly speed up cloning for +You can specify the depth of fetching and cloning using `GIT_DEPTH`. This does a +shallow clone of the repository and can significantly speed up cloning for repositories with a large number of commits or old, large binaries. The value is passed to `git fetch` and `git clone`. @@ -4103,7 +4300,7 @@ NOTE: **Note:** If you use a depth of 1 and have a queue of jobs or retry jobs, jobs may fail. -Since Git fetching and cloning is based on a ref, such as a branch name, Runners +Since Git fetching and cloning is based on a ref, such as a branch name, runners can't clone a specific commit SHA. If there are multiple jobs in the queue, or you're retrying an old job, the commit to be tested needs to be within the Git history that is cloned. Setting too small a value for `GIT_DEPTH` can make @@ -4124,18 +4321,18 @@ You can set it globally or per-job in the [`variables`](#variables) section. ### Custom build directories -> [Introduced](https://gitlab.com/gitlab-org/gitlab-runner/-/issues/2211) in GitLab Runner 11.10 +> [Introduced](https://gitlab.com/gitlab-org/gitlab-runner/-/issues/2211) in GitLab Runner 11.10. NOTE: **Note:** -This can only be used when `custom_build_dir` is enabled in the [Runner's +This can only be used when `custom_build_dir` is enabled in the [runner's configuration](https://docs.gitlab.com/runner/configuration/advanced-configuration.html#the-runnerscustom_build_dir-section). This is the default configuration for `docker` and `kubernetes` executor. By default, GitLab Runner clones the repository in a unique subpath of the `$CI_BUILDS_DIR` directory. However, your project might require the code in a specific directory (Go projects, for example). In that case, you can specify -the `GIT_CLONE_PATH` variable to tell the Runner in which directory to clone the -repository: +the `GIT_CLONE_PATH` variable to tell the runner the directory to clone the +repository in: ```yaml variables: @@ -4156,9 +4353,9 @@ An executor using a concurrency greater than `1` might lead to failures because multiple jobs might be working on the same directory if the `builds_dir` is shared between jobs. GitLab Runner does not try to prevent this situation. It's up to the administrator -and developers to comply with the requirements of Runner configuration. +and developers to comply with the requirements of runner configuration. -To avoid this scenario, you can use a unique path within `$CI_BUILDS_DIR`, because Runner +To avoid this scenario, you can use a unique path within `$CI_BUILDS_DIR`, because runner exposes two additional variables that provide a unique `ID` of concurrency: - `$CI_CONCURRENT_ID`: Unique ID for all jobs running within the given executor. @@ -4209,7 +4406,7 @@ because `$CI_BUILDS_DIR` is not expanded. ## Special YAML features It's possible to use special YAML features like anchors (`&`), aliases (`*`) -and map merging (`<<`), which will allow you to greatly reduce the complexity +and map merging (`<<`), which allows you to greatly reduce the complexity of `.gitlab-ci.yml`. Read more about the various [YAML features](https://learnxinyminutes.com/docs/yaml/). |
