diff options
Diffstat (limited to 'doc/ci')
73 files changed, 1625 insertions, 598 deletions
diff --git a/doc/ci/README.md b/doc/ci/README.md index fce0ad15b70..150f160b762 100644 --- a/doc/ci/README.md +++ b/doc/ci/README.md @@ -1,4 +1,7 @@ --- +stage: Verify +group: Continuous Integration +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers comments: false description: "Learn how to use GitLab CI/CD, the GitLab built-in Continuous Integration, Continuous Deployment, and Continuous Delivery toolset to build, test, and deploy your application." type: index @@ -57,8 +60,10 @@ the following documents: - [GitLab CI/CD basic workflow](introduction/index.md#basic-cicd-workflow). - [Step-by-step guide for writing `.gitlab-ci.yml` for the first time](../user/project/pages/getting_started_part_four.md). -If you're coming over from Jenkins, you can also check out our handy [reference](jenkins/index.md) -for converting your pipelines. +If you're migrating from another CI/CD tool, check out our handy references: + +- [Migrating from CircleCI](migration/circleci.md) +- [Migrating from Jenkins](jenkins/index.md) You can also get started by using one of the [`.gitlab-ci.yml` templates](https://gitlab.com/gitlab-org/gitlab-foss/tree/master/lib/gitlab/ci/templates) @@ -66,7 +71,7 @@ available through the UI. You can use them by creating a new file, choosing a template that suits your application, and adjusting it to your needs: - + For a broader overview, see the [CI/CD getting started](quick_start/README.md) guide. @@ -75,7 +80,7 @@ Once you're familiar with how GitLab CI/CD works, see the for all the attributes you can set and use. NOTE: **Note:** -GitLab CI/CD and [shared runners](runners/README.md#shared-specific-and-group-runners) are enabled in GitLab.com and available for all users, limited only to the [user's pipelines quota](../user/gitlab_com/index.md#shared-runners). +GitLab CI/CD and [shared runners](runners/README.md#shared-runners) are enabled in GitLab.com and available for all users, limited only to the [user's pipelines quota](../user/gitlab_com/index.md#shared-runners). ## Concepts @@ -103,7 +108,7 @@ GitLab CI/CD supports numerous configuration options: | [Pipeline triggers](triggers/README.md) | Trigger pipelines through the API. | | [Pipelines for Merge Requests](merge_request_pipelines/index.md) | Design a pipeline structure for running a pipeline in merge requests. | | [Integrate with Kubernetes clusters](../user/project/clusters/index.md) | Connect your project to Google Kubernetes Engine (GKE) or an existing Kubernetes cluster. | -| [Optimize GitLab and Runner for large repositories](large_repositories/index.md) | Recommended strategies for handling large repos. | +| [Optimize GitLab and Runner for large repositories](large_repositories/index.md) | Recommended strategies for handling large repositories. | | [`.gitlab-ci.yml` full reference](yaml/README.md) | All the attributes you can use with GitLab CI/CD. | Note that certain operations can only be performed according to the @@ -183,22 +188,43 @@ See also the [Why CI/CD?](https://docs.google.com/presentation/d/1OGgk2Tcxbpl7DJ As GitLab CI/CD has evolved, certain breaking changes have been necessary. These are: +#### 13.0 + +- [Remove Backported + `os.Expand`](https://gitlab.com/gitlab-org/gitlab-runner/-/issues/4915) +- [Remove Fedora 29 package + support](https://gitlab.com/gitlab-org/gitlab-runner/-/issues/16158) +- [Remove macOS 32-bit + support](https://gitlab.com/gitlab-org/gitlab-runner/-/issues/25466) +- [Removed `debug/jobs/list?v=1` + endpoint](https://gitlab.com/gitlab-org/gitlab-runner/-/issues/6361) +- [Remove support for array of strings when defining services for Docker + executor](https://gitlab.com/gitlab-org/gitlab-runner/-/issues/4922) +- [Remove `--docker-services` flag on register + command](https://gitlab.com/gitlab-org/gitlab-runner/-/issues/6404) +- [Remove legacy build directory + caching](https://gitlab.com/gitlab-org/gitlab-runner/-/issues/4180) +- [Remove `FF_USE_LEGACY_VOLUMES_MOUNTING_ORDER` feature + flag](https://gitlab.com/gitlab-org/gitlab-runner/-/issues/6581) +- [Remove support for Windows Server + 1803](https://gitlab.com/gitlab-org/gitlab-runner/-/issues/6553) + #### 12.0 - [Use refspec to clone/fetch Git - repository](https://gitlab.com/gitlab-org/gitlab-runner/issues/4069). + repository](https://gitlab.com/gitlab-org/gitlab-runner/-/issues/4069). - [Old cache - configuration](https://gitlab.com/gitlab-org/gitlab-runner/issues/4070). + configuration](https://gitlab.com/gitlab-org/gitlab-runner/-/issues/4070). - [Old metrics server - configuration](https://gitlab.com/gitlab-org/gitlab-runner/issues/4072). + configuration](https://gitlab.com/gitlab-org/gitlab-runner/-/issues/4072). - [Remove - `FF_K8S_USE_ENTRYPOINT_OVER_COMMAND`](https://gitlab.com/gitlab-org/gitlab-runner/issues/4073). + `FF_K8S_USE_ENTRYPOINT_OVER_COMMAND`](https://gitlab.com/gitlab-org/gitlab-runner/-/issues/4073). - [Remove Linux distributions that reach EOL](https://gitlab.com/gitlab-org/gitlab-runner/-/merge_requests/1130). - [Update command line API for helper - images](https://gitlab.com/gitlab-org/gitlab-runner/issues/4013). + images](https://gitlab.com/gitlab-org/gitlab-runner/-/issues/4013). - [Remove old `git clean` - flow](https://gitlab.com/gitlab-org/gitlab-runner/issues/4175). + flow](https://gitlab.com/gitlab-org/gitlab-runner/-/issues/4175). #### 11.0 diff --git a/doc/ci/caching/index.md b/doc/ci/caching/index.md index 16cabae353e..392dd4fdeba 100644 --- a/doc/ci/caching/index.md +++ b/doc/ci/caching/index.md @@ -1,4 +1,7 @@ --- +stage: Verify +group: Continuous Integration +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers type: index, concepts, howto --- @@ -83,9 +86,9 @@ which type of Runner you are using, cache can act differently. From the perspective of the developer, to ensure maximum availability of the cache, when declaring `cache` in your jobs, use one or a mix of the following: -- [Tag your Runners](../runners/README.md#using-tags) and use the tag on jobs +- [Tag your Runners](../runners/README.md#use-tags-to-limit-the-number-of-jobs-using-the-runner) and use the tag on jobs that share their cache. -- [Use sticky Runners](../runners/README.md#locking-a-specific-runner-from-being-enabled-for-other-projects) +- [Use sticky Runners](../runners/README.md#prevent-a-specific-runner-from-being-enabled-for-other-projects) that will be only available to a particular project. - [Use a `key`](../yaml/README.md#cachekey) that fits your workflow (for example, different caches on each branch). For that, you can take advantage of the @@ -556,7 +559,7 @@ next run of the pipeline, the cache will be stored in a different location. ### Clearing the cache manually -> [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/issues/41249) in GitLab 10.4. +> [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/41249) in GitLab 10.4. If you want to avoid editing `.gitlab-ci.yml`, you can easily clear the cache via GitLab's UI: diff --git a/doc/ci/chatops/README.md b/doc/ci/chatops/README.md index 5488834a24e..a8fea0836c1 100644 --- a/doc/ci/chatops/README.md +++ b/doc/ci/chatops/README.md @@ -1,4 +1,7 @@ --- +stage: Configure +group: Configure +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers type: index, concepts, howto --- diff --git a/doc/ci/ci_cd_for_external_repos/bitbucket_integration.md b/doc/ci/ci_cd_for_external_repos/bitbucket_integration.md index 2836f9932c6..ba801950c40 100644 --- a/doc/ci/ci_cd_for_external_repos/bitbucket_integration.md +++ b/doc/ci/ci_cd_for_external_repos/bitbucket_integration.md @@ -1,4 +1,7 @@ --- +stage: Verify +group: Continuous Integration +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers type: howto --- diff --git a/doc/ci/ci_cd_for_external_repos/github_integration.md b/doc/ci/ci_cd_for_external_repos/github_integration.md index 399205d66cf..5365ef49944 100644 --- a/doc/ci/ci_cd_for_external_repos/github_integration.md +++ b/doc/ci/ci_cd_for_external_repos/github_integration.md @@ -1,4 +1,7 @@ --- +stage: Verify +group: Continuous Integration +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers type: howto --- @@ -12,7 +15,7 @@ GitLab. Watch a video on [Using GitLab CI/CD pipelines with GitHub repositories](https://www.youtube.com/watch?v=qgl3F2j-1cI). NOTE: **Note:** -Because of [GitHub limitations](https://gitlab.com/gitlab-org/gitlab/issues/9147), +Because of [GitHub limitations](https://gitlab.com/gitlab-org/gitlab/-/issues/9147), [GitHub OAuth](../../integration/github.md#enabling-github-oauth) cannot be used to authenticate with GitHub as an external CI/CD repository. @@ -20,7 +23,7 @@ cannot be used to authenticate with GitHub as an external CI/CD repository. NOTE: **Note:** Personal access tokens can only be used to connect GitHub.com -repositories to GitLab. +repositories to GitLab, and the GitHub user must have the [owner role](https://help.github.com/en/github/getting-started-with-github/access-permissions-on-github). To perform a one-off authorization with GitHub to grant GitLab access your repositories: @@ -33,11 +36,9 @@ repositories: your project, update commit statuses, and create a web hook to notify GitLab of new commits. -1. In GitLab create a **CI/CD for external repo** project and select +1. In GitLab, go to the [new project page](../../gitlab-basics/create-project.md#create-a-project-in-gitlab), select the **CI/CD for external repo** tab, and then click **GitHub**. -  - 1. Paste the token into the **Personal access token** field and click **List Repositories**. Click **Connect** to select the repository. @@ -91,7 +92,7 @@ To manually enable GitLab CI/CD for your repository: https://gitlab.com/api/v4/projects/<NAMESPACE>%2F<PROJECT>/mirror/pull?private_token=<PERSONAL_ACCESS_TOKEN> ``` -  + Select the **Let me select individual events** option, then check the **Pull requests** and **Pushes** checkboxes. These settings are needed for [pipelines for external pull requests](index.md#pipelines-for-external-pull-requests). 1. In GitHub add a `.gitlab-ci.yml` to configure GitLab CI/CD. diff --git a/doc/ci/ci_cd_for_external_repos/index.md b/doc/ci/ci_cd_for_external_repos/index.md index cc6645b72c2..78988e8a057 100644 --- a/doc/ci/ci_cd_for_external_repos/index.md +++ b/doc/ci/ci_cd_for_external_repos/index.md @@ -1,4 +1,7 @@ --- +stage: Verify +group: Continuous Integration +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers type: index, howto --- @@ -31,7 +34,7 @@ To connect to an external repository: ## Pipelines for external pull requests -> [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/issues/65139) in GitLab Premium 12.3. +> [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/65139) in GitLab Premium 12.3. When using GitLab CI/CD with an [external repository on GitHub](github_integration.md), it's possible to run a pipeline in the context of a Pull Request. @@ -87,10 +90,10 @@ The variable names are prefixed with `CI_EXTERNAL_PULL_REQUEST_`. ### Limitations -This feature currently does not support Pull Requests from fork repositories. Any Pull Requests from fork repositories will be ignored. [Read more](https://gitlab.com/gitlab-org/gitlab/issues/5667). +This feature currently does not support Pull Requests from fork repositories. Any Pull Requests from fork repositories will be ignored. [Read more](https://gitlab.com/gitlab-org/gitlab/-/issues/5667). Given that GitLab will create 2 pipelines, if changes are pushed to a remote branch that references an open Pull Request, both will contribute to the status of the Pull Request via GitHub integration. If you want to exclusively run pipelines on external pull requests and not on branches you can add `except: [branches]` to the job specs. -[Read more](https://gitlab.com/gitlab-org/gitlab/issues/24089#workaround). +[Read more](https://gitlab.com/gitlab-org/gitlab/-/issues/24089#workaround). diff --git a/doc/ci/cloud_deployment/index.md b/doc/ci/cloud_deployment/index.md index 8883ad5dfd6..26eb99bd55a 100644 --- a/doc/ci/cloud_deployment/index.md +++ b/doc/ci/cloud_deployment/index.md @@ -7,7 +7,7 @@ type: howto # Cloud deployment -Interacting with a major cloud provider such as Amazon AWS may have become a much needed task that's +Interacting with a major cloud provider may have become a much needed task that's part of your delivery process. GitLab is making this process less painful by providing Docker images that come with the needed libraries and tools pre-installed. By referencing them in your CI/CD pipeline, you'll be able to interact with your chosen @@ -15,7 +15,12 @@ cloud provider more easily. ## AWS -> [Introduced](https://gitlab.com/gitlab-org/gitlab/issues/31167) in GitLab 12.6. +GitLab provides Docker images to simplify working with AWS, and a template to make +it easier to [deploy to AWS](#deploy-your-application-to-the-aws-elastic-container-service-ecs). + +### Run AWS commands from GitLab CI/CD + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/31167) in GitLab 12.6. GitLab's AWS Docker image provides the [AWS Command Line Interface](https://aws.amazon.com/cli/), which enables you to run `aws` commands. As part of your deployment strategy, you can run `aws` commands directly from @@ -30,12 +35,21 @@ Some credentials are required to be able to run `aws` commands: NOTE: **Note:** A new **Access key ID** and **Secret access key** pair will be generated. Please take a note of them right away. -1. In your GitLab project, go to **Settings > CI / CD**. Set the Access key ID and Secret access key as [environment variables](../variables/README.md#gitlab-cicd-environment-variables), using the following variable names: +1. In your GitLab project, go to **Settings > CI / CD**. Set the following as + [environment variables](../variables/README.md#gitlab-cicd-environment-variables) + (see table below): - | Env. variable name | Value | - |:------------------------|:-------------------------| - | `AWS_ACCESS_KEY_ID` | Your "Access key ID" | - | `AWS_SECRET_ACCESS_KEY` | Your "Secret access key" | + - Access key ID. + - Secret access key. + - Region code. You can check the [list of AWS regional endpoints](https://docs.aws.amazon.com/general/latest/gr/rande.html#regional-endpoints). + You might want to check if the AWS service you intend to use is + [available in the chosen region](https://aws.amazon.com/about-aws/global-infrastructure/regional-product-services/). + + | Env. variable name | Value | + |:------------------------|:-----------------------| + | `AWS_ACCESS_KEY_ID` | Your Access key ID | + | `AWS_SECRET_ACCESS_KEY` | Your Secret access key | + | `AWS_DEFAULT_REGION` | Your region code | 1. You can now use `aws` commands in the `.gitlab-ci.yml` file of this project: @@ -49,25 +63,25 @@ Some credentials are required to be able to run `aws` commands: ``` NOTE: **Note:** - Please note that the image used in the example above + The image used in the example above (`registry.gitlab.com/gitlab-org/cloud-deploy/aws-base:latest`) is hosted on the [GitLab Container Registry](../../user/packages/container_registry/index.md) and is - ready to use. Alternatively, replace the image with another one hosted on [AWS ECR](#aws-ecr). + ready to use. Alternatively, replace the image with one hosted on AWS ECR. -### AWS ECR +### Use an AWS Elastic Container Registry (ECR) image in your CI/CD -Instead of referencing an image hosted on the GitLab Registry, you are free to -reference any other image hosted on any third-party registry, such as +Instead of referencing an image hosted on the GitLab Registry, you can +reference an image hosted on any third-party registry, such as the [Amazon Elastic Container Registry (ECR)](https://aws.amazon.com/ecr/). -To do so, please make sure to [push your image into your ECR -repository](https://docs.aws.amazon.com/AmazonECR/latest/userguide/docker-push-ecr-image.html) -before referencing it in your `.gitlab-ci.yml` file and replace the `image` -path to point to your ECR. +To do so, [push your image into your ECR +repository](https://docs.aws.amazon.com/AmazonECR/latest/userguide/docker-push-ecr-image.html). +Then reference it in your `.gitlab-ci.yml` file and replace the `image` +path to point to your ECR image. -### Deploy your application to AWS Elastic Container Service (ECS) +### Deploy your application to the AWS Elastic Container Service (ECS) -> [Introduced](https://gitlab.com/gitlab-org/gitlab/issues/207962) in GitLab 12.9. +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/207962) in GitLab 12.9. GitLab provides a series of [CI templates that you can include in your project](../yaml/README.md#include). To automate deployments of your application to your [Amazon Elastic Container Service](https://aws.amazon.com/ecs/) (AWS ECS) @@ -80,7 +94,7 @@ components, like an ECS service, ECS task definition, a database on AWS RDS, etc After you're all set up on AWS ECS, follow these steps: 1. Make sure your AWS credentials are set up as environment variables for your - project. You can follow [the steps above](#aws) to complete this setup. + project. You can follow [the steps above](#run-aws-commands-from-gitlab-cicd) to complete this setup. 1. Add these variables to your project's `.gitlab-ci.yml` file: ```yaml @@ -107,11 +121,11 @@ After you're all set up on AWS ECS, follow these steps: ```yaml include: - - template: Deploy-ECS.gitlab-ci.yml + - template: AWS/Deploy-ECS.gitlab-ci.yml ``` The `Deploy-ECS` template ships with GitLab and is available [on - GitLab.com](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/ci/templates/Deploy-ECS.gitlab-ci.yml). + GitLab.com](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/ci/templates/AWS/Deploy-ECS.gitlab-ci.yml). 1. Commit and push your updated `.gitlab-ci.yml` to your project's repository, and you're done! @@ -123,6 +137,17 @@ After you're all set up on AWS ECS, follow these steps: task definition, making the cluster pull the newest version of your application. +CAUTION: **Warning:** +The [`Deploy-ECS.gitlab-ci.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/ci/templates/AWS/Deploy-ECS.gitlab-ci.yml) +template includes both the [`Jobs/Build.gitlab-ci.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/ci/templates/Jobs/Build.gitlab-ci.yml) +and [`Jobs/Deploy/ECS.gitlab-ci.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/ci/templates/Jobs/Deploy/ECS.gitlab-ci.yml) +"sub-templates". Do not include these "sub-templates" on their own, and only include the main +`Deploy-ECS.gitlab-ci.yml` template. The "sub-templates" are designed to only be +used along with the main template. They may move or change unexpectedly causing your +pipeline to fail if you didn't include the main template. Also, the job names within +these templates may change. Do not override these jobs names in your own pipeline, +as the override will stop working when the name changes. + Alternatively, if you don't wish to use the `Deploy-ECS.gitlab-ci.yml` template to deploy to AWS ECS, you can always use our `aws-base` Docker image to run your own [AWS CLI commands for ECS](https://docs.aws.amazon.com/cli/latest/reference/ecs/index.html#cli-aws-ecs). diff --git a/doc/ci/directed_acyclic_graph/img/dag_graph_example_clicked_v13_1.png b/doc/ci/directed_acyclic_graph/img/dag_graph_example_clicked_v13_1.png Binary files differnew file mode 100644 index 00000000000..3610d471ef4 --- /dev/null +++ b/doc/ci/directed_acyclic_graph/img/dag_graph_example_clicked_v13_1.png diff --git a/doc/ci/directed_acyclic_graph/img/dag_graph_example_v13_1.png b/doc/ci/directed_acyclic_graph/img/dag_graph_example_v13_1.png Binary files differnew file mode 100644 index 00000000000..bd5215d31c8 --- /dev/null +++ b/doc/ci/directed_acyclic_graph/img/dag_graph_example_v13_1.png diff --git a/doc/ci/directed_acyclic_graph/index.md b/doc/ci/directed_acyclic_graph/index.md index 8722efd3b40..fff0fda0ab4 100644 --- a/doc/ci/directed_acyclic_graph/index.md +++ b/doc/ci/directed_acyclic_graph/index.md @@ -1,10 +1,13 @@ --- +stage: Verify +group: Continuous Integration +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers type: reference --- # Directed Acyclic Graph -> - [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/issues/47063) in GitLab 12.2. +> - [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/47063) in GitLab 12.2. > - [Feature flag removed](https://gitlab.com/gitlab-org/gitlab/-/issues/206902) in GitLab 12.10. A [directed acyclic graph](https://www.techopedia.com/definition/5739/directed-acyclic-graph-dag) can be @@ -75,3 +78,34 @@ are certain use cases that you may need to work around. For more information: - [`needs` requirements and limitations](../yaml/README.md#requirements-and-limitations). - Related epic [tracking planned improvements](https://gitlab.com/groups/gitlab-org/-/epics/1716). + +## DAG Visualization + +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/215517) in GitLab 13.1 as a [Beta feature](https://about.gitlab.com/handbook/product/#beta). +> - It's deployed behind a feature flag, disabled by default. +> - It's enabled on GitLab.com. +> - It's not recommended for production use. +> - For GitLab self-managed instances, GitLab administrators can opt to [enable it](#enable-or-disable-dag-visualization-core-only) + +The DAG visualization makes it easier to visualize the relationships between dependent jobs in a DAG. This graph will display all the jobs in a pipeline that need or are needed by other jobs. Jobs with no relationships are not displayed in this view. + + + +Clicking a node will highlight all the job paths it depends on. + + + +### Enable or disable DAG Visualization **(CORE ONLY)** + +DAG Visualization is under development and requires more testing, but is being made available as a beta features so users can check its limitations and uses. + +It is deployed behind a feature flag that is **disabled by default**. +[GitLab administrators with access to the GitLab Rails console](../../administration/feature_flags.md) +can opt to enable it for your instance: + +```ruby +# Instance-wide +Feature.enable(:dag_pipeline_tab) +# or by project +Feature.enable(:dag_pipeline_tab, Project.find(<project id>)) +``` diff --git a/doc/ci/docker/README.md b/doc/ci/docker/README.md index f76471b50f2..4e7d9015e85 100644 --- a/doc/ci/docker/README.md +++ b/doc/ci/docker/README.md @@ -1,4 +1,7 @@ --- +stage: Verify +group: Continuous Integration +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers comments: false type: index --- diff --git a/doc/ci/docker/using_docker_build.md b/doc/ci/docker/using_docker_build.md index f992af6c8a5..65b9c03186b 100644 --- a/doc/ci/docker/using_docker_build.md +++ b/doc/ci/docker/using_docker_build.md @@ -1,10 +1,13 @@ --- +stage: Verify +group: Continuous Integration +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers type: concepts, howto --- # Building Docker images with GitLab CI/CD -GitLab CI/CD allows you to use Docker Engine to build and test docker-based projects. +GitLab CI/CD allows you to use Docker Engine to build and test Docker-based projects. One of the new trends in Continuous Integration/Deployment is to: @@ -91,15 +94,15 @@ NOTE: **Note:** By adding `gitlab-runner` to the `docker` group you are effectively granting `gitlab-runner` full root permissions. For more information please read [On Docker security: `docker` group considered harmful](https://www.andreas-jung.com/contents/on-docker-security-docker-group-considered-harmful). -### Use docker-in-docker workflow with Docker executor +### Use Docker-in-Docker workflow with Docker executor -The second approach is to use the special docker-in-docker (dind) +The second approach is to use the special Docker-in-Docker (dind) [Docker image](https://hub.docker.com/_/docker/) with all tools installed (`docker`) and run the job script in context of that image in privileged mode. NOTE: **Note:** -`docker-compose` is not part of docker-in-docker (dind). To use `docker-compose` in your +`docker-compose` is not part of Docker-in-Docker (dind). To use `docker-compose` in your CI builds, follow the `docker-compose` [installation instructions](https://docs.docker.com/compose/install/). @@ -113,19 +116,19 @@ out the official Docker documentation on Docker-in-Docker works well, and is the recommended configuration, but it is not without its own challenges: -- When using docker-in-docker, each job is in a clean environment without the past +- When using Docker-in-Docker, each job is in a clean environment without the past history. Concurrent jobs work fine because every build gets its own instance of Docker engine so they won't conflict with each other. But this also means that jobs can be slower because there's no caching of layers. - By default, Docker 17.09 and higher uses `--storage-driver overlay2` which is the recommended storage driver. See [Using the overlayfs driver](#use-the-overlayfs-driver) for details. -- Since the `docker:19.03.8-dind` container and the Runner container don't share their +- Since the `docker:19.03.11-dind` container and the Runner container don't share their root filesystem, the job's working directory can be used as a mount point for child containers. For example, if you have files you want to share with a child container, you may create a subdirectory under `/builds/$CI_PROJECT_PATH` and use it as your mount point (for a more thorough explanation, check [issue - #41227](https://gitlab.com/gitlab-org/gitlab-foss/issues/41227)): + #41227](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/41227)): ```yaml variables: @@ -139,7 +142,7 @@ not without its own challenges: An example project using this approach can be found here: <https://gitlab.com/gitlab-examples/docker>. In the examples below, we are using Docker images tags to specify a -specific version, such as `docker:19.03.8`. If tags like `docker:stable` +specific version, such as `docker:19.03.11`. If tags like `docker:stable` are used, you have no control over what version is going to be used and this can lead to unpredictable behavior, especially when new versions are released. @@ -151,12 +154,12 @@ Requires GitLab Runner 11.11 or later, but is not supported if GitLab Runner is installed using the [Helm chart](https://docs.gitlab.com/runner/install/kubernetes.html). See the [related -issue](https://gitlab.com/gitlab-org/charts/gitlab-runner/issues/83) for +issue](https://gitlab.com/gitlab-org/charts/gitlab-runner/-/issues/83) for details. The Docker daemon supports connection over TLS and it's done by default -for Docker 19.03.8 or higher. This is the **suggested** way to use the -docker-in-docker service and +for Docker 19.03.11 or higher. This is the **suggested** way to use the +Docker-in-Docker service and [GitLab.com Shared Runners](../../user/gitlab_com/index.md#shared-runners) support this. @@ -171,19 +174,19 @@ support this. --registration-token REGISTRATION_TOKEN \ --executor docker \ --description "My Docker Runner" \ - --docker-image "docker:19.03.8" \ + --docker-image "docker:19.03.11" \ --docker-privileged \ --docker-volumes "/certs/client" ``` The above command will register a new Runner to use the special - `docker:19.03.8` image, which is provided by Docker. **Notice that it's + `docker:19.03.11` image, which is provided by Docker. **Notice that it's using the `privileged` mode to start the build and service - containers.** If you want to use [docker-in-docker](https://www.docker.com/blog/docker-can-now-run-within-docker/) mode, you always + containers.** If you want to use [Docker-in-Docker](https://www.docker.com/blog/docker-can-now-run-within-docker/) mode, you always have to use `privileged = true` in your Docker containers. This will also mount `/certs/client` for the service and build - container, which is needed for the docker client to use the + container, which is needed for the Docker client to use the certificates inside of that directory. For more information how Docker with TLS works check <https://hub.docker.com/_/docker/#tls>. @@ -196,7 +199,7 @@ support this. executor = "docker" [runners.docker] tls_verify = false - image = "docker:19.03.8" + image = "docker:19.03.11" privileged = true disable_cache = false volumes = ["/certs/client", "/cache"] @@ -206,10 +209,10 @@ support this. ``` 1. You can now use `docker` in the build script (note the inclusion of the - `docker:19.03.8-dind` service): + `docker:19.03.11-dind` service): ```yaml - image: docker:19.03.8 + image: docker:19.03.11 variables: # When using dind service, we need to instruct docker, to talk with @@ -234,7 +237,7 @@ support this. DOCKER_TLS_CERTDIR: "/certs" services: - - docker:19.03.8-dind + - docker:19.03.11-dind before_script: - docker info @@ -261,7 +264,7 @@ Assuming that the Runner `config.toml` is similar to: executor = "docker" [runners.docker] tls_verify = false - image = "docker:19.03.8" + image = "docker:19.03.11" privileged = true disable_cache = false volumes = ["/cache"] @@ -271,10 +274,10 @@ Assuming that the Runner `config.toml` is similar to: ``` You can now use `docker` in the build script (note the inclusion of the -`docker:19.03.8-dind` service): +`docker:19.03.11-dind` service): ```yaml -image: docker:19.03.8 +image: docker:19.03.11 variables: # When using dind service we need to instruct docker, to talk with the @@ -295,7 +298,7 @@ variables: DOCKER_TLS_CERTDIR: "" services: - - docker:19.03.8-dind + - docker:19.03.11-dind before_script: - docker info @@ -315,7 +318,7 @@ container so that Docker is available in the context of that image. NOTE: **Note:** If you bind the Docker socket [when using GitLab Runner 11.11 or newer](https://gitlab.com/gitlab-org/gitlab-runner/-/merge_requests/1261), -you can no longer use `docker:19.03.8-dind` as a service because volume bindings +you can no longer use `docker:19.03.11-dind` as a service because volume bindings are done to the services as well, making these incompatible. In order to do that, follow the steps: @@ -330,12 +333,12 @@ In order to do that, follow the steps: --registration-token REGISTRATION_TOKEN \ --executor docker \ --description "My Docker Runner" \ - --docker-image "docker:19.03.8" \ + --docker-image "docker:19.03.11" \ --docker-volumes /var/run/docker.sock:/var/run/docker.sock ``` The above command will register a new Runner to use the special - `docker:19.03.8` image which is provided by Docker. **Notice that it's using + `docker:19.03.11` image which is provided by Docker. **Notice that it's using the Docker daemon of the Runner itself, and any containers spawned by Docker commands will be siblings of the Runner rather than children of the Runner.** This may have complications and limitations that are unsuitable for your workflow. @@ -349,7 +352,7 @@ In order to do that, follow the steps: executor = "docker" [runners.docker] tls_verify = false - image = "docker:19.03.8" + image = "docker:19.03.11" privileged = false disable_cache = false volumes = ["/var/run/docker.sock:/var/run/docker.sock", "/cache"] @@ -358,11 +361,11 @@ In order to do that, follow the steps: ``` 1. You can now use `docker` in the build script (note that you don't need to - include the `docker:19.03.8-dind` service as when using the Docker in Docker + include the `docker:19.03.11-dind` service as when using the Docker in Docker executor): ```yaml - image: docker:19.03.8 + image: docker:19.03.11 before_script: - docker info @@ -377,7 +380,7 @@ In order to do that, follow the steps: While the above method avoids using Docker in privileged mode, you should be aware of the following implications: -- By sharing the docker daemon, you are effectively disabling all +- By sharing the Docker daemon, you are effectively disabling all the security mechanisms of containers and exposing your host to privilege escalation which can lead to container breakout. For example, if a project ran `docker rm -f $(docker ps -a -q)` it would remove the GitLab Runner @@ -392,9 +395,9 @@ aware of the following implications: docker run --rm -t -i -v $(pwd)/src:/home/app/src test-image:latest run_app_tests ``` -## Making docker-in-docker builds faster with Docker layer caching +## Making Docker-in-Docker builds faster with Docker layer caching -When using docker-in-docker, Docker will download all layers of your image every +When using Docker-in-Docker, Docker will download all layers of your image every time you create a build. Recent versions of Docker (Docker 1.13 and above) can use a pre-existing image as a cache during the `docker build` step, considerably speeding up the build process. @@ -416,10 +419,10 @@ any image that's used with the `--cache-from` argument must first be pulled Here's a `.gitlab-ci.yml` file showing how Docker caching can be used: ```yaml -image: docker:19.03.8 +image: docker:19.03.11 services: - - docker:19.03.8-dind + - docker:19.03.11-dind variables: # Use TLS https://docs.gitlab.com/ee/ci/docker/using_docker_build.html#tls-enabled @@ -514,7 +517,7 @@ Once you've built a Docker image, you can push it up to the built-in ## Troubleshooting -### docker: Cannot connect to the Docker daemon at tcp://docker:2375. Is the docker daemon running? +### `docker: Cannot connect to the Docker daemon at tcp://docker:2375. Is the docker daemon running?` This is a common error when you are using [Docker in Docker](#use-docker-in-docker-workflow-with-docker-executor) diff --git a/doc/ci/docker/using_docker_images.md b/doc/ci/docker/using_docker_images.md index 51139da2d16..2448bb536ab 100644 --- a/doc/ci/docker/using_docker_images.md +++ b/doc/ci/docker/using_docker_images.md @@ -1,4 +1,7 @@ --- +stage: Verify +group: Runner +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers type: concepts, howto --- @@ -364,7 +367,7 @@ For example, the following two definitions are equal: | `alias` | no | 9.4 |Additional alias that can be used to access the service from the job's container. Read [Accessing the services](#accessing-the-services) for more information. | NOTE: **Note:** -Alias support for the Kubernetes executor was [introduced](https://gitlab.com/gitlab-org/gitlab-runner/issues/2229) in GitLab Runner 12.8, and is only available for Kubernetes version 1.7 or later. +Alias support for the Kubernetes executor was [introduced](https://gitlab.com/gitlab-org/gitlab-runner/-/issues/2229) in GitLab Runner 12.8, and is only available for Kubernetes version 1.7 or later. ### Starting multiple services from the same image @@ -543,7 +546,7 @@ runtime. of credentials on runner's host. We recommend to upgrade your Runner to at least version **1.8** if you want to use private registries. - Not available for [Kubernetes executor](https://docs.gitlab.com/runner/executors/kubernetes.html), - follow <https://gitlab.com/gitlab-org/gitlab-runner/issues/2673> for + follow <https://gitlab.com/gitlab-org/gitlab-runner/-/issues/2673> for details. ### Using statically-defined credentials @@ -591,7 +594,7 @@ There are two ways to determine the value of `DOCKER_AUTH_CONFIG`: ``` - **Second way -** In some setups, it's possible that Docker client - will use the available system keystore to store the result of `docker + will use the available system key store to store the result of `docker login`. In that case, it's impossible to read `~/.docker/config.json`, so you will need to prepare the required base64-encoded version of `${username}:${password}` and create the Docker configuration JSON manually. @@ -709,7 +712,7 @@ To configure credentials store, follow these steps: ``` - Or, if you are running self-managed Runners, add the above JSON to - `${GITLAB_RUNNER_HOME}/.docker/config.json`. GitLab Runner will read this config file + `${GITLAB_RUNNER_HOME}/.docker/config.json`. GitLab Runner will read this configuration file and will use the needed helper for this specific repository. NOTE: **Note:** `credsStore` is used to access ALL the registries. @@ -727,6 +730,9 @@ To configure access for `aws_account_id.dkr.ecr.region.amazonaws.com`, follow th 1. Make sure `docker-credential-ecr-login` is available in GitLab Runner's `$PATH`. +1. Have any of the following [AWS credentials setup](https://github.com/awslabs/amazon-ecr-credential-helper#aws-credentials). + Make sure that GitLab Runner can access the credentials. + 1. Make GitLab Runner use it. There are two ways to accomplish this. Either: - Create a [variable](../variables/README.md#gitlab-cicd-environment-variables) @@ -741,9 +747,21 @@ To configure access for `aws_account_id.dkr.ecr.region.amazonaws.com`, follow th } ``` + This configures Docker to use the credential helper for a specific registry. + + or + + ```json + { + "credsStore": "ecr-login" + } + ``` + + This configures Docker to use the credential helper for all Amazon ECR registries. + - Or, if you are running self-managed Runners, add the above JSON to `${GITLAB_RUNNER_HOME}/.docker/config.json`. - GitLab Runner will read this config file and will use the needed helper for this + GitLab Runner will read this configuration file and will use the needed helper for this specific repository. 1. You can now use any private image from `aws_account_id.dkr.ecr.region.amazonaws.com` defined in diff --git a/doc/ci/docker/using_kaniko.md b/doc/ci/docker/using_kaniko.md index 587f1f91f72..d53430400ec 100644 --- a/doc/ci/docker/using_kaniko.md +++ b/doc/ci/docker/using_kaniko.md @@ -1,21 +1,24 @@ --- +stage: Verify +group: Continuous Integration +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers type: howto --- # Building images with kaniko and GitLab CI/CD -> [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/issues/45512) in GitLab 11.2. Requires GitLab Runner 11.2 and above. +> [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/45512) in GitLab 11.2. Requires GitLab Runner 11.2 and above. [kaniko](https://github.com/GoogleContainerTools/kaniko) is a tool to build container images from a Dockerfile, inside a container or Kubernetes cluster. kaniko solves two problems with using the -[docker-in-docker +[Docker-in-Docker build](using_docker_build.md#use-docker-in-docker-workflow-with-docker-executor) method: -- Docker-in-docker requires [privileged mode](https://docs.docker.com/engine/reference/run/#runtime-privilege-and-linux-capabilities) +- Docker-in-Docker requires [privileged mode](https://docs.docker.com/engine/reference/run/#runtime-privilege-and-linux-capabilities) in order to function, which is a significant security concern. -- Docker-in-docker generally incurs a performance penalty and can be quite slow. +- Docker-in-Docker generally incurs a performance penalty and can be quite slow. ## Requirements diff --git a/doc/ci/enable_or_disable_ci.md b/doc/ci/enable_or_disable_ci.md index 12ed41e81ee..46cf76637a4 100644 --- a/doc/ci/enable_or_disable_ci.md +++ b/doc/ci/enable_or_disable_ci.md @@ -1,4 +1,7 @@ --- +stage: Verify +group: Continuous Integration +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers type: howto --- diff --git a/doc/ci/environments/deployment_safety.md b/doc/ci/environments/deployment_safety.md new file mode 100644 index 00000000000..055baa78743 --- /dev/null +++ b/doc/ci/environments/deployment_safety.md @@ -0,0 +1,106 @@ +# Deployment safety + +Deployment jobs can be more sensitive than other jobs in a pipeline, +and might need to be treated with extra care. GitLab has several features +that help maintain deployment security and stability. + +You can: + +- [Restrict write-access to a critical environment](#restrict-write-access-to-a-critical-environment) +- [Restrict deployments for a particular period](#restrict-deployments-for-a-particular-period) + +If you are using a continuous deployment workflow and want to ensure that concurrent deployments to the same environment do not happen, you should enable the following options: + +- [Ensure only one deployment job runs at a time](#ensure-only-one-deployment-job-runs-at-a-time) +- [Skip outdated deployment jobs](#skip-outdated-deployment-jobs) + +## Restrict write access to a critical environment + +By default, environments can be modified by any team member that has [Developer permission or higher](../../user/permissions.md#project-members-permissions). +If you want to restrict write access to a critical environment (for example a `production` environment), +you can set up [protected environments](protected_environments.md). + +## Ensure only one deployment job runs at a time + +Pipeline jobs in GitLab CI/CD run in parallel, so it's possible that two deployment +jobs in two different pipelines attempt to deploy to the same environment at the same +time. This is not desired behavior as deployments should happen sequentially. + +You can ensure only one deployment job runs at a time with the [`resource_group` keyword](../yaml/README.md#resource_group) keyword in your `.gitlab-ci.yml`. + +For example: + +```yaml +deploy: + script: deploy-to-prod + resource_group: prod +``` + +Example of a problematic pipeline flow **before** using the resource group: + +1. `deploy` job in Pipeline-A starts running. +1. `deploy` job in Pipeline-B starts running. *This is a concurrent deployment that could cause an unexpected result.* +1. `deploy` job in Pipeline-A finished. +1. `deploy` job in Pipeline-B finished. + +The improved pipeline flow **after** using the resource group: + +1. `deploy` job in Pipeline-A starts running. +1. `deploy` job in Pipeline-B attempts to start, but waits for the first `deploy` job to finish. +1. `deploy` job in Pipeline-A finishes. +1. `deploy` job in Pipeline-B starts running. + +For more information, see [`resource_group` keyword in `.gitlab-ci.yml`](../yaml/README.md#resource_group). + +## Skip outdated deployment jobs + +The execution order of pipeline jobs can vary from run to run, which could cause +undesired behavior. For example, a deployment job in a newer pipeline could +finish before a deployment job in an older pipeline. +This creates a race condition where the older deployment finished later, +overwriting the "newer" deployment. + +You can ensure that older deployment jobs are cancelled automatically when a newer deployment +runs by enabling the [Skip outdated deployment jobs](../pipelines/settings.md#skip-outdated-deployment-jobs) feature. + +Example of a problematic pipeline flow **before** enabling Skip outdated deployment jobs: + +1. Pipeline-A is created on the master branch. +1. Later, Pipeline-B is created on the master branch (with a newer commit SHA). +1. The `deploy` job in Pipeline-B finishes first, and deploys the newer code. +1. The `deploy` job in Pipeline-A finished later, and deploys the older code, **overwriting** the newer (latest) deployment. + +The improved pipeline flow **after** enabling Skip outdated deployment jobs: + +1. Pipeline-A is created on the `master` branch. +1. Later, Pipeline-B is created on the `master` branch (with a newer SHA). +1. The `deploy` job in Pipeline-B finishes first, and deploys the newer code. +1. The `deploy` job in Pipeline-A is automatically cancelled, so that it doesn't overwrite the deployment from the newer pipeline. + +## Restrict deployments for a particular period + +If you want to prevent deployments for a particular period, for example during a planned +vacation period when most employees are out, you can set up a [Deploy Freeze](../../user/project/releases/index.md#set-a-deploy-freeze). +During a deploy freeze period, no deployment can be executed. This is helpful to +ensure that deployments do not happen unexpectedly. + +## Troubleshooting + +### Pipelines jobs fail with `The deployment job is older than the previously succeeded deployment job...` + +This is caused by the [Skip outdated deployment jobs](../pipelines/settings.md#skip-outdated-deployment-jobs) feature. +If you have multiple jobs for the same environment (including non-deployment jobs), you might encounter this problem, for example: + +```yaml +build:service-a: + environment: + name: production + +build:service-b: + environment: + name: production +``` + +The [Skip outdated deployment jobs](../pipelines/settings.md#skip-outdated-deployment-jobs) might not work well with this configuration, and will need to be disabled. + +There is a [plan to introduce a new annotation for environments](https://gitlab.com/gitlab-org/gitlab/-/issues/208655) to address this issue. diff --git a/doc/ci/environments/environments_dashboard.md b/doc/ci/environments/environments_dashboard.md index 4a72c0d6d62..e920e0d2400 100644 --- a/doc/ci/environments/environments_dashboard.md +++ b/doc/ci/environments/environments_dashboard.md @@ -7,7 +7,7 @@ type: reference # Environments Dashboard **(PREMIUM)** -> [Introduced](https://gitlab.com/gitlab-org/gitlab/issues/3713) in [GitLab Premium](https://about.gitlab.com/pricing/) 12.5. +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/3713) in [GitLab Premium](https://about.gitlab.com/pricing/) 12.5. The Environments Dashboard provides a cross-project environment-based view that lets you see the big picture diff --git a/doc/ci/environments/incremental_rollouts.md b/doc/ci/environments/incremental_rollouts.md index 016a6ac7cad..5da5c8e0a87 100644 --- a/doc/ci/environments/incremental_rollouts.md +++ b/doc/ci/environments/incremental_rollouts.md @@ -1,4 +1,7 @@ --- +stage: Release +group: Progressive Delivery +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers type: concepts, howto --- @@ -34,7 +37,7 @@ use as examples to build your own: ## Manual Rollouts -> [Introduced](https://gitlab.com/gitlab-org/gitlab/issues/5415) in GitLab 10.8. +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/5415) in GitLab 10.8. It is possible to configure GitLab to do incremental rollouts manually through `.gitlab-ci.yml`. Manual configuration allows more control over the this feature. The steps in an incremental rollout depend on the @@ -74,7 +77,7 @@ available, demonstrating manually triggered incremental rollouts. ## Timed Rollouts -> [Introduced](https://gitlab.com/gitlab-org/gitlab/issues/7545) in GitLab 11.4. +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/7545) in GitLab 11.4. Timed rollouts behave in the same way as manual rollouts, except that each job is defined with a delay in minutes before it will deploy. Clicking on the job will reveal the countdown. @@ -111,3 +114,26 @@ timed rollout 30%: A [deployable application](https://gitlab.com/gl-release/timed-rollout-example) is available, [demonstrating configuration of timed rollouts](https://gitlab.com/gl-release/timed-rollout-example/blob/master/.gitlab-ci.yml#L86-95). + +## Blue-Green Deployment + +Also sometimes known as A/B deployment or red-black deployment, this technique is used to reduce +downtime and risk during a deployment. When combined with incremental rollouts, you can +minimize the impact of a deployment causing an issue. + +With this technique there are two deployments ("blue" and "green", but any naming can be used). +Only one of these deployments is live at any given time, except during an incremental rollout. + +For example, your blue deployment can be currently active on production, while the +green deployment is "live" for testing, but not deployed to production. If issues +are found, the green deployment can be updated without affecting the production +deployment (currently blue). If testing finds no issues, you switch production to the green +deployment, and blue is now available to test the next release. + +This process reduces downtime as there is no need to take down the production deployment +to switch to a different deployment. Both deployments are running in parallel, and +can be switched to at any time. + +An [example deployable application](https://gitlab.com/gl-release/blue-green-example) +is available, with a [`gitlab-ci.yml` CI/CD configuration file](https://gitlab.com/gl-release/blue-green-example/blob/master/.gitlab-ci.yml) +that demonstrates blue-green deployments. diff --git a/doc/ci/environments/index.md b/doc/ci/environments/index.md index b6ec30b5571..84480b836f8 100644 --- a/doc/ci/environments/index.md +++ b/doc/ci/environments/index.md @@ -162,7 +162,7 @@ Starting with GitLab 9.3, the environment URL is exposed to the Runner via #### Set dynamic environment URLs after a job finishes -> [Introduced](https://gitlab.com/gitlab-org/gitlab/issues/17066) in GitLab 12.9. +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/17066) in GitLab 12.9. In a job script, you can specify a static [environment URL](#using-the-environment-url). However, there may be times when you want a dynamic URL. For example, @@ -384,7 +384,7 @@ feature. ### Configuring Kubernetes deployments -> [Introduced](https://gitlab.com/gitlab-org/gitlab/issues/27630) in GitLab 12.6. +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/27630) in GitLab 12.6. If you are deploying to a [Kubernetes cluster](../../user/project/clusters/index.md) associated with your project, you can configure these deployments from your @@ -421,7 +421,18 @@ NOTE: **Note:** Kubernetes configuration is not supported for Kubernetes clusters that are [managed by GitLab](../../user/project/clusters/index.md#gitlab-managed-clusters). To follow progress on support for GitLab-managed clusters, see the -[relevant issue](https://gitlab.com/gitlab-org/gitlab/issues/38054). +[relevant issue](https://gitlab.com/gitlab-org/gitlab/-/issues/38054). + +### Deployment safety + +Deployment jobs can be more sensitive than other jobs in a pipeline, +and might need to be treated with an extra care. There are multiple features +in GitLab that helps maintain deployment security and stability. + +- [Restrict write-access to a critical environment](deployment_safety.md#restrict-write-access-to-a-critical-environment) +- [Limit the job-concurrency for deployment jobs](deployment_safety.md#ensure-only-one-deployment-job-runs-at-a-time) +- [Skip outdated deployment jobs](deployment_safety.md#skip-outdated-deployment-jobs) +- [Restrict deployments for a particular period](deployment_safety.md#restrict-deployments-for-a-particular-period) ### Complete example @@ -699,11 +710,17 @@ stop action when the associated branch is deleted. The `stop_review` job must be in the same `stage` as the `deploy_review` job in order for the environment to automatically stop. +Additionally, both jobs should have matching [`rules`](../yaml/README.md#onlyexcept-basic) +or [`only/except`](../yaml/README.md#onlyexcept-basic) configuration. In the example +above, if the configuration is not identical, the `stop_review` job might not be +included in all pipelines that include the `deploy_review` job, and it will not be +possible to trigger the `action: stop` to stop the environment automatically. + You can read more in the [`.gitlab-ci.yml` reference](../yaml/README.md#environmenton_stop). #### Environments auto-stop -> [Introduced](https://gitlab.com/gitlab-org/gitlab/issues/20956) in GitLab 12.8. +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/20956) in GitLab 12.8. You can set a expiry time to environments and stop them automatically after a certain period. @@ -849,10 +866,6 @@ versions of the app, all without leaving GitLab.  -#### Linking to external dashboard - -Add a [button to the Monitoring dashboard](../../user/project/operations/linking_to_an_external_dashboard.md) linking directly to your existing external dashboards. - #### Embedding metrics in GitLab Flavored Markdown Metric charts can be embedded within GitLab Flavored Markdown. See [Embedding Metrics within GitLab Flavored Markdown](../../user/project/integrations/prometheus.md#embedding-metric-charts-within-gitlab-flavored-markdown) for more details. @@ -912,7 +925,7 @@ fetch = +refs/environments/*:refs/remotes/origin/environments/* ### Scoping environments with specs > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/2112) in [GitLab Premium](https://about.gitlab.com/pricing/) 9.4. -> - [Scoping for environment variables was moved to Core](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/30779) to Core in GitLab 12.2. +> - [Scoping for environment variables was moved to Core](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/30779) in GitLab 12.2. You can limit the environment scope of a variable by defining which environments it can be available for. diff --git a/doc/ci/environments/protected_environments.md b/doc/ci/environments/protected_environments.md index 43ac42ea0c7..3a44fcfb182 100644 --- a/doc/ci/environments/protected_environments.md +++ b/doc/ci/environments/protected_environments.md @@ -55,6 +55,8 @@ Maintainers can: **Allowed to Deploy** dropdown menu. - Unprotect a protected environment by clicking the **Unprotect** button for that environment. +For more information, see [Deployment safety](deployment_safety.md). + <!-- ## Troubleshooting Include any troubleshooting steps that you can foresee. If you know beforehand what issues diff --git a/doc/ci/examples/README.md b/doc/ci/examples/README.md index 1b3c4c887f4..c54e4b96332 100644 --- a/doc/ci/examples/README.md +++ b/doc/ci/examples/README.md @@ -1,4 +1,7 @@ --- +stage: Verify +group: Continuous Integration +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers comments: false type: index --- @@ -10,9 +13,8 @@ implement [GitLab CI/CD](../README.md) for your specific use case. Examples are available in several forms. As a collection of: -- `.gitlab-ci.yml` [template files](https://gitlab.com/gitlab-org/gitlab-foss/tree/master/lib/gitlab/ci/templates) maintained in GitLab. When you create a new file via the UI, - GitLab will give you the option to choose one of these templates. This will allow you to start using CI/CD with your project quickly. - If your favorite programming language or framework are missing, we would love your help by sending a merge request with a new `.gitlab-ci.yml` to this project. +- `.gitlab-ci.yml` [template files](#cicd-templates) maintained in GitLab, for many + common frameworks and programming languages. - Repositories with [example projects](https://gitlab.com/gitlab-examples) for various languages. You can fork and adjust them to your own needs. Projects include demonstrations of [multi-project pipelines](https://gitlab.com/gitlab-examples/multi-project-pipelines) and using [Review Apps with a static site served by NGINX](https://gitlab.com/gitlab-examples/review-apps-nginx/). - Examples and [other resources](#other-resources) listed below. @@ -47,9 +49,55 @@ language users and GitLab by sending a merge request with a guide for that langu You may want to apply for the [GitLab Community Writers Program](https://about.gitlab.com/community/writers/) to get paid for writing complete articles for GitLab. -## Adding templates to your GitLab installation **(PREMIUM ONLY)** - -If you want to have customized examples and templates for your own self-managed GitLab instance available to your team, your GitLab administrator can [designate an instance template repository](../../user/admin_area/settings/instance_template_repository.md) that contains examples and templates specific to your enterprise. +## CI/CD templates + +Get started with GitLab CI/CD and your favorite programming language or framework by using a +`.gitlab-ci.yml` [template](https://gitlab.com/gitlab-org/gitlab/tree/master/lib/gitlab/ci/templates). + +When you create a `gitlab-ci.yml` file in the UI, you can +choose one of these templates: + +- [Android (`Android.gitlab-ci.yml`)](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/ci/templates/Android.gitlab-ci.yml) +- [Android with fastlane (`Android-Fastlane.gitlab-ci.yml`)](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/ci/templates/Android-Fastlane.gitlab-ci.yml) +- [Bash (`Bash.gitlab-ci.yml`)](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/ci/templates/Bash.gitlab-ci.yml) +- [C++ (`C++.gitlab-ci.yml`)](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/ci/templates/C++.gitlab-ci.yml) +- [Chef (`Chef.gitlab-ci.yml`)](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/ci/templates/Chef.gitlab-ci.yml) +- [Clojure (`Clojure.gitlab-ci.yml`)](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/ci/templates/Clojure.gitlab-ci.yml) +- [Crystal (`Crystal.gitlab-ci.yml`)](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/ci/templates/Crystal.gitlab-ci.yml) +- [Django (`Django.gitlab-ci.yml`)](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/ci/templates/Django.gitlab-ci.yml) +- [Docker (`Docker.gitlab-ci.yml`)](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/ci/templates/Docker.gitlab-ci.yml) +- [dotNET (`dotNET.gitlab-ci.yml`)](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/ci/templates/dotNET.gitlab-ci.yml) +- [dotNET Core (`dotNET-Core.yml`)](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/ci/templates/dotNET-Core.yml) +- [Elixir (`Elixir.gitlab-ci.yml`)](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/ci/templates/Elixir.gitlab-ci.yml) +- [goLang (`Go.gitlab-ci.yml`)](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/ci/templates/Go.gitlab-ci.yml) +- [Gradle (`Gradle.gitlab-ci.yml`)](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/ci/templates/Gradle.gitlab-ci.yml) +- [Grails (`Grails.gitlab-ci.yml`)](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/ci/templates/Grails.gitlab-ci.yml) +- [iOS with fastlane (`iOS-Fastlane.gitlab-ci.yml`)](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/ci/templates/iOS-Fastlane.gitlab-ci.yml) +- [Julia (`Julia.gitlab-ci.yml`)](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/ci/templates/Julia.gitlab-ci.yml) +- [Laravel (`Laravel.gitlab-ci.yml`)](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/ci/templates/Laravel.gitlab-ci.yml) +- [LaTeX (`LaTeX.gitlab-ci.yml`)](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/ci/templates/LaTeX.gitlab-ci.yml) +- [Maven (`Maven.gitlab-ci.yml`)](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/ci/templates/Maven.gitlab-ci.yml) +- [Mono (`Mono.gitlab-ci.yml`)](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/ci/templates/Mono.gitlab-ci.yml) +- [Node.js (`Nodejs.gitlab-ci.yml`)](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/ci/templates/Nodejs.gitlab-ci.yml) +- [OpenShift (`OpenShift.gitlab-ci.yml`)](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/ci/templates/OpenShift.gitlab-ci.yml) +- [Packer (`Packer.gitlab-ci.yml`)](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/ci/templates/Packer.gitlab-ci.yml) +- [PHP (`PHP.gitlab-ci.yml`)](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/ci/templates/PHP.gitlab-ci.yml) +- [Python (`Python.gitlab-ci.yml`)](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/ci/templates/Python.gitlab-ci.yml) +- [Ruby (`Ruby.gitlab-ci.yml`)](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/ci/templates/Ruby.gitlab-ci.yml) +- [Rust (`Rust.gitlab-ci.yml`)](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/ci/templates/Rust.gitlab-ci.yml) +- [Scala (`Scala.gitlab-ci.yml`)](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/ci/templates/Scala.gitlab-ci.yml) +- [Swift (`Swift.gitlab-ci.yml`)](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/ci/templates/Swift.gitlab-ci.yml) +- [Terraform (`Terraform.gitlab-ci.yml`)](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/ci/templates/Terraform.gitlab-ci.yml) + +If a programming language or framework template is not in this list, you can contribute +one. To create a template, submit a merge request +to <https://gitlab.com/gitlab-org/gitlab/tree/master/lib/gitlab/ci/templates>. + +### Adding templates to your GitLab installation **(PREMIUM ONLY)** + +You can add custom examples and templates to your self-managed GitLab instance. +Your GitLab administrator can [designate an instance template repository](../../user/admin_area/settings/instance_template_repository.md) +that contains examples and templates specific to your organization. ## Other resources diff --git a/doc/ci/examples/artifactory_and_gitlab/index.md b/doc/ci/examples/artifactory_and_gitlab/index.md index 85dcc199e2b..c1b3ddec1b9 100644 --- a/doc/ci/examples/artifactory_and_gitlab/index.md +++ b/doc/ci/examples/artifactory_and_gitlab/index.md @@ -1,4 +1,7 @@ --- +stage: Verify +group: Continuous Integration +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers disqus_identifier: 'https://docs.gitlab.com/ee/articles/artifactory_and_gitlab/index.html' author: Fabio Busatto author_gitlab: bikebilly @@ -106,7 +109,7 @@ parameter in `.gitlab-ci.yml` to use the custom location instead of the default Now it's time we set up [GitLab CI/CD](https://about.gitlab.com/stages-devops-lifecycle/continuous-integration/) to automatically build, test and deploy the dependency! -GitLab CI/CD uses a file in the root of the repo, named `.gitlab-ci.yml`, to read the definitions for jobs +GitLab CI/CD uses a file in the root of the repository, named `.gitlab-ci.yml`, to read the definitions for jobs that will be executed by the configured GitLab Runners. You can read more about this file in the [GitLab Documentation](../../yaml/README.md). First of all, remember to set up variables for your deployment. Navigate to your project's **Settings > CI/CD > Environment variables** page @@ -116,7 +119,7 @@ and add the following ones (replace them with your current values, of course): - **MAVEN_REPO_USER**: `gitlab` (your Artifactory username) - **MAVEN_REPO_PASS**: `AKCp2WXr3G61Xjz1PLmYa3arm3yfBozPxSta4taP3SeNu2HPXYa7FhNYosnndFNNgoEds8BCS` (your Artifactory Encrypted Password) -Now it's time to define jobs in `.gitlab-ci.yml` and push it to the repo: +Now it's time to define jobs in `.gitlab-ci.yml` and push it to the repository: ```yaml image: maven:latest @@ -151,7 +154,7 @@ deploy: GitLab Runner will use the latest [Maven Docker image](https://hub.docker.com/_/maven/), which already contains all the tools and the dependencies you need to manage the project, in order to run the jobs. -Environment variables are set to instruct Maven to use the `homedir` of the repo instead of the user's home when searching for configuration and dependencies. +Environment variables are set to instruct Maven to use the `homedir` of the repository instead of the user's home when searching for configuration and dependencies. Caching the `.m2/repository folder` (where all the Maven files are stored), and the `target` folder (where our application will be created), is useful for speeding up the process by running all Maven phases in a sequential order, therefore, executing `mvn test` will automatically run `mvn compile` if necessary. @@ -161,7 +164,7 @@ Both `build` and `test` jobs leverage the `mvn` command to compile the applicati Deploy to Artifactory is done as defined by the variables we have just set up. The deployment occurs only if we're pushing or merging to `master` branch, so that the development versions are tested but not published. -Done! Now you have all the changes in the GitLab repo, and a pipeline has already been started for this commit. In the **Pipelines** tab you can see what's happening. +Done! Now you have all the changes in the GitLab repository, and a pipeline has already been started for this commit. In the **Pipelines** tab you can see what's happening. If the deployment has been successful, the deploy job log will output: ```plaintext @@ -174,7 +177,7 @@ If the deployment has been successful, the deploy job log will output: >**Note**: the `mvn` command downloads a lot of files from the internet, so you'll see a lot of extra activity in the log the first time you run it. -Yay! You did it! Checking in Artifactory will confirm that you have a new artifact available in the `libs-release-local` repo. +Yay! You did it! Checking in Artifactory will confirm that you have a new artifact available in the `libs-release-local` repository. ## Create the main Maven application @@ -225,7 +228,7 @@ Here is how you can get the content of the file directly from Artifactory: 1. Click on **Generate Maven Settings** 1. Click on **Generate Settings** 1. Copy to clipboard the configuration file -1. Save the file as `.m2/settings.xml` in your repo +1. Save the file as `.m2/settings.xml` in your repository Now you are ready to use the Artifactory repository to resolve dependencies and use `simple-maven-dep` in your main application! @@ -236,7 +239,7 @@ You need a last step to have everything in place: configure the `.gitlab-ci.yml` You want to leverage [GitLab CI/CD](https://about.gitlab.com/stages-devops-lifecycle/continuous-integration/) to automatically build, test and run your awesome application, and see if you can get the greeting as expected! -All you need to do is to add the following `.gitlab-ci.yml` to the repo: +All you need to do is to add the following `.gitlab-ci.yml` to the repository: ```yaml image: maven:latest diff --git a/doc/ci/examples/authenticating-with-hashicorp-vault/index.md b/doc/ci/examples/authenticating-with-hashicorp-vault/index.md index 089babc8945..3b8be54ae59 100644 --- a/doc/ci/examples/authenticating-with-hashicorp-vault/index.md +++ b/doc/ci/examples/authenticating-with-hashicorp-vault/index.md @@ -1,4 +1,7 @@ --- +stage: Release +group: Release Management +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers type: tutorial --- diff --git a/doc/ci/examples/deploy_spring_boot_to_cloud_foundry/index.md b/doc/ci/examples/deploy_spring_boot_to_cloud_foundry/index.md index 49f4a14c5ac..87cee1820bc 100644 --- a/doc/ci/examples/deploy_spring_boot_to_cloud_foundry/index.md +++ b/doc/ci/examples/deploy_spring_boot_to_cloud_foundry/index.md @@ -1,4 +1,7 @@ --- +stage: Release +group: Progressive Delivery +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers author: Dylan Griffith author_gitlab: DylanGriffith level: intermediate @@ -100,7 +103,7 @@ production: - master ``` -We've used the `java:8` [docker +We've used the `java:8` [Docker image](../../docker/using_docker_images.md) to build our application as it provides the up-to-date Java 8 JDK on [Docker Hub](https://hub.docker.com/). We've also added the [`only` diff --git a/doc/ci/examples/deployment/README.md b/doc/ci/examples/deployment/README.md index 2ed1a99e0d9..ec02fb6dd43 100644 --- a/doc/ci/examples/deployment/README.md +++ b/doc/ci/examples/deployment/README.md @@ -1,4 +1,7 @@ --- +stage: Release +group: Progressive Delivery +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers type: tutorial --- @@ -53,7 +56,7 @@ To use different provider take a look at long list of [Supported Providers](http ## Using Dpl with Docker In most cases, you will have configured [GitLab Runner](https://docs.gitlab.com/runner/) to use your server's shell commands. -This means that all commands are run in the context of local user (e.g. gitlab_runner or gitlab_ci_multi_runner). +This means that all commands are run in the context of local user (e.g. `gitlab_runner` or `gitlab_ci_multi_runner`). It also means that most probably in your Docker container you don't have the Ruby runtime installed. You will have to install it: diff --git a/doc/ci/examples/deployment/composer-npm-deploy.md b/doc/ci/examples/deployment/composer-npm-deploy.md index c5f49fd6e59..cea6f26181f 100644 --- a/doc/ci/examples/deployment/composer-npm-deploy.md +++ b/doc/ci/examples/deployment/composer-npm-deploy.md @@ -1,4 +1,7 @@ --- +stage: Release +group: Progressive Delivery +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers type: tutorial --- @@ -44,7 +47,7 @@ All these operations will put all files into a `build` folder, which is ready to ## How to transfer files to a live server -You have multiple options: rsync, scp, sftp, and so on. For now, we will use scp. +You have multiple options: rsync, SCP, SFTP, and so on. For now, we will use SCP. To make this work, you need to add a GitLab CI/CD Variable (accessible on `gitlab.example/your-project-name/variables`). That variable will be called `STAGING_PRIVATE_KEY` and it's the **private** SSH key of your server. @@ -74,7 +77,7 @@ And this is basically all you need in the `before_script` section. ## How to deploy -As we stated above, we need to deploy the `build` folder from the docker image to our server. To do so, we create a new job: +As we stated above, we need to deploy the `build` folder from the Docker image to our server. To do so, we create a new job: ```yaml stage_deploy: @@ -94,7 +97,7 @@ stage_deploy: Here's the breakdown: 1. `only:dev` means that this build will run only when something is pushed to the `dev` branch. You can remove this block completely and have everything be ran on every push (but probably this is something you don't want) -1. `ssh-add ...` we will add that private key you added on the web UI to the docker container +1. `ssh-add ...` we will add that private key you added on the web UI to the Docker container 1. We will connect via `ssh` and create a new `_tmp` folder 1. We will connect via `scp` and upload the `build` folder (which was generated by a `npm` script) to our previously created `_tmp` folder 1. We will connect again via `ssh` and move the `live` folder to an `_old` folder, then move `_tmp` to `live`. @@ -120,7 +123,7 @@ Therefore, for a production environment we use additional steps to ensure that a Since this was a WordPress project, I gave real life code snippets. Some further ideas you can pursue: - Having a slightly different script for `master` branch will allow you to deploy to a production server from that branch and to a stage server from any other branches. -- Instead of pushing it live, you can push it to WordPress official repo (with creating a SVN commit, etc.). +- Instead of pushing it live, you can push it to WordPress official repository (with creating a SVN commit, etc.). - You could generate i18n text domains on the fly. --- diff --git a/doc/ci/examples/devops_and_game_dev_with_gitlab_ci_cd/index.md b/doc/ci/examples/devops_and_game_dev_with_gitlab_ci_cd/index.md index 610220a6bff..51d9f169939 100644 --- a/doc/ci/examples/devops_and_game_dev_with_gitlab_ci_cd/index.md +++ b/doc/ci/examples/devops_and_game_dev_with_gitlab_ci_cd/index.md @@ -1,4 +1,7 @@ --- +stage: Verify +group: Continuous Integration +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers author: Ryan Hall author_gitlab: blitzgren level: intermediate @@ -353,7 +356,7 @@ test: ### Run your CI/CD pipeline -That's it! Add all your new files, commit, and push. For a reference of what our repo should +That's it! Add all your new files, commit, and push. For a reference of what our repository should look like at this point, please refer to the [final commit related to this article on my sample repository](https://gitlab.com/blitzgren/gitlab-game-demo/commit/8b36ef0ecebcf569aeb251be4ee13743337fcfe2). By applying both build and test stages, GitLab will run them sequentially at every push to our repository. If all goes well you'll end up with a green check mark on each job for the pipeline: @@ -419,15 +422,15 @@ fully understand [IAM Best Practices in AWS](https://docs.aws.amazon.com/IAM/lat 1. Log into your AWS account and go to the [Security Credentials page](https://console.aws.amazon.com/iam/home#/security_credential) 1. Click the **Access Keys** section and **Create New Access Key**. Create the key and keep the ID and secret around, you'll need them later -  +  1. Go to your GitLab project, click **Settings > CI/CD** on the left sidebar 1. Expand the **Variables** section  -1. Add a key named `AWS_KEY_ID` and copy the key ID from Step 2 into the **Value** textbox -1. Add a key named `AWS_KEY_SECRET` and copy the key secret from Step 2 into the **Value** textbox +1. Add a key named `AWS_KEY_ID` and copy the key ID from Step 2 into the **Value** field +1. Add a key named `AWS_KEY_SECRET` and copy the key secret from Step 2 into the **Value** field ### Deploy your game with GitLab CI/CD @@ -526,7 +529,7 @@ a lot of breathing room in quickly getting changes to players. Here are some ideas to further investigate that can speed up or improve your pipeline: - [Yarn](https://yarnpkg.com) instead of npm -- Set up a custom [Docker](../../../ci/docker/using_docker_images.md#define-image-and-services-from-gitlab-ciyml) image that can preload dependencies and tools (like AWS CLI) +- Set up a custom [Docker](../../../ci/docker/using_docker_images.md#define-image-and-services-from-gitlab-ciyml) image that can pre-load dependencies and tools (like AWS CLI) - Forward a [custom domain](https://docs.aws.amazon.com/AmazonS3/latest/dev/website-hosting-custom-domain-walkthrough.html) to your game's S3 static website - Combine jobs if you find it unnecessary for a small project - Avoid the queues and set up your own [custom GitLab CI/CD runner](https://about.gitlab.com/blog/2016/03/01/gitlab-runner-with-docker/) diff --git a/doc/ci/examples/end_to_end_testing_webdriverio/index.md b/doc/ci/examples/end_to_end_testing_webdriverio/index.md index 8491881570c..05b3cc257e3 100644 --- a/doc/ci/examples/end_to_end_testing_webdriverio/index.md +++ b/doc/ci/examples/end_to_end_testing_webdriverio/index.md @@ -1,4 +1,7 @@ --- +stage: Verify +group: Testing +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers author: Vincent Tunru author_gitlab: Vinnl level: advanced diff --git a/doc/ci/examples/laravel_with_gitlab_and_envoy/index.md b/doc/ci/examples/laravel_with_gitlab_and_envoy/index.md index 0331fa70f06..5b442436baf 100644 --- a/doc/ci/examples/laravel_with_gitlab_and_envoy/index.md +++ b/doc/ci/examples/laravel_with_gitlab_and_envoy/index.md @@ -1,4 +1,7 @@ --- +stage: Verify +group: Continuous Integration +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers disqus_identifier: 'https://docs.gitlab.com/ee/articles/laravel_with_gitlab_and_envoy/index.html' author: Mehran Rasulian author_gitlab: mehranrasulian @@ -29,7 +32,7 @@ It uses a clean, minimal [Blade syntax](https://laravel.com/docs/master/blade) t ## Initialize our Laravel app on GitLab -We assume [you have installed a new laravel project](https://laravel.com/docs/master/installation#installation), so let's start with a unit test, and initialize Git for the project. +We assume [you have installed a new Laravel project](https://laravel.com/docs/master/installation#installation), so let's start with a unit test, and initialize Git for the project. ### Unit Test @@ -664,7 +667,7 @@ If something doesn't work as expected, you can roll back to the latest working v By clicking on the external link icon specified on the right side, GitLab opens the production website. Our deployment successfully was done and we can see the application is live. - + In the case that you're interested to know how is the application directory structure on the production server after deployment, here are three directories named `current`, `releases` and `storage`. As you know, the `current` directory is a symbolic link that points to the latest release. diff --git a/doc/ci/examples/php.md b/doc/ci/examples/php.md index 6a6c0e8fde1..e7768868c15 100644 --- a/doc/ci/examples/php.md +++ b/doc/ci/examples/php.md @@ -1,4 +1,7 @@ --- +stage: Verify +group: Continuous Integration +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers type: tutorial --- @@ -13,7 +16,7 @@ using the Shell executor. While it is possible to test PHP apps on any system, this would require manual configuration from the developer. To overcome this we will be using the -official [PHP docker image](https://hub.docker.com/_/php) that can be found in Docker Hub. +official [PHP Docker image](https://hub.docker.com/_/php) that can be found in Docker Hub. This will allow us to test PHP projects against different versions of PHP. However, not everything is plug 'n' play, you still need to configure some @@ -62,7 +65,7 @@ docker-php-ext-install pdo_mysql ``` You might wonder what `docker-php-ext-install` is. In short, it is a script -provided by the official php docker image that you can use to easily install +provided by the official PHP Docker image that you can use to easily install extensions. For more information read the documentation at <https://hub.docker.com/_/php>. @@ -111,7 +114,7 @@ test:app: ### Test against different PHP versions in Docker builds Testing against multiple versions of PHP is super easy. Just add another job -with a different docker image version and the runner will do the rest: +with a different Docker image version and the runner will do the rest: ```yaml before_script: @@ -171,7 +174,7 @@ Finally, push to GitLab and let the tests begin! ### Test against different PHP versions in Shell builds The [phpenv](https://github.com/phpenv/phpenv) project allows you to easily manage different versions of PHP -each with its own config. This is especially useful when testing PHP projects +each with its own configuration. This is especially useful when testing PHP projects with the Shell executor. You will have to install it on your build machine under the `gitlab-runner` diff --git a/doc/ci/examples/test-and-deploy-python-application-to-heroku.md b/doc/ci/examples/test-and-deploy-python-application-to-heroku.md index 2c626cb458a..30a64e65607 100644 --- a/doc/ci/examples/test-and-deploy-python-application-to-heroku.md +++ b/doc/ci/examples/test-and-deploy-python-application-to-heroku.md @@ -1,4 +1,7 @@ --- +stage: Verify +group: Continuous Integration +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers type: tutorial --- diff --git a/doc/ci/examples/test-and-deploy-ruby-application-to-heroku.md b/doc/ci/examples/test-and-deploy-ruby-application-to-heroku.md index 5df407f19fe..bf589c5991d 100644 --- a/doc/ci/examples/test-and-deploy-ruby-application-to-heroku.md +++ b/doc/ci/examples/test-and-deploy-ruby-application-to-heroku.md @@ -1,4 +1,7 @@ --- +stage: Verify +group: Continuous Integration +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers type: tutorial --- diff --git a/doc/ci/examples/test-clojure-application.md b/doc/ci/examples/test-clojure-application.md index 6ea38f22bca..31dacd34730 100644 --- a/doc/ci/examples/test-clojure-application.md +++ b/doc/ci/examples/test-clojure-application.md @@ -1,4 +1,7 @@ --- +stage: Verify +group: Continuous Integration +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers type: tutorial --- diff --git a/doc/ci/examples/test-scala-application.md b/doc/ci/examples/test-scala-application.md index 1831417e48e..fec376915c9 100644 --- a/doc/ci/examples/test-scala-application.md +++ b/doc/ci/examples/test-scala-application.md @@ -1,4 +1,7 @@ --- +stage: Verify +group: Continuous Integration +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers type: tutorial --- diff --git a/doc/ci/examples/test_phoenix_app_with_gitlab_ci_cd/index.md b/doc/ci/examples/test_phoenix_app_with_gitlab_ci_cd/index.md index cd1ad923249..c901f6c9c66 100644 --- a/doc/ci/examples/test_phoenix_app_with_gitlab_ci_cd/index.md +++ b/doc/ci/examples/test_phoenix_app_with_gitlab_ci_cd/index.md @@ -1,4 +1,7 @@ --- +stage: Verify +group: Continuous Integration +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers author: Alexandre S Hostert author_gitlab: Hostert level: beginner @@ -43,9 +46,9 @@ Phoenix can run in any OS where Erlang is supported: - Debian - Windows - Fedora -- Raspbian +- Raspberry Pi OS -Check the [Phoenix learning guide](https://hexdocs.pm/phoenix/learning.html) for more information. +Check the [Phoenix learning guide](https://hexdocs.pm/phoenix/overview.html) for more information. ### What is Elixir? @@ -151,7 +154,7 @@ point `localhost` to `127.0.0.1`. Great, now we have a local Phoenix Server running our app. -Locally, our application is running in an `iex` session. [iex](https://elixir-lang.org/getting-started/introduction.html#interactive-mode) stands for Interactive Elixir. +Locally, our application is running in an [`iex`](https://elixir-lang.org/getting-started/introduction.html#interactive-mode) session, which stands for Interactive Elixir. In this interactive mode, we can type any Elixir expression and get its result. To exit `iex`, we need to press `Ctrl+C` twice. So, when we need to stop the Phoenix server, we have to hit `Ctrl+C` twice. @@ -161,7 +164,7 @@ twice. With GitLab, we can manage our development workflow, improve our productivity, track issues, perform code review, and much more from a single platform. With GitLab CI/CD, we can be much more productive, because every time we, or our co-workers push any code, GitLab CI/CD will build and -test the changes, telling us in realtime if anything goes wrong. +test the changes, telling us in real time if anything goes wrong. Certainly, when our application starts to grow, we'll need more developers working on the same project and this process of building and testing can easily become a mess without proper management. @@ -262,7 +265,7 @@ project. our application? This virtual machine must have all dependencies to run our application. This is where a Docker image is needed. The correct image will provide the entire system for us. - As we are focusing on testing (not deploying), you can use the [elixir:latest](https://hub.docker.com/_/elixir) docker image, which already has the + As we are focusing on testing (not deploying), you can use the [elixir:latest](https://hub.docker.com/_/elixir) Docker image, which already has the dependencies for running Phoenix tests installed, such as Elixir and Erlang: ```yaml diff --git a/doc/ci/git_submodules.md b/doc/ci/git_submodules.md index 4797251129b..b12ac59625f 100644 --- a/doc/ci/git_submodules.md +++ b/doc/ci/git_submodules.md @@ -1,4 +1,7 @@ --- +stage: Verify +group: Continuous Integration +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers type: reference --- diff --git a/doc/ci/interactive_web_terminal/index.md b/doc/ci/interactive_web_terminal/index.md index 661e32b1571..be3741332e0 100644 --- a/doc/ci/interactive_web_terminal/index.md +++ b/doc/ci/interactive_web_terminal/index.md @@ -1,10 +1,13 @@ --- +stage: Verify +group: Runner +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers type: reference --- # Interactive Web Terminals -> [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/issues/50144) in GitLab 11.3. +> [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/50144) in GitLab 11.3. Interactive web terminals give the user access to a terminal in GitLab for running one-off commands for their CI pipeline. Since this is giving the user @@ -31,7 +34,7 @@ Two things need to be configured for the interactive web terminal to work: NOTE: **Note:** Interactive web terminals are not yet supported by [`gitlab-runner` Helm chart](https://docs.gitlab.com/charts/charts/gitlab/gitlab-runner/index.html), -but support [is planned](https://gitlab.com/gitlab-org/charts/gitlab-runner/issues/79). +but support [is planned](https://gitlab.com/gitlab-org/charts/gitlab-runner/-/issues/79). ## Debugging a running job @@ -41,7 +44,7 @@ NOTE: **Note:** Not all executors are NOTE: **Note:** The `docker` executor does not keep running after the build script is finished. At that point, the terminal will automatically disconnect and will not wait for the user to finish. Please follow [this -issue](https://gitlab.com/gitlab-org/gitlab-runner/issues/3605) for updates on +issue](https://gitlab.com/gitlab-org/gitlab-runner/-/issues/3605) for updates on improving this behavior. Sometimes, when a job is running, things don't go as you would expect, and it diff --git a/doc/ci/introduction/index.md b/doc/ci/introduction/index.md index b7f1837d83e..7ba24f9c861 100644 --- a/doc/ci/introduction/index.md +++ b/doc/ci/introduction/index.md @@ -1,4 +1,7 @@ --- +stage: Verify +group: Continuous Integration +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers description: "An overview of Continuous Integration, Continuous Delivery, and Continuous Deployment, as well as an introduction to GitLab CI/CD." type: concepts --- diff --git a/doc/ci/jenkins/index.md b/doc/ci/jenkins/index.md index c4346005138..37e0522e74d 100644 --- a/doc/ci/jenkins/index.md +++ b/doc/ci/jenkins/index.md @@ -1,4 +1,7 @@ --- +stage: Verify +group: Continuous Integration +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers comments: false type: index, howto --- @@ -72,13 +75,13 @@ There are some high level differences between the products worth mentioning: with the [`rules` syntax](../yaml/README.md#rules). - GitLab [pipeline scheduling concepts](../pipelines/schedules.md) are also different than with Jenkins. - You can reuse pipeline configurations using the [`include` keyword](../yaml/README.md#include) - and [templates](#templates). Your templates can be kept in a central repo (with different + and [templates](#templates). Your templates can be kept in a central repository (with different permissions), and then any project can use them. This central project could also contain scripts or other reusable code. - You can also use the [`extends` keyword](../yaml/README.md#extends) to reuse configuration within a single pipeline configuration. - All jobs within a single stage always run in parallel, and all stages run in sequence. We are planning - to allow certain jobs to break this sequencing as needed with our [directed acyclic graph](https://gitlab.com/gitlab-org/gitlab-foss/issues/47063) + to allow certain jobs to break this sequencing as needed with our [directed acyclic graph](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/47063) feature. - The [`parallel`](../yaml/README.md#parallel) keyword can automatically parallelize tasks, like tests that support parallelization. @@ -90,7 +93,7 @@ There are some high level differences between the products worth mentioning: - One important difference is that jobs run independently of each other and have a fresh environment in each job. Passing artifacts between jobs is controlled using the [`artifacts`](../yaml/README.md#artifacts) and [`dependencies`](../yaml/README.md#dependencies) - keywords. When finished, the planned [Workspaces](https://gitlab.com/gitlab-org/gitlab/issues/29265) + keywords. When finished, the planned [Workspaces](https://gitlab.com/gitlab-org/gitlab/-/issues/29265) feature will allow you to more easily persist a common workspace between serial jobs. - The `.gitlab-ci.yml` file is checked in to the root of your repository, much like a Jenkinsfile, but is in the YAML format (see [complete reference](../yaml/README.md)) instead of a Groovy DSL. It's most @@ -115,13 +118,13 @@ agents you were using. There are some important differences in the way Runners work in comparison to agents: -- Runners can be set up as [shared across an instance, be added at the group level, or set up at the project level](../runners/README.md#shared-specific-and-group-runners). +- Runners can be set up as [shared across an instance, be added at the group level, or set up at the project level](../runners/README.md#types-of-runners). They will self-select jobs from the scopes you've defined automatically. -- You can also [use tags](../runners/README.md#using-tags) for finer control, and +- You can also [use tags](../runners/README.md#use-tags-to-limit-the-number-of-jobs-using-the-runner) for finer control, and associate runners with specific jobs. For example, you can use a tag for jobs that require dedicated, more powerful, or specific hardware. - GitLab has [autoscaling for Runners](https://docs.gitlab.com/runner/configuration/autoscale.html) - which will let configure them to be provisioned as needed, and scaled down when not. + which will let you configure them to be provisioned as needed, and scaled down when not. This is similar to ephemeral agents in Jenkins. If you are using `gitlab.com`, you can take advantage of our [shared Runner fleet](../../user/gitlab_com/index.md#shared-runners) @@ -136,7 +139,7 @@ GitLab works a bit differently, we use the more highly structured [YAML](https:/ places scripting elements inside of `script:` blocks separate from the pipeline specification itself. This is a strength of GitLab, in that it helps keep the learning curve much simpler to get up and running -and avoids some of the problem of unconstrained complexity which can make your Jenkinsfiles hard to understand +and avoids some of the problem of unconstrained complexity which can make your Jenkinsfile hard to understand and manage. That said, we do of course still value DRY (don't repeat yourself) principles and want to ensure that @@ -202,12 +205,12 @@ be used by all projects in the group. An instance administrator can set a group the source for [instance project templates](../../user/group/custom_project_templates.md), which can be used by projects in that instance. -## Converting Declarative Jenkinsfiles +## Converting a declarative Jenkinsfile -Declarative Jenkinsfiles contain "Sections" and "Directives" which are used to control the behavior of your +A declarative Jenkinsfile contains "Sections" and "Directives" which are used to control the behavior of your pipelines. There are equivalents for all of these in GitLab, which we've documented below. -This section is based on the [Jenkinsfile syntax documentation](https://jenkins.io/doc/book/pipeline/syntax/) +This section is based on the [Jenkinsfile syntax documentation](https://www.jenkins.io/doc/book/pipeline/syntax/) and is meant to be a mapping of concepts there to concepts in GitLab. ### Sections @@ -217,7 +220,7 @@ and is meant to be a mapping of concepts there to concepts in GitLab. The agent section is used to define how a pipeline will be executed. For GitLab, we use the [GitLab Runner](../runners/README.md) to provide this capability. You can configure your own runners in Kubernetes or on any host, or take advantage of our shared runner fleet (note that the shared runner fleet is only available for GitLab.com users.) The link above will bring you to the documentation which will describe how to get -up and running quickly. We also support using [tags](../runners/README.md#using-tags) to direct different jobs +up and running quickly. We also support using [tags](../runners/README.md#use-tags-to-limit-the-number-of-jobs-using-the-runner) to direct different jobs to different Runners (execution agents). The `agent` section also allows you to define which Docker images should be used for execution, for which we use diff --git a/doc/ci/junit_test_reports.md b/doc/ci/junit_test_reports.md index a77044e849d..aa0d40a4d06 100644 --- a/doc/ci/junit_test_reports.md +++ b/doc/ci/junit_test_reports.md @@ -1,10 +1,13 @@ --- +stage: Verify +group: Testing +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers type: reference --- # JUnit test reports -> [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/issues/45318) in GitLab 11.2. Requires GitLab Runner 11.2 and above. +> [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/45318) in GitLab 11.2. Requires GitLab Runner 11.2 and above. ## Overview @@ -189,6 +192,20 @@ cpp: junit: report.xml ``` +#### CUnit + +[CUnit](https://cunity.gitlab.io/cunit/) can be made to produce [JUnit XML reports](https://cunity.gitlab.io/cunit/group__CI.html) automatically when run using its `CUnitCI.h` macros: + +```yaml +cunit: + stage: test + script: + - ./my-cunit-test + artifacts: + reports: + junit: ./my-cunit-test.xml +``` + ### .Net example The [JunitXML.TestLogger](https://www.nuget.org/packages/JunitXml.TestLogger/) NuGet @@ -215,17 +232,9 @@ Test: - ./**/*test-result.xml ``` -## Limitations - -Currently, the following tools might not work because their XML formats are unsupported in GitLab. - -|Case|Tool|Issue| -|---|---|---| -|`<testcase>` does not have `classname` attribute|ESlint, sass-lint|<https://gitlab.com/gitlab-org/gitlab-foss/issues/50964>| - ## Viewing JUnit test reports on GitLab -> [Introduced](https://gitlab.com/gitlab-org/gitlab/issues/24792) in GitLab 12.5. +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/24792) in GitLab 12.5. If JUnit XML files are generated and uploaded as part of a pipeline, these reports can be viewed inside the pipelines details page. The **Tests** tab on this page will @@ -250,6 +259,9 @@ following command: ```ruby Feature.enable(:junit_pipeline_view) + +# Enable the feature for a specific project +Feature.enable(:junit_pipeline_view, Project.find(<your-project-id-here>)) ``` ## Viewing JUnit screenshots on GitLab diff --git a/doc/ci/large_repositories/index.md b/doc/ci/large_repositories/index.md index b4059fc252b..b4652050654 100644 --- a/doc/ci/large_repositories/index.md +++ b/doc/ci/large_repositories/index.md @@ -1,4 +1,7 @@ --- +stage: Verify +group: Runner +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers type: reference --- @@ -114,6 +117,16 @@ For exact parameters accepted by for [`git clean`](https://git-scm.com/docs/git-clean). The available parameters are dependent on Git version. +## Git fetch extra flags + +> [Introduced](https://gitlab.com/gitlab-org/gitlab-runner/-/issues/4142) in GitLab Runner 13.1. + +[`GIT_FETCH_EXTRA_FLAGS`](../yaml/README.md#git-fetch-extra-flags) allows you +to modify `git fetch` behavior by passing extra flags. + +See the [`GIT_FETCH_EXTRA_FLAGS` documentation](../yaml/README.md#git-fetch-extra-flags) +for more information. + ## Fork-based workflow > Introduced in GitLab Runner 11.10. diff --git a/doc/ci/merge_request_pipelines/index.md b/doc/ci/merge_request_pipelines/index.md index a724bf416b6..7a4ca989fb6 100644 --- a/doc/ci/merge_request_pipelines/index.md +++ b/doc/ci/merge_request_pipelines/index.md @@ -1,11 +1,14 @@ --- +stage: Verify +group: Continuous Integration +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers type: reference, index last_update: 2019-07-03 --- # Pipelines for Merge Requests -> [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/issues/15310) in GitLab 11.6. +> [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/15310) in GitLab 11.6. In a [basic configuration](../pipelines/pipeline_architectures.md#basic-pipelines), GitLab runs a pipeline each time changes are pushed to a branch. @@ -143,7 +146,7 @@ test: Instead, you can use the [`$CI_COMMIT_REF_NAME` predefined environment -variable](../variables/predefined_variables.md#variables-reference) in +variable](../variables/predefined_variables.md) in combination with [`only:variables`](../yaml/README.md#onlyvariablesexceptvariables) to accomplish this behavior: diff --git a/doc/ci/merge_request_pipelines/pipelines_for_merged_results/index.md b/doc/ci/merge_request_pipelines/pipelines_for_merged_results/index.md index c35a5d0a07e..a43085c8e7a 100644 --- a/doc/ci/merge_request_pipelines/pipelines_for_merged_results/index.md +++ b/doc/ci/merge_request_pipelines/pipelines_for_merged_results/index.md @@ -1,11 +1,14 @@ --- +stage: Verify +group: Continuous Integration +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers type: reference last_update: 2019-07-03 --- # Pipelines for Merged Results **(PREMIUM)** -> [Introduced](https://gitlab.com/gitlab-org/gitlab/issues/7380) in [GitLab Premium](https://about.gitlab.com/pricing/) 11.10. +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/7380) in [GitLab Premium](https://about.gitlab.com/pricing/) 11.10. When you submit a merge request, you are requesting to merge changes from a source branch into a target branch. By default, the CI pipeline runs jobs @@ -44,7 +47,7 @@ To enable pipelines for merge results: - You must have maintainer [permissions](../../../user/permissions.md). - You must be using [GitLab Runner](https://gitlab.com/gitlab-org/gitlab-runner) 11.9 or later. - You must not be forking or using cross-repo workflows. To follow progress, - see [#11934](https://gitlab.com/gitlab-org/gitlab/issues/11934). + see [#11934](https://gitlab.com/gitlab-org/gitlab/-/issues/11934). - You must not be using [fast forward merges](../../../user/project/merge_requests/fast_forward_merge.md) yet. To follow progress, see [#58226](https://gitlab.com/gitlab-org/gitlab/-/issues/26996). @@ -56,7 +59,7 @@ To enable pipelines for merged results for your project: 1. [Configure your CI/CD configuration file](../index.md#configuring-pipelines-for-merge-requests) so that the pipeline or individual jobs run for merge requests. 1. Visit your project's **Settings > General** and expand **Merge requests**. -1. Check **Merge pipelines will try to validate the post-merge result prior to merging**. +1. Check **Enable merge trains and pipelines for merged results**. 1. Click **Save changes**. CAUTION: **Caution:** @@ -76,9 +79,9 @@ merge happens. For more information, read the [documentation on Merge Trains](merge_trains/index.md). -## Automatic pipeline cancelation +## Automatic pipeline cancellation -> [Introduced](https://gitlab.com/gitlab-org/gitlab/issues/12996) in [GitLab Premium](https://about.gitlab.com/pricing/) 12.3. +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/12996) in [GitLab Premium](https://about.gitlab.com/pricing/) 12.3. GitLab CI/CD can detect the presence of redundant pipelines, and will cancel them automatically in order to conserve CI resources. diff --git a/doc/ci/merge_request_pipelines/pipelines_for_merged_results/merge_trains/index.md b/doc/ci/merge_request_pipelines/pipelines_for_merged_results/merge_trains/index.md index d921b75aa44..fc8e6368d72 100644 --- a/doc/ci/merge_request_pipelines/pipelines_for_merged_results/merge_trains/index.md +++ b/doc/ci/merge_request_pipelines/pipelines_for_merged_results/merge_trains/index.md @@ -1,12 +1,15 @@ --- +stage: Verify +group: Continuous Integration +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers type: reference last_update: 2019-07-03 --- # Merge Trains **(PREMIUM)** -> - [Introduced](https://gitlab.com/gitlab-org/gitlab/issues/9186) in [GitLab Premium](https://about.gitlab.com/pricing/) 12.0. -> - [Squash and merge](../../../../user/project/merge_requests/squash_and_merge.md) support [introduced](https://gitlab.com/gitlab-org/gitlab/issues/13001) in [GitLab Premium](https://about.gitlab.com/pricing/) 12.6. +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/9186) in [GitLab Premium](https://about.gitlab.com/pricing/) 12.0. +> - [Squash and merge](../../../../user/project/merge_requests/squash_and_merge.md) support [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/13001) in [GitLab Premium](https://about.gitlab.com/pricing/) 12.6. When [pipelines for merged results](../index.md#pipelines-for-merged-results-premium) are enabled, the pipeline jobs run as if the changes from your source branch have already @@ -79,7 +82,7 @@ To enable merge trains for your project: 1. [Configure your CI/CD configuration file](../../index.md#configuring-pipelines-for-merge-requests) so that the pipeline or individual jobs run for merge requests. 1. Visit your project's **Settings > General** and expand **Merge requests**. -1. Check **Merge pipelines will try to validate the post-merge result prior to merging**. +1. Check **Enable merge trains and pipelines for merged results**. 1. Click **Save changes**. CAUTION: **Caution:** @@ -164,7 +167,7 @@ To check the reason: [Merge When Pipeline Succeeds](../../../../user/project/merge_requests/merge_when_pipeline_succeeds.md) is currently unavailable when Merge Trains are enabled. -See [the related issue](https://gitlab.com/gitlab-org/gitlab/issues/12267) +See [the related issue](https://gitlab.com/gitlab-org/gitlab/-/issues/12267) for more information. ### Merge Train Pipeline cannot be retried @@ -184,14 +187,14 @@ run a new successful pipeline before you can re-add a merge request to a merge t Merge trains ensure that each pipeline has succeeded before a merge happens, so you can clear the **Pipelines must succeed** check box and keep -**Merge pipelines will try to validate the post-merge result prior to merging** (merge trains) enabled. +**Enable merge trains and pipelines for merged results** (merge trains) enabled. If you want to keep the **Pipelines must succeed** option enabled along with Merge Trains, you can create a new pipeline for merged results when this error occurs by going to the **Pipelines** tab and clicking **Run pipeline**. Then click **Start/Add to merge train when pipeline succeeds**. -See [the related issue](https://gitlab.com/gitlab-org/gitlab/issues/35135) +See [the related issue](https://gitlab.com/gitlab-org/gitlab/-/issues/35135) for more information. ### Merge Trains feature flag **(PREMIUM ONLY)** diff --git a/doc/ci/metrics_reports.md b/doc/ci/metrics_reports.md index f353aa2670f..67f9ad00c90 100644 --- a/doc/ci/metrics_reports.md +++ b/doc/ci/metrics_reports.md @@ -1,14 +1,17 @@ --- +stage: Verify +group: Testing +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers type: reference --- # Metrics Reports **(PREMIUM)** -> [Introduced](https://gitlab.com/gitlab-org/gitlab/issues/9788) in [GitLab Premium](https://about.gitlab.com/pricing/) 11.10. Requires GitLab Runner 11.10 and above. +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/9788) in [GitLab Premium](https://about.gitlab.com/pricing/) 11.10. Requires GitLab Runner 11.10 and above. ## Overview -GitLab provides a lot of great reporting tools for [merge requests](../user/project/merge_requests/index.md) - [JUnit reports](junit_test_reports.md), [codequality](../user/project/merge_requests/code_quality.md), performance tests, etc. While JUnit is a great open framework for tests that "pass" or "fail", it is also important to see other types of metrics from a given change. +GitLab provides a lot of great reporting tools for [merge requests](../user/project/merge_requests/index.md) - [JUnit reports](junit_test_reports.md), [code quality](../user/project/merge_requests/code_quality.md), performance tests, etc. While JUnit is a great open framework for tests that "pass" or "fail", it is also important to see other types of metrics from a given change. You can configure your job to use custom Metrics Reports, and GitLab will display a report on the merge request so that it's easier and faster to identify changes without having to check the entire log. diff --git a/doc/ci/migration/circleci.md b/doc/ci/migration/circleci.md new file mode 100644 index 00000000000..f6868abc334 --- /dev/null +++ b/doc/ci/migration/circleci.md @@ -0,0 +1,332 @@ +--- +comments: false +type: index, howto +--- + +# Migrating from CircleCI + +If you are currently using CircleCI, you can migrate your CI/CD pipelines to [GitLab CI/CD](../introduction/index.md), +and start making use of all its powerful features. Check out our +[CircleCI vs GitLab](https://about.gitlab.com/devops-tools/circle-ci-vs-gitlab.html) +comparison to see what's different. + +We have collected several resources that you may find useful before starting to migrate. + +The [Quick Start Guide](../quick_start/README.md) is a good overview of how GitLab CI/CD works. You may also be interested in [Auto DevOps](../../topics/autodevops/index.md) which can be used to build, test, and deploy your applications with little to no configuration needed at all. + +For advanced CI/CD teams, [custom project templates](../../user/admin_area/custom_project_templates.md) can enable the reuse of pipeline configurations. + +If you have questions that are not answered here, the [GitLab community forum](https://forum.gitlab.com/) can be a great resource. + +## `config.yml` vs `gitlab-ci.yml` + +CircleCI's `config.yml` configuration file defines scripts, jobs, and workflows (known as "stages" in GitLab). In GitLab, a similar approach is used with a `.gitlab-ci.yml` file in the root directory of your repository. + +### Jobs + +In CircleCI, jobs are a collection of steps to perform a specific task. In GitLab, [jobs](../yaml/README.md#introduction) are also a fundamental element in the configuration file. The `checkout` parameter is not necessary in GitLab CI/CD as the repository is automatically fetched. + +CircleCI example job definition: + +```yaml +jobs: + job1: + steps: + - checkout + - run: "execute-script-for-job1" +``` + +Example of the same job definition in GitLab CI/CD: + +``` yaml +job1: + script: "execute-script-for-job1" +``` + +### Docker image definition + +CircleCI defines images at the job level, which is also supported by GitLab CI/CD. Additionally, GitLab CI/CD supports setting this globally to be used by all jobs that don't have `image` defined. + +CircleCI example image definition: + +```yaml +jobs: + job1: + docker: + - image: ruby:2.6 +``` + +Example of the same image definition in GitLab CI/CD: + +```yaml +job1: + image: ruby:2.6 +``` + +### Workflows + +CircleCI determines the run order for jobs with `workflows`. This is also used to determine concurrent, sequential, scheduled, or manual runs. The equivalent function in GitLab CI/CD is called [stages](../yaml/README.md#stages). Jobs on the same stage run in parallel, and only run after previous stages complete. Execution of the next stage is skipped when a job fails by default, but this can be allowed to continue even [after a failed job](../yaml/README.md#allow_failure). + +See [the Pipeline Architecture Overview](../pipelines/pipeline_architectures.md) for guidance on different types of pipelines that you can use. Pipelines can be tailored to meet your needs, such as for a large complex project or a monorepo with independent defined components. + +#### Parallel and sequential job execution + +The following examples show how jobs can run in parallel, or sequentially: + +1. `job1` and `job2` run in parallel (in the `build` stage for GitLab CI/CD). +1. `job3` runs only after `job1` and `job2` complete successfully (in the `test` stage). +1. `job4` runs only after `job3` completes successfully (in the `deploy` stage). + +CircleCI example with `workflows`: + +```yaml +version: 2 +jobs: + job1: + steps: + - checkout + - run: make build dependencies + job2: + steps: + - run: make build artifacts + job3: + steps: + - run: make test + job4: + steps: + - run: make deploy + +workflows: + version: 2 + jobs: + - job1 + - job2 + - job3: + requires: + - job1 + - job2 + - job4: + requires: + - job3 +``` + +Example of the same workflow as `stages` in GitLab CI/CD: + +```yaml +stages: + - build + - test + - deploy + +job 1: + stage: build + script: make build dependencies + +job 2: + stage: build + script: make build artifacts + +job3: + stage: test + script: make test + +job4: + stage: deploy + script: make deploy +``` + +#### Scheduled run + +GitLab CI/CD has an easy to use UI to [schedule pipelines](../pipelines/schedules.md). Also, [rules](../yaml/README.md#rules) can be used to determine if jobs should be included or excluded from a scheduled pipeline. + +CircleCI example of a scheduled workflow: + +```yaml +commit-workflow: + jobs: + - build +scheduled-workflow: + triggers: + - schedule: + cron: "0 1 * * *" + filters: + branches: + only: try-schedule-workflow + jobs: + - build +``` + +Example of the same scheduled pipeline using [`rules`](../yaml/README.md#rules) in GitLab CI/CD: + +```yaml +job1: + script: + - make build + rules: + - if: '$CI_PIPELINE_SOURCE == "schedule" && $CI_COMMIT_REF_NAME == "try-schedule-workflow"' +``` + +After the pipeline configuration is saved, you configure the cron schedule in the [GitLab UI](../pipelines/schedules.md#configuring-pipeline-schedules), and can enable or disable schedules in the UI as well. + +#### Manual run + +CircleCI example of a manual workflow: + +```yaml +release-branch-workflow: + jobs: + - build + - testing: + requires: + - build + - deploy: + type: approval + requires: + - testing +``` + +Example of the same workflow using [`when: manual`](../yaml/README.md#whenmanual) in GitLab CI/CD: + +```yaml +deploy_prod: + stage: deploy + script: + - echo "Deploy to production server" + when: manual +``` + +### Filter job by branch + +[Rules](../yaml/README.md#rules) are a mechanism to determine if the job will or will not run for a specific branch. + +CircleCI example of a job filtered by branch: + +```yaml +jobs: + deploy: + branches: + only: + - master + - /rc-.*/ +``` + +Example of the same workflow using `rules` in GitLab CI/CD: + +```yaml +deploy_prod: + stage: deploy + script: + - echo "Deploy to production server" + rules: + - if: '$CI_COMMIT_BRANCH == "master"' +``` + +### Caching + +GitLab provides a caching mechanism to speed up build times for your jobs by reusing previously downloaded dependencies. It's important to know the different between [cache and artifacts](../caching/index.md#cache-vs-artifacts) to make the best use of these features. + +CircleCI example of a job using a cache: + +```yaml +jobs: + job1: + steps: + - restore_cache: + key: source-v1-< .Revision > + - checkout + - run: npm install + - save_cache: + key: source-v1-< .Revision > + paths: + - "node_modules" +``` + +Example of the same pipeline using `cache` in GitLab CI/CD: + +```yaml +image: node:latest + +# Cache modules in between jobs +cache: + key: $CI_COMMIT_REF_SLUG + paths: + - .npm/ + +before_script: + - npm ci --cache .npm --prefer-offline + +test_async: + script: + - node ./specs/start.js ./specs/async.spec.js +``` + +## Contexts and variables + +CircleCI provides [Contexts](https://circleci.com/docs/2.0/contexts/) to securely pass environment variables across project pipelines. In GitLab, a [Group](../../user/group/index.md) can be created to assemble related projects together. At the group level, [variables](../variables/README.md#group-level-environment-variables) can be stored outside the individual projects, and securely passed into pipelines across multiple projects. + +## Orbs + +There are two GitLab issues open addressing CircleCI Orbs and how GitLab can achieve similar functionality. + +- <https://gitlab.com/gitlab-com/Product/-/issues/1151> +- <https://gitlab.com/gitlab-org/gitlab/-/issues/195173> + +## Build environments + +CircleCI offers `executors` as the underlying technology to run a specific job. In GitLab, this is done by [Runners](https://docs.gitlab.com/runner/). + +The following environments are supported: + +Self-Managed Runners: + +- Linux +- Windows +- macOS + +GitLab.com Shared Runners: + +- Linux +- Windows +- [Planned: macOS](https://gitlab.com/gitlab-com/gl-infra/infrastructure/-/issues/5720) + +### Machine and specific build environments + +[Tags](../yaml/README.md#tags) can be used to run jobs on different platforms, by telling GitLab which Runners should run the jobs. + +CircleCI example of a job running on a specific environment: + +```yaml +jobs: + ubuntuJob: + machine: + image: ubuntu-1604:201903-01 + steps: + - checkout + - run: echo "Hello, $USER!" + osxJob: + macos: + xcode: 11.3.0 + steps: + - checkout + - run: echo "Hello, $USER!" +``` + +Example of the same job using `tags` in GitLab CI/CD: + +```yaml +windows job: + stage: + - build + tags: + - windows + script: + - echo Hello, %USERNAME%! + +osx job: + stage: + - build + tags: + - osx + script: + - echo "Hello, $USER!" +``` diff --git a/doc/ci/multi_project_pipelines.md b/doc/ci/multi_project_pipelines.md index 2627f83c043..40d495a69f1 100644 --- a/doc/ci/multi_project_pipelines.md +++ b/doc/ci/multi_project_pipelines.md @@ -1,11 +1,14 @@ --- +stage: Verify +group: Continuous Integration +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers type: reference --- # Multi-project pipelines > - [Introduced](https://about.gitlab.com/releases/2015/08/22/gitlab-7-14-released/#build-triggers-api-gitlab-ci) in GitLab 7.14, as Build Triggers. -> - [Made available](https://gitlab.com/gitlab-org/gitlab/issues/199224) in all tiers in GitLab 12.8. +> - [Made available](https://gitlab.com/gitlab-org/gitlab/-/issues/199224) in all tiers in GitLab 12.8. You can set up [GitLab CI/CD](README.md) across multiple projects, so that a pipeline in one project can trigger a pipeline in another project. @@ -35,7 +38,7 @@ With Multi-Project Pipelines you can visualize the entire pipeline, including al ## Multi-project pipeline visualization **(PREMIUM)** -> [Introduced](https://gitlab.com/gitlab-org/gitlab/issues/2121) in [GitLab Premium 9.3](https://about.gitlab.com/releases/2017/06/22/gitlab-9-3-released/#multi-project-pipeline-graphs). +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/2121) in [GitLab Premium 9.3](https://about.gitlab.com/releases/2017/06/22/gitlab-9-3-released/#multi-project-pipeline-graphs). When you configure GitLab CI/CD for your project, you can visualize the stages of your [jobs](pipelines/index.md#configure-a-pipeline) on a [pipeline graph](pipelines/index.md#visualize-pipelines). @@ -50,7 +53,7 @@ and when hovering or tapping (on touchscreen devices) they will expand and be sh ## Triggering multi-project pipelines through API > - Use of `CI_JOB_TOKEN` for multi-project pipelines was [introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/2017) in [GitLab Premium](https://about.gitlab.com/pricing/) 9.3. -> - Use of `CI_JOB_TOKEN` for multi-project pipelines was [made available](https://gitlab.com/gitlab-org/gitlab/issues/31573) in all tiers in GitLab 12.4. +> - Use of `CI_JOB_TOKEN` for multi-project pipelines was [made available](https://gitlab.com/gitlab-org/gitlab/-/issues/31573) in all tiers in GitLab 12.4. When you use the [`CI_JOB_TOKEN` to trigger pipelines](triggers/README.md#ci-job-token), GitLab recognizes the source of the job token, and thus internally ties these pipelines @@ -61,8 +64,8 @@ outbound connections for upstream and downstream pipeline dependencies. ## Creating multi-project pipelines from `.gitlab-ci.yml` -> - [Introduced](https://gitlab.com/gitlab-org/gitlab/issues/8997) in [GitLab Premium](https://about.gitlab.com/pricing/) 11.8. -> - [Made available](https://gitlab.com/gitlab-org/gitlab/issues/199224) in all tiers in 12.8. +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/8997) in [GitLab Premium](https://about.gitlab.com/pricing/) 11.8. +> - [Made available](https://gitlab.com/gitlab-org/gitlab/-/issues/199224) in all tiers in 12.8. ### Triggering a downstream pipeline using a bridge job @@ -129,7 +132,7 @@ Use: - The `project` keyword to specify the full path to a downstream project. - The `branch` keyword to specify the name of a branch in the project specified by `project`. - [From GitLab 12.4](https://gitlab.com/gitlab-org/gitlab/issues/10126), variable expansion is + [From GitLab 12.4](https://gitlab.com/gitlab-org/gitlab/-/issues/10126), variable expansion is supported. GitLab will use a commit that is currently on the HEAD of the branch when @@ -192,8 +195,8 @@ the ones defined in the upstream project will take precedence. ### Mirroring status from triggered pipeline -> - [Introduced](https://gitlab.com/gitlab-org/gitlab/issues/11238) in [GitLab Premium](https://about.gitlab.com/pricing/) 12.3. -> - [Moved](https://gitlab.com/gitlab-org/gitlab/issues/199224) to GitLab Core in 12.8. +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/11238) in [GitLab Premium](https://about.gitlab.com/pricing/) 12.3. +> - [Moved](https://gitlab.com/gitlab-org/gitlab/-/issues/199224) to GitLab Core in 12.8. You can mirror the pipeline status from the triggered pipeline to the source bridge job by using `strategy: depend`. For example: @@ -240,7 +243,7 @@ Some features are not implemented yet. For example, support for environments. ## Trigger a pipeline when an upstream project is rebuilt -> [Introduced](https://gitlab.com/gitlab-org/gitlab/issues/9045) in [GitLab Premium](https://about.gitlab.com/pricing/) 12.8. +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/9045) in [GitLab Premium](https://about.gitlab.com/pricing/) 12.8. You can trigger a pipeline in your project whenever a pipeline finishes for a new tag in a different project: diff --git a/doc/ci/parent_child_pipelines.md b/doc/ci/parent_child_pipelines.md index e28adc2bc01..c936bd8f5fe 100644 --- a/doc/ci/parent_child_pipelines.md +++ b/doc/ci/parent_child_pipelines.md @@ -1,10 +1,13 @@ --- +stage: Verify +group: Continuous Integration +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers type: reference --- # Parent-child pipelines -> [Introduced](https://gitlab.com/gitlab-org/gitlab/issues/16094) in GitLab 12.7. +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/16094) in GitLab 12.7. As pipelines grow more complex, a few related problems start to emerge: @@ -131,7 +134,7 @@ microservice_a: ## Dynamic child pipelines -> [Introduced](https://gitlab.com/gitlab-org/gitlab/issues/35632) in GitLab 12.9. +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/35632) in GitLab 12.9. Instead of running a child pipeline from a static YAML file, you can define a job that runs your own script to generate a YAML file, which is then [used to trigger a child pipeline](yaml/README.md#trigger-child-pipeline-with-generated-configuration-file). @@ -148,5 +151,5 @@ This is [resolved in GitLab 12.10](https://gitlab.com/gitlab-org/gitlab/-/issues ## Limitations A parent pipeline can trigger many child pipelines, but a child pipeline cannot trigger -further child pipelines. See the [related issue](https://gitlab.com/gitlab-org/gitlab/issues/29651) +further child pipelines. See the [related issue](https://gitlab.com/gitlab-org/gitlab/-/issues/29651) for discussion on possible future improvements. diff --git a/doc/ci/pipelines/img/code_coverage_graph_v13_1.png b/doc/ci/pipelines/img/code_coverage_graph_v13_1.png Binary files differnew file mode 100644 index 00000000000..da92a1b4a86 --- /dev/null +++ b/doc/ci/pipelines/img/code_coverage_graph_v13_1.png diff --git a/doc/ci/pipelines/index.md b/doc/ci/pipelines/index.md index 731cb196eaa..48aaebc3456 100644 --- a/doc/ci/pipelines/index.md +++ b/doc/ci/pipelines/index.md @@ -1,4 +1,7 @@ --- +stage: Verify +group: Continuous Integration +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers disqus_identifier: 'https://docs.gitlab.com/ee/ci/pipelines.html' type: reference --- @@ -88,7 +91,7 @@ Clicking a pipeline will bring you to the **Pipeline Details** page and show the jobs that were run for that pipeline. From here you can cancel a running pipeline, retry jobs on a failed pipeline, or [delete a pipeline](#delete-a-pipeline). -[Starting in GitLab 12.3](https://gitlab.com/gitlab-org/gitlab-foss/issues/50499), a link to the +[Starting in GitLab 12.3](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/50499), a link to the latest pipeline for the last commit of a given branch is available at `/project/pipelines/[branch]/latest`. Also, `/project/pipelines/latest` will redirect you to the latest pipeline for the last commit on the project's default branch. @@ -98,6 +101,8 @@ you can filter the pipeline list by: - Trigger author - Branch name +- Status ([since GitLab 13.1](https://gitlab.com/gitlab-org/gitlab/-/issues/217617)) +- Tag ([since GitLab 13.1](https://gitlab.com/gitlab-org/gitlab/-/issues/217617)) ### Run a pipeline manually @@ -119,7 +124,7 @@ The pipeline will execute the jobs as configured. ### Run a pipeline by using a URL query string -> [Introduced](https://gitlab.com/gitlab-org/gitlab/issues/24146) in GitLab 12.5. +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/24146) in GitLab 12.5. You can use a query string to pre-populate the **Run Pipeline** page. For example, the query string `.../pipelines/new?ref=my_branch&var[foo]=bar&file_var[file_foo]=file_bar` will pre-populate the @@ -179,7 +184,7 @@ This functionality is only available: ### Delete a pipeline -> [Introduced](https://gitlab.com/gitlab-org/gitlab/issues/24851) in GitLab 12.7. +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/24851) in GitLab 12.7. Users with [owner permissions](../../user/permissions.md) in a project can delete a pipeline by clicking on the pipeline in the **CI/CD > Pipelines** to get to the **Pipeline Details** @@ -403,7 +408,7 @@ For example, if you start rolling out new code and: ### Expand and collapse job log sections -> [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/issues/14664) in GitLab 12.0. +> [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/14664) in GitLab 12.0. Job logs are divided into sections that can be collapsed or expanded. Each section will display the duration. @@ -500,7 +505,7 @@ Stages in pipeline mini graphs are collapsible. Hover your mouse over them and c ### Pipeline success and duration charts > - Introduced in GitLab 3.1.1 as Commit Stats, and later renamed to Pipeline Charts. -> - [Renamed](https://gitlab.com/gitlab-org/gitlab/issues/38318) to CI / CD Analytics in GitLab 12.8. +> - [Renamed](https://gitlab.com/gitlab-org/gitlab/-/issues/38318) to CI / CD Analytics in GitLab 12.8. GitLab tracks the history of your pipeline successes and failures, as well as how long each pipeline ran. To view this information, go to **Analytics > CI / CD Analytics**. diff --git a/doc/ci/pipelines/job_artifacts.md b/doc/ci/pipelines/job_artifacts.md index d4774016d9c..1d60f412e5e 100644 --- a/doc/ci/pipelines/job_artifacts.md +++ b/doc/ci/pipelines/job_artifacts.md @@ -1,4 +1,7 @@ --- +stage: Verify +group: Continuous Integration +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers disqus_identifier: 'https://docs.gitlab.com/ee/user/project/pipelines/job_artifacts.html' type: reference, howto --- @@ -103,7 +106,7 @@ combination thereof (`junit: [rspec.xml, test-results/TEST-*.xml]`). #### `artifacts:reports:dotenv` -> - [Introduced](https://gitlab.com/gitlab-org/gitlab/issues/17066) in GitLab 12.9. +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/17066) in GitLab 12.9. > - Requires GitLab Runner 11.5 and later. The `dotenv` report collects a set of environment variables as artifacts. @@ -122,7 +125,7 @@ There are a couple of limitations on top of the [original dotenv rules](https:// #### `artifacts:reports:cobertura` -> - [Introduced](https://gitlab.com/gitlab-org/gitlab/issues/3708) in GitLab 12.9. +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/3708) in GitLab 12.9. > - Requires [GitLab Runner](https://docs.gitlab.com/runner/) 11.5 and above. The `cobertura` report collects [Cobertura coverage XML files](../../user/project/merge_requests/test_coverage_visualization.md). @@ -137,9 +140,10 @@ third party ports for other languages like JavaScript, Python, Ruby, and so on. > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/207528) in GitLab 13.0. > - Requires [GitLab Runner](https://docs.gitlab.com/runner/) 11.5 and above. -The `terraform` report obtains a Terraform `tfplan.json` file. The collected Terraform +The `terraform` report obtains a Terraform `tfplan.json` file. [JQ processing required to remove credentials](../../user/infrastructure/index.md#output-terraform-plan-information-into-a-merge-request). The collected Terraform plan report will be uploaded to GitLab as an artifact and will be automatically shown -in merge requests. +in merge requests. For more information, see +[Output `terraform plan` information into a merge request](../../user/infrastructure/index.md#output-terraform-plan-information-into-a-merge-request). #### `artifacts:reports:codequality` **(STARTER)** @@ -164,6 +168,18 @@ The collected SAST report will be uploaded to GitLab as an artifact and will be in the merge requests and pipeline view. It's also used to provide data for security dashboards. +#### `artifacts:reports:secret_detection` **(ULTIMATE)** + +> - Introduced in GitLab 13.1. +> - Requires GitLab Runner 11.5 and above. + +The `secret-detection` report collects [detected secrets](../../user/application_security/secret_detection/index.md) +as artifacts. + +The collected Secret Detection report is uploaded to GitLab as an artifact and summarized +in the merge requests and pipeline view. It's also used to provide data for security +dashboards. + #### `artifacts:reports:dependency_scanning` **(ULTIMATE)** > - Introduced in GitLab 11.5. @@ -250,6 +266,17 @@ as artifacts. The collected Metrics report will be uploaded to GitLab as an artifact and will be automatically shown in merge requests. +#### `artifacts:reports:requirements` **(ULTIMATE)** + +> - [Introduced](https://gitlab.com/groups/gitlab-org/-/epics/2859) in GitLab 13.1. +> - Requires GitLab Runner 11.5 and above. + +The `requirements` report collects `requirements.json` files as artifacts. + +The collected Requirements report will be uploaded to GitLab as an artifact and +existing [requirements](../../user/project/requirements/index.md) will be +marked as Satisfied. + ## Browsing artifacts > - From GitLab 9.2, PDFs, images, videos, and other formats can be previewed directly in the job artifacts browser without the need to download them. diff --git a/doc/ci/pipelines/pipeline_architectures.md b/doc/ci/pipelines/pipeline_architectures.md index 4872e2595f9..9176444660d 100644 --- a/doc/ci/pipelines/pipeline_architectures.md +++ b/doc/ci/pipelines/pipeline_architectures.md @@ -1,4 +1,7 @@ --- +stage: Verify +group: Continuous Integration +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers type: reference --- diff --git a/doc/ci/pipelines/schedules.md b/doc/ci/pipelines/schedules.md index 0c0a613c628..e488179ffee 100644 --- a/doc/ci/pipelines/schedules.md +++ b/doc/ci/pipelines/schedules.md @@ -1,4 +1,7 @@ --- +stage: Verify +group: Continuous Integration +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers disqus_identifier: 'https://docs.gitlab.com/ee/user/project/pipelines/schedules.html' type: reference, howto --- diff --git a/doc/ci/pipelines/settings.md b/doc/ci/pipelines/settings.md index 0a859b5b68f..2c9d4ccf2c4 100644 --- a/doc/ci/pipelines/settings.md +++ b/doc/ci/pipelines/settings.md @@ -1,4 +1,7 @@ --- +stage: Verify +group: Continuous Integration +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers disqus_identifier: 'https://docs.gitlab.com/ee/user/project/pipelines/settings.html' type: reference, howto --- @@ -59,7 +62,7 @@ if the job surpasses the threshold, it is marked as failed. > [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/17221) in GitLab 10.7. Project defined timeout (either specific timeout set by user or the default -60 minutes timeout) may be [overridden on Runner level](../runners/README.md#setting-maximum-job-timeout-for-a-runner). +60 minutes timeout) may be [overridden on Runner level](../runners/README.md#set-maximum-job-timeout-for-a-runner). ## Maximum artifacts size **(CORE ONLY)** @@ -69,7 +72,7 @@ For information about setting a maximum artifact size for a project, see ## Custom CI configuration path > - [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/12509) in GitLab 9.4. -> - [Support for external `.gitlab-ci.yml` locations](https://gitlab.com/gitlab-org/gitlab/issues/14376) introduced in GitLab 12.6. +> - [Support for external `.gitlab-ci.yml` locations](https://gitlab.com/gitlab-org/gitlab/-/issues/14376) introduced in GitLab 12.6. By default we look for the `.gitlab-ci.yml` file in the project's root directory. If needed, you can specify an alternate path and file name, including locations outside the project. @@ -118,7 +121,8 @@ job log using a regular expression. In the pipelines settings, search for the  Leave blank if you want to disable it or enter a Ruby regular expression. You -can use <https://rubular.com> to test your regex. +can use <https://rubular.com> to test your regex. The regex returns the **last** +match found in the output. If the pipeline succeeds, the coverage is shown in the merge request widget and in the jobs table. @@ -130,15 +134,18 @@ in the jobs table. A few examples of known coverage tools for a variety of languages can be found in the pipelines settings page. -### Download test coverage history +### Code Coverage history -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/209121) in GitLab 12.10. +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/209121) the ability to download a `.csv` in GitLab 12.10. +> - [Graph introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/33743) in GitLab 13.1. If you want to see the evolution of your project code coverage over time, -you can download a CSV file with this data. From your project: +you can view a graph or download a CSV file with this data. From your project: -1. Go to **{chart}** **Project Analytics > Repository**. -1. Click **Download raw data (.csv)** +1. Go to **{chart}** **Project Analytics > Repository** to see the historic data for each job listed in the dropdown above the graph. +1. If you want a CSV file of that data, click **Download raw data (.csv)** + + ### Removing color codes @@ -205,9 +212,11 @@ you can enable this in the project settings: 1. Check the **Auto-cancel redundant, pending pipelines** checkbox. 1. Click **Save changes**. +Note that only jobs with [interruptible](../yaml/README.md#interruptible) set to `true` will be cancelled. + ## Skip outdated deployment jobs -> [Introduced](https://gitlab.com/gitlab-org/gitlab/issues/25276) in GitLab 12.9. +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/25276) in GitLab 12.9. Your project may have multiple concurrent deployment jobs that are scheduled to run within the same time frame. @@ -224,6 +233,8 @@ To avoid this scenario: The pending deployment jobs will be skipped. +For more information, see [Deployment safety](../environments/deployment_safety.md). + ## Pipeline Badges In the pipelines settings page you can find pipeline status and test coverage @@ -289,7 +300,7 @@ https://example.gitlab.com/<namespace>/<project>/badges/<branch>/coverage.svg?st #### Flat square -> [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/issues/30120) in GitLab 11.8. +> [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/30120) in GitLab 11.8. ```plaintext https://example.gitlab.com/<namespace>/<project>/badges/<branch>/coverage.svg?style=flat-square @@ -297,6 +308,16 @@ https://example.gitlab.com/<namespace>/<project>/badges/<branch>/coverage.svg?st  +### Custom badge text + +The text for a badge can be customized. This can be useful to differentiate between multiple coverage jobs that run in the same pipeline. Customize the badge text and width by adding the `key_text=custom_text` and `key_width=custom_key_width` parameters to the URL: + +```plaintext +https://gitlab.com/gitlab-org/gitlab-foss/badges/master/coverage.svg?job=karma&key_text=Frontend+Coverage&key_width=100 +``` + + + ## Environment Variables [Environment variables](../variables/README.md#gitlab-cicd-environment-variables) can be set in an environment to be available to a runner. diff --git a/doc/ci/quick_start/README.md b/doc/ci/quick_start/README.md index a14cfcfa9ed..25469570603 100644 --- a/doc/ci/quick_start/README.md +++ b/doc/ci/quick_start/README.md @@ -1,4 +1,7 @@ --- +stage: Verify +group: Continuous Integration +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers type: reference --- @@ -165,7 +168,7 @@ The next step is to configure a Runner so that it picks the pending jobs. ## Configuring a Runner In GitLab, Runners run the jobs that you define in `.gitlab-ci.yml`. A Runner -can be a virtual machine, a VPS, a bare-metal machine, a docker container or +can be a virtual machine, a VPS, a bare-metal machine, a Docker container or even a cluster of containers. GitLab and the Runners communicate through an API, so the only requirement is that the Runner's machine has network access to the GitLab server. @@ -184,7 +187,7 @@ can be found at <https://docs.gitlab.com/runner/>. In order to have a functional Runner you need to follow two steps: 1. [Install it](https://docs.gitlab.com/runner/install/) -1. [Configure it](../runners/README.md#registering-a-specific-runner) +1. [Configure it](https://docs.gitlab.com/runner/configuration/) Follow the links above to set up your own Runner or use a Shared Runner as described in the next section. diff --git a/doc/ci/review_apps/index.md b/doc/ci/review_apps/index.md index ad0a4270f43..67fac70c8de 100644 --- a/doc/ci/review_apps/index.md +++ b/doc/ci/review_apps/index.md @@ -7,7 +7,7 @@ type: reference # Review Apps -> - [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/issues/21971) in GitLab 8.12. Further additions were made in GitLab 8.13 and 8.14. +> - [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/21971) in GitLab 8.12. Further additions were made in GitLab 8.13 and 8.14. > - Inspired by [Heroku's Review Apps](https://devcenter.heroku.com/articles/github-integration-review-apps), which itself was inspired by [Fourchette](https://github.com/rainforestapp/fourchette). Review Apps is a collaboration tool that takes the hard work out of providing an environment to showcase product changes. @@ -65,7 +65,7 @@ The process of configuring Review Apps is as follows: ### Enable Review Apps button -> [Introduced](https://gitlab.com/gitlab-org/gitlab/issues/118844) in GitLab 12.8. +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/118844) in GitLab 12.8. When configuring Review Apps for a project, you need to add a new job to `.gitlab-ci.yml`, as mentioned above. To facilitate this and if you are using Kubernetes, you can click @@ -95,8 +95,11 @@ The following are example projects that demonstrate Review App configuration: - [NGINX](https://gitlab.com/gitlab-examples/review-apps-nginx). - [OpenShift](https://gitlab.com/gitlab-examples/review-apps-openshift). -<i class="fa fa-youtube-play youtube" aria-hidden="true"></i> -See also the video [Demo: Cloud Native Development with GitLab](https://www.youtube.com/watch?v=jfIyQEwrocw), which includes a Review Apps example. +Other examples of Review Apps: + +- <i class="fa fa-youtube-play youtube" aria-hidden="true"></i> +[Cloud Native Development with GitLab](https://www.youtube.com/watch?v=jfIyQEwrocw). +- [Review Apps for Android](https://about.gitlab.com/blog/2020/05/06/how-to-create-review-apps-for-android-with-gitlab-fastlane-and-appetize-dot-io/). ## Route Maps @@ -185,7 +188,7 @@ Once you have the route mapping set up, it will take effect in the following loc ## Visual Reviews **(STARTER)** -> [Introduced](https://gitlab.com/gitlab-org/gitlab/issues/10761) in GitLab Starter 12.0. +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/10761) in GitLab Starter 12.0. With Visual Reviews, you can provide a feedback form to your Review Apps so that reviewers can post comments directly from the app back to the merge request @@ -258,7 +261,7 @@ Then, when your app is deployed via GitLab CI/CD, those variables should get replaced with their real values. NOTE: **Note:** -Future enhancements [are planned](https://gitlab.com/gitlab-org/gitlab/issues/11322) +Future enhancements [are planned](https://gitlab.com/gitlab-org/gitlab/-/issues/11322) to make this process even easier. ### Determining merge request ID diff --git a/doc/ci/runners/README.md b/doc/ci/runners/README.md index deefd2f0e73..86fabd5d54f 100644 --- a/doc/ci/runners/README.md +++ b/doc/ci/runners/README.md @@ -1,190 +1,59 @@ --- +stage: Verify +group: Runner +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers type: reference --- # Configuring GitLab Runners In GitLab CI/CD, Runners run the code defined in [`.gitlab-ci.yml`](../yaml/README.md). -They are isolated (virtual) machines that pick up jobs through the coordinator -API of GitLab CI/CD. - -A Runner can be specific to a certain project or serve any project -in GitLab CI/CD. A Runner that serves all projects is called a shared Runner. - -Ideally, the GitLab Runner should not be installed on the same machine as GitLab. -Read the [requirements documentation](../../install/requirements.md#gitlab-runner) -for more information. - -## Shared, specific and group Runners - -After [installing the Runner](https://docs.gitlab.com/runner/install/), you can either register it as shared or -specific. You can only register a shared Runner if you have admin access to -the GitLab instance. The main differences between a shared and a specific Runner -are: - -- **Shared Runners** are useful for jobs that have similar requirements, - between multiple projects. Rather than having multiple Runners idling for - many projects, you can have a single or a small number of Runners that handle - multiple projects. This makes it easier to maintain and update them. - Shared Runners process jobs using a [fair usage queue](#how-shared-runners-pick-jobs). - In contrast to specific Runners that use a FIFO queue, this prevents - cases where projects create hundreds of jobs which can lead to eating all - available shared Runners resources. -- **Specific Runners** are useful for jobs that have special requirements or for - projects with a specific demand. If a job has certain requirements, you can set - up the specific Runner with this in mind, while not having to do this for all - Runners. For example, if you want to deploy a certain project, you can set up - a specific Runner to have the right credentials for this. The [usage of tags](#using-tags) - may be useful in this case. Specific Runners process jobs using a [FIFO](https://en.wikipedia.org/wiki/FIFO_(computing_and_electronics)) queue. -- **Group Runners** are useful when you have multiple projects under one group - and would like all projects to have access to a set of Runners. Group Runners - process jobs using a [FIFO](https://en.wikipedia.org/wiki/FIFO_(computing_and_electronics)) queue. - -A Runner that is specific only runs for the specified project(s). A shared Runner -can run jobs for every project that has enabled the option **Allow shared Runners** -under **Settings > CI/CD**. - -Projects with high demand of CI activity can also benefit from using specific -Runners. By having dedicated Runners you are guaranteed that the Runner is not -being held up by another project's jobs. - -You can set up a specific Runner to be used by multiple projects. The difference -with a shared Runner is that you have to enable each project explicitly for -the Runner to be able to run its jobs. +A GitLab Runner is a lightweight, highly-scalable agent that picks up a CI job through +the coordinator API of GitLab CI/CD, runs the job, and sends the result back to the GitLab instance. -Specific Runners do not get shared with forked projects automatically. -A fork does copy the CI settings (jobs, allow shared, etc) of the cloned -repository. - -## Registering a shared Runner - -You can only register a shared Runner if you are an admin of the GitLab instance. - -1. Grab the shared-Runner token on the `admin/runners` page - -  - -1. [Register the Runner](https://docs.gitlab.com/runner/register/) - -Shared Runners are enabled by default as of GitLab 8.2, but can be disabled -with the **Disable shared Runners** button which is present under each project's -**Settings > CI/CD** page. Previous versions of GitLab defaulted shared -Runners to disabled. - -## Registering a specific Runner - -Registering a specific Runner can be done in two ways: - -1. Creating a Runner with the project registration token -1. Converting a shared Runner into a specific Runner (one-way, admin only) - -### Registering a specific Runner with a project registration token - -To create a specific Runner without having admin rights to the GitLab instance, -visit the project you want to make the Runner work for in GitLab: - -1. Go to **Settings > CI/CD** to obtain the token -1. [Register the Runner](https://docs.gitlab.com/runner/register/) - -## Registering a group Runner - -Creating a group Runner requires Owner permissions for the group. To create a -group Runner visit the group you want to make the Runner work for in GitLab: - -1. Go to **Settings > CI/CD** to obtain the token -1. [Register the Runner](https://docs.gitlab.com/runner/register/) - -### Making an existing shared Runner specific - -If you are an admin on your GitLab instance, you can turn any shared Runner into -a specific one, but not the other way around. Keep in mind that this is a one -way transition. - -1. Go to the Runners in the **Admin Area > Overview > Runners** (`/admin/runners`) - and find your Runner -1. Enable any projects under **Restrict projects for this Runner** to be used - with the Runner - -From now on, the shared Runner will be specific to those projects. +Runners are created by an administrator and are visible in the GitLab UI. +Runners can be specific to certain projects or available to all projects. -## Locking a specific Runner from being enabled for other projects +## Types of Runners -You can configure a Runner to assign it exclusively to a project. When a -Runner is locked this way, it can no longer be enabled for other projects. -This setting can be enabled the first time you [register a Runner](https://docs.gitlab.com/runner/register/) and -can be changed afterwards under each Runner's settings. +There are three types of Runners: -To lock/unlock a Runner: +- [Shared](#shared-runners) (for all projects) +- [Group](#group-runners) (for all projects in a group) +- [Specific](#specific-runners) (for specific projects) -1. Visit your project's **Settings > CI/CD** -1. Find the Runner you wish to lock/unlock and make sure it's enabled -1. Click the pencil button -1. Check the **Lock to current projects** option -1. Click **Save changes** for the changes to take effect +If you are running self-managed GitLab, you can create your own Runners. -## Assigning a Runner to another project +If you are using GitLab.com, you can use the shared Runners provided by GitLab or +create your own group or specific Runners. -If you are an Owner on a project where a specific Runner is assigned to, and the -Runner is not [locked only to that project](#locking-a-specific-runner-from-being-enabled-for-other-projects), -you can enable the Runner also on any other project where you have Owner permissions. - -To enable/disable a Runner in your project: - -1. Visit your project's **Settings > CI/CD** -1. Find the Runner you wish to enable/disable -1. Click **Enable for this project** or **Disable for this project** - -> **Note**: -Consider that if you don't lock your specific Runner to a specific project, any -user with Maintainer role in you project can assign your Runner to another arbitrary -project without requiring your authorization, so use it with caution. - -CAUTION: **Caution:** -Never add a private Runner that you're using in your private projects to a -project that you share with other people. - -CAUTION: **Caution:** -Never use a Runner from a project which has multiple maintainers in your -private project. - -An admin can enable/disable a specific Runner for projects: - -1. Navigate to **Admin > Runners** -1. Find the Runner you wish to enable/disable -1. Click edit on the Runner -1. Click **Enable** or **Disable** on the project - -## Protected Runners - -> [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/13194) in GitLab 10.0. +### Shared Runners -You can protect Runners from revealing sensitive information. -Whenever a Runner is protected, the Runner picks only jobs created on -[protected branches](../../user/project/protected_branches.md) or [protected tags](../../user/project/protected_tags.md), and ignores other jobs. +*Shared Runners* are available to every project in a GitLab instance. -To protect/unprotect Runners: +Use shared Runners when you have multiple jobs with similar requirements. Rather than +having multiple Runners idling for many projects, you can have a few Runners that handle +multiple projects. -1. Visit your project's **Settings > CI/CD** -1. Find a Runner you want to protect/unprotect and make sure it's enabled -1. Click the pencil button besides the Runner name -1. Check the **Protected** option -1. Click **Save changes** for the changes to take effect +If you are using a self-managed instance of GitLab, your administrator can create +shared Runners and configure them to use the +[executor](https://docs.gitlab.com/runner/executors/README.html) you want. - +If you are using GitLab.com, you can select from a list of +[shared Runners that GitLab maintains](../../user/gitlab_com/index.md#shared-runners). -## Manually clearing the Runners cache +#### How shared Runners pick jobs -Read [clearing the cache](../caching/index.md#clearing-the-cache). +Shared Runners process jobs by using a fair usage queue. This queue prevents +projects from creating hundreds of jobs and using all available +shared Runner resources. -## How shared Runners pick jobs - -Shared Runners abide to a process queue we call fair usage. The fair usage -algorithm tries to assign jobs to shared Runners from projects that have the -lowest number of jobs currently running on shared Runners. +The fair usage queue algorithm assigns jobs based on the projects that have the +fewest number of jobs already running on shared Runners. **Example 1** -We have following jobs in queue: +If these jobs are in the queue: - Job 1 for Project 1 - Job 2 for Project 1 @@ -193,20 +62,20 @@ We have following jobs in queue: - Job 5 for Project 2 - Job 6 for Project 3 -With the fair usage algorithm jobs are assigned in following order: +The fair usage algorithm assigns jobs in this order: -1. Job 1 is chosen first, because it has the lowest job number from projects with no running jobs (that is, all projects) -1. Job 4 is next, because 4 is now the lowest job number from projects with no running jobs (Project 1 has a job running) -1. Job 6 is next, because 6 is now the lowest job number from projects with no running jobs (Projects 1 and 2 have jobs running) -1. Job 2 is next, because, of projects with the lowest number of jobs running (each has 1), it is the lowest job number +1. Job 1 is chosen first, because it has the lowest job number from projects with no running jobs (that is, all projects). +1. Job 4 is next, because 4 is now the lowest job number from projects with no running jobs (Project 1 has a job running). +1. Job 6 is next, because 6 is now the lowest job number from projects with no running jobs (Projects 1 and 2 have jobs running). +1. Job 2 is next, because, of projects with the lowest number of jobs running (each has 1), it is the lowest job number. 1. Job 5 is next, because Project 1 now has 2 jobs running and Job 5 is the lowest remaining job number between Projects 2 and 3. -1. Lastly we choose Job 3... because it's the only job left +1. Finally is Job 3... because it's the only job left. --- **Example 2** -We have following jobs in queue: +If these jobs are in the queue: - Job 1 for project 1 - Job 2 for project 1 @@ -215,103 +84,129 @@ We have following jobs in queue: - Job 5 for project 2 - Job 6 for project 3 -With the fair usage algorithm jobs are assigned in following order: +The fair usage algorithm assigns jobs in this order: -1. Job 1 is chosen first, because it has the lowest job number from projects with no running jobs (that is, all projects) -1. We finish job 1 -1. Job 2 is next, because, having finished Job 1, all projects have 0 jobs running again, and 2 is the lowest available job number -1. Job 4 is next, because with Project 1 running a job, 4 is the lowest number from projects running no jobs (Projects 2 and 3) -1. We finish job 4 -1. Job 5 is next, because having finished Job 4, Project 2 has no jobs running again -1. Job 6 is next, because Project 3 is the only project left with no running jobs -1. Lastly we choose Job 3... because, again, it's the only job left (who says 1 is the loneliest number?) +1. Job 1 is chosen first, because it has the lowest job number from projects with no running jobs (that is, all projects). +1. We finish job 1. +1. Job 2 is next, because, having finished Job 1, all projects have 0 jobs running again, and 2 is the lowest available job number. +1. Job 4 is next, because with Project 1 running a job, 4 is the lowest number from projects running no jobs (Projects 2 and 3). +1. We finish job 4. +1. Job 5 is next, because having finished Job 4, Project 2 has no jobs running again. +1. Job 6 is next, because Project 3 is the only project left with no running jobs. +1. Lastly we choose Job 3... because, again, it's the only job left. -## Using shared Runners effectively +#### Enable a shared Runner -If you are planning to use shared Runners, there are several things you -should keep in mind. +By default, all projects can use shared Runners, and they are enabled by default. -### Using tags +However, you can enable or disable shared Runners for individual projects. -You must set up a Runner to be able to run all the different types of jobs -that it may encounter on the projects it's shared over. This would be -problematic for large amounts of projects, if it wasn't for tags. +To enable or disable a shared Runner: -By tagging a Runner for the types of jobs it can handle, you can make sure -shared Runners will [only run the jobs they are equipped to run](../yaml/README.md#tags). +1. Go to the project's **{settings}** **Settings > CI/CD** and expand the **Runners** section. +1. Click **Allow shared Runners** or **Disable shared Runners**. -For instance, at GitLab we have Runners tagged with "rails" if they contain -the appropriate dependencies to run Rails test suites. +### Group Runners -### Allowing Runners with tags to pick jobs without tags +Use *Group Runners* when you want all projects in a group +to have access to a set of Runners. -When you [register a Runner](https://docs.gitlab.com/runner/register/), its default behavior is to **only pick** -[tagged jobs](../yaml/README.md#tags). +Group Runners process jobs by using a first in, first out ([FIFO](https://en.wikipedia.org/wiki/FIFO_(computing_and_electronics))) queue. -NOTE: **Note:** -Owner [permissions](../../user/permissions.md) are required to change the -Runner settings. +#### Create a group Runner -To make a Runner pick untagged jobs: +You can create a group Runner for your self-managed GitLab instance or for GitLab.com. +You must have [Owner permissions](../../user/permissions.md#group-members-permissions) for the group. -1. Visit your project's **Settings > CI/CD > Runners**. -1. Find the Runner you want to pick untagged jobs and make sure it's enabled. -1. Click the pencil button. -1. Check the **Run untagged jobs** option. -1. Click the **Save changes** button for the changes to take effect. +To create a group Runner: + +1. [Install Runner](https://docs.gitlab.com/runner/install/). +1. Go to the group you want to make the Runner work for. +1. Go to **{settings}** **Settings > CI/CD** and expand the **Runners** section. +1. Note the URL and token. +1. [Register the Runner](https://docs.gitlab.com/runner/register/). + +#### Pause or remove a group Runner + +You can pause or remove a group Runner. +You must have [Owner permissions](../../user/permissions.md#group-members-permissions) for the group. + +1. Go to the group you want to remove or pause the Runner for. +1. Go to **{settings}** **Settings > CI/CD** and expand the **Runners** section. +1. Click **Pause** or **Remove Runner**. +1. On the confirmation dialog, click **OK**. + +### Specific Runners + +Use *Specific Runners* when you want to use Runners for specific projects. For example, +when you have: + +- Jobs with specific requirements, like a deploy job that requires credentials. +- Projects with a lot of CI activity that can benefit from being separate from other Runners. + +You can set up a specific Runner to be used by multiple projects. Specific Runners +must be enabled for each project explicitly. + +Specific Runners process jobs by using a first in, first out ([FIFO](https://en.wikipedia.org/wiki/FIFO_(computing_and_electronics))) queue. NOTE: **Note:** -The Runner tags list can not be empty when it's not allowed to pick untagged jobs. +Specific Runners do not get shared with forked projects automatically. +A fork *does* copy the CI / CD settings of the cloned repository. -Below are some example scenarios of different variations. +#### Create a specific Runner -#### Runner runs only tagged jobs +You can create a specific Runner for your self-managed GitLab instance or for GitLab.com. +You must have [Owner permissions](../../user/permissions.md#project-members-permissions) for the project. -The following examples illustrate the potential impact of the Runner being set -to run only tagged jobs. +To create a specific Runner: -Example 1: +1. [Install Runner](https://docs.gitlab.com/runner/install/). +1. Go to the project's **{settings}** **Settings > CI/CD** and expand the **Runners** section. +1. Note the URL and token. +1. [Register the Runner](https://docs.gitlab.com/runner/register/). -1. The Runner is configured to run only tagged jobs and has the `docker` tag. -1. A job that has a `hello` tag is executed and stuck. +#### Enable a specific Runner for a specific project -Example 2: +A specific Runner is available in the project it was created for. An administrator can +enable a specific Runner to apply to additional projects. -1. The Runner is configured to run only tagged jobs and has the `docker` tag. -1. A job that has a `docker` tag is executed and run. +- You must have Owner permissions for the project. +- The specific Runner must not be [locked](#prevent-a-specific-runner-from-being-enabled-for-other-projects). -Example 3: +To enable or disable a specific Runner for a project: -1. The Runner is configured to run only tagged jobs and has the `docker` tag. -1. A job that has no tags defined is executed and stuck. +1. Go to the project's **{settings}** **Settings > CI/CD** and expand the **Runners** section. +1. Click **Enable for this project** or **Disable for this project**. -#### Runner is allowed to run untagged jobs +#### Prevent a specific Runner from being enabled for other projects -The following examples illustrate the potential impact of the Runner being set -to run tagged and untagged jobs. +You can configure a specific Runner so it is "locked" and cannot be enabled for other projects. +This setting can be enabled when you first [register a Runner](https://docs.gitlab.com/runner/register/), +but can also be changed later. -Example 1: +To lock or unlock a Runner: -1. The Runner is configured to run untagged jobs and has the `docker` tag. -1. A job that has no tags defined is executed and run. -1. A second job that has a `docker` tag defined is executed and run. +1. Go to the project's **{settings}** **Settings > CI/CD** and expand the **Runners** section. +1. Find the Runner you want to lock or unlock. Make sure it's enabled. +1. Click the pencil button. +1. Check the **Lock to current projects** option. +1. Click **Save changes**. -Example 2: +## Manually clear the Runner cache -1. The Runner is configured to run untagged jobs and has no tags defined. -1. A job that has no tags defined is executed and run. -1. A second job that has a `docker` tag defined is stuck. +Read [clearing the cache](../caching/index.md#clearing-the-cache). + +## Set maximum job timeout for a Runner -### Setting maximum job timeout for a Runner +For each Runner, you can specify a *maximum job timeout*. This timeout, +if smaller than the [project defined timeout](../pipelines/settings.md#timeout), takes precedence. -For each Runner you can specify a _maximum job timeout_. Such timeout, -if smaller than [project defined timeout](../pipelines/settings.md#timeout), will take the precedence. This -feature can be used to prevent Shared Runner from being appropriated -by a project by setting a ridiculous big timeout (for example, one week). +This feature can be used to prevent your shared Runner from being overwhelmed +by a project that has jobs with a long timeout (for example, one week). -When not configured, Runner will not override project timeout. +When not configured, Runners will not override the project timeout. -How this feature will work: +How this feature works: **Example 1 - Runner timeout bigger than project timeout** @@ -334,7 +229,7 @@ How this feature will work: 1. You start a job 1. The job, if running longer, will be timed out after **30 minutes** -### Be careful with sensitive information +## Be careful with sensitive information With some [Runner Executors](https://docs.gitlab.com/runner/executors/README.html), if you can run a job on the Runner, you can get full access to the file system, @@ -349,6 +244,25 @@ The above is easily avoided by restricting the usage of shared Runners on large public GitLab instances, controlling access to your GitLab instance, and using more secure [Runner Executors](https://docs.gitlab.com/runner/executors/README.html). +### Prevent Runners from revealing sensitive information + +> [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/13194) in GitLab 10.0. + +You can protect Runners so they don't reveal sensitive information. +When a Runner is protected, the Runner picks jobs created on +[protected branches](../../user/project/protected_branches.md) or [protected tags](../../user/project/protected_tags.md) only, +and ignores other jobs. + +To protect or unprotect a Runner: + +1. Go to the project's **{settings}** **Settings > CI/CD** and expand the **Runners** section. +1. Find the Runner you want to protect or unprotect. Make sure it's enabled. +1. Click the pencil button. +1. Check the **Protected** option. +1. Click **Save changes**. + + + ### Forks Whenever a project is forked, it copies the settings of the jobs that relate @@ -356,23 +270,21 @@ to it. This means that if you have shared Runners set up for a project and someone forks that project, the shared Runners will also serve jobs of this project. -## Attack vectors in Runners +### Attack vectors in Runners Mentioned briefly earlier, but the following things of Runners can be exploited. We're always looking for contributions that can mitigate these [Security Considerations](https://docs.gitlab.com/runner/security/). -### Resetting the registration token for a Project +### Reset the Runner registration token for a project -If you think that registration token for a Project was revealed, you should -reset them. It's recommended because such a token can be used to register another -Runner to the Project. It may then be used to obtain the values of secret -variables or clone the project code, that normally may be unavailable for the -attacker. +If you think that a registration token for a project was revealed, you should +reset it. A token can be used to register another Runner for the project. That new Runner +may then be used to obtain the values of secret variables or to clone project code. To reset the token: -1. Go to **Settings > CI/CD** for a specified Project. +1. Go to the project's **{settings}** **Settings > CI/CD**. 1. Expand the **General pipelines settings** section. 1. Find the **Runner token** form field and click the **Reveal value** button. 1. Delete the value and save the form. @@ -384,7 +296,7 @@ any new Runners to the project. If you are using any tools to provision and register new Runners, the tokens used in those tools should be updated to reflect the value of the new token. -## Determining the IP address of a Runner +## Determine the IP address of a Runner > [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/17286) in GitLab 10.6. @@ -397,22 +309,88 @@ automatically updated in GitLab. The IP address for shared Runners and specific Runners can be found in different places. -### Shared Runners +### Determine the IP address of a shared Runner To view the IP address of a shared Runner you must have admin access to the GitLab instance. To determine this: -1. Visit **Admin Area > Overview > Runners** -1. Look for the Runner in the table and you should see a column for "IP Address" +1. Visit **{admin}** **Admin Area > Overview > Runners**. +1. Look for the Runner in the table and you should see a column for **IP Address**.  -### Specific Runners +### Determine the IP address of a specific Runner -You can find the IP address of a Runner for a specific project by: +To can find the IP address of a Runner for a specific project, +you must have Owner [permissions](../../user/permissions.md#project-members-permissions) for the project. -1. Visit your project's **Settings > CI/CD** -1. Find the Runner and click on it's ID which links you to the details page -1. On the details page you should see a row for "IP Address" +1. Go to the project's **{settings}** **Settings > CI/CD** and expand the **Runners** section. +1. On the details page you should see a row for **IP Address**.  + +## Use tags to limit the number of jobs using the Runner + +You must set up a Runner to be able to run all the different types of jobs +that it may encounter on the projects it's shared over. This would be +problematic for large amounts of projects, if it weren't for tags. + +By tagging a Runner for the types of jobs it can handle, you can make sure +shared Runners will [only run the jobs they are equipped to run](../yaml/README.md#tags). + +For instance, at GitLab we have Runners tagged with `rails` if they contain +the appropriate dependencies to run Rails test suites. + +When you [register a Runner](https://docs.gitlab.com/runner/register/), its default behavior is to **only pick** +[tagged jobs](../yaml/README.md#tags). +To change this, you must have Owner [permissions](../../user/permissions.md#project-members-permissions) for the project. + +To make a Runner pick untagged jobs: + +1. Go to the project's **{settings}** **Settings > CI/CD** and expand the **Runners** section. +1. Find the Runner you want to pick untagged jobs and make sure it's enabled. +1. Click the pencil button. +1. Check the **Run untagged jobs** option. +1. Click the **Save changes** button for the changes to take effect. + +NOTE: **Note:** +The Runner tags list can not be empty when it's not allowed to pick untagged jobs. + +Below are some example scenarios of different variations. + +### Runner runs only tagged jobs + +The following examples illustrate the potential impact of the Runner being set +to run only tagged jobs. + +Example 1: + +1. The Runner is configured to run only tagged jobs and has the `docker` tag. +1. A job that has a `hello` tag is executed and stuck. + +Example 2: + +1. The Runner is configured to run only tagged jobs and has the `docker` tag. +1. A job that has a `docker` tag is executed and run. + +Example 3: + +1. The Runner is configured to run only tagged jobs and has the `docker` tag. +1. A job that has no tags defined is executed and stuck. + +### Runner is allowed to run untagged jobs + +The following examples illustrate the potential impact of the Runner being set +to run tagged and untagged jobs. + +Example 1: + +1. The Runner is configured to run untagged jobs and has the `docker` tag. +1. A job that has no tags defined is executed and run. +1. A second job that has a `docker` tag defined is executed and run. + +Example 2: + +1. The Runner is configured to run untagged jobs and has no tags defined. +1. A job that has no tags defined is executed and run. +1. A second job that has a `docker` tag defined is stuck. diff --git a/doc/ci/services/README.md b/doc/ci/services/README.md index 7fe12eb53e7..fda453b7aa0 100644 --- a/doc/ci/services/README.md +++ b/doc/ci/services/README.md @@ -1,4 +1,7 @@ --- +stage: Verify +group: Runner +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers comments: false type: index --- diff --git a/doc/ci/services/mysql.md b/doc/ci/services/mysql.md index dcfd863709e..1b017111d22 100644 --- a/doc/ci/services/mysql.md +++ b/doc/ci/services/mysql.md @@ -1,4 +1,7 @@ --- +stage: Verify +group: Runner +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers type: reference --- @@ -43,7 +46,7 @@ Database: <your_mysql_database> If you are wondering why we used `mysql` for the `Host`, read more at [How services are linked to the job](../docker/using_docker_images.md#how-services-are-linked-to-the-job). -You can also use any other docker image available on [Docker Hub](https://hub.docker.com/_/mysql/). +You can also use any other Docker image available on [Docker Hub](https://hub.docker.com/_/mysql/). For example, to use MySQL 5.5 the service becomes `mysql:5.5`. The `mysql` image can accept some environment variables. For more details diff --git a/doc/ci/services/postgres.md b/doc/ci/services/postgres.md index 2f92bd969ff..aadbce5a50a 100644 --- a/doc/ci/services/postgres.md +++ b/doc/ci/services/postgres.md @@ -1,4 +1,7 @@ --- +stage: Verify +group: Runner +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers type: reference --- @@ -45,7 +48,7 @@ Database: nice_marmot If you are wondering why we used `postgres` for the `Host`, read more at [How services are linked to the job](../docker/using_docker_images.md#how-services-are-linked-to-the-job). -You can also use any other docker image available on [Docker Hub](https://hub.docker.com/_/postgres). +You can also use any other Docker image available on [Docker Hub](https://hub.docker.com/_/postgres). For example, to use PostgreSQL 9.3 the service becomes `postgres:9.3`. The `postgres` image can accept some environment variables. For more details diff --git a/doc/ci/services/redis.md b/doc/ci/services/redis.md index f22ee87a9d3..193f78f240b 100644 --- a/doc/ci/services/redis.md +++ b/doc/ci/services/redis.md @@ -1,4 +1,7 @@ --- +stage: Verify +group: Runner +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers type: reference --- @@ -30,7 +33,7 @@ Host: redis And that's it. Redis will now be available to be used within your testing framework. -You can also use any other docker image available on [Docker Hub](https://hub.docker.com/_/redis). +You can also use any other Docker image available on [Docker Hub](https://hub.docker.com/_/redis). For example, to use Redis 2.8 the service becomes `redis:2.8`. ## Use Redis with the Shell executor diff --git a/doc/ci/ssh_keys/README.md b/doc/ci/ssh_keys/README.md index 16e87e92a07..4ad4758b281 100644 --- a/doc/ci/ssh_keys/README.md +++ b/doc/ci/ssh_keys/README.md @@ -1,4 +1,7 @@ --- +stage: Verify +group: Continuous Integration +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers last_updated: 2017-12-13 type: tutorial --- diff --git a/doc/ci/triggers/README.md b/doc/ci/triggers/README.md index db3d2352f09..c6ac87ef888 100644 --- a/doc/ci/triggers/README.md +++ b/doc/ci/triggers/README.md @@ -1,4 +1,7 @@ --- +stage: Verify +group: Continuous Integration +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers type: tutorial --- @@ -20,7 +23,7 @@ The following methods of authentication are supported: - [Trigger token](#trigger-token) - [CI job token](#ci-job-token) -If using the `$CI_PIPELINE_SOURCE` [predefined environment variable](../variables/predefined_variables.md#variables-reference) +If using the `$CI_PIPELINE_SOURCE` [predefined environment variable](../variables/predefined_variables.md) to limit which jobs run in a pipeline, the value could be either `pipeline` or `trigger`, depending on which trigger method is used. @@ -49,7 +52,7 @@ with the [GitLab Container Registry](../../user/packages/container_registry/inde #### When used with multi-project pipelines > - Use of `CI_JOB_TOKEN` for multi-project pipelines was [introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/2017) in [GitLab Premium](https://about.gitlab.com/pricing/) 9.3. -> - Use of `CI_JOB_TOKEN` for multi-project pipelines was [made available](https://gitlab.com/gitlab-org/gitlab/issues/31573) in all tiers in GitLab 12.4. +> - Use of `CI_JOB_TOKEN` for multi-project pipelines was [made available](https://gitlab.com/gitlab-org/gitlab/-/issues/31573) in all tiers in GitLab 12.4. This way of triggering can only be used when invoked inside `.gitlab-ci.yml`, and it creates a dependent pipeline relation visible on the diff --git a/doc/ci/variables/README.md b/doc/ci/variables/README.md index 3e31a2169e2..541e739082f 100644 --- a/doc/ci/variables/README.md +++ b/doc/ci/variables/README.md @@ -1,4 +1,7 @@ --- +stage: Verify +group: Continuous Integration +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers type: reference --- @@ -17,6 +20,13 @@ that can be reused in different scripts. Variables are useful for customizing your jobs in GitLab CI/CD. When you use variables, you don't have to hard-code values. +> For more information about advanced use of GitLab CI/CD: +> +> - <i class="fa fa-youtube-play youtube" aria-hidden="true"></i> Get to productivity faster with these [7 advanced GitLab CI workflow hacks](https://about.gitlab.com/webcast/7cicd-hacks/) +> shared by GitLab engineers. +> - <i class="fa fa-youtube-play youtube" aria-hidden="true"></i> Learn how the Cloud Native Computing Foundation (CNCF) [eliminates the complexity](https://about.gitlab.com/customers/cncf/) +> of managing projects across many cloud providers with GitLab CI/CD. + ## Predefined environment variables GitLab CI/CD has a [default set of predefined variables](predefined_variables.md) @@ -106,7 +116,7 @@ From within the UI, you can add or update custom environment variables: 1. Go to your project's **Settings > CI/CD** and expand the **Variables** section. 1. Click the **Add Variable** button. In the **Add variable** modal, fill in the details: - - **Key**: Must be one line, with no spaces, using only letters, numbers, `-` or `_`. + - **Key**: Must be one line, with no spaces, using only letters, numbers, or `_`. - **Value**: No limitations. - **Type**: `File` or `Variable`. - **Environment scope**: `All`, or specific environments. @@ -133,7 +143,7 @@ The output will be: ### Custom environment variables of type Variable -> [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/issues/46806) in GitLab 11.11. +> [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/46806) in GitLab 11.11. For variables with the type **Variable**, the Runner creates an environment variable that uses the key for the name and the value for the value. @@ -143,13 +153,13 @@ which may be further validated. They appear when you add or update a variable in ### Custom environment variables of type File -> [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/issues/46806) in GitLab 11.11. +> [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/46806) in GitLab 11.11. For variables with the type **File**, the Runner creates an environment variable that uses the key for the name. For the value, the Runner writes the variable value to a temporary file and uses this path. You can use tools like [the AWS CLI](https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-envvars.html) -and [kubectl](https://kubernetes.io/docs/concepts/configuration/organize-cluster-access-kubeconfig/#the-kubeconfig-environment-variable) +and [`kubectl`](https://kubernetes.io/docs/concepts/configuration/organize-cluster-access-kubeconfig/#the-kubeconfig-environment-variable) to customize your configuration by using **File** type variables. In the past, a common pattern was to read the value of a CI variable, save it in a file, and then @@ -175,7 +185,7 @@ kubectl config set-cluster e2e --server="$KUBE_URL" --certificate-authority="$KU ### Mask a custom variable -> [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/issues/13784) in GitLab 11.10 +> [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/13784) in GitLab 11.10 Variables can be masked so that the value of the variable will be hidden in job logs. @@ -195,7 +205,7 @@ The value of the variable must: - Be at least 8 characters long. - Not be a predefined or custom environment variable. - Consist only of characters from the Base64 alphabet (RFC4648). - [In GitLab 12.2](https://gitlab.com/gitlab-org/gitlab-foss/issues/63043) + [In GitLab 12.2](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/63043) and newer, `@` and `:` are also valid values. You can't mask variables that don't meet these requirements. @@ -242,13 +252,15 @@ In most cases `bash` or `sh` is used to execute the job script. To access environment variables, use the syntax for your Runner's [shell](https://docs.gitlab.com/runner/executors/). -| Shell | Usage | -|----------------------|-----------------| -| bash/sh | `$variable` | -| windows batch | `%variable%` | -| PowerShell | `$env:variable` | +| Shell | Usage | +|----------------------|------------------------------------------| +| bash/sh | `$variable` | +| PowerShell | `$env:variable` (primary) or `$variable` | +| Windows Batch | `%variable%` | + +### Bash -To access environment variables in bash, prefix the variable name with (`$`): +To access environment variables in **bash**, prefix the variable name with (`$`): ```yaml job_name: @@ -256,32 +268,54 @@ job_name: - echo $CI_JOB_ID ``` -To access environment variables in **Windows Batch**, surround the variable -with (`%`): +### PowerShell + +To access environment variables in a **Windows PowerShell** environment, prefix +the variable name with (`$env:`). For environment variables set by GitLab CI, including those set by [`variables`](https://gitlab.com/gitlab-org/gitlab/blob/master/doc/ci/yaml/README.md#variables) +parameter, they can also be accessed by prefixing the variable name with (`$`) +as of [GitLab Runner 1.0.0](https://gitlab.com/gitlab-org/gitlab-runner/-/commit/abc44bb158008cd3a49c0d8173717c38dadb29ae#47afd7e8f12afdb8f0246262488f24e6dd071a22). +System set environment variables however must be accessed using (`$env:`). ```yaml job_name: script: - - echo %CI_JOB_ID% + - echo $env:CI_JOB_ID + - echo $CI_JOB_ID + - echo $env:PATH ``` -To access environment variables in a **Windows PowerShell** environment, prefix -the variable name with (`$env:`): +In [some cases](https://gitlab.com/gitlab-org/gitlab-runner/-/issues/4115#note_157692820) +environment variables may need to be surrounded by quotes to expand properly: ```yaml job_name: script: - - echo $env:CI_JOB_ID + - D:\\qislsf\\apache-ant-1.10.5\\bin\\ant.bat "-DsosposDailyUsr=$env:SOSPOS_DAILY_USR" portal_test ``` -You can also list all environment variables with the `export` command, -but be aware that this will also expose the values of all the variables +### Windows Batch + +To access environment variables in **Windows Batch**, surround the variable +with (`%`): + +```yaml +job_name: + script: + - echo %CI_JOB_ID% +``` + +### List all environment variables + +You can also list all environment variables with the `export` command in Bash +or `dir env:` command in PowerShell. +Be aware that this will also expose the values of all the variables you set, in the job log: ```yaml job_name: script: - export + # - 'dir env:' # use this for PowerShell ``` Example values: @@ -377,9 +411,8 @@ script: You can define per-project or per-group variables that are set in the pipeline environment. Group-level variables are stored out of -the repository (not in `.gitlab-ci.yml`) and are securely passed to GitLab Runner -making them available during a pipeline run. It's the **recommended method** to -use for storing things like passwords, SSH keys, and credentials. +the repository (not in `.gitlab-ci.yml`) and are securely passed to GitLab Runner, +which makes them available during a pipeline run. For Premium users who do **not** use an external key store or who use GitLab's [integration with HashiCorp Vault](../examples/authenticating-with-hashicorp-vault/index.md), we recommend using group environment variables to store secrets like passwords, SSH keys, and credentials. Group-level variables can be added by: @@ -394,10 +427,55 @@ Once you set them, they will be available for all subsequent pipelines. Any grou  -### Inherit environment variables +## Instance-level CI/CD environment variables + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/14108) in GitLab 13.0. + +Instance variables are useful for no longer needing to manually enter the same credentials repeatedly for all your projects. Instance-level variables are available to all projects and groups on the instance. -> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/22638) in GitLab 13.0. -> - It's deployed behind a feature flag (`ci_dependency_variables`), disabled by default. +NOTE: **Note:** +The maximum number of instance-level variables is [planned to be 25](https://gitlab.com/gitlab-org/gitlab/-/issues/216097). + +You can define instance-level variables via the UI or [API](../../api/instance_level_ci_variables.md). + +To add an instance-level variable: + +1. Navigate to your admin area's **Settings > CI/CD** and expand the **Variables** section. +1. Click the **Add variable** button, and fill in the details: + + - **Key**: Must be one line, using only letters, numbers, or `_` (underscore), with no spaces. + - **Value**: 700 characters allowed. + - **Type**: `File` or `Variable`. + - **Protect variable** (Optional): If selected, the variable will only be available in pipelines that run on protected branches or tags. + - **Mask variable** (Optional): If selected, the variable's **Value** will not be shown in job logs. The variable will not be saved if the value does not meet the [masking requirements](#masked-variable-requirements). + +After a variable is created, you can update any of the details by clicking the **{pencil}** **Edit** button. + +### Enable or disable UI interface for instance-level CI/CD variables + +The UI interface for Instance-level CI/CD variables is under development but ready for production use. +It is deployed behind a feature flag that is **enabled by default**. +[GitLab administrators with access to the GitLab Rails console](../../administration/feature_flags.md) can opt to disable it for your instance. + +NOTE: **Note:** +This feature will not work if the [instance-level CI/CD variables API feature flag is disabled](../../api/instance_level_ci_variables.md#enable-or-disable-instance-level-cicd-variables-core-only). + +To disable it: + +```ruby +Feature.disable(:instance_variables_ui) +``` + +To enable it: + +```ruby +Feature.enable(:instance_variables_ui) +``` + +## Inherit environment variables + +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/22638) in GitLab 13.0 behind a disabled [feature flag](../../administration/feature_flags.md): `ci_dependency_variables`. +> - [Feature flag removed](https://gitlab.com/gitlab-org/gitlab/-/issues/217834) in GitLab 13.1. You can inherit environment variables from dependent jobs. @@ -442,25 +520,6 @@ deploy: artifacts: true ``` -### Enable inherited environment variables **(CORE ONLY)** - -The Inherited Environment Variables feature is under development and not ready for production use. It is -deployed behind a feature flag that is **disabled by default**. -[GitLab administrators with access to the GitLab Rails console](../../administration/feature_flags.md) -can enable it for your instance. - -To enable it: - -```ruby -Feature.enable(:ci_dependency_variables) -``` - -To disable it: - -```ruby -Feature.disable(:ci_dependency_variables) -``` - ## Priority of environment variables Variables of different types can take precedence over other @@ -468,7 +527,8 @@ variables, depending on where they are defined. The order of precedence for variables is (from highest to lowest): -1. [Trigger variables](../triggers/README.md#making-use-of-trigger-variables) or [scheduled pipeline variables](../pipelines/schedules.md#using-variables). +1. [Trigger variables](../triggers/README.md#making-use-of-trigger-variables), [scheduled pipeline variables](../pipelines/schedules.md#using-variables), + and [manual pipeline run variables](#override-a-variable-by-manually-running-a-pipeline). 1. Project-level [variables](#custom-environment-variables) or [protected variables](#protect-a-custom-variable). 1. Group-level [variables](#group-level-environment-variables) or [protected variables](#protect-a-custom-variable). 1. [Inherited environment variables](#inherit-environment-variables). @@ -487,8 +547,10 @@ variables take precedence over those defined in `.gitlab-ci.yml`. ## Unsupported variables -There are cases where some variables cannot be used in the context of a -`.gitlab-ci.yml` definition (for example under `script`). Read more about which variables are [not supported](where_variables_can_be_used.md). +Variable names are limited by the underlying shell used to execute scripts (see [available shells](https://docs.gitlab.com/runner/shells/index.html). +Each shell has its own unique set of reserved variable names. +You will also want to keep in mind the [scope of environment variables](where_variables_can_be_used.md) to ensure a variable is defined in the scope +in which you wish to use it. ## Where variables can be used @@ -518,7 +580,7 @@ An example integration that defines deployment variables is the ### Auto DevOps environment variables -> [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/issues/49056) in GitLab 11.7. +> [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/49056) in GitLab 11.7. You can configure [Auto DevOps](../../topics/autodevops/index.md) to pass CI variables to the running application by prefixing the key of the @@ -530,12 +592,12 @@ then be available as environment variables on the running application container. CAUTION: **Caution:** -Variables with multiline values are not currently supported due to +Variables with multi-line values are not currently supported due to limitations with the current Auto DevOps scripting environment. ### Override a variable by manually running a pipeline -> [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/issues/44059) in GitLab 10.8. +> [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/44059) in GitLab 10.8. You can override the value of a current variable by [running a pipeline manually](../pipelines/index.md#run-a-pipeline-manually). @@ -555,8 +617,8 @@ value for this specific pipeline. ## Environment variables expressions -> - [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/issues/37397) in GitLab 10.7 for [the `only` and `except` CI keywords](../yaml/README.md#onlyexcept-advanced) -> - [Expanded](https://gitlab.com/gitlab-org/gitlab/issues/27863) in GitLab 12.3 with [the `rules` keyword](../yaml/README.md#rules) +> - [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/37397) in GitLab 10.7 for [the `only` and `except` CI keywords](../yaml/README.md#onlyexcept-advanced) +> - [Expanded](https://gitlab.com/gitlab-org/gitlab/-/issues/27863) in GitLab 12.3 with [the `rules` keyword](../yaml/README.md#rules) Use variable expressions to limit which jobs are created within a pipeline after changes are pushed to GitLab. @@ -612,8 +674,8 @@ Below you can find supported syntax reference: It sometimes happens that you want to check whether a variable is defined or not. To do that, you can compare a variable to `null` keyword, like - `$VARIABLE == null`. This expression is going to evaluate to truth if - variable is not defined when `==` is used, or to falsey if `!=` is used. + `$VARIABLE == null`. This expression evaluates to true if + variable is not defined when `==` is used, or to false if `!=` is used. 1. Checking for an empty variable @@ -644,7 +706,7 @@ Below you can find supported syntax reference: which means that it is defined and non-empty, you can simply use variable name as an expression, like `$STAGING`. If `$STAGING` variable is defined, and is non empty, expression will evaluate to truth. - `$STAGING` value needs to a string, with length higher than zero. + `$STAGING` value needs to be a string, with length higher than zero. Variable that contains only whitespace characters is not an empty variable. 1. Pattern matching (introduced in GitLab 11.0) @@ -652,7 +714,7 @@ Below you can find supported syntax reference: Examples: - `=~`: True if pattern is matched. Ex: `$VARIABLE =~ /^content.*/` - - `!~`: True if pattern is not matched. Ex: `$VARIABLE_1 !~ /^content.*/` ([Introduced](https://gitlab.com/gitlab-org/gitlab-foss/issues/61900) in GitLab 11.11) + - `!~`: True if pattern is not matched. Ex: `$VARIABLE_1 !~ /^content.*/` ([Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/61900) in GitLab 11.11) Variable pattern matching with regular expressions uses the [RE2 regular expression syntax](https://github.com/google/re2/wiki/Syntax). @@ -694,7 +756,7 @@ deploy_staging: ``` NOTE: **Note:** -The available regular expression syntax is limited. See [related issue](https://gitlab.com/gitlab-org/gitlab/issues/35438) +The available regular expression syntax is limited. See [related issue](https://gitlab.com/gitlab-org/gitlab/-/issues/35438) for more details. If needed, you can use a test pipeline to determine whether a regular expression will diff --git a/doc/ci/variables/deprecated_variables.md b/doc/ci/variables/deprecated_variables.md index 543da481938..94ec8439605 100644 --- a/doc/ci/variables/deprecated_variables.md +++ b/doc/ci/variables/deprecated_variables.md @@ -1,4 +1,7 @@ --- +stage: Verify +group: Continuous Integration +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers type: reference --- diff --git a/doc/ci/variables/img/ci_job_stage_output_example.png b/doc/ci/variables/img/ci_job_stage_output_example.png Binary files differindex e333da57121..2344b19d9b3 100644 --- a/doc/ci/variables/img/ci_job_stage_output_example.png +++ b/doc/ci/variables/img/ci_job_stage_output_example.png diff --git a/doc/ci/variables/img/inherited_group_variables_v12_5.png b/doc/ci/variables/img/inherited_group_variables_v12_5.png Binary files differindex a13ba711083..cce024cfab8 100644 --- a/doc/ci/variables/img/inherited_group_variables_v12_5.png +++ b/doc/ci/variables/img/inherited_group_variables_v12_5.png diff --git a/doc/ci/variables/img/override_value_via_manual_pipeline_output.png b/doc/ci/variables/img/override_value_via_manual_pipeline_output.png Binary files differindex 8a13bb3849e..2d4c4d24520 100644 --- a/doc/ci/variables/img/override_value_via_manual_pipeline_output.png +++ b/doc/ci/variables/img/override_value_via_manual_pipeline_output.png diff --git a/doc/ci/variables/img/override_variable_manual_pipeline.png b/doc/ci/variables/img/override_variable_manual_pipeline.png Binary files differindex de768105aec..2b242466297 100644 --- a/doc/ci/variables/img/override_variable_manual_pipeline.png +++ b/doc/ci/variables/img/override_variable_manual_pipeline.png diff --git a/doc/ci/variables/predefined_variables.md b/doc/ci/variables/predefined_variables.md index d4d3a13bb2a..8463bbeb582 100644 --- a/doc/ci/variables/predefined_variables.md +++ b/doc/ci/variables/predefined_variables.md @@ -1,4 +1,7 @@ --- +stage: Verify +group: Continuous Integration +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers type: reference --- @@ -7,8 +10,6 @@ type: reference For an introduction on this subject, read through the [getting started with environment variables](README.md) document. -## Overview - Some of the predefined environment variables are available only if a minimum version of [GitLab Runner](https://docs.gitlab.com/runner/) is used. Consult the table below to find the version of Runner required. @@ -19,7 +20,8 @@ Starting with GitLab 9.0, we have deprecated some variables. Read the strongly advised to use the new variables as we will remove the old ones in future GitLab releases.** -## Variables reference +You can add a command to your `.gitlab-ci.yml` file to +[output the values of all variables available for a job](README.md#list-all-environment-variables). | Variable | GitLab | Runner | Description | |-----------------------------------------------|--------|--------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| @@ -42,7 +44,7 @@ future GitLab releases.** | `CI_COMMIT_TITLE` | 10.8 | all | The title of the commit - the full first line of the message | | `CI_CONCURRENT_ID` | all | 11.10 | Unique ID of build execution within a single executor. | | `CI_CONCURRENT_PROJECT_ID` | all | 11.10 | Unique ID of build execution within a single executor and project. | -| `CI_CONFIG_PATH` | 9.4 | 0.5 | The path to CI config file. Defaults to `.gitlab-ci.yml` | +| `CI_CONFIG_PATH` | 9.4 | 0.5 | The path to CI configuration file. Defaults to `.gitlab-ci.yml` | | `CI_DEBUG_TRACE` | all | 1.7 | Whether [debug logging (tracing)](README.md#debug-logging) is enabled | | `CI_DEFAULT_BRANCH` | 12.4 | all | The name of the default branch for the project. | | `CI_DEPLOY_PASSWORD` | 10.8 | all | Authentication password of the [GitLab Deploy Token](../../user/project/deploy_tokens/index.md#gitlab-deploy-token), only present if the Project has one related. | @@ -56,18 +58,17 @@ future GitLab releases.** | `CI_EXTERNAL_PULL_REQUEST_SOURCE_BRANCH_SHA` | 12.3 | all | The HEAD SHA of the source branch of the pull request if [the pipelines are for external pull requests](../ci_cd_for_external_repos/index.md#pipelines-for-external-pull-requests). Available only if `only: [external_pull_requests]` or [`rules`](../yaml/README.md#rules) syntax is used and the pull request is open. | | `CI_EXTERNAL_PULL_REQUEST_TARGET_BRANCH_NAME` | 12.3 | all | The target branch name of the pull request if [the pipelines are for external pull requests](../ci_cd_for_external_repos/index.md#pipelines-for-external-pull-requests). Available only if `only: [external_pull_requests]` or [`rules`](../yaml/README.md#rules) syntax is used and the pull request is open. | | `CI_EXTERNAL_PULL_REQUEST_TARGET_BRANCH_SHA` | 12.3 | all | The HEAD SHA of the target branch of the pull request if [the pipelines are for external pull requests](../ci_cd_for_external_repos/index.md#pipelines-for-external-pull-requests). Available only if `only: [external_pull_requests]` or [`rules`](../yaml/README.md#rules) syntax is used and the pull request is open. | +| `CI_HAS_OPEN_REQUIREMENTS` | 13.1 | all | Included with the value `true` only if the pipeline's project has any open [requirements](../../user/project/requirements/index.md). Not included if there are no open requirements for the pipeline's project. | | `CI_JOB_ID` | 9.0 | all | The unique ID of the current job that GitLab CI/CD uses internally | | `CI_JOB_IMAGE` | 12.9 | 12.9 | The name of the image running the CI job | | `CI_JOB_MANUAL` | 8.12 | all | The flag to indicate that job was manually started | | `CI_JOB_NAME` | 9.0 | 0.5 | The name of the job as defined in `.gitlab-ci.yml` | | `CI_JOB_STAGE` | 9.0 | 0.5 | The name of the stage as defined in `.gitlab-ci.yml` | -| `CI_JOB_TOKEN` | 9.0 | 1.2 | Token used for authenticating with the [GitLab Container Registry](../../user/packages/container_registry/index.md) and downloading [dependent repositories](../../user/project/new_ci_build_permissions_model.md#dependent-repositories) | +| `CI_JOB_TOKEN` | 9.0 | 1.2 | Token used for authenticating with the [GitLab Container Registry](../../user/packages/container_registry/index.md), downloading [dependent repositories](../../user/project/new_ci_build_permissions_model.md#dependent-repositories), and accessing [GitLab-managed Terraform state](../../user/infrastructure/index.md#gitlab-managed-terraform-state). | | `CI_JOB_JWT` | 12.10 | all | RS256 JSON web token that can be used for authenticating with third party systems that support JWT authentication, for example [HashiCorp's Vault](../examples/authenticating-with-hashicorp-vault). | | `CI_JOB_URL` | 11.1 | 0.5 | Job details URL | -| `CI_KUBERNETES_ACTIVE` | 13.0 | all | Included with the value `true` only if the pipeline has a Kubernetes cluster available for deployments. Not included if no cluster is availble. Can be used as an alternative to [`only:kubernetes`/`except:kubernetes`](../yaml/README.md#onlykubernetesexceptkubernetes) with [`rules:if`](../yaml/README.md#rulesif) | +| `CI_KUBERNETES_ACTIVE` | 13.0 | all | Included with the value `true` only if the pipeline has a Kubernetes cluster available for deployments. Not included if no cluster is available. Can be used as an alternative to [`only:kubernetes`/`except:kubernetes`](../yaml/README.md#onlykubernetesexceptkubernetes) with [`rules:if`](../yaml/README.md#rulesif) | | `CI_MERGE_REQUEST_ASSIGNEES` | 11.9 | all | Comma-separated list of username(s) of assignee(s) for the merge request if [the pipelines are for merge requests](../merge_request_pipelines/index.md). Available only if `only: [merge_requests]` or [`rules`](../yaml/README.md#rules) syntax is used and the merge request is created. | -| `CI_MERGE_REQUEST_CHANGED_PAGE_PATHS` | 12.9 | all | Comma-separated list of paths of changed pages in a deployed [Review App](../review_apps/index.md) for a [Merge Request](../merge_request_pipelines/index.md). A [Route Map](../review_apps/index.md#route-maps) must be configured. | -| `CI_MERGE_REQUEST_CHANGED_PAGE_URLS` | 12.9 | all | Comma-separated list of URLs of changed pages in a deployed [Review App](../review_apps/index.md) for a [Merge Request](../merge_request_pipelines/index.md). A [Route Map](../review_apps/index.md#route-maps) must be configured. | | `CI_MERGE_REQUEST_ID` | 11.6 | all | The ID of the merge request if [the pipelines are for merge requests](../merge_request_pipelines/index.md). Available only if `only: [merge_requests]` or [`rules`](../yaml/README.md#rules) syntax is used and the merge request is created. | | `CI_MERGE_REQUEST_IID` | 11.6 | all | The IID of the merge request if [the pipelines are for merge requests](../merge_request_pipelines/index.md). Available only if `only: [merge_requests]` or [`rules`](../yaml/README.md#rules) syntax is used and the merge request is created. | | `CI_MERGE_REQUEST_LABELS` | 11.9 | all | Comma-separated label names of the merge request if [the pipelines are for merge requests](../merge_request_pipelines/index.md). Available only if `only: [merge_requests]` or [`rules`](../yaml/README.md#rules) syntax is used and the merge request is created. | @@ -97,7 +98,7 @@ future GitLab releases.** | `CI_PROJECT_DIR` | all | all | The full path where the repository is cloned and where the job is run. If the GitLab Runner `builds_dir` parameter is set, this variable is set relative to the value of `builds_dir`. For more information, see [Advanced configuration](https://docs.gitlab.com/runner/configuration/advanced-configuration.html#the-runners-section) for GitLab Runner. | | `CI_PROJECT_ID` | all | all | The unique ID of the current project that GitLab CI/CD uses internally | | `CI_PROJECT_NAME` | 8.10 | 0.5 | The name of the directory for the project that is currently being built. For example, if the project URL is `gitlab.example.com/group-name/project-1`, the `CI_PROJECT_NAME` would be `project-1`. | -| `CI_PROJECT_NAMESPACE` | 8.10 | 0.5 | The project namespace (username or groupname) that is currently being built | +| `CI_PROJECT_NAMESPACE` | 8.10 | 0.5 | The project namespace (username or group name) that is currently being built | | `CI_PROJECT_PATH` | 8.10 | 0.5 | The namespace with project name | | `CI_PROJECT_PATH_SLUG` | 9.3 | all | `$CI_PROJECT_PATH` lowercased and with everything except `0-9` and `a-z` replaced with `-`. Use in URLs and domain names. | | `CI_PROJECT_REPOSITORY_LANGUAGES` | 12.3 | all | Comma-separated, lowercased list of the languages used in the repository (e.g. `ruby,javascript,html,css`) | diff --git a/doc/ci/variables/where_variables_can_be_used.md b/doc/ci/variables/where_variables_can_be_used.md index 0c40ac960e5..2f26fddc808 100644 --- a/doc/ci/variables/where_variables_can_be_used.md +++ b/doc/ci/variables/where_variables_can_be_used.md @@ -1,4 +1,7 @@ --- +stage: Verify +group: Continuous Integration +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers type: reference --- @@ -72,10 +75,10 @@ ordering of variables definitions. ### Execution shell environment This is an expansion that takes place during the `script` execution. -How it works depends on the used shell (bash/sh/cmd/PowerShell). For example, if the job's +How it works depends on the used shell (`bash`, `sh`, `cmd`, PowerShell). For example, if the job's `script` contains a line `echo $MY_VARIABLE-${MY_VARIABLE_2}`, it should be properly handled by bash/sh (leaving empty strings or some values depending whether the variables were -defined or not), but will not work with Windows' cmd/PowerShell, since these shells +defined or not), but will not work with Windows' `cmd` or PowerShell, since these shells are using a different variables syntax. Supported: diff --git a/doc/ci/yaml/README.md b/doc/ci/yaml/README.md index c40580cbfb7..7ed5a8fec01 100644 --- a/doc/ci/yaml/README.md +++ b/doc/ci/yaml/README.md @@ -1,4 +1,7 @@ --- +stage: Verify +group: Continuous Integration +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers type: reference --- @@ -22,6 +25,14 @@ We have complete examples of configuring pipelines: - For a collection of examples, see [GitLab CI/CD Examples](../examples/README.md). - To see a large `.gitlab-ci.yml` file used in an enterprise, see the [`.gitlab-ci.yml` file for `gitlab`](https://gitlab.com/gitlab-org/gitlab/blob/master/.gitlab-ci.yml). +> For some additional information about GitLab CI/CD: +> +> - <i class="fa fa-youtube-play youtube" aria-hidden="true"></i> Watch the [CI/CD Ease of configuration](https://www.youtube.com/embed/opdLqwz6tcE) video. +> - Watch the [Making the case for CI/CD in your organization](https://about.gitlab.com/compare/github-actions-alternative/) +> webcast to learn the benefits of CI/CD and how to measure the results of CI/CD automation. +> - <i class="fa fa-youtube-play youtube" aria-hidden="true"></i> Learn how [Verizon reduced rebuilds](https://about.gitlab.com/blog/2019/02/14/verizon-customer-story/) +> from 30 days to under 8 hours with GitLab. + NOTE: **Note:** If you have a [mirrored repository where GitLab pulls from](../../user/project/repository/repository_mirroring.md#pulling-from-a-remote-repository-starter), you may need to enable pipeline triggering in your project's @@ -93,8 +104,8 @@ The following table lists available parameters for jobs: | Keyword | Description | |:---------------------------------------------------|:------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | [`script`](#script) | Shell script which is executed by Runner. | -| [`image`](#image) | Use docker images. Also available: `image:name` and `image:entrypoint`. | -| [`services`](#services) | Use docker services images. Also available: `services:name`, `services:alias`, `services:entrypoint`, and `services:command`. | +| [`image`](#image) | Use Docker images. Also available: `image:name` and `image:entrypoint`. | +| [`services`](#services) | Use Docker services images. Also available: `services:name`, `services:alias`, `services:entrypoint`, and `services:command`. | | [`before_script`](#before_script-and-after_script) | Override a set of commands that are executed before job. | | [`after_script`](#before_script-and-after_script) | Override a set of commands that are executed after job. | | [`stage`](#stage) | Defines a job stage (default: `test`). | @@ -106,7 +117,7 @@ The following table lists available parameters for jobs: | [`when`](#when) | When to run job. Also available: `when:manual` and `when:delayed`. | | [`environment`](#environment) | Name of an environment to which the job deploys. Also available: `environment:name`, `environment:url`, `environment:on_stop`, `environment:auto_stop_in` and `environment:action`. | | [`cache`](#cache) | List of files that should be cached between subsequent runs. Also available: `cache:paths`, `cache:key`, `cache:untracked`, and `cache:policy`. | -| [`artifacts`](#artifacts) | List of files and directories to attach to a job on success. Also available: `artifacts:paths`, `artifacts:expose_as`, `artifacts:name`, `artifacts:untracked`, `artifacts:when`, `artifacts:expire_in`, `artifacts:reports`, `artifacts:reports:junit`, `artifacts:reports:cobertura`, and `artifacts:reports:terraform`.<br><br>In GitLab [Enterprise Edition](https://about.gitlab.com/pricing/), these are available: `artifacts:reports:codequality`, `artifacts:reports:sast`, `artifacts:reports:dependency_scanning`, `artifacts:reports:container_scanning`, `artifacts:reports:dast`, `artifacts:reports:license_scanning`, `artifacts:reports:license_management` (removed in GitLab 13.0),`artifacts:reports:performance` and `artifacts:reports:metrics`. | +| [`artifacts`](#artifacts) | List of files and directories to attach to a job on success. Also available: `artifacts:paths`, `artifacts:exclude`, `artifacts:expose_as`, `artifacts:name`, `artifacts:untracked`, `artifacts:when`, `artifacts:expire_in`, `artifacts:reports`, `artifacts:reports:junit`, `artifacts:reports:cobertura`, and `artifacts:reports:terraform`.<br><br>In GitLab [Enterprise Edition](https://about.gitlab.com/pricing/), these are available: `artifacts:reports:codequality`, `artifacts:reports:sast`, `artifacts:reports:dependency_scanning`, `artifacts:reports:container_scanning`, `artifacts:reports:dast`, `artifacts:reports:license_scanning`, `artifacts:reports:license_management` (removed in GitLab 13.0),`artifacts:reports:performance` and `artifacts:reports:metrics`. | | [`dependencies`](#dependencies) | Restrict which artifacts are passed to a specific job by providing a list of jobs to fetch artifacts from. | | [`coverage`](#coverage) | Code coverage settings for a given job. | | [`retry`](#retry) | When and how many times a job can be auto-retried in case of a failure. | @@ -163,7 +174,7 @@ rspec 2.6: #### `inherit` -> [Introduced](https://gitlab.com/gitlab-org/gitlab/issues/207484) in GitLab 12.9. +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/207484) in GitLab 12.9. You can disable inheritance of globally defined defaults and variables with the `inherit:` parameter. @@ -279,7 +290,7 @@ There are also two edge cases worth mentioning: ### `workflow:rules` -> [Introduced](https://gitlab.com/gitlab-org/gitlab/issues/29654) in GitLab 12.5 +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/29654) in GitLab 12.5 The top-level `workflow:` key applies to the entirety of a pipeline, and will determine whether or not a pipeline is created. It currently accepts a single @@ -310,7 +321,6 @@ include: - template: 'Workflows/Branch-Pipelines.gitlab-ci.yml' ``` -The [`MergeRequest-Pipelines` include](https://gitlab.com/gitlab-org/gitlab/-/tree/master/lib/gitlab/ci/templates/Workflows/MergeRequest-Pipelines.gitlab-ci.yml) sets your pipelines to run for the default branch (usually `master`), tags, and The [`MergeRequest-Pipelines` template](https://gitlab.com/gitlab-org/gitlab/-/tree/master/lib/gitlab/ci/templates/Workflows/MergeRequest-Pipelines.gitlab-ci.yml) makes your pipelines run for the default branch (usually `master`), tags, and all types of merge request pipelines. Use this template if you use any of the @@ -418,7 +428,7 @@ include: '.gitlab-ci-production.yml' #### `include:file` -> [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/issues/53903) in GitLab 11.7. +> [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/53903) in GitLab 11.7. To include files from another private project under the same GitLab instance, use `include:file`. This file is referenced using full paths relative to the @@ -468,7 +478,7 @@ or public project, or template, is allowed. #### `include:template` -> [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/issues/53445) in GitLab 11.7. +> [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/53445) in GitLab 11.7. `include:template` can be used to include `.gitlab-ci.yml` templates that are [shipped with GitLab](https://gitlab.com/gitlab-org/gitlab/tree/master/lib/gitlab/ci/templates). @@ -494,13 +504,13 @@ so it's possible to use project, remote or template includes. #### Nested includes -> [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/issues/56836) in GitLab 11.9. +> [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/56836) in GitLab 11.9. Nested includes allow you to compose a set of includes. A total of 100 includes is allowed, but duplicate includes are considered a configuration error. -Since [GitLab 12.4](https://gitlab.com/gitlab-org/gitlab/issues/28212), the time limit +Since [GitLab 12.4](https://gitlab.com/gitlab-org/gitlab/-/issues/28212), the time limit for resolving all files is 30 seconds. #### Additional `includes` examples @@ -522,13 +532,13 @@ For: #### `image:name` -An [extended docker configuration option](../docker/using_docker_images.md#extended-docker-configuration-options). +An [extended Docker configuration option](../docker/using_docker_images.md#extended-docker-configuration-options). For more information, see [Available settings for `image`](../docker/using_docker_images.md#available-settings-for-image). #### `image:entrypoint` -An [extended docker configuration option](../docker/using_docker_images.md#extended-docker-configuration-options). +An [extended Docker configuration option](../docker/using_docker_images.md#extended-docker-configuration-options). For more information, see [Available settings for `image`](../docker/using_docker_images.md#available-settings-for-image). @@ -544,25 +554,25 @@ For: ##### `services:name` -An [extended docker configuration option](../docker/using_docker_images.md#extended-docker-configuration-options). +An [extended Docker configuration option](../docker/using_docker_images.md#extended-docker-configuration-options). For more information, see [Available settings for `services`](../docker/using_docker_images.md#available-settings-for-services). ##### `services:alias` -An [extended docker configuration option](../docker/using_docker_images.md#extended-docker-configuration-options). +An [extended Docker configuration option](../docker/using_docker_images.md#extended-docker-configuration-options). For more information, see [Available settings for `services`](../docker/using_docker_images.md#available-settings-for-services). ##### `services:entrypoint` -An [extended docker configuration option](../docker/using_docker_images.md#extended-docker-configuration-options). +An [extended Docker configuration option](../docker/using_docker_images.md#extended-docker-configuration-options). For more information, see [Available settings for `services`](../docker/using_docker_images.md#available-settings-for-services). ##### `services:command` -An [extended docker configuration option](../docker/using_docker_images.md#extended-docker-configuration-options). +An [extended Docker configuration option](../docker/using_docker_images.md#extended-docker-configuration-options). For more information, see [Available settings for `services`](../docker/using_docker_images.md#available-settings-for-services). @@ -628,7 +638,7 @@ Scripts specified in `after_script` are executed in a new shell, separate from a - Changes outside of the working tree (depending on the Runner executor), like software installed by a `before_script` or `script` script. - Have a separate timeout, which is hard coded to 5 minutes. See - [related issue](https://gitlab.com/gitlab-org/gitlab-runner/issues/2716) for details. + [related issue](https://gitlab.com/gitlab-org/gitlab-runner/-/issues/2716) for details. - Don't affect the job's exit code. If the `script` section succeeds and the `after_script` times out or fails, the job will exit with code `0` (`Job Succeeded`). @@ -651,6 +661,44 @@ job: [YAML anchors for `before_script` and `after_script`](#yaml-anchors-for-before_script-and-after_script) are available. +#### Coloring script output + +Script output can be colored using [ANSI escape codes](https://en.wikipedia.org/wiki/ANSI_escape_code#Colors), +or by running commands or programs that output ANSI escape codes. + +For example, using [Bash with color codes](https://misc.flogisoft.com/bash/tip_colors_and_formatting): + +```yaml +job: + script: + - echo -e "\e[31mThis text is red,\e[0m but this text isn't\e[31m however this text is red again." +``` + +You can define the color codes in Shell variables, or even [custom environment variables](../variables/README.md#custom-environment-variables), +which makes the commands easier to read and reusable. + +For example, using the same example as above and variables defined in a `before_script`: + +```yaml +job: + before_script: + - TXT_RED="\e[31m" && TXT_CLEAR="\e[0m" + script: + - echo -e "${TXT_RED}This text is red,${TXT_CLEAR} but this part isn't${TXT_RED} however this part is again." + - echo "This text is not colored" +``` + +Or with [PowerShell color codes](https://superuser.com/a/1259916): + +```yaml +job: + before_script: + - $esc="$([char]27)"; $TXT_RED="$esc[31m"; $TXT_CLEAR="$esc[0m" + script: + - Write-Host $TXT_RED"This text is red,"$TXT_CLEAR" but this text isn't"$TXT_RED" however this text is red again." + - Write-Host "This text is not colored" +``` + ### `stage` `stage` is defined per-job and relies on [`stages`](#stages) which is defined @@ -701,7 +749,7 @@ Jobs will run on your own Runners in parallel only if: #### `.pre` and `.post` -> [Introduced](https://gitlab.com/gitlab-org/gitlab/issues/31441) in GitLab 12.4. +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/31441) in GitLab 12.4. The following stages are available to every pipeline: @@ -910,7 +958,7 @@ can use one of the [`workflow: rules` templates](#workflowrules-templates) to ge This will ensure that the behavior is more stable as you start adding additional `rules` blocks, and will avoid issues like creating a duplicate, merge request (detached) pipeline. -We don't recomment mixing `only/except` jobs with `rules` jobs in the same pipeline. +We don't recommend mixing `only/except` jobs with `rules` jobs in the same pipeline. It may not cause YAML errors, but debugging the exact execution behavior can be complex due to the different default behaviors of `only/except` and `rules`. @@ -966,10 +1014,14 @@ job: - if: '$CI_MERGE_REQUEST_SOURCE_BRANCH_NAME' # If neither of the first two match but the simple presence does, we set to "on_success" by default ``` -If none of the provided rules match, the job will be set to `when:never`, and -not included in the pipeline. If `rules:when` is not included in the configuration -at all, the behavior defaults to `job:when`, which continues to default to -`on_success`. +Some details regarding the logic that determines the `when` for the job: + +- If none of the provided rules match, the job is set to `when: never`, and is + not included in the pipeline. +- A rule without any conditional clause, such as a `when` or `allow_failure` + rule without `if` or `changes`, always matches, and is always used if reached. +- If a rule matches and has no `when` defined, the rule will use the `when` + defined for the job, which defaults to `on_success` if not defined. #### `rules:changes` @@ -1032,12 +1084,16 @@ checks. After the 10000th check, rules with patterned globs will always match. #### `rules:allow_failure` -> [Introduced](https://gitlab.com/gitlab-org/gitlab/issues/30235) in GitLab 12.8. +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/30235) in GitLab 12.8. You can use [`allow_failure: true`](#allow_failure) within `rules:` to allow a job to fail, or a manual job to wait for action, without stopping the pipeline itself. All jobs using `rules:` default to `allow_failure: false` if `allow_failure:` is not defined. +The rule-level `rules:allow_failure` option overrides the job-level +[`allow_failure`](#allow_failure) option, and is only applied when the job is +triggered by the particular rule. + ```yaml job: script: "echo Hello, Rules!" @@ -1111,7 +1167,7 @@ docker build: Additional job configuration may be added to rules in the future. If something useful is not available, please -[open an issue](https://gitlab.com/gitlab-org/gitlab/issues). +[open an issue](https://gitlab.com/gitlab-org/gitlab/-/issues). ### `only`/`except` (basic) @@ -1402,7 +1458,7 @@ Learn more about [variables expressions](../variables/README.md#environment-vari #### `only:changes`/`except:changes` -> `changes` policy [introduced](https://gitlab.com/gitlab-org/gitlab-foss/issues/19232) in GitLab 11.4. +> `changes` policy [introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/19232) in GitLab 11.4. Using the `changes` keyword with `only` or `except` makes it possible to define if a job should be created based on files modified by a Git push event. @@ -1414,7 +1470,7 @@ This means the `only:changes` policy is useful for pipelines where: - `$CI_PIPELINE_SOURCE == 'external_pull_request_event'` If there is no Git push event, such as for pipelines with -[sources other than the three above](../variables/predefined_variables.md#variables-reference), +[sources other than the three above](../variables/predefined_variables.md), `changes` can't determine if a given file is new or old, and will always return true. @@ -1548,9 +1604,9 @@ runs. ### `needs` -> - [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/issues/47063) in GitLab 12.2. +> - [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/47063) in GitLab 12.2. > - In GitLab 12.3, maximum number of jobs in `needs` array raised from five to 50. -> - [Introduced](https://gitlab.com/gitlab-org/gitlab/issues/30631) in GitLab 12.8, `needs: []` lets jobs start immediately. +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/30631) in GitLab 12.8, `needs: []` lets jobs start immediately. The `needs:` keyword enables executing jobs out-of-order, allowing you to implement a [directed acyclic graph](../directed_acyclic_graph/index.md) in your `.gitlab-ci.yml`. @@ -1612,7 +1668,7 @@ This example creates four paths of execution: pipeline will be created with YAML error. - The maximum number of jobs that a single job can need in the `needs:` array is limited: - For GitLab.com, the limit is ten. For more information, see our - [infrastructure issue](https://gitlab.com/gitlab-com/gl-infra/infrastructure/issues/7541). + [infrastructure issue](https://gitlab.com/gitlab-com/gl-infra/infrastructure/-/issues/7541). - For self-managed instances, the limit is: - 10, if the `ci_dag_limit_needs` feature flag is enabled (default). - 50, if the `ci_dag_limit_needs` feature flag is disabled. @@ -1620,7 +1676,7 @@ This example creates four paths of execution: the current job will depend on all parallel jobs created. - `needs:` is similar to `dependencies:` in that it needs to use jobs from prior stages, meaning it's impossible to create circular dependencies. Depending on jobs in the - current stage is not possible either, but support [is planned](https://gitlab.com/gitlab-org/gitlab/issues/30632). + current stage is not possible either, but support [is planned](https://gitlab.com/gitlab-org/gitlab/-/issues/30632). - Related to the above, stages must be explicitly defined for all jobs that have the keyword `needs:` or are referred to by one. @@ -1643,7 +1699,7 @@ Feature::enable(:ci_dag_limit_needs) #### Artifact downloads with `needs` -> [Introduced](https://gitlab.com/gitlab-org/gitlab/issues/14311) in GitLab v12.6. +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/14311) in GitLab v12.6. When using `needs`, artifact downloads are controlled with `artifacts: true` or `artifacts: false`. The `dependencies` keyword should not be used with `needs`, as this is deprecated since GitLab 12.6. @@ -1686,7 +1742,7 @@ rspec: #### Cross project artifact downloads with `needs` **(PREMIUM)** -> [Introduced](https://gitlab.com/gitlab-org/gitlab/issues/14311) in GitLab v12.7. +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/14311) in GitLab v12.7. `needs` can be used to download artifacts from up to five jobs in pipelines on [other refs in the same project](#artifact-downloads-between-pipelines-in-the-same-project), @@ -2142,9 +2198,15 @@ The `stop_review_app` job is **required** to have the following keywords defined - `stage` should be the same as the `review_app` in order for the environment to stop automatically when the branch is deleted +Additionally, both jobs should have matching [`rules`](../yaml/README.md#onlyexcept-basic) +or [`only/except`](../yaml/README.md#onlyexcept-basic) configuration. In the example +above, if the configuration is not identical, the `stop_review_app` job might not be +included in all pipelines that include the `review_app` job, and it will not be +possible to trigger the `action: stop` to stop the environment automatically. + #### `environment:auto_stop_in` -> [Introduced](https://gitlab.com/gitlab-org/gitlab/issues/20956) in GitLab 12.8. +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/20956) in GitLab 12.8. The `auto_stop_in` keyword is for specifying life period of the environment, that when expired, GitLab automatically stops them. @@ -2167,7 +2229,7 @@ For more information, see #### `environment:kubernetes` -> [Introduced](https://gitlab.com/gitlab-org/gitlab/issues/27630) in GitLab 12.6. +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/27630) in GitLab 12.6. The `kubernetes` block is used to configure deployments to a [Kubernetes cluster](../../user/project/clusters/index.md) that is associated with your project. @@ -2195,7 +2257,7 @@ NOTE: **Note:** Kubernetes configuration is not supported for Kubernetes clusters that are [managed by GitLab](../../user/project/clusters/index.md#gitlab-managed-clusters). To follow progress on support for GitLab-managed clusters, see the -[relevant issue](https://gitlab.com/gitlab-org/gitlab/issues/38054). +[relevant issue](https://gitlab.com/gitlab-org/gitlab/-/issues/38054). #### Dynamic environments @@ -2336,7 +2398,7 @@ cache: ##### `cache:key:files` -> [Introduced](https://gitlab.com/gitlab-org/gitlab/issues/18986) in GitLab v12.5. +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/18986) in GitLab v12.5. The `cache:key:files` keyword extends the `cache:key` functionality by making it easier to reuse some caches, and rebuild them less often, which will speed up subsequent pipeline @@ -2366,7 +2428,7 @@ use the new cache, instead of rebuilding the dependencies. ##### `cache:key:prefix` -> [Introduced](https://gitlab.com/gitlab-org/gitlab/issues/18986) in GitLab v12.5. +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/18986) in GitLab v12.5. The `prefix` parameter adds extra functionality to `key:files` by allowing the key to be composed of the given `prefix` combined with the SHA computed for `cache:key:files`. @@ -2545,9 +2607,36 @@ job: - path/*xyz/* ``` +#### `artifacts:exclude` + +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/15122) in GitLab 13.1 +> - Requires GitLab Runner 13.1 + +`exclude` makes it possible to prevent files from being added to an artifacts +archive. + +Similar to [`artifacts:paths`](#artifactspaths), `exclude` paths are relative +to the project directory. Wildcards can be used that follow the +[glob](https://en.wikipedia.org/wiki/Glob_(programming)) patterns and +[`filepath.Match`](https://golang.org/pkg/path/filepath/#Match). + +For example, to store all files in `binaries/`, but not `*.o` files located in +subdirectories of `binaries/`: + +```yaml +artifacts: + paths: + - binaries/ + exclude: + - binaries/**/*.o +``` + +Files matched by [`artifacts:untracked`](#artifactsuntracked) can be excluded using +`artifacts:exclude` too. + #### `artifacts:expose_as` -> [Introduced](https://gitlab.com/gitlab-org/gitlab/issues/15018) in GitLab 12.5. +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/15018) in GitLab 12.5. The `expose_as` keyword can be used to expose [job artifacts](../pipelines/job_artifacts.md) in the [merge request](../../user/project/merge_requests/index.md) UI. @@ -2691,6 +2780,15 @@ artifacts: - binaries/ ``` +Send all untracked files but [exclude](#artifactsexclude) `*.txt`: + +```yaml +artifacts: + untracked: true + exclude: + - *.txt +``` + #### `artifacts:when` > Introduced in GitLab 8.9 and GitLab Runner v1.3.0. @@ -2748,7 +2846,7 @@ job: ``` NOTE: **Note:** -For artifacts created in [GitLab 13.0](https://gitlab.com/gitlab-org/gitlab/-/issues/16267) +For artifacts created in [GitLab 13.1](https://gitlab.com/gitlab-org/gitlab/-/issues/16267) and later, the latest artifact for a ref is always kept, regardless of the expiry time. #### `artifacts:reports` @@ -2948,7 +3046,7 @@ You can specify the number of [retry attempts for certain stages of job executio ### `timeout` -> [Introduced](https://gitlab.com/gitlab-org/gitlab/issues/14887) in GitLab 12.3. +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/14887) in GitLab 12.3. `timeout` allows you to configure a timeout for a specific job. For example: @@ -3017,8 +3115,8 @@ job split into three separate jobs. ### `trigger` -> - [Introduced](https://gitlab.com/gitlab-org/gitlab/issues/8997) in [GitLab Premium](https://about.gitlab.com/pricing/) 11.8. -> - [Moved](https://gitlab.com/gitlab-org/gitlab/issues/199224) to GitLab Core in 12.8. +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/8997) in [GitLab Premium](https://about.gitlab.com/pricing/) 11.8. +> - [Moved](https://gitlab.com/gitlab-org/gitlab/-/issues/199224) to GitLab Core in 12.8. `trigger` allows you to define downstream pipeline trigger. When a job created from `trigger` definition is started by GitLab, a downstream pipeline gets @@ -3086,7 +3184,7 @@ upstream_bridge: #### `trigger` syntax for child pipeline -> [Introduced](https://gitlab.com/gitlab-org/gitlab/issues/16094) in GitLab 12.7. +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/16094) in GitLab 12.7. To create a [child pipeline](../parent_child_pipelines.md), specify the path to the YAML file containing the CI config of the child pipeline: @@ -3110,7 +3208,7 @@ trigger_job: ##### Trigger child pipeline with generated configuration file -> [Introduced](https://gitlab.com/gitlab-org/gitlab/issues/35632) in GitLab 12.9. +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/35632) in GitLab 12.9. You can also trigger a child pipeline from a [dynamically generated configuration file](../parent_child_pipelines.md#dynamic-child-pipelines): @@ -3217,7 +3315,7 @@ Once an uninterruptible job is running, the pipeline will never be canceled, reg ### `resource_group` -> [Introduced](https://gitlab.com/gitlab-org/gitlab/issues/15536) in GitLab 12.7. +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/15536) in GitLab 12.7. Sometimes running multiples jobs or pipelines at the same time in an environment can lead to errors during the deployment. @@ -3252,6 +3350,8 @@ NOTE: **Note:** This key can only contain letters, digits, `-`, `_`, `/`, `$`, `{`, `}`, `.`, and spaces. It can't start or end with `/`. +For more information, see [Deployments Safety](../environments/deployment_safety.md). + ### `pages` `pages` is a special job that is used to upload static content to GitLab that @@ -3351,7 +3451,7 @@ variables: `none` also re-uses the local working copy, but skips all Git operations (including GitLab Runner's pre-clone script, if present). It's mostly useful -for jobs that operate exclusively on artifacts (for examples `deploy`). Git repository +for jobs that operate exclusively on artifacts (for example, `deploy`). Git repository data may be present, but it's certain to be out of date, so you should only rely on files brought into the local working copy from cache or artifacts. @@ -3362,7 +3462,7 @@ variables: NOTE: **Note:** `GIT_STRATEGY` is not supported for [Kubernetes executor](https://docs.gitlab.com/runner/executors/kubernetes.html), -but may be in the future. See the [support Git strategy with Kubernetes executor feature proposal](https://gitlab.com/gitlab-org/gitlab-runner/issues/3847) +but may be in the future. See the [support Git strategy with Kubernetes executor feature proposal](https://gitlab.com/gitlab-org/gitlab-runner/-/issues/3847) for updates. ### Git submodule strategy @@ -3459,6 +3559,43 @@ script: - ls -al cache/ ``` +### Git fetch extra flags + +> [Introduced](https://gitlab.com/gitlab-org/gitlab-runner/-/issues/4142) in GitLab Runner 13.1. + +The `GIT_FETCH_EXTRA_FLAGS` variable is used to control the behavior of +`git fetch`. You can set it globally or per-job in the [`variables`](#variables) section. + +`GIT_FETCH_EXTRA_FLAGS` accepts all possible options of the [`git fetch`](https://git-scm.com/docs/git-fetch) command, but please note that `GIT_FETCH_EXTRA_FLAGS` flags will be appended after the default flags that can't be modified. + +The default flags are: + +- [GIT_DEPTH](#shallow-cloning). +- The list of [refspecs](https://git-scm.com/book/en/v2/Git-Internals-The-Refspec). +- A remote called `origin`. + +If `GIT_FETCH_EXTRA_FLAGS` is: + +- Not specified, `git fetch` flags default to `--prune --quiet` along with the default flags. +- Given the value `none`, `git fetch` is executed only with the default flags. + +For example, the default flags are `--prune --quiet`, so you can make `git fetch` more verbose by overriding this with just `--prune`: + +```yaml +variables: + GIT_FETCH_EXTRA_FLAGS: --prune +script: + - ls -al cache/ +``` + +The configurtion above will result in `git fetch` being called this way: + +```shell +git fetch origin $REFSPECS --depth 50 --prune +``` + +Where `$REFSPECS` is a value provided to the Runner internally by GitLab. + ### Job stages attempts > Introduced in GitLab, it requires GitLab Runner v1.9+. @@ -3488,8 +3625,8 @@ You can set them globally or per-job in the [`variables`](#variables) section. > Introduced in GitLab 8.9 as an experimental feature. -CAUTION: **Caution:** -May change in future releases or be removed completely. +NOTE: **Note**: +As of GitLab 12.0, newly created projects will automatically have a [default `git depth` value of `50`](../pipelines/settings.md#git-shallow-clone). You can specify the depth of fetching and cloning using `GIT_DEPTH`. This allows shallow cloning of the repository which can significantly speed up cloning for @@ -3739,7 +3876,7 @@ feature. Anchors are only valid within the file they were defined in. #### YAML anchors for `before_script` and `after_script` -> [Introduced](https://gitlab.com/gitlab-org/gitlab/issues/23005) in GitLab 12.5. +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/23005) in GitLab 12.5. You can use [YAML anchors](#anchors) with `before_script` and `after_script`, which makes it possible to include a predefined list of commands in multiple @@ -3753,7 +3890,7 @@ Example: .something_after: &something_after - echo 'something after' - +- echo 'another thing after' job_name: before_script: @@ -3766,7 +3903,7 @@ job_name: #### YAML anchors for `script` -> [Introduced](https://gitlab.com/gitlab-org/gitlab/issues/23005) in GitLab 12.5. +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/23005) in GitLab 12.5. You can use [YAML anchors](#anchors) with scripts, which makes it possible to include a predefined list of commands in multiple jobs. @@ -3796,6 +3933,7 @@ the use of the `SAMPLE_VARIABLE` variable: # global variables variables: &global-variables SAMPLE_VARIABLE: sample_variable_value + ANOTHER_SAMPLE_VARIABLE: another_sample_variable_value # a job that needs to set the GIT_STRATEGY variable, yet depend on global variables job_no_git_strategy: @@ -3819,7 +3957,7 @@ lines where the job is defined: # - run test ``` -you can instead start its name with a dot (`.`) and it won't be processed by +You can instead start its name with a dot (`.`) and it won't be processed by GitLab CI/CD. In the following example, `.hidden_job` will be ignored: ```yaml @@ -3842,11 +3980,11 @@ if using Git 2.10 or newer. ## Processing Git pushes -GitLab will create at most 4 branch and tags pipelines when -doing pushing multiple changes in single `git push` invocation. +GitLab will create at most 4 branch and tag pipelines when +pushing multiple changes in single `git push` invocation. -This limitation does not affect any of the updated Merge Request pipelines, -all updated Merge Requests will have a pipeline created when using +This limitation does not affect any of the updated Merge Request pipelines. +All updated Merge Requests will have a pipeline created when using [pipelines for merge requests](../merge_request_pipelines/index.md). ## Deprecated parameters diff --git a/doc/ci/yaml/includes.md b/doc/ci/yaml/includes.md index a7b626bdd7c..969b54d5be8 100644 --- a/doc/ci/yaml/includes.md +++ b/doc/ci/yaml/includes.md @@ -1,3 +1,10 @@ +--- +stage: Verify +group: Continuous Integration +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +type: reference +--- + # GitLab CI/CD YAML includes In addition to the [`includes` examples](README.md#include) listed in the |
