diff options
Diffstat (limited to 'doc/ci')
| -rw-r--r-- | doc/ci/environments.md | 15 | ||||
| -rw-r--r-- | doc/ci/examples/browser_performance.md | 93 | ||||
| -rw-r--r-- | doc/ci/examples/code_climate.md | 7 | ||||
| -rw-r--r-- | doc/ci/examples/container_scanning.md | 4 | ||||
| -rw-r--r-- | doc/ci/examples/dast.md | 25 | ||||
| -rw-r--r-- | doc/ci/examples/laravel_with_gitlab_and_envoy/index.md | 2 | ||||
| -rw-r--r-- | doc/ci/pipelines.md | 8 | ||||
| -rw-r--r-- | doc/ci/quick_start/README.md | 5 | ||||
| -rw-r--r-- | doc/ci/runners/README.md | 19 | ||||
| -rw-r--r-- | doc/ci/variables/README.md | 103 | ||||
| -rw-r--r-- | doc/ci/yaml/README.md | 31 |
11 files changed, 271 insertions, 41 deletions
diff --git a/doc/ci/environments.md b/doc/ci/environments.md index 58c4a71cef9..b3d9f0bc96c 100644 --- a/doc/ci/environments.md +++ b/doc/ci/environments.md @@ -247,10 +247,19 @@ declaring their names dynamically in `.gitlab-ci.yml`. Dynamic environments is the basis of [Review apps](review_apps/index.md). >**Note:** -The `name` and `url` parameters can use any of the defined CI variables, +The `name` and `url` parameters can use most of the defined CI variables, including predefined, secure variables and `.gitlab-ci.yml` -[`variables`](yaml/README.md#variables). -You however cannot use variables defined under `script` or on the Runner's side. +[`variables`](yaml/README.md#variables). You however cannot use variables +defined under `script` or on the Runner's side. There are other variables that +are unsupported in environment name context: +- `CI_JOB_ID` +- `CI_JOB_TOKEN` +- `CI_BUILD_ID` +- `CI_BUILD_TOKEN` +- `CI_REGISTRY_USER` +- `CI_REGISTRY_PASSWORD` +- `CI_REPOSITORY_URL` +- `CI_ENVIRONMENT_URL` GitLab Runner exposes various [environment variables][variables] when a job runs, and as such, you can use them as environment names. Let's add another job in diff --git a/doc/ci/examples/browser_performance.md b/doc/ci/examples/browser_performance.md index 42dc6ef36ba..691370d7195 100644 --- a/doc/ci/examples/browser_performance.md +++ b/doc/ci/examples/browser_performance.md @@ -1,22 +1,28 @@ # Browser Performance Testing with the Sitespeed.io container -This example shows how to run the [Sitespeed.io container](https://hub.docker.com/r/sitespeedio/sitespeed.io/) on your code by using -GitLab CI/CD and [Sitespeed.io](https://www.sitespeed.io) using Docker-in-Docker. +This example shows how to run the +[Sitespeed.io container](https://hub.docker.com/r/sitespeedio/sitespeed.io/) on +your code by using GitLab CI/CD and [Sitespeed.io](https://www.sitespeed.io) +using Docker-in-Docker. -First, you need a GitLab Runner with the [docker-in-docker executor](../docker/using_docker_build.md#use-docker-in-docker-executor). - -Once you set up the Runner, add a new job to `.gitlab-ci.yml`, called `performance`: +First, you need a GitLab Runner with the +[docker-in-docker executor](../docker/using_docker_build.md#use-docker-in-docker-executor). +Once you set up the Runner, add a new job to `.gitlab-ci.yml`, called +`performance`: ```yaml +performance: stage: performance image: docker:git + variables: + URL: https://example.com services: - docker:dind script: - mkdir gitlab-exporter - - wget -O ./gitlab-exporter/index.js https://gitlab.com/gitlab-org/gl-performance/raw/10-5/index.js + - wget -O ./gitlab-exporter/index.js https://gitlab.com/gitlab-org/gl-performance/raw/master/index.js - mkdir sitespeed-results - - docker run --shm-size=1g --rm -v "$(pwd)":/sitespeed.io sitespeedio/sitespeed.io:6.3.1 --plugins.add ./gitlab-exporter --outputFolder sitespeed-results https://my.website.com + - docker run --shm-size=1g --rm -v "$(pwd)":/sitespeed.io sitespeedio/sitespeed.io:6.3.1 --plugins.add ./gitlab-exporter --outputFolder sitespeed-results $URL - mv sitespeed-results/data/performance.json performance.json artifacts: paths: @@ -24,37 +30,84 @@ Once you set up the Runner, add a new job to `.gitlab-ci.yml`, called `performan - sitespeed-results/ ``` -This will create a `performance` job in your CI/CD pipeline and will run Sitespeed.io against the webpage you define. The GitLab plugin for Sitespeed.io is downloaded in order to export key metrics to JSON. The full HTML Sitespeed.io report will also be saved as an artifact, and if you have Pages enabled it can be viewed directly in your browser. For further customization options of Sitespeed.io, including the ability to provide a list of URLs to test, please consult their [documentation](https://www.sitespeed.io/documentation/sitespeed.io/configuration/). +The above example will: + +1. Create a `performance` job in your CI/CD pipeline and will run + Sitespeed.io against the webpage you defined in `URL`. +1. The [GitLab plugin](https://gitlab.com/gitlab-org/gl-performance) for + Sitespeed.io is downloaded in order to export key metrics to JSON. The full + HTML Sitespeed.io report will also be saved as an artifact, and if you have + [GitLab Pages](../../user/project/pages/index.md) enabled, it can be viewed + directly in your browser. + +For further customization options of Sitespeed.io, including the ability to +provide a list of URLs to test, please consult +[their documentation](https://www.sitespeed.io/documentation/sitespeed.io/configuration/). -For [GitLab Premium](https://about.gitlab.com/products/) users, key metrics are automatically -extracted and shown right in the merge request widget. Learn more about [Browser Performance Testing](https://docs.gitlab.com/ee/user/project/merge_requests/browser_performance_testing.html). +TIP: **Tip:** +For [GitLab Premium](https://about.gitlab.com/pricing/) users, key metrics are automatically +extracted and shown right in the merge request widget. Learn more about +[Browser Performance Testing](https://docs.gitlab.com/ee/user/project/merge_requests/browser_performance_testing.html). ## Performance testing on Review Apps -The above CI YML is great for testing against static environments, and it can be extended for dynamic environments. There are a few extra steps to take to set this up: -1. The `performance` job should run after the environment has started. -1. In the `deploy` job, persist the hostname so it is available to the `performance` job. The same can be done for static environments like staging and production to unify the code path. Saving it as an artifact is as simple as `echo $CI_ENVIRONMENT_URL > environment_url.txt`. -1. In the `performance` job read the artifact into an environment variable, like `$CI_ENVIRONMENT_URL`, and use it to parameterize the test URL's. -1. Now you can run the Sitespeed.io container against the desired hostname and paths. +The above CI YML is great for testing against static environments, and it can +be extended for dynamic environments. There are a few extra steps to take to +set this up: -A simple `performance` job would look like: +1. The `performance` job should run after the dynamic environment has started. +1. In the `review` job, persist the hostname and upload it as an artifact so + it's available to the `performance` job (the same can be done for static + environments like staging and production to unify the code path). Saving it + as an artifact is as simple as `echo $CI_ENVIRONMENT_URL > environment_url.txt` + in your job's `script`. +1. In the `performance` job, read the previous artifact into an environment + variable, like `$CI_ENVIRONMENT_URL`, and use it to parameterize the test + URLs. +1. You can now run the Sitespeed.io container against the desired hostname and + paths. + +Your `.gitlab-ci.yml` file would look like: ```yaml +stages: + - deploy + - performance + +review: + stage: deploy + environment: + name: review/$CI_COMMIT_REF_SLUG + url: http://$CI_COMMIT_REF_SLUG.$APPS_DOMAIN + script: + - run_deploy_script + - echo $CI_ENVIRONMENT_URL > environment_url.txt + artifacts: + paths: + - environment_url.txt + only: + - branches + except: + - master + +performance: stage: performance image: docker:git services: - docker:dind + dependencies: + - review script: - export CI_ENVIRONMENT_URL=$(cat environment_url.txt) - mkdir gitlab-exporter - - wget -O ./gitlab-exporter/index.js https://gitlab.com/gitlab-org/gl-performance/raw/10-5/index.js + - wget -O ./gitlab-exporter/index.js https://gitlab.com/gitlab-org/gl-performance/raw/master/index.js - mkdir sitespeed-results - docker run --shm-size=1g --rm -v "$(pwd)":/sitespeed.io sitespeedio/sitespeed.io:6.3.1 --plugins.add ./gitlab-exporter --outputFolder sitespeed-results "$CI_ENVIRONMENT_URL" - mv sitespeed-results/data/performance.json performance.json artifacts: paths: - - performance.json - - sitespeed-results/ + - performance.json + - sitespeed-results/ ``` -A complete example can be found in our [Auto DevOps CI YML](https://gitlab.com/gitlab-org/gitlab-ci-yml/blob/master/Auto-DevOps.gitlab-ci.yml).
\ No newline at end of file +A complete example can be found in our [Auto DevOps CI YML](https://gitlab.com/gitlab-org/gitlab-ci-yml/blob/master/Auto-DevOps.gitlab-ci.yml). diff --git a/doc/ci/examples/code_climate.md b/doc/ci/examples/code_climate.md index 64a759a9a99..92317c77427 100644 --- a/doc/ci/examples/code_climate.md +++ b/doc/ci/examples/code_climate.md @@ -9,11 +9,12 @@ Once you set up the Runner, add a new job to `.gitlab-ci.yml`, called `codequali ```yaml codequality: - image: docker:latest + image: docker:stable variables: - DOCKER_DRIVER: overlay + DOCKER_DRIVER: overlay2 + allow_failure: true services: - - docker:dind + - docker:stable-dind script: - export SP_VERSION=$(echo "$CI_SERVER_VERSION" | sed 's/^\([0-9]*\)\.\([0-9]*\).*/\1-\2-stable/') - docker run --env SOURCE_CODE="$PWD" --volume "$PWD":/code --volume /var/run/docker.sock:/var/run/docker.sock "registry.gitlab.com/gitlab-org/security-products/codequality:$SP_VERSION" /code diff --git a/doc/ci/examples/container_scanning.md b/doc/ci/examples/container_scanning.md index 3437b63748a..c58efc7392a 100644 --- a/doc/ci/examples/container_scanning.md +++ b/doc/ci/examples/container_scanning.md @@ -11,7 +11,7 @@ called `sast:container`: ```yaml sast:container: - image: docker:latest + image: docker:stable variables: DOCKER_DRIVER: overlay2 ## Define two new variables based on GitLab's CI/CD predefined variables @@ -20,7 +20,7 @@ sast:container: CI_APPLICATION_TAG: $CI_COMMIT_SHA allow_failure: true services: - - docker:dind + - docker:stable-dind script: - docker run -d --name db arminc/clair-db:latest - docker run -p 6060:6060 --link db:postgres -d --name clair arminc/clair-local-scan:v2.0.1 diff --git a/doc/ci/examples/dast.md b/doc/ci/examples/dast.md index 96de0f5ff5c..8df223ee560 100644 --- a/doc/ci/examples/dast.md +++ b/doc/ci/examples/dast.md @@ -14,9 +14,10 @@ called `dast`: ```yaml dast: - image: owasp/zap2docker-stable + image: registry.gitlab.com/gitlab-org/security-products/zaproxy variables: website: "https://example.com" + allow_failure: true script: - mkdir /zap/wrk/ - /zap/zap-baseline.py -J gl-dast-report.json -t $website || true @@ -30,6 +31,28 @@ the tests on the URL defined in the `website` variable (change it to use your own) and finally write the results in the `gl-dast-report.json` file. You can then download and analyze the report artifact in JSON format. +It's also possible to authenticate the user before performing DAST checks: + +```yaml +dast: + image: registry.gitlab.com/gitlab-org/security-products/zaproxy + variables: + website: "https://example.com" + login_url: "https://example.com/sign-in" + allow_failure: true + script: + - mkdir /zap/wrk/ + - /zap/zap-baseline.py -J gl-dast-report.json -t $website \ + --auth-url $login_url \ + --auth-username "john.doe@example.com" \ + --auth-password "john-doe-password" || true + - cp /zap/wrk/gl-dast-report.json . + artifacts: + paths: [gl-dast-report.json] +``` +See [zaproxy documentation](https://gitlab.com/gitlab-org/security-products/zaproxy) +to learn more about authentication settings. + TIP: **Tip:** Starting with [GitLab Ultimate][ee] 10.4, this information will be automatically extracted and shown right in the merge request widget. To do diff --git a/doc/ci/examples/laravel_with_gitlab_and_envoy/index.md b/doc/ci/examples/laravel_with_gitlab_and_envoy/index.md index b62874ef029..1f9b9d53fc1 100644 --- a/doc/ci/examples/laravel_with_gitlab_and_envoy/index.md +++ b/doc/ci/examples/laravel_with_gitlab_and_envoy/index.md @@ -190,7 +190,7 @@ To start, we create an `Envoy.blade.php` in the root of our app with a simple ta ```php @servers(['web' => 'remote_username@remote_host']) -@task('list', [on => 'web']) +@task('list', ['on' => 'web']) ls -l @endtask ``` diff --git a/doc/ci/pipelines.md b/doc/ci/pipelines.md index 856d7f264e4..301cccc80a3 100644 --- a/doc/ci/pipelines.md +++ b/doc/ci/pipelines.md @@ -2,6 +2,11 @@ > Introduced in GitLab 8.8. +NOTE: **Note:** +If you have a [mirrored repository where GitLab pulls from](https://docs.gitlab.com/ee/workflow/repository_mirroring.html#pulling-from-a-remote-repository), +you may need to enable pipeline triggering in your project's +**Settings > Repository > Pull from a remote repository > Trigger pipelines for mirror updates**. + ## Pipelines A pipeline is a group of [jobs][] that get executed in [stages][](batches). @@ -121,9 +126,8 @@ The basic requirements is that there are two numbers separated with one of the following (you can even use them interchangeably): - a space -- a forward slash (`/`) +- a slash (`/`) - a colon (`:`) -- a dot (`.`) >**Note:** More specifically, [it uses][regexp] this regular expression: `\d+[\s:\/\\]+\d+\s*`. diff --git a/doc/ci/quick_start/README.md b/doc/ci/quick_start/README.md index f64e868d390..fec0ff87326 100644 --- a/doc/ci/quick_start/README.md +++ b/doc/ci/quick_start/README.md @@ -126,6 +126,11 @@ git push origin master Now if you go to the **Pipelines** page you will see that the pipeline is pending. +NOTE: **Note:** +If you have a [mirrored repository where GitLab pulls from](https://docs.gitlab.com/ee/workflow/repository_mirroring.html#pulling-from-a-remote-repository), +you may need to enable pipeline triggering in your project's +**Settings > Repository > Pull from a remote repository > Trigger pipelines for mirror updates**. + You can also go to the **Commits** page and notice the little pause icon next to the commit SHA. diff --git a/doc/ci/runners/README.md b/doc/ci/runners/README.md index b91aa334ff3..60dc2ef9ac5 100644 --- a/doc/ci/runners/README.md +++ b/doc/ci/runners/README.md @@ -35,7 +35,7 @@ are: A Runner that is specific only runs for the specified project(s). A shared Runner can run jobs for every project that has enabled the option **Allow shared Runners** -under **Settings ➔ CI/CD**. +under **Settings > CI/CD**. Projects with high demand of CI activity can also benefit from using specific Runners. By having dedicated Runners you are guaranteed that the Runner is not @@ -76,7 +76,7 @@ Registering a specific can be done in two ways: To create a specific Runner without having admin rights to the GitLab instance, visit the project you want to make the Runner work for in GitLab: -1. Go to **Settings ➔ CI/CD** to obtain the token +1. Go to **Settings > CI/CD** to obtain the token 1. [Register the Runner][register] ### Making an existing shared Runner specific @@ -85,7 +85,7 @@ If you are an admin on your GitLab instance, you can turn any shared Runner into a specific one, but not the other way around. Keep in mind that this is a one way transition. -1. Go to the Runners in the admin area **Overview ➔ Runners** (`/admin/runners`) +1. Go to the Runners in the admin area **Overview > Runners** (`/admin/runners`) and find your Runner 1. Enable any projects under **Restrict projects for this Runner** to be used with the Runner @@ -101,7 +101,7 @@ can be changed afterwards under each Runner's settings. To lock/unlock a Runner: -1. Visit your project's **Settings ➔ CI/CD** +1. Visit your project's **Settings > CI/CD** 1. Find the Runner you wish to lock/unlock and make sure it's enabled 1. Click the pencil button 1. Check the **Lock to current projects** option @@ -115,7 +115,7 @@ you can enable the Runner also on any other project where you have Master permis To enable/disable a Runner in your project: -1. Visit your project's **Settings ➔ CI/CD** +1. Visit your project's **Settings > CI/CD** 1. Find the Runner you wish to enable/disable 1. Click **Enable for this project** or **Disable for this project** @@ -124,6 +124,13 @@ Consider that if you don't lock your specific Runner to a specific project, any user with Master role in you project can assign your runner to another arbitrary project without requiring your authorization, so use it with caution. +An admin can enable/disable a specific Runner for projects: + +1. Navigate to **Admin > Runners** +2. Find the Runner you wish to enable/disable +3. Click edit on the Runner +4. Click **Enable** or **Disable** on the project + ## Protected Runners > @@ -136,7 +143,7 @@ Whenever a Runner is protected, the Runner picks only jobs created on To protect/unprotect Runners: -1. Visit your project's **Settings ➔ CI/CD** +1. Visit your project's **Settings > CI/CD** 1. Find a Runner you want to protect/unprotect and make sure it's enabled 1. Click the pencil button besides the Runner name 1. Check the **Protected** option diff --git a/doc/ci/variables/README.md b/doc/ci/variables/README.md index bd4aeb006bd..4a504a98902 100644 --- a/doc/ci/variables/README.md +++ b/doc/ci/variables/README.md @@ -449,6 +449,107 @@ export CI_REGISTRY_USER="gitlab-ci-token" export CI_REGISTRY_PASSWORD="longalfanumstring" ``` +## Variables expressions + +> Variables expressions were added in GitLab 10.7. + +It is possible to use variables expressions with only / except policies in +`.gitlab-ci.yml`. By using this approach you can limit what jobs are going to +be created within a pipeline after pushing a code to GitLab. + +This is particularly useful in combination with secret variables and triggered +pipeline variables. + +```yaml +deploy: + script: cap staging deploy + environment: staging + only: + variables: + - $RELEASE == "staging" + - $STAGING +``` + +Each expression provided is going to be evaluated before creating a pipeline. + +If any of the conditions in `variables` evaluates to truth when using `only`, +a new job is going to be created. If any of the expressions evaluates to truth +when `except` is being used, a job is not going to be created. + +This follows usual rules for [`only` / `except` policies][builds-policies]. + +### Supported syntax + +Below you can find supported syntax reference: + +1. Equality matching using a string + + > Example: `$VARIABLE == "some value"` + + You can use equality operator `==` to compare a variable content to a + string. We support both, double quotes and single quotes to define a string + value, so both `$VARIABLE == "some value"` and `$VARIABLE == 'some value'` + are supported. `"some value" == $VARIABLE` is correct too. + +1. Checking for an undefined value + + > Example: `$VARIABLE == null` + + It sometimes happens that you want to check whether a variable is defined + or not. To do that, you can compare a variable to `null` keyword, like + `$VARIABLE == null`. This expression is going to evaluate to truth if + variable is not defined. + +1. Checking for an empty variable + + > Example: `$VARIABLE == ""` + + If you want to check whether a variable is defined, but is empty, you can + simply compare it against an empty string, like `$VAR == ''`. + +1. Comparing two variables + + > Example: `$VARIABLE_1 == $VARIABLE_2` + + It is possible to compare two variables. This is going to compare values + of these variables. + +1. Variable presence check + + > Example: `$STAGING` + + If you only want to create a job when there is some variable present, + which means that it is defined and non-empty, you can simply use + variable name as an expression, like `$STAGING`. If `$STAGING` variable + is defined, and is non empty, expression will evaluate to truth. + `$STAGING` value needs to a string, with length higher than zero. + Variable that contains only whitespace characters is not an empty variable. + +### Unsupported predefined variables + +Because GitLab evaluates variables before creating jobs, we do not support a +few variables that depend on persistence layer, like `$CI_JOB_ID`. + +Environments (like `production` or `staging`) are also being created based on +what jobs pipeline consists of, thus some environment-specific variables are +not supported as well. + +We do not support variables containing tokens because of security reasons. + +You can find a full list of unsupported variables below: + +- `CI_JOB_ID` +- `CI_JOB_TOKEN` +- `CI_BUILD_ID` +- `CI_BUILD_TOKEN` +- `CI_REGISTRY_USER` +- `CI_REGISTRY_PASSWORD` +- `CI_REPOSITORY_URL` +- `CI_ENVIRONMENT_URL` + +These variables are also not supported in a contex of a +[dynamic environment name][dynamic-environments]. + [ce-13784]: https://gitlab.com/gitlab-org/gitlab-ce/issues/13784 "Simple protection of CI secret variables" [eep]: https://about.gitlab.com/products/ "Available only in GitLab Premium" [envs]: ../environments.md @@ -459,3 +560,5 @@ export CI_REGISTRY_PASSWORD="longalfanumstring" [triggered]: ../triggers/README.md [triggers]: ../triggers/README.md#pass-job-variables-to-a-trigger [subgroups]: ../../user/group/subgroups/index.md +[builds-policies]: ../yaml/README.md#only-and-except-complex +[dynamic-environments]: ../environments.md#dynamic-environments diff --git a/doc/ci/yaml/README.md b/doc/ci/yaml/README.md index c2b06e53c2f..68aa64b3834 100644 --- a/doc/ci/yaml/README.md +++ b/doc/ci/yaml/README.md @@ -10,6 +10,11 @@ of your repository and contains definitions of how your project should be built. If you want a quick introduction to GitLab CI, follow our [quick start guide](../quick_start/README.md). +NOTE: **Note:** +If you have a [mirrored repository where GitLab pulls from](https://docs.gitlab.com/ee/workflow/repository_mirroring.html#pulling-from-a-remote-repository), +you may need to enable pipeline triggering in your project's +**Settings > Repository > Pull from a remote repository > Trigger pipelines for mirror updates**. + ## Jobs The YAML file defines a set of jobs with constraints stating when they should @@ -315,9 +320,14 @@ policy configuration. GitLab now supports both, simple and complex strategies, so it is possible to use an array and a hash configuration scheme. -Two keys are now available: `refs` and `kubernetes`. Refs strategy equals to -simplified only/except configuration, whereas kubernetes strategy accepts only -`active` keyword. +Three keys are now available: `refs`, `kubernetes` and `variables`. +Refs strategy equals to simplified only/except configuration, whereas +kubernetes strategy accepts only `active` keyword. + +`variables` keyword is used to define variables expressions. In other words +you can use predefined variables / secret variables / project / group or +environment-scoped variables to define an expression GitLab is going to +evaluate in order to decide whether a job should be created or not. See the example below. Job is going to be created only when pipeline has been scheduled or runs for a `master` branch, and only if kubernetes service is @@ -332,6 +342,20 @@ job: kubernetes: active ``` +Example of using variables expressions: + +```yaml +deploy: + only: + refs: + - branches + variables: + - $RELEASE == "staging" + - $STAGING +``` + +Learn more about variables expressions on [a separate page][variables-expressions]. + ## `tags` `tags` is used to select specific Runners from the list of all Runners that are @@ -1550,3 +1574,4 @@ CI with various languages. [ce-7447]: https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/7447 [ce-12909]: https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/12909 [schedules]: ../../user/project/pipelines/schedules.md +[variables-expressions]: ../variables/README.md#variables-expressions |