Merge branch 'docs/refactor-ci-variables' into 'master'

Refactor CI variables docs

See merge request !7751

3 files changed, 218 insertions, 61 deletions

diff --git a/doc/ci/README.md b/doc/ci/README.md
index db236ce2a66..73bd2516d46 100644
--- a/doc/ci/README.md
+++ b/doc/ci/README.md
@@ -11,7 +11,8 @@
 - [Configure a Runner, the application that runs your builds](runners/README.md)
 - [Use Docker images with GitLab Runner](docker/using_docker_images.md)
 - [Use CI to build Docker images](docker/using_docker_build.md)
-- [Use variables in your `.gitlab-ci.yml`](variables/README.md)
+- [CI Variables](variables/README.md) - Learn how to use variables defined in
+  your `.gitlab-ci.yml` or secured ones defined in your project's settings
 - [Use SSH keys in your build environment](ssh_keys/README.md)
 - [Trigger builds through the API](triggers/README.md)
 - [Build artifacts](../user/project/builds/artifacts.md)
@@ -25,4 +26,6 @@
 
 ## Breaking changes
 
-- [New CI build permissions model](../user/project/new_ci_build_permissions_model.md) Read about what changed in GitLab 8.12 and how that affects your builds. There's a new way to access your Git submodules and LFS objects in builds.
+- [New CI build permissions model](../user/project/new_ci_build_permissions_model.md)
+  Read about what changed in GitLab 8.12 and how that affects your builds.
+  There's a new way to access your Git submodules and LFS objects in builds.
diff --git a/doc/ci/variables/README.md b/doc/ci/variables/README.md
index a4c3a731a20..0a0ea2e96e1 100644
--- a/doc/ci/variables/README.md
+++ b/doc/ci/variables/README.md
@@ -1,22 +1,30 @@
-## Variables
+# Variables
 
-When receiving a build from GitLab CI, the runner prepares the build environment.
-It starts by setting a list of **predefined variables** (Environment Variables) and a list of **user-defined variables**
+When receiving a build from GitLab CI, the [Runner] prepares the build environment.
+It starts by setting a list of **predefined variables** (environment variables)
+and a list of **user-defined variables**.
 
-The variables can be overwritten. They take precedence over each other in this order:
-1. Trigger variables
-1. Secure variables
-1. YAML-defined job-level variables
-1. YAML-defined global variables
-1. Predefined variables
+## Priority of variables
 
-For example, if you define:
-1. `API_TOKEN=SECURE` as Secure Variable
-1. `API_TOKEN=YAML` as YAML-defined variable
+The variables can be overwritten and they take precedence over each other in
+this order:
 
-The `API_TOKEN` will take the Secure Variable value: `SECURE`.
+1. [Trigger variables][triggers] (take precedence over all)
+1. [Secure variables](#secure-variables)
+1. YAML-defined [job-level variables](../yaml/README.md#job-variables)
+1. YAML-defined [global variables](../yaml/README.md#variables)
+1. [Predefined variables](#predefined-variables-environment-variables) (are the
+   lowest in the chain)
 
-### Predefined variables (Environment Variables)
+For example, if you define `API_TOKEN=secure` as a secure variable and
+`API_TOKEN=yaml` in your `.gitlab-ci.yml`, the `API_TOKEN` will take the value
+`secure` as the secure variables are higher in the chain.
+
+## Predefined variables (Environment variables)
+
+Some of the predefined environment variables are available only if a minimum
+version of [GitLab Runner][runner] is used. Consult the table below to find the
+version of Runner required.
 
 | Variable                | GitLab | Runner | Description |
 |-------------------------|--------|--------|-------------|
@@ -52,7 +60,6 @@ The `API_TOKEN` will take the Secure Variable value: `SECURE`.
 | **GITLAB_USER_ID**      | 8.12   | all    | The id of the user who started the build |
 | **GITLAB_USER_EMAIL**   | 8.12   | all    | The email of the user who started the build |
 
-**Some of the variables are only available when using runner with at least defined version.**
 
 Example values:
 
@@ -60,7 +67,7 @@ Example values:
 export CI_BUILD_ID="50"
 export CI_BUILD_REF="1ecfd275763eff1d6b4844ea3168962458c9f27a"
 export CI_BUILD_REF_NAME="master"
-export CI_BUILD_REPO="https://gitab-ci-token:abcde-1234ABCD5678ef@gitlab.com/gitlab-org/gitlab-ce.git"
+export CI_BUILD_REPO="https://gitab-ci-token:abcde-1234ABCD5678ef@example.com/gitlab-org/gitlab-ce.git"
 export CI_BUILD_TAG="1.0.0"
 export CI_BUILD_NAME="spec:other"
 export CI_BUILD_STAGE="test"
@@ -73,9 +80,9 @@ export CI_PROJECT_DIR="/builds/gitlab-org/gitlab-ce"
 export CI_PROJECT_NAME="gitlab-ce"
 export CI_PROJECT_NAMESPACE="gitlab-org"
 export CI_PROJECT_PATH="gitlab-org/gitlab-ce"
-export CI_PROJECT_URL="https://gitlab.com/gitlab-org/gitlab-ce"
-export CI_REGISTRY="registry.gitlab.com"
-export CI_REGISTRY_IMAGE="registry.gitlab.com/gitlab-org/gitlab-ce"
+export CI_PROJECT_URL="https://example.com/gitlab-org/gitlab-ce"
+export CI_REGISTRY="registry.example.com"
+export CI_REGISTRY_IMAGE="registry.example.com/gitlab-org/gitlab-ce"
 export CI_RUNNER_ID="10"
 export CI_RUNNER_DESCRIPTION="my runner"
 export CI_RUNNER_TAGS="docker, linux"
@@ -84,30 +91,64 @@ export CI_SERVER_NAME="GitLab"
 export CI_SERVER_REVISION="70606bf"
 export CI_SERVER_VERSION="8.9.0"
 export GITLAB_USER_ID="42"
-export GITLAB_USER_EMAIL="alexzander@sporer.com"
+export GITLAB_USER_EMAIL="user@example.com"
 ```
 
-### YAML-defined variables
-**This feature requires GitLab Runner 0.5.0 or higher and GitLab CI 7.14 or higher **
+## `.gitlab-ci.yaml` defined variables
+
+>**Note:**
+This feature requires GitLab Runner 0.5.0 or higher and GitLab CI 7.14 or higher.
+
+GitLab CI allows you to add to `.gitlab-ci.yml` variables that are set in the
+build environment. The variables are hence saved in the repository, and they
+are meant to store non-sensitive project configuration, e.g., `RAILS_ENV` or
+`DATABASE_URL`.
 
-GitLab CI allows you to add to `.gitlab-ci.yml` variables that are set in build environment.
-The variables are stored in repository and are meant to store non-sensitive project configuration, ie. RAILS_ENV or DATABASE_URL.
+For example, if you set the variable below globally (not inside a job), it will
+be used in all executed commands and scripts:
 
 ```yaml
 variables:
   DATABASE_URL: "postgres://postgres@postgres/my_database"
 ```
 
-These variables can be later used in all executed commands and scripts.
+The YAML-defined variables are also set to all created
+[service containers](../docker/using_docker_images.md), thus allowing to fine
+tune them.
+
+Variables can be defined at a global level, but also at a job level. To turn off
+global defined variables in your job, define an empty array:
+
+```yaml
+job_name:
+  variables: []
+```
+
+## Secure variables
 
-The YAML-defined variables are also set to all created service containers, thus allowing to fine tune them.
+>**Notes:**
+- This feature requires GitLab Runner 0.4.0 or higher.
+- Be aware that secure variables are not masked, and their values can be shown
+  in the build logs if explicitly asked to do so. If your project is public or
+  internal, you can set the pipelines private from your project's Pipelines
+  settings. Follow the discussion in issue [#13784][ce-13784] for masking the
+  secure variables.
 
-Variables can be defined at a global level, but also at a job level.
+GitLab CI allows you to define per-project **Secure variables** that are set in
+the build environment. The secure variables are stored out of the repository
+(`.gitlab-ci.yml`) and are securely passed to GitLab Runner making them
+available in the build environment. It's the recommended method to use for
+storing things like passwords, secret keys and credentials.
 
-More information about Docker integration can be found in [Using Docker Images](../docker/using_docker_images.md).
+Secure variables can be added by going to your project's
+**Settings ➔ Variables ➔ Add variable**.
 
-#### Debug tracing
+Once you set them, they will be available for all subsequent builds.
 
+## Debug tracing
+
+> Introduced in GitLab Runner 1.7.
+>
 > **WARNING:** Enabling debug tracing can have severe security implications. The
   output **will** contain the content of all your secure variables and any other
   secrets! The output **will** be uploaded to the GitLab server and made visible
@@ -124,55 +165,162 @@ trace, resulting in a verbose build trace listing all commands that were run,
 variables that were set, etc.
 
 Before enabling this, you should ensure builds are visible to
-[team members only](../../../user/permissions.md#project-features). You should
-also [erase](../pipelines.md#seeing-build-traces) all generated build traces
+[team members only](../../user/permissions.md#project-features). You should
+also [erase](../pipelines.md#seeing-build-status) all generated build traces
 before making them visible again.
 
 To enable debug traces, set the `CI_DEBUG_TRACE` variable to `true`:
 
 ```yaml
-job1:
+job_name:
   variables:
     CI_DEBUG_TRACE: "true"
 ```
 
-The [example project](https://gitlab.com/gitlab-examples/ci-debug-trace)
-demonstrates a working configuration, including build trace examples.
+Example truncated output with debug trace set to true:
 
-### User-defined variables (Secure Variables)
-**This feature requires GitLab Runner 0.4.0 or higher**
-
-GitLab CI allows you to define per-project **Secure Variables** that are set in
-the build environment.
-The secure variables are stored out of the repository (the `.gitlab-ci.yml`).
-The variables are securely passed to GitLab Runner and are available in the
-build environment.
-It's desired method to use them for storing passwords, secret keys or whatever
-you want.
+```bash
+...
+
+export CI_SERVER_TLS_CA_FILE="/builds/gitlab-examples/ci-debug-trace.tmp/CI_SERVER_TLS_CA_FILE"
+if [[ -d "/builds/gitlab-examples/ci-debug-trace/.git" ]]; then
+  echo $'\''\x1b[32;1mFetching changes...\x1b[0;m'\''
+  $'\''cd'\'' "/builds/gitlab-examples/ci-debug-trace"
+  $'\''git'\'' "config" "fetch.recurseSubmodules" "false"
+  $'\''rm'\'' "-f" ".git/index.lock"
+  $'\''git'\'' "clean" "-ffdx"
+  $'\''git'\'' "reset" "--hard"
+  $'\''git'\'' "remote" "set-url" "origin" "https://gitlab-ci-token:xxxxxxxxxxxxxxxxxxxx@example.com/gitlab-examples/ci-debug-trace.git"
+  $'\''git'\'' "fetch" "origin" "--prune" "+refs/heads/*:refs/remotes/origin/*" "+refs/tags/*:refs/tags/*"
+else
+  $'\''mkdir'\'' "-p" "/builds/gitlab-examples/ci-debug-trace.tmp/git-template"
+  $'\''rm'\'' "-r" "-f" "/builds/gitlab-examples/ci-debug-trace"
+  $'\''git'\'' "config" "-f" "/builds/gitlab-examples/ci-debug-trace.tmp/git-template/config" "fetch.recurseSubmodules" "false"
+  echo $'\''\x1b[32;1mCloning repository...\x1b[0;m'\''
+  $'\''git'\'' "clone" "--no-checkout" "https://gitlab-ci-token:xxxxxxxxxxxxxxxxxxxx@example.com/gitlab-examples/ci-debug-trace.git" "/builds/gitlab-examples/ci-debug-trace" "--template" "/builds/gitlab-examples/ci-debug-trace.tmp/git-template"
+  $'\''cd'\'' "/builds/gitlab-examples/ci-debug-trace"
+fi
+echo $'\''\x1b[32;1mChecking out dd648b2e as master...\x1b[0;m'\''
+$'\''git'\'' "checkout" "-f" "-q" "dd648b2e48ce6518303b0bb580b2ee32fadaf045"
+'
++++ hostname
+++ echo 'Running on runner-8a2f473d-project-1796893-concurrent-0 via runner-8a2f473d-machine-1480971377-317a7d0f-digital-ocean-4gb...'
+Running on runner-8a2f473d-project-1796893-concurrent-0 via runner-8a2f473d-machine-1480971377-317a7d0f-digital-ocean-4gb...
+++ export CI=true
+++ CI=true
+++ export CI_DEBUG_TRACE=false
+++ CI_DEBUG_TRACE=false
+++ export CI_BUILD_REF=dd648b2e48ce6518303b0bb580b2ee32fadaf045
+++ CI_BUILD_REF=dd648b2e48ce6518303b0bb580b2ee32fadaf045
+++ export CI_BUILD_BEFORE_SHA=dd648b2e48ce6518303b0bb580b2ee32fadaf045
+++ CI_BUILD_BEFORE_SHA=dd648b2e48ce6518303b0bb580b2ee32fadaf045
+++ export CI_BUILD_REF_NAME=master
+++ CI_BUILD_REF_NAME=master
+++ export CI_BUILD_ID=7046507
+++ CI_BUILD_ID=7046507
+++ export CI_BUILD_REPO=https://gitlab-ci-token:xxxxxxxxxxxxxxxxxxxx@example.com/gitlab-examples/ci-debug-trace.git
+++ CI_BUILD_REPO=https://gitlab-ci-token:xxxxxxxxxxxxxxxxxxxx@example.com/gitlab-examples/ci-debug-trace.git
+++ export CI_BUILD_TOKEN=xxxxxxxxxxxxxxxxxxxx
+++ CI_BUILD_TOKEN=xxxxxxxxxxxxxxxxxxxx
+++ export CI_PROJECT_ID=1796893
+++ CI_PROJECT_ID=1796893
+++ export CI_PROJECT_DIR=/builds/gitlab-examples/ci-debug-trace
+++ CI_PROJECT_DIR=/builds/gitlab-examples/ci-debug-trace
+++ export CI_SERVER=yes
+++ CI_SERVER=yes
+++ export 'CI_SERVER_NAME=GitLab CI'
+++ CI_SERVER_NAME='GitLab CI'
+++ export CI_SERVER_VERSION=
+++ CI_SERVER_VERSION=
+++ export CI_SERVER_REVISION=
+++ CI_SERVER_REVISION=
+++ export GITLAB_CI=true
+++ GITLAB_CI=true
+++ export CI=true
+++ CI=true
+++ export GITLAB_CI=true
+++ GITLAB_CI=true
+++ export CI_BUILD_ID=7046507
+++ CI_BUILD_ID=7046507
+++ export CI_BUILD_TOKEN=xxxxxxxxxxxxxxxxxxxx
+++ CI_BUILD_TOKEN=xxxxxxxxxxxxxxxxxxxx
+++ export CI_BUILD_REF=dd648b2e48ce6518303b0bb580b2ee32fadaf045
+++ CI_BUILD_REF=dd648b2e48ce6518303b0bb580b2ee32fadaf045
+++ export CI_BUILD_BEFORE_SHA=dd648b2e48ce6518303b0bb580b2ee32fadaf045
+++ CI_BUILD_BEFORE_SHA=dd648b2e48ce6518303b0bb580b2ee32fadaf045
+++ export CI_BUILD_REF_NAME=master
+++ CI_BUILD_REF_NAME=master
+++ export CI_BUILD_NAME=debug_trace
+++ CI_BUILD_NAME=debug_trace
+++ export CI_BUILD_STAGE=test
+++ CI_BUILD_STAGE=test
+++ export CI_SERVER_NAME=GitLab
+++ CI_SERVER_NAME=GitLab
+++ export CI_SERVER_VERSION=8.14.3-ee
+++ CI_SERVER_VERSION=8.14.3-ee
+++ export CI_SERVER_REVISION=82823
+++ CI_SERVER_REVISION=82823
+++ export CI_PROJECT_ID=17893
+++ CI_PROJECT_ID=17893
+++ export CI_PROJECT_NAME=ci-debug-trace
+++ CI_PROJECT_NAME=ci-debug-trace
+++ export CI_PROJECT_PATH=gitlab-examples/ci-debug-trace
+++ CI_PROJECT_PATH=gitlab-examples/ci-debug-trace
+++ export CI_PROJECT_NAMESPACE=gitlab-examples
+++ CI_PROJECT_NAMESPACE=gitlab-examples
+++ export CI_PROJECT_URL=https://example.com/gitlab-examples/ci-debug-trace
+++ CI_PROJECT_URL=https://example.com/gitlab-examples/ci-debug-trace
+++ export CI_PIPELINE_ID=52666
+++ CI_PIPELINE_ID=52666
+++ export CI_RUNNER_ID=1337
+++ CI_RUNNER_ID=1337
+++ export CI_RUNNER_DESCRIPTION=shared-runners-manager-1.example.com
+++ CI_RUNNER_DESCRIPTION=shared-runners-manager-1.example.com
+++ export 'CI_RUNNER_TAGS=shared, docker, linux, ruby, mysql, postgres, mongo, git-annex'
+++ CI_RUNNER_TAGS='shared, docker, linux, ruby, mysql, postgres, mongo, git-annex'
+++ export CI_REGISTRY=registry.example.com
+++ CI_REGISTRY=registry.example.com
+++ export CI_DEBUG_TRACE=true
+++ CI_DEBUG_TRACE=true
+++ export GITLAB_USER_ID=42
+++ GITLAB_USER_ID=42
+++ export GITLAB_USER_EMAIL=user@example.com
+++ GITLAB_USER_EMAIL=axilleas@axilleas.me
+++ export VERY_SECURE_VARIABLE=imaverysecurevariable
+++ VERY_SECURE_VARIABLE=imaverysecurevariable
+++ mkdir -p /builds/gitlab-examples/ci-debug-trace.tmp
+++ echo -n '-----BEGIN CERTIFICATE-----
+MIIFQzCCBCugAwIBAgIRAL/ElDjuf15xwja1ZnCocWAwDQYJKoZIhvcNAQELBQAw'
+
+...
+```
 
-**The value of the variable can be shown in build log if explicitly asked to do so.**
-If your project is public or internal you can make the builds private.
+## Using the CI variables in your job scripts
 
-Secure Variables can added by going to `Project > Variables > Add Variable`.
+All variables are set as environment variables in the build environment, and
+they are accessible with normal methods that are used to access such variables.
+In most cases `bash` or `sh` is used to execute the build script.
 
-They will be available for all subsequent builds.
+To access the variables (predefined and user-defined) in a `bash`/`sh` environment,
+prefix the variable name with the dollar sign (`$`):
 
-### Use variables
-The variables are set as environment variables in build environment and are accessible with normal methods that are used to access such variables.
-In most cases the **bash** is used to execute build script.
-To access variables (predefined and user-defined) in bash environment, prefix the variable name with `$`:
 ```
 job_name:
   script:
     - echo $CI_BUILD_ID
 ```
 
-You can also list all environment variables with `export` command,
-but be aware that this will also expose value of all **Secure Variables** in build log:
+You can also list all environment variables with the `export` command,
+but be aware that this will also expose the values of all the secure variables
+you set, in the build log:
+
 ```
 job_name:
   script:
     - export
 ```
 
+[ce-13784]: https://gitlab.com/gitlab-org/gitlab-ce/issues/13784
+[runner]: https://docs.gitlab.com/runner/
 [triggered]: ../triggers/README.md
+[triggers]: ../triggers/README.md#pass-build-variables-to-a-trigger
diff --git a/doc/ci/yaml/README.md b/doc/ci/yaml/README.md
index 338c9a27789..5f88974d360 100644
--- a/doc/ci/yaml/README.md
+++ b/doc/ci/yaml/README.md
@@ -406,14 +406,20 @@ except master.
 ### job variables
 
 It is possible to define build variables using a `variables` keyword on a job
-level. It works basically the same way as its global-level equivalent but
-allows you to define job-specific build variables.
+level. It works basically the same way as its [global-level equivalent](#variables)
+but allows you to define job-specific build variables.
 
 When the `variables` keyword is used on a job level, it overrides global YAML
-build variables and predefined variables.
+build variables and predefined variables. To turn off global defined variables
+in your job, define an empty array:
 
-Build variables priority is defined in
-[variables documentation](../variables/README.md).
+```yaml
+job_name:
+  variables: []
+```
+
+Build variables priority is defined in the
+[variables documentation][variables].
 
 ### tags