Document where and how you can used variables in GitLab CI38325-variables-usage-in-gitlab-ci-yml-docs

4 files changed, 121 insertions, 39 deletions

diff --git a/doc/ci/README.md b/doc/ci/README.md
index 8d1d72c2a2b..b82ae9c3d89 100644
--- a/doc/ci/README.md
+++ b/doc/ci/README.md
@@ -47,6 +47,8 @@ you don't need to set up anything to start to use them with GitLab CI/CD.
 
 - [CI/CD Variables](variables/README.md) - Learn how to use variables defined in
   your `.gitlab-ci.yml` or secured ones defined in your project's settings
+  - [Where variables can be used](variables/where_variables_can_be_used.md) deeper look on where
+    and how defined variables can be used
 - **The permissions model** - Learn about the access levels a user can have for
   performing certain CI actions
   - [User permissions](../user/permissions.md#gitlab-ci)
diff --git a/doc/ci/environments.md b/doc/ci/environments.md
index 3a491f0073c..d7e8cbd4d3b 100644
--- a/doc/ci/environments.md
+++ b/doc/ci/environments.md
@@ -250,19 +250,9 @@ the basis of [Review apps](review_apps/index.md).
 The `name` and `url` parameters can use most of the defined CI variables,
 including predefined, secure variables and `.gitlab-ci.yml`
 [`variables`](yaml/README.md#variables). You however cannot use variables
-defined under `script` or on the Runner's side. There are other variables that
-are unsupported in environment name context:
-- `CI_PIPELINE_ID`
-- `CI_JOB_ID`
-- `CI_JOB_TOKEN`
-- `CI_BUILD_ID`
-- `CI_BUILD_TOKEN`
-- `CI_REGISTRY_USER`
-- `CI_REGISTRY_PASSWORD`
-- `CI_REPOSITORY_URL`
-- `CI_ENVIRONMENT_URL`
-- `CI_DEPLOY_USER`
-- `CI_DEPLOY_PASSWORD`
+defined under `script` or on the Runner's side. There are also other variables that
+are unsupported in environment name context. You can read more about this in
+[where variables can be used][where variables can be used].
 
 GitLab Runner exposes various [environment variables][variables] when a job runs,
 and as such, you can use them as environment names. Let's add another job in
@@ -676,6 +666,7 @@ Below are some links you may find interesting:
 [deployments]: #deployments
 [permissions]: ../user/permissions.md
 [variables]: variables/README.md
+[where variables can be used]: variables/where_variables_can_be_used.md
 [env-name]: yaml/README.md#environment-name
 [only]: yaml/README.md#only-and-except
 [onstop]: yaml/README.md#environment-on_stop
diff --git a/doc/ci/variables/README.md b/doc/ci/variables/README.md
index aedf7958c8a..11ada4dc50a 100644
--- a/doc/ci/variables/README.md
+++ b/doc/ci/variables/README.md
@@ -542,31 +542,8 @@ Below you can find supported syntax reference:
 
 ### Unsupported predefined variables
 
-Because GitLab evaluates variables before creating jobs, we do not support a
-few variables that depend on persistence layer, like `$CI_JOB_ID`.
-
-Environments (like `production` or `staging`) are also being created based on
-what jobs pipeline consists of, thus some environment-specific variables are
-not supported as well.
-
-We do not support variables containing tokens because of security reasons.
-
-You can find a full list of unsupported variables below:
-
-- `CI_PIPELINE_ID`
-- `CI_JOB_ID`
-- `CI_JOB_TOKEN`
-- `CI_BUILD_ID`
-- `CI_BUILD_TOKEN`
-- `CI_REGISTRY_USER`
-- `CI_REGISTRY_PASSWORD`
-- `CI_REPOSITORY_URL`
-- `CI_ENVIRONMENT_URL`
-- `CI_DEPLOY_USER`
-- `CI_DEPLOY_PASSWORD`
-
-These variables are also not supported in a context of a
-[dynamic environment name][dynamic-environments].
+You can read more about which variables are not supported in
+[where variables can be used][where variables can be used]
 
 [ce-13784]: https://gitlab.com/gitlab-org/gitlab-ce/issues/13784 "Simple protection of CI secret variables"
 [eep]: https://about.gitlab.com/products/ "Available only in GitLab Premium"
@@ -579,5 +556,5 @@ These variables are also not supported in a context of a
 [triggers]: ../triggers/README.md#pass-job-variables-to-a-trigger
 [subgroups]: ../../user/group/subgroups/index.md
 [builds-policies]: ../yaml/README.md#only-and-except-complex
-[dynamic-environments]: ../environments.md#dynamic-environments
 [gitlab-deploy-token]: ../../user/project/deploy_tokens/index.md#gitlab-deploy-token
+[where variables can be used]: where_variables_can_be_used.md
diff --git a/doc/ci/variables/where_variables_can_be_used.md b/doc/ci/variables/where_variables_can_be_used.md
new file mode 100644
index 00000000000..67444f10c96
--- /dev/null
+++ b/doc/ci/variables/where_variables_can_be_used.md
@@ -0,0 +1,112 @@
+# Where variables can be used
+
+As it is described in [variables][variables] document, in GitLab CI you can
+define many different variables. Some of them can be used for all GitLab CI
+features, but some of them are more or less limited.
+
+This document describes where and how the different types of variables can be used.
+
+## Variables usage
+
+### `.gitlab-ci.yml` file
+
+| Feature                                  | Can be expanded?  | Expansion place | Comments     |
+|------------------------------------------|-------------------|-----------------|--------------|
+| `job:environment:url`                    | yes               | Rails           | <ul><li>Variable expansion is made by the GitLab Rails application internal variable expansion mechanism ([described bellow](#gitlab-rails-application-internal-variable-expansion-mechanism))</li><li>Works with all variables defined for a job (Secret Variables, variables from `.gitlab-ci.yml`, variables from triggers, variables from Pipeline Schedules)</li><li>Doesn't work with variables defined in Runner's `config.toml` nor variables created in job's script</li></ul> |
+| `job:environment:name`                   | yes               | Rails           | Similar to the above, but variables expansion doesn't support: <ul><li>variables based itself on environment's name (`CI_ENVIRONMENT_NAME`, `CI_ENVIRONMENT_SLUG`)</li><li>any other variables related to environment (currently only `CI_ENVIRONMENT_URL`),</li><li>[persisted variables](#persisted-variables)</li></ul> |
+| `job:only:variables:[]`                  | no                | n/a             | The variable needs to be in form of `$variable` - only that one will work<br/>Feature doesn't support:<ul><li>variables based itself on environment's name (`CI_ENVIRONMENT_NAME`, `CI_ENVIRONMENT_SLUG`)</li><li>any other variables related to environment (currently only `CI_ENVIRONMENT_URL`),</li><li>[persisted variables](#persisted-variables)</li></ul> |
+| `job:except:variables:[]`                | no                | n/a             | Exactly like `job:only:variables:[]` |
+| `job:variables` (and global `variables`) | yes               | Runner          | <ul><li>Variable expansion is made by GitLab Runner's internal variable expansion mechanism</li><li>All variables are expanded once, before they are exported inside of the job script</li></ul> |
+| `job:image` (and global `image`)         | yes               | Runner          | Variable expansion is made by GitLab Runner's internal variable expansion mechanism ([described bellow](#gitlab-runners-internal-variable-expansion-mechanism)) |
+| `job:services:[]` (and global `services:[]`) | yes           | Runner          | Variable expansion is made by GitLab Runner's internal variable expansion mechanism |
+| `job:services:[]:name` (and global `services:[]:name` | yes  | Runner          | Variable expansion is made by GitLab Runner's internal variable expansion mechanism |
+| `job:cache:key` (and global `cache:key`) | yes               | Runner          | Variable expansion is made by GitLab Runner's internal variable expansion mechanism |
+| `job:artifacts:name`                     | yes               | Runner          | Variable expansion is made by GitLab Runner's shell environment  |
+| `job:script`, `job:before_script` (and global `before_script`), `job:after_script` | yes | Script execution shell | Variable expansion is made by the execution shell environment ([described bellow](#execution-shell-environment)) |
+
+### `config.toml` file
+
+| Feature                              | Can be expanded? | Comments |
+|--------------------------------------|------------------|----------|
+| `runners.environment`                | yes              | Variable expansion is made by the runner's internal variable expansion mechanism |
+| `runners.kubernetes.pod_labels`      | yes              | Variable expansion is made by the runner's internal variable expansion mechanism |
+| `runners.kubernetes.pod_annotations` | yes              | Variable expansion is made by the runner's internal variable expansion mechanism |
+
+## Expansion mechanisms
+
+### GitLab Rails application internal variable expansion mechanism
+
+Expanded part needs to be in a form of `$variable`, or `${variable}` or `%variable%` - each one form is handled in the same way, no matter which OS/shell will finally handle the job (since the expansion is done in Rails before any Runner will get this job)
+
+### GitLab Runner's internal variable expansion mechanism
+
+Runner uses Go's `os.Expand()` method for variable expansion. It means that it will handle
+only variables defined as `$variable` and `${variable}`. What's also important, is that
+the expansion is done only once, so nested variables may or may not work, depending on the
+ordering of variables definitions.
+
+Runner's internal expansion will use only variables defined by the CI (so Secret Variables,
+`.gitlab-ci.yml` variables, `config.toml` variables and variables from triggers and pipeline
+schedules). Variables defined inside of scripts (e.g. by `export MY_VARIABLE="test"`)
+**are not supported here**.
+
+### Execution shell environment
+
+This is an expansion that takes a place during script execution.
+
+How it works it depends on used shell (bash/sh/cmd/powershell). For example, if the job's
+script contains a line `echo $MY_VARIABLE-${MY_VARIABLE_2}`, it should be properly handled
+by bash/sh (leaving empty strings or some values depending whether the variables were
+defined or not), but will not work with Windows' CMD nor PowerShell, since these shells
+are using different way of variables syntax.
+
+The script may use all available variables that are default for the shell (e.g. `$PATH` which
+should be present in all bash/sh shells) and all variables defined by GitLab CI (so Secret Variables,
+`.gitlab-ci.yml` variables, `config.toml` variables and variables from triggers and pipeline schedules).
+
+The script may also use all variables defined in the lines before, until they are defined in the
+same context, so:
+
+- variable defined by for example `export MY_VARIABLE="test"` in `before_script` will work in
+  following lines of `before_script` and all lines of related `script`,
+- variable defined in the same way in `script` will work in following lines of `script`,
+- variable defined in the same way in `after_script` will work in following lines of `after_script`.
+
+## Persisted variables
+
+Most of the predefined variables that are created by GitLab CI, are basing on data available
+before the pipeline and job were created. They may be also not related to job data at all, like
+`CI_PROJECT_DIR` which is created by GitLab Runner, basing on Runner's configuration and
+used executor.
+
+However, some of the variables are basing on pipeline's or job's data that is available
+after the entity in Database is created. Because GitLab evaluates variables before creating
+jobs, we Can't support a few variables that depend on persistence layer.
+
+Environments (like production or staging) are also being created based on
+what jobs pipeline consists of, thus some environment-specific variables are
+not supported as well.
+
+We do not support variables containing tokens because of security reasons.
+
+The variables that are defined as `persisted` and are unsupported in some cases
+are:
+
+- `CI_PIPELINE_ID`
+- `CI_JOB_ID`
+- `CI_JOB_TOKEN`
+- `CI_BUILD_ID`
+- `CI_BUILD_TOKEN`
+- `CI_REGISTRY_USER`
+- `CI_REGISTRY_PASSWORD`
+- `CI_REPOSITORY_URL`
+- `CI_DEPLOY_USER`
+- `CI_DEPLOY_PASSWORD`
+
+These variables are available on GitLab Runner when the job is started, so they can be used
+by all features described in the table as `Expansion place: Runner`. But they can't be
+used by features described in the table as `Expansion place: Rails` nor in the `only` nor `except`
+[variables expressions][variables expressions].
+
+[variables]: README.md
+[variables expressions]: README.md#variables-expressions