diff options
author | Grzegorz Bizon <grzegorz@gitlab.com> | 2016-04-18 12:28:19 +0000 |
---|---|---|
committer | Grzegorz Bizon <grzegorz@gitlab.com> | 2016-04-18 12:28:19 +0000 |
commit | 06952aaf24633550f57fd54b70d27732509935c2 (patch) | |
tree | 08245bdf4c9ba3469d8bd20b4f617b4acd0f79dd | |
parent | 2b8fc1387b97dc8ae7857458cc61c6f7a21f6d54 (diff) | |
parent | 1339fda1cd3325c0186b5f1b53444e7319ad3cb6 (diff) | |
download | gitlab-ce-06952aaf24633550f57fd54b70d27732509935c2.tar.gz |
Merge branch 'feature/ci-job-env-variables' into 'master'
Add environment variables on a job level in CI
Make it possible to define environment variables on a job level.
Closes #14716
See merge request !3612
-rw-r--r-- | CHANGELOG | 1 | ||||
-rw-r--r-- | app/models/ci/build.rb | 18 | ||||
-rw-r--r-- | doc/ci/variables/README.md | 22 | ||||
-rw-r--r-- | doc/ci/yaml/README.md | 16 | ||||
-rw-r--r-- | lib/ci/gitlab_ci_yaml_processor.rb | 25 | ||||
-rw-r--r-- | spec/lib/ci/gitlab_ci_yaml_processor_spec.rb | 88 | ||||
-rw-r--r-- | spec/models/build_spec.rb | 16 | ||||
-rw-r--r-- | spec/support/gitlab_stubs/gitlab_ci.yml | 17 |
8 files changed, 168 insertions, 35 deletions
diff --git a/CHANGELOG b/CHANGELOG index 7ef374b04d8..b929f1fd7b9 100644 --- a/CHANGELOG +++ b/CHANGELOG @@ -3,6 +3,7 @@ Please view this file on the master branch, on stable branches it's out of date. v 8.7.0 (unreleased) - Method instrumentation now uses Module#prepend instead of aliasing methods - Repository.clean_old_archives is now instrumented + - Add support for environment variables on a job level in CI configuration file - The Projects::HousekeepingService class has extra instrumentation - All service classes (those residing in app/services) are now instrumented - Developers can now add custom tags to transactions diff --git a/app/models/ci/build.rb b/app/models/ci/build.rb index 7d33838044b..85ef0523b31 100644 --- a/app/models/ci/build.rb +++ b/app/models/ci/build.rb @@ -365,11 +365,23 @@ module Ci self.update(erased_by: user, erased_at: Time.now) end - private - def yaml_variables + global_yaml_variables + job_yaml_variables + end + + def global_yaml_variables + if commit.config_processor + commit.config_processor.global_variables.map do |key, value| + { key: key, value: value, public: true } + end + else + [] + end + end + + def job_yaml_variables if commit.config_processor - commit.config_processor.variables.map do |key, value| + commit.config_processor.job_variables(name).map do |key, value| { key: key, value: value, public: true } end else diff --git a/doc/ci/variables/README.md b/doc/ci/variables/README.md index b0e53cbc261..70fb81492d6 100644 --- a/doc/ci/variables/README.md +++ b/doc/ci/variables/README.md @@ -1,17 +1,20 @@ ## Variables + When receiving a build from GitLab CI, the runner prepares the build environment. It starts by setting a list of **predefined variables** (Environment Variables) and a list of **user-defined variables** The variables can be overwritten. They take precedence over each other in this order: +1. Trigger variables 1. Secure variables -1. YAML-defined variables +1. YAML-defined job-level variables +1. YAML-defined global variables 1. Predefined variables For example, if you define: -1. API_TOKEN=SECURE as Secure Variable -1. API_TOKEN=YAML as YAML-defined variable +1. `API_TOKEN=SECURE` as Secure Variable +1. `API_TOKEN=YAML` as YAML-defined variable -The API_TOKEN will take the Secure Variable value: `SECURE`. +The `API_TOKEN` will take the Secure Variable value: `SECURE`. ### Predefined variables (Environment Variables) @@ -70,15 +73,20 @@ These variables can be later used in all executed commands and scripts. The YAML-defined variables are also set to all created service containers, thus allowing to fine tune them. +Variables can be defined at a global level, but also at a job level. + More information about Docker integration can be found in [Using Docker Images](../docker/using_docker_images.md). ### User-defined variables (Secure Variables) **This feature requires GitLab Runner 0.4.0 or higher** -GitLab CI allows you to define per-project **Secure Variables** that are set in build environment. +GitLab CI allows you to define per-project **Secure Variables** that are set in +the build environment. The secure variables are stored out of the repository (the `.gitlab-ci.yml`). -The variables are securely passed to GitLab Runner and are available in build environment. -It's desired method to use them for storing passwords, secret keys or whatever you want. +The variables are securely passed to GitLab Runner and are available in the +build environment. +It's desired method to use them for storing passwords, secret keys or whatever +you want. **The value of the variable can be shown in build log if explicitly asked to do so.** If your project is public or internal you can make the builds private. diff --git a/doc/ci/yaml/README.md b/doc/ci/yaml/README.md index abb6e97e5e6..61475b45988 100644 --- a/doc/ci/yaml/README.md +++ b/doc/ci/yaml/README.md @@ -23,6 +23,7 @@ If you want a quick introduction to GitLab CI, follow our - [Jobs](#jobs) - [script](#script) - [stage](#stage) + - [job variables](#job-variables) - [only and except](#only-and-except) - [tags](#tags) - [when](#when) @@ -174,6 +175,8 @@ These variables can be later used in all executed commands and scripts. The YAML-defined variables are also set to all created service containers, thus allowing to fine tune them. +Variables can be also defined on [job level](#job-variables). + ### cache >**Note:** @@ -324,6 +327,7 @@ job_name: | services | no | Use docker services, covered in [Using Docker Images](../docker/using_docker_images.md#define-image-and-services-from-gitlab-ciyml) | | stage | no | Defines a build stage (default: `test`) | | type | no | Alias for `stage` | +| variables | no | Define build variables on a job level | | only | no | Defines a list of git refs for which build is created | | except | no | Defines a list of git refs for which build is not created | | tags | no | Defines a list of tags which are used to select Runner | @@ -414,6 +418,18 @@ job: The above example will run `job` for all branches on `gitlab-org/gitlab-ce`, except master. +### job variables + +It is possible to define build variables using a `variables` keyword on a job +level. It works basically the same way as its global-level equivalent but +allows you to define job-specific build variables. + +When the `variables` keyword is used on a job level, it overrides global YAML +build variables and predefined variables. + +Build variables priority is defined in +[variables documentation](../variables/README.md). + ### tags `tags` is used to select specific Runners from the list of all Runners that are diff --git a/lib/ci/gitlab_ci_yaml_processor.rb b/lib/ci/gitlab_ci_yaml_processor.rb index b7209c14148..b8ede3a7edc 100644 --- a/lib/ci/gitlab_ci_yaml_processor.rb +++ b/lib/ci/gitlab_ci_yaml_processor.rb @@ -7,9 +7,9 @@ module Ci ALLOWED_YAML_KEYS = [:before_script, :image, :services, :types, :stages, :variables, :cache] ALLOWED_JOB_KEYS = [:tags, :script, :only, :except, :type, :image, :services, :allow_failure, :type, :stage, :when, :artifacts, :cache, - :dependencies] + :dependencies, :variables] - attr_reader :before_script, :image, :services, :variables, :path, :cache + attr_reader :before_script, :image, :services, :path, :cache def initialize(config, path = nil) @config = YAML.safe_load(config, [Symbol], [], true) @@ -40,6 +40,17 @@ module Ci @stages || DEFAULT_STAGES end + def global_variables + @variables + end + + def job_variables(name) + job = @jobs[name.to_sym] + return [] unless job + + job.fetch(:variables, []) + end + private def initial_parsing @@ -115,7 +126,7 @@ module Ci end unless @variables.nil? || validate_variables(@variables) - raise ValidationError, "variables should be a map of key-valued strings" + raise ValidationError, "variables should be a map of key-value strings" end if @cache @@ -145,6 +156,7 @@ module Ci validate_job_types!(name, job) validate_job_stage!(name, job) if job[:stage] + validate_job_variables!(name, job) if job[:variables] validate_job_cache!(name, job) if job[:cache] validate_job_artifacts!(name, job) if job[:artifacts] validate_job_dependencies!(name, job) if job[:dependencies] @@ -206,6 +218,13 @@ module Ci end end + def validate_job_variables!(name, job) + unless validate_variables(job[:variables]) + raise ValidationError, + "#{name} job: variables should be a map of key-value strings" + end + end + def validate_job_cache!(name, job) if job[:cache][:key] && !validate_string(job[:cache][:key]) raise ValidationError, "#{name} job: cache:key parameter should be a string" diff --git a/spec/lib/ci/gitlab_ci_yaml_processor_spec.rb b/spec/lib/ci/gitlab_ci_yaml_processor_spec.rb index dcb8a3451bd..5f4b63bcafb 100644 --- a/spec/lib/ci/gitlab_ci_yaml_processor_spec.rb +++ b/spec/lib/ci/gitlab_ci_yaml_processor_spec.rb @@ -345,20 +345,76 @@ module Ci end end - describe "Variables" do - it "returns variables when defined" do - variables = { - var1: "value1", - var2: "value2", - } - config = YAML.dump({ - variables: variables, - before_script: ["pwd"], - rspec: { script: "rspec" } - }) + describe 'Variables' do + context 'when global variables are defined' do + it 'returns global variables' do + variables = { + VAR1: 'value1', + VAR2: 'value2', + } - config_processor = GitlabCiYamlProcessor.new(config, path) - expect(config_processor.variables).to eq(variables) + config = YAML.dump({ + variables: variables, + before_script: ['pwd'], + rspec: { script: 'rspec' } + }) + + config_processor = GitlabCiYamlProcessor.new(config, path) + + expect(config_processor.global_variables).to eq(variables) + end + end + + context 'when job variables are defined' do + context 'when syntax is correct' do + it 'returns job variables' do + variables = { + KEY1: 'value1', + SOME_KEY_2: 'value2' + } + + config = YAML.dump( + { before_script: ['pwd'], + rspec: { + variables: variables, + script: 'rspec' } + }) + + config_processor = GitlabCiYamlProcessor.new(config, path) + + expect(config_processor.job_variables(:rspec)).to eq variables + end + end + + context 'when syntax is incorrect' do + it 'raises error' do + variables = [:KEY1, 'value1', :KEY2, 'value2'] + + config = YAML.dump( + { before_script: ['pwd'], + rspec: { + variables: variables, + script: 'rspec' } + }) + + expect { GitlabCiYamlProcessor.new(config, path) } + .to raise_error(GitlabCiYamlProcessor::ValidationError, + /job: variables should be a map/) + end + end + end + + context 'when job variables are not defined' do + it 'returns empty array' do + config = YAML.dump({ + before_script: ['pwd'], + rspec: { script: 'rspec' } + }) + + config_processor = GitlabCiYamlProcessor.new(config, path) + + expect(config_processor.job_variables(:rspec)).to eq [] + end end end @@ -730,14 +786,14 @@ EOT config = YAML.dump({ variables: "test", rspec: { script: "test" } }) expect do GitlabCiYamlProcessor.new(config, path) - end.to raise_error(GitlabCiYamlProcessor::ValidationError, "variables should be a map of key-valued strings") + end.to raise_error(GitlabCiYamlProcessor::ValidationError, "variables should be a map of key-value strings") end - it "returns errors if variables is not a map of key-valued strings" do + it "returns errors if variables is not a map of key-value strings" do config = YAML.dump({ variables: { test: false }, rspec: { script: "test" } }) expect do GitlabCiYamlProcessor.new(config, path) - end.to raise_error(GitlabCiYamlProcessor::ValidationError, "variables should be a map of key-valued strings") + end.to raise_error(GitlabCiYamlProcessor::ValidationError, "variables should be a map of key-value strings") end it "returns errors if job when is not on_success, on_failure or always" do diff --git a/spec/models/build_spec.rb b/spec/models/build_spec.rb index b7457808040..b5d356aa066 100644 --- a/spec/models/build_spec.rb +++ b/spec/models/build_spec.rb @@ -238,6 +238,22 @@ describe Ci::Build, models: true do it { is_expected.to eq(predefined_variables + predefined_trigger_variable + yaml_variables + secure_variables + trigger_variables) } end + + context 'when job variables are defined' do + ## + # Job-level variables are defined in gitlab_ci.yml fixture + # + context 'when job variables are unique' do + let(:build) { create(:ci_build, name: 'staging') } + + it 'includes job variables' do + expect(subject).to include( + { key: :KEY1, value: 'value1', public: true }, + { key: :KEY2, value: 'value2', public: true } + ) + end + end + end end end end diff --git a/spec/support/gitlab_stubs/gitlab_ci.yml b/spec/support/gitlab_stubs/gitlab_ci.yml index a5b256bd3ec..e55a61b2b94 100644 --- a/spec/support/gitlab_stubs/gitlab_ci.yml +++ b/spec/support/gitlab_stubs/gitlab_ci.yml @@ -4,7 +4,7 @@ services: before_script: - gem install bundler - - bundle install + - bundle install - bundle exec rake db:create variables: @@ -17,7 +17,7 @@ types: rspec: script: "rake spec" - tags: + tags: - ruby - postgres only: @@ -26,27 +26,32 @@ rspec: spinach: script: "rake spinach" allow_failure: true - tags: + tags: - ruby - mysql except: - tags staging: + variables: + KEY1: value1 + KEY2: value2 script: "cap deploy stating" type: deploy - tags: + tags: - ruby - mysql except: - stable production: + variables: + DB_NAME: mysql type: deploy - script: + script: - cap deploy production - cap notify - tags: + tags: - ruby - mysql only: |