diff options
27 files changed, 1016 insertions, 66 deletions
diff --git a/app/models/clusters/applications/prometheus.rb b/app/models/clusters/applications/prometheus.rb index d24a298b0a6..adce55cb61b 100644 --- a/app/models/clusters/applications/prometheus.rb +++ b/app/models/clusters/applications/prometheus.rb @@ -99,6 +99,8 @@ module Clusters def configured? kube_client.present? && available? + rescue Gitlab::UrlBlocker::BlockedUrlError + false end private diff --git a/changelogs/unreleased/fix_prometheus_configured_blocked.yml b/changelogs/unreleased/fix_prometheus_configured_blocked.yml new file mode 100644 index 00000000000..5676c41ffd3 --- /dev/null +++ b/changelogs/unreleased/fix_prometheus_configured_blocked.yml @@ -0,0 +1,5 @@ +--- +title: Fix job page not loading because kuberenetes/prometheus URL is blocked +merge_request: 24743 +author: +type: fixed diff --git a/changelogs/unreleased/port-trigger-keyword-to-ce.yml b/changelogs/unreleased/port-trigger-keyword-to-ce.yml new file mode 100644 index 00000000000..d518bd03021 --- /dev/null +++ b/changelogs/unreleased/port-trigger-keyword-to-ce.yml @@ -0,0 +1,5 @@ +--- +title: Port `trigger` keyword in CI config to Core +merge_request: 24191 +author: +type: fixed diff --git a/doc/ci/multi_project_pipelines.md b/doc/ci/multi_project_pipelines.md index 6b84c3d6d12..1876681e372 100644 --- a/doc/ci/multi_project_pipelines.md +++ b/doc/ci/multi_project_pipelines.md @@ -51,7 +51,8 @@ outbound connections for upstream and downstream pipeline dependencies. ## Creating multi-project pipelines from `.gitlab-ci.yml` -> [Introduced](https://gitlab.com/gitlab-org/gitlab/issues/8997) in [GitLab Premium](https://about.gitlab.com/pricing/) 11.8. +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/issues/8997) in [GitLab Premium](https://about.gitlab.com/pricing/) 11.8. +> - [Moved](https://gitlab.com/gitlab-org/gitlab/issues/199224) to GitLab Core in 12.8. ### Triggering a downstream pipeline using a bridge job @@ -181,7 +182,8 @@ the ones defined in the upstream project will take precedence. ### Mirroring status from triggered pipeline -> [Introduced](https://gitlab.com/gitlab-org/gitlab/issues/11238) in [GitLab Premium](https://about.gitlab.com/pricing/) 12.3. +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/issues/11238) in [GitLab Premium](https://about.gitlab.com/pricing/) 12.3. +> - [Moved](https://gitlab.com/gitlab-org/gitlab/issues/199224) to GitLab Core in 12.8. You can mirror the pipeline status from the triggered pipeline to the source bridge job by using `strategy: depend`. For example: diff --git a/doc/ci/parent_child_pipelines.md b/doc/ci/parent_child_pipelines.md index 95a364d9a94..387f2e69606 100644 --- a/doc/ci/parent_child_pipelines.md +++ b/doc/ci/parent_child_pipelines.md @@ -45,7 +45,7 @@ the child pipeline configuration. ## Examples -The simplest case is [triggering a child pipeline](yaml/README.md#trigger-premium) using a +The simplest case is [triggering a child pipeline](yaml/README.md#trigger) using a local YAML file to define the pipeline configuration. In this case, the parent pipeline will trigger the child pipeline, and continue without waiting: diff --git a/doc/ci/variables/README.md b/doc/ci/variables/README.md index f09dbf8c133..643ccd45898 100644 --- a/doc/ci/variables/README.md +++ b/doc/ci/variables/README.md @@ -168,6 +168,9 @@ let's say you want to output `HELLO WORLD` for a `TEST` variable. You can either set the variable directly in the `.gitlab-ci.yml` file or through the UI. +NOTE: **Note:** +It is possible to [specify variables when running manual jobs](../pipelines.md#specifying-variables-when-running-manual-jobs). + #### Via `.gitlab-ci.yml` To create a new custom `env_var` variable via [`.gitlab-ci.yml`](../yaml/README.md#variables), define their variable/value pair under diff --git a/doc/ci/yaml/README.md b/doc/ci/yaml/README.md index ccc2a1893db..881f3297695 100644 --- a/doc/ci/yaml/README.md +++ b/doc/ci/yaml/README.md @@ -113,7 +113,7 @@ The following table lists available parameters for jobs: | [`retry`](#retry) | When and how many times a job can be auto-retried in case of a failure. | | [`timeout`](#timeout) | Define a custom job-level timeout that takes precedence over the project-wide setting. | | [`parallel`](#parallel) | How many instances of a job should be run in parallel. | -| [`trigger`](#trigger-premium) | Defines a downstream pipeline trigger. | +| [`trigger`](#trigger) | Defines a downstream pipeline trigger. | | [`include`](#include) | Allows this job to include external YAML files. Also available: `include:local`, `include:file`, `include:template`, and `include:remote`. | | [`extends`](#extends) | Configuration entries that this job is going to inherit from. | | [`pages`](#pages) | Upload the result of a job to use with GitLab Pages. | @@ -2572,9 +2572,10 @@ Please be aware that semaphore_test_boosters reports usages statistics to the au You can then navigate to the **Jobs** tab of a new pipeline build and see your RSpec job split into three separate jobs. -### `trigger` **(PREMIUM)** +### `trigger` -> [Introduced](https://gitlab.com/gitlab-org/gitlab/issues/8997) in [GitLab Premium](https://about.gitlab.com/pricing/) 11.8. +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/issues/8997) in [GitLab Premium](https://about.gitlab.com/pricing/) 11.8. +> - [Moved](https://gitlab.com/gitlab-org/gitlab/issues/199224) to GitLab Core in 12.8. `trigger` allows you to define downstream pipeline trigger. When a job created from `trigger` definition is started by GitLab, a downstream pipeline gets @@ -3892,7 +3893,7 @@ job_no_git_strategy: Triggers can be used to force a rebuild of a specific branch, tag or commit, with an API call when a pipeline gets created using a trigger token. -Not to be confused with [`trigger`](#trigger-premium). +Not to be confused with [`trigger`](#trigger). [Read more in the triggers documentation.](../triggers/README.md) diff --git a/doc/development/README.md b/doc/development/README.md index 7511221b246..1e5e1cdce5f 100644 --- a/doc/development/README.md +++ b/doc/development/README.md @@ -147,6 +147,7 @@ Complementary reads: ## Integration guides - [Jira Connect app](integrations/jira_connect.md) +- [Security Scanners](integrations/secure.md) ## Testing guides diff --git a/doc/development/import_project.md b/doc/development/import_project.md index 37cf07ff702..b969cb5f1c4 100644 --- a/doc/development/import_project.md +++ b/doc/development/import_project.md @@ -53,8 +53,18 @@ As part of this script we also disable direct and background upload to avoid sit We can simply run this script from the terminal: +Parameters: + +| Attribute | Type | Required | Description | +| --------- | ---- | -------- | ----------- | +| `username` | string | yes | User name | +| `namespace_path` | string | yes | Namespace path | +| `project_path` | string | yes | Project name | +| `archive_path` | string | yes | Path to the exported project tarball you want to import | +| `measurement_enabled` | boolean | no | Measure execution time, number of SQL calls and GC count | + ```shell -bundle exec rake "gitlab:import_export:import[root, root, testingprojectimport, /path/to/file.tar.gz]" +bundle exec rake "gitlab:import_export:import[root, root, testingprojectimport, /path/to/file.tar.gz, true]" ``` ### Importing via the Rails console diff --git a/doc/development/integrations/secure.md b/doc/development/integrations/secure.md index b9b37a7e298..c54c2050790 100644 --- a/doc/development/integrations/secure.md +++ b/doc/development/integrations/secure.md @@ -116,9 +116,9 @@ the scanner with all the libraries and tools it depends on. ### Image size -Depending on the CI infrastucture, +Depending on the CI infrastructure, the CI may have to fetch the Docker image every time the job runs. -To make the scanning job run fast, and to avoid wasting bandwith, +To make the scanning job run fast, and to avoid wasting bandwidth, it is important to make Docker images as small as possible, ideally smaller than 50 MB. @@ -189,7 +189,7 @@ then `artifacts:reports:dependency_scanning` must be set to `depscan.json`. ### Exit code Following the POSIX exit code standard, the scanner will exit with 0 for success and any number from 1 to 255 for anything else. -This also includes the case when vulnerabilities are found. +Success also includes the case when vulnerabilities are found. ### Logging @@ -275,7 +275,7 @@ It should not repeat the other fields of the vulnerability object. In particular, the `description` should not repeat the `location` (what is affected) or the `solution` (how to mitigate the risk). -There is a proposal to remove either the `name` or the `message`, to remove abmiguities. +There is a proposal to remove either the `name` or the `message`, to remove ambiguities. See [issue #36779](https://gitlab.com/gitlab-org/gitlab/issues/36779). #### Solution diff --git a/doc/subscriptions/index.md b/doc/subscriptions/index.md index 5e3ce934e87..1c541c77737 100644 --- a/doc/subscriptions/index.md +++ b/doc/subscriptions/index.md @@ -225,13 +225,21 @@ The following table describes details of your subscription for groups: #### CI pipeline minutes -CI pipeline minutes are the execution time for your [pipelines](../ci/pipelines.md) on our shared runners. Each [GitLab.com tier](https://about.gitlab.com/pricing/) includes a monthly quota of CI pipeline minutes. The quota is applied per group, shared across all members of that group, its subgroups and nested projects. To view the usage, navigate to the group's page, then **Settings > Usage Quotas**. +CI pipeline minutes are the execution time for your [pipelines](../ci/pipelines.md) on GitLab's shared runners. Each [GitLab.com tier](https://about.gitlab.com/pricing/) includes a monthly quota of CI pipeline minutes. -Only pipeline minutes for our shared runners are restricted. If you have a specific runner setup for your projects, there is no limit to your build time on GitLab.com. +Quotas apply to: -The minutes limit only applies to private projects. The available quota is reset on the first of each calendar month at midnight UTC. +- Groups, where the minutes are shared across all members of the group, its subgroups, and nested projects. To view the group's usage, navigate to the group's page, then **Settings > Usage Quotas**. -If you reach your limit, you can [purchase additional CI minutes](#extra-shared-runners-pipeline-minutes), or upgrade your account to [Silver or Gold](https://about.gitlab.com/pricing/). Note, your own runners can still be used even if you reach your limits. +- Your personal account, where the minutes are available for your personal projects. To view and buy personal minutes, click your avatar, then **Settings > Pipeline quota**. + +Only pipeline minutes for GitLab shared runners are restricted. If you have a specific runner set up for your projects, there is no limit to your build time on GitLab.com. + +The minutes limit does not apply to public projects. + +The available quota is reset on the first of each calendar month at midnight UTC. + +If you reach your limit, you can [purchase additional CI minutes](#extra-shared-runners-pipeline-minutes), or upgrade your account to [Silver or Gold](https://about.gitlab.com/pricing/). Your own runners can still be used even if you reach your limits. ##### How pipeline quota usage is calculated @@ -285,14 +293,16 @@ For example: ##### Purchasing additional minutes -In order to purchase additional minutes, you should follow these steps: +To purchase additional minutes, follow these steps. + +1. For group minutes, go to **Group > Settings > Pipelines quota**. -1. Go to **Group > Settings > Pipelines quota**. Once you are on that page, click on **Buy additional minutes**. + For personal project minutes, click your avatar, then **Settings > Pipeline quota**. -  +1. Click **Buy additional minutes**. 1. Locate the subscription card that is linked to your group on GitLab.com, - click on **Buy more CI minutes**, and complete the details about the transaction. + click **Buy more CI minutes**, and complete the details about the transaction.  diff --git a/doc/user/application_security/dast/index.md b/doc/user/application_security/dast/index.md index a96a17aa45a..c96f0f8b0d3 100644 --- a/doc/user/application_security/dast/index.md +++ b/doc/user/application_security/dast/index.md @@ -359,7 +359,7 @@ The DAST job can emit various reports. CAUTION: **Caution:** The JSON report artifacts are not a public API of DAST and their format is expected to change in the future. -The DAST tool always emits a JSON report report file called `gl-dast-report.json` and sample reports can be found in the [DAST repository](https://gitlab.com/gitlab-org/security-products/dast/tree/master/test/end-to-end/expect). +The DAST tool always emits a JSON report report file called `gl-dast-report.json` and sample reports can be found in the [DAST repository](https://gitlab.com/gitlab-org/security-products/dast/-/tree/master/test/end-to-end/expect). There are two formats of data in the JSON report that are used side by side: the proprietary ZAP format which will be eventually deprecated, and a "common" format which will be the default in the future. diff --git a/doc/user/group/index.md b/doc/user/group/index.md index f3f65ae9f75..a5383da2e1f 100644 --- a/doc/user/group/index.md +++ b/doc/user/group/index.md @@ -163,6 +163,24 @@ If you change your mind before your request is approved, just click the  +## Changing the owner of a group + +Ownership of a group means at least one of its members has +[Owner permission](../permissions.md#group-members-permissions). Groups must have at +least one owner. + +Changing the owner of a group with only one owner is possible. To change the sole owner +of a group: + +- As an administrator: + 1. Go to the group's **{users}** **Members** tab. + 1. Give a different member **Owner** permissions. + 1. Refresh the page. You can now remove **Owner** permissions from the original owner. +- As the current group's owner: + 1. Go to the group's **{users}** **Members** tab. + 1. Give a different member **Owner** permissions. + 1. Have the new owner sign in and remove **Owner** permissions from you. + ## Add projects to a group There are two different ways to add a new project to a group: diff --git a/lib/gitlab/ci/config/entry/bridge.rb b/lib/gitlab/ci/config/entry/bridge.rb new file mode 100644 index 00000000000..7a6840218e1 --- /dev/null +++ b/lib/gitlab/ci/config/entry/bridge.rb @@ -0,0 +1,151 @@ +# frozen_string_literal: true + +module Gitlab + module Ci + class Config + module Entry + ## + # Entry that represents a CI/CD Bridge job that is responsible for + # defining a downstream project trigger. + # + class Bridge < ::Gitlab::Config::Entry::Node + include ::Gitlab::Config::Entry::Configurable + include ::Gitlab::Config::Entry::Attributable + include ::Gitlab::Config::Entry::Inheritable + + ALLOWED_KEYS = %i[trigger stage allow_failure only except + when extends variables needs rules].freeze + + validations do + validates :config, allowed_keys: ALLOWED_KEYS + validates :config, presence: true + validates :name, presence: true + validates :name, type: Symbol + validates :config, disallowed_keys: { + in: %i[only except when start_in], + message: 'key may not be used with `rules`' + }, + if: :has_rules? + + with_options allow_nil: true do + validates :when, + inclusion: { in: %w[on_success on_failure always], + message: 'should be on_success, on_failure or always' } + validates :extends, type: String + validates :rules, array_of_hashes: true + end + + validate on: :composed do + unless trigger.present? || bridge_needs.present? + errors.add(:config, 'should contain either a trigger or a needs:pipeline') + end + end + + validate on: :composed do + next unless bridge_needs.present? + next if bridge_needs.one? + + errors.add(:config, 'should contain at most one bridge need') + end + end + + entry :trigger, ::Gitlab::Ci::Config::Entry::Trigger, + description: 'CI/CD Bridge downstream trigger definition.', + inherit: false + + entry :needs, ::Gitlab::Ci::Config::Entry::Needs, + description: 'CI/CD Bridge needs dependency definition.', + inherit: false, + metadata: { allowed_needs: %i[job bridge] } + + entry :stage, ::Gitlab::Ci::Config::Entry::Stage, + description: 'Pipeline stage this job will be executed into.', + inherit: false + + entry :only, ::Gitlab::Ci::Config::Entry::Policy, + description: 'Refs policy this job will be executed for.', + default: ::Gitlab::Ci::Config::Entry::Policy::DEFAULT_ONLY, + inherit: false + + entry :except, ::Gitlab::Ci::Config::Entry::Policy, + description: 'Refs policy this job will be executed for.', + inherit: false + + entry :rules, ::Gitlab::Ci::Config::Entry::Rules, + description: 'List of evaluable Rules to determine job inclusion.', + inherit: false, + metadata: { + allowed_when: %w[on_success on_failure always never manual delayed].freeze + } + + entry :variables, ::Gitlab::Ci::Config::Entry::Variables, + description: 'Environment variables available for this job.', + inherit: false + + helpers(*ALLOWED_KEYS) + attributes(*ALLOWED_KEYS) + + def self.matching?(name, config) + !name.to_s.start_with?('.') && + config.is_a?(Hash) && + (config.key?(:trigger) || config.key?(:needs)) + end + + def self.visible? + true + end + + def compose!(deps = nil) + super do + has_workflow_rules = deps&.workflow&.has_rules? + + # If workflow:rules: or rules: are used + # they are considered not compatible + # with `only/except` defaults + # + # Context: https://gitlab.com/gitlab-org/gitlab/merge_requests/21742 + if has_rules? || has_workflow_rules + # Remove only/except defaults + # defaults are not considered as defined + @entries.delete(:only) unless only_defined? + @entries.delete(:except) unless except_defined? + end + end + end + + def has_rules? + @config&.key?(:rules) + end + + def name + @metadata[:name] + end + + def value + { name: name, + trigger: (trigger_value if trigger_defined?), + needs: (needs_value if needs_defined?), + ignore: !!allow_failure, + stage: stage_value, + when: when_value, + extends: extends_value, + variables: (variables_value if variables_defined?), + rules: (rules_value if has_rules?), + only: only_value, + except: except_value }.compact + end + + def bridge_needs + needs_value[:bridge] if needs_value + end + + private + + def overwrite_entry(deps, key, current_entry) + deps.default[key] unless current_entry.specified? + end + end + end + end + end +end diff --git a/lib/gitlab/ci/config/entry/jobs.rb b/lib/gitlab/ci/config/entry/jobs.rb index b517dae4d2e..1d3036189b0 100644 --- a/lib/gitlab/ci/config/entry/jobs.rb +++ b/lib/gitlab/ci/config/entry/jobs.rb @@ -36,7 +36,7 @@ module Gitlab end end - TYPES = [Entry::Hidden, Entry::Job].freeze + TYPES = [Entry::Hidden, Entry::Job, Entry::Bridge].freeze private_constant :TYPES @@ -77,5 +77,3 @@ module Gitlab end end end - -::Gitlab::Ci::Config::Entry::Jobs.prepend_if_ee('::EE::Gitlab::Ci::Config::Entry::Jobs') diff --git a/lib/gitlab/ci/config/entry/trigger.rb b/lib/gitlab/ci/config/entry/trigger.rb new file mode 100644 index 00000000000..7202784842a --- /dev/null +++ b/lib/gitlab/ci/config/entry/trigger.rb @@ -0,0 +1,93 @@ +# frozen_string_literal: true + +module Gitlab + module Ci + class Config + module Entry + ## + # Entry that represents a cross-project downstream trigger. + # + class Trigger < ::Gitlab::Config::Entry::Simplifiable + strategy :SimpleTrigger, if: -> (config) { config.is_a?(String) } + strategy :ComplexTrigger, if: -> (config) { config.is_a?(Hash) } + + class SimpleTrigger < ::Gitlab::Config::Entry::Node + include ::Gitlab::Config::Entry::Validatable + + validations { validates :config, presence: true } + + def value + { project: @config } + end + end + + class ComplexTrigger < ::Gitlab::Config::Entry::Simplifiable + strategy :CrossProjectTrigger, if: -> (config) { !config.key?(:include) } + + strategy :SameProjectTrigger, if: -> (config) do + ::Feature.enabled?(:ci_parent_child_pipeline, default_enabled: true) && + config.key?(:include) + end + + class CrossProjectTrigger < ::Gitlab::Config::Entry::Node + include ::Gitlab::Config::Entry::Validatable + include ::Gitlab::Config::Entry::Attributable + + ALLOWED_KEYS = %i[project branch strategy].freeze + attributes :project, :branch, :strategy + + validations do + validates :config, presence: true + validates :config, allowed_keys: ALLOWED_KEYS + validates :project, presence: true + validates :branch, type: String, allow_nil: true + validates :strategy, type: String, inclusion: { in: %w[depend], message: 'should be depend' }, allow_nil: true + end + end + + class SameProjectTrigger < ::Gitlab::Config::Entry::Node + include ::Gitlab::Config::Entry::Validatable + include ::Gitlab::Config::Entry::Attributable + include ::Gitlab::Config::Entry::Configurable + + INCLUDE_MAX_SIZE = 3 + ALLOWED_KEYS = %i[strategy include].freeze + attributes :strategy + + validations do + validates :config, presence: true + validates :config, allowed_keys: ALLOWED_KEYS + validates :strategy, type: String, inclusion: { in: %w[depend], message: 'should be depend' }, allow_nil: true + end + + entry :include, ::Gitlab::Ci::Config::Entry::Includes, + description: 'List of external YAML files to include.', + reserved: true, + metadata: { max_size: INCLUDE_MAX_SIZE } + + def value + @config + end + end + + class UnknownStrategy < ::Gitlab::Config::Entry::Node + def errors + if ::Feature.enabled?(:ci_parent_child_pipeline, default_enabled: true) + ['config must specify either project or include'] + else + ['config must specify project'] + end + end + end + end + + class UnknownStrategy < ::Gitlab::Config::Entry::Node + def errors + ["#{location} has to be either a string or a hash"] + end + end + end + end + end + end +end diff --git a/lib/tasks/gitlab/import_export/import.rake b/lib/tasks/gitlab/import_export/import.rake index 8fe61df605a..c832cba0287 100644 --- a/lib/tasks/gitlab/import_export/import.rake +++ b/lib/tasks/gitlab/import_export/import.rake @@ -7,12 +7,12 @@ # 2. Performs Sidekiq job synchronously # # @example -# bundle exec rake "gitlab:import_export:import[root, root, imported_project, /path/to/file.tar.gz]" +# bundle exec rake "gitlab:import_export:import[root, root, imported_project, /path/to/file.tar.gz, true]" # namespace :gitlab do namespace :import_export do desc 'GitLab | Import/Export | EXPERIMENTAL | Import large project archives' - task :import, [:username, :namespace_path, :project_path, :archive_path] => :gitlab_environment do |_t, args| + task :import, [:username, :namespace_path, :project_path, :archive_path, :measurement_enabled] => :gitlab_environment do |_t, args| # Load it here to avoid polluting Rake tasks with Sidekiq test warnings require 'sidekiq/testing' @@ -26,7 +26,8 @@ namespace :gitlab do namespace_path: args.namespace_path, project_path: args.project_path, username: args.username, - file_path: args.archive_path + file_path: args.archive_path, + measurement_enabled: args.measurement_enabled == 'true' ).import end end @@ -38,6 +39,7 @@ class GitlabProjectImport @file_path = opts.fetch(:file_path) @namespace = Namespace.find_by_full_path(opts.fetch(:namespace_path)) @current_user = User.find_by_username(opts.fetch(:username)) + @measurement_enabled = opts.fetch(:measurement_enabled) end def import @@ -72,6 +74,54 @@ class GitlabProjectImport RequestStore.clear! end + def with_count_queries(&block) + count = 0 + + counter_f = ->(name, started, finished, unique_id, payload) { + unless payload[:name].in? %w[CACHE SCHEMA] + count += 1 + end + } + + ActiveSupport::Notifications.subscribed(counter_f, "sql.active_record", &block) + + puts "Number of sql calls: #{count}" + end + + def with_gc_counter + gc_counts_before = GC.stat.select { |k, v| k =~ /count/ } + yield + gc_counts_after = GC.stat.select { |k, v| k =~ /count/ } + stats = gc_counts_before.merge(gc_counts_after) { |k, vb, va| va - vb } + puts "Total GC count: #{stats[:count]}" + puts "Minor GC count: #{stats[:minor_gc_count]}" + puts "Major GC count: #{stats[:major_gc_count]}" + end + + def with_measure_time + timing = Benchmark.realtime do + yield + end + + time = Time.at(timing).utc.strftime("%H:%M:%S") + puts "Time to finish: #{time}" + end + + def with_measuring + puts "Measuring enabled..." + with_gc_counter do + with_count_queries do + with_measure_time do + yield + end + end + end + end + + def measurement_enabled? + @measurement_enabled != false + end + # We want to ensure that all Sidekiq jobs are executed # synchronously as part of that process. # This ensures that all expensive operations do not escape @@ -79,8 +129,13 @@ class GitlabProjectImport def with_isolated_sidekiq_job Sidekiq::Testing.fake! do with_request_store do + # If you are attempting to import a large project into a development environment, + # you may see Gitaly throw an error about too many calls or invocations. + # This is due to a n+1 calls limit being set for development setups (not enforced in production) + # https://gitlab.com/gitlab-org/gitlab/-/merge_requests/24475#note_283090635 + # For development setups, this code-path will be excluded from n+1 detection. ::Gitlab::GitalyClient.allow_n_plus_1_calls do - yield + measurement_enabled? ? with_measuring { yield } : yield end end diff --git a/locale/gitlab.pot b/locale/gitlab.pot index c64785c7e44..12716eaa888 100644 --- a/locale/gitlab.pot +++ b/locale/gitlab.pot @@ -16773,6 +16773,9 @@ msgstr "" msgid "Security Reports|Oops, something doesn't seem right." msgstr "" +msgid "Security Reports|Security reports can only be accessed by authorized users." +msgstr "" + msgid "Security Reports|There was an error adding the comment." msgstr "" @@ -16797,6 +16800,12 @@ msgstr "" msgid "Security Reports|Undo dismiss" msgstr "" +msgid "Security Reports|You do not have sufficient permissions to access this report" +msgstr "" + +msgid "Security Reports|You must sign in as an authorized user to see this report" +msgstr "" + msgid "Security configuration help link" msgstr "" diff --git a/spec/lib/gitlab/ci/config/entry/bridge_spec.rb b/spec/lib/gitlab/ci/config/entry/bridge_spec.rb new file mode 100644 index 00000000000..07590556db8 --- /dev/null +++ b/spec/lib/gitlab/ci/config/entry/bridge_spec.rb @@ -0,0 +1,227 @@ +# frozen_string_literal: true + +require 'spec_helper' + +describe Gitlab::Ci::Config::Entry::Bridge do + subject { described_class.new(config, name: :my_bridge) } + + it_behaves_like 'with inheritable CI config' do + let(:inheritable_key) { 'default' } + let(:inheritable_class) { Gitlab::Ci::Config::Entry::Default } + + # These are entries defined in Default + # that we know that we don't want to inherit + # as they do not have sense in context of Bridge + let(:ignored_inheritable_columns) do + %i[before_script after_script image services cache interruptible timeout + retry tags artifacts] + end + end + + describe '.matching?' do + subject { described_class.matching?(name, config) } + + context 'when config is not a hash' do + let(:name) { :my_trigger } + let(:config) { 'string' } + + it { is_expected.to be_falsey } + end + + context 'when config is a regular job' do + let(:name) { :my_trigger } + let(:config) do + { script: 'ls -al' } + end + + it { is_expected.to be_falsey } + + context 'with rules' do + let(:config) do + { + script: 'ls -al', + rules: [{ if: '$VAR == "value"', when: 'always' }] + } + end + + it { is_expected.to be_falsey } + end + end + + context 'when config is a bridge job' do + let(:name) { :my_trigger } + let(:config) do + { trigger: 'other-project' } + end + + it { is_expected.to be_truthy } + + context 'with rules' do + let(:config) do + { + trigger: 'other-project', + rules: [{ if: '$VAR == "value"', when: 'always' }] + } + end + + it { is_expected.to be_truthy } + end + end + + context 'when config is a hidden job' do + let(:name) { '.my_trigger' } + let(:config) do + { trigger: 'other-project' } + end + + it { is_expected.to be_falsey } + end + end + + describe '.new' do + before do + subject.compose! + end + + let(:base_config) do + { + trigger: { project: 'some/project', branch: 'feature' }, + extends: '.some-key', + stage: 'deploy', + variables: { VARIABLE: '123' } + } + end + + context 'when trigger config is a non-empty string' do + let(:config) { { trigger: 'some/project' } } + + describe '#valid?' do + it { is_expected.to be_valid } + end + + describe '#value' do + it 'is returns a bridge job configuration' do + expect(subject.value).to eq(name: :my_bridge, + trigger: { project: 'some/project' }, + ignore: false, + stage: 'test', + only: { refs: %w[branches tags] }) + end + end + end + + context 'when bridge trigger is a hash' do + let(:config) do + { trigger: { project: 'some/project', branch: 'feature' } } + end + + describe '#valid?' do + it { is_expected.to be_valid } + end + + describe '#value' do + it 'is returns a bridge job configuration hash' do + expect(subject.value).to eq(name: :my_bridge, + trigger: { project: 'some/project', + branch: 'feature' }, + ignore: false, + stage: 'test', + only: { refs: %w[branches tags] }) + end + end + end + + context 'when bridge configuration contains trigger, when, extends, stage, only, except, and variables' do + let(:config) do + base_config.merge({ + when: 'always', + only: { variables: %w[$SOMEVARIABLE] }, + except: { refs: %w[feature] } + }) + end + + it { is_expected.to be_valid } + end + + context 'when bridge configuration uses rules' do + let(:config) { base_config.merge({ rules: [{ if: '$VAR == null', when: 'never' }] }) } + + it { is_expected.to be_valid } + end + + context 'when bridge configuration uses rules with job:when' do + let(:config) do + base_config.merge({ + when: 'always', + rules: [{ if: '$VAR == null', when: 'never' }] + }) + end + + it { is_expected.not_to be_valid } + end + + context 'when bridge configuration uses rules with only' do + let(:config) do + base_config.merge({ + only: { variables: %w[$SOMEVARIABLE] }, + rules: [{ if: '$VAR == null', when: 'never' }] + }) + end + + it { is_expected.not_to be_valid } + end + + context 'when bridge configuration uses rules with except' do + let(:config) do + base_config.merge({ + except: { refs: %w[feature] }, + rules: [{ if: '$VAR == null', when: 'never' }] + }) + end + + it { is_expected.not_to be_valid } + end + + context 'when bridge has only job needs' do + let(:config) do + { + needs: ['some_job'] + } + end + + describe '#valid?' do + it { is_expected.not_to be_valid } + end + end + + context 'when bridge config contains unknown keys' do + let(:config) { { unknown: 123 } } + + describe '#valid?' do + it { is_expected.not_to be_valid } + end + + describe '#errors' do + it 'is returns an error about unknown config key' do + expect(subject.errors.first) + .to match /config contains unknown keys: unknown/ + end + end + end + + context 'when bridge config contains build-specific attributes' do + let(:config) { { script: 'something' } } + + describe '#valid?' do + it { is_expected.not_to be_valid } + end + + describe '#errors' do + it 'returns an error message' do + expect(subject.errors.first) + .to match /contains unknown keys: script/ + end + end + end + end +end diff --git a/spec/lib/gitlab/ci/config/entry/jobs_spec.rb b/spec/lib/gitlab/ci/config/entry/jobs_spec.rb index 61c8956d41f..05249b7c717 100644 --- a/spec/lib/gitlab/ci/config/entry/jobs_spec.rb +++ b/spec/lib/gitlab/ci/config/entry/jobs_spec.rb @@ -5,27 +5,31 @@ require 'spec_helper' describe Gitlab::Ci::Config::Entry::Jobs do let(:entry) { described_class.new(config) } + let(:config) do + { + '.hidden_job'.to_sym => { script: 'something' }, + '.hidden_bridge'.to_sym => { trigger: 'my/project' }, + regular_job: { script: 'something' }, + my_trigger: { trigger: 'my/project' } + } + end + describe '.all_types' do subject { described_class.all_types } it { is_expected.to include(::Gitlab::Ci::Config::Entry::Hidden) } it { is_expected.to include(::Gitlab::Ci::Config::Entry::Job) } + it { is_expected.to include(::Gitlab::Ci::Config::Entry::Bridge) } end describe '.find_type' do using RSpec::Parameterized::TableSyntax - let(:config) do - { - '.hidden_job'.to_sym => { script: 'something' }, - regular_job: { script: 'something' }, - invalid_job: 'text' - } - end - where(:name, :type) do :'.hidden_job' | ::Gitlab::Ci::Config::Entry::Hidden + :'.hidden_bridge' | ::Gitlab::Ci::Config::Entry::Hidden :regular_job | ::Gitlab::Ci::Config::Entry::Job + :my_trigger | ::Gitlab::Ci::Config::Entry::Bridge :invalid_job | nil end @@ -42,8 +46,6 @@ describe Gitlab::Ci::Config::Entry::Jobs do end context 'when entry config value is correct' do - let(:config) { { rspec: { script: 'rspec' } } } - describe '#valid?' do it 'is valid' do expect(entry).to be_valid @@ -88,43 +90,41 @@ describe Gitlab::Ci::Config::Entry::Jobs do entry.compose! end - let(:config) do - { rspec: { script: 'rspec' }, - spinach: { script: 'spinach' }, - '.hidden'.to_sym => {} } - end - describe '#value' do it 'returns key value' do expect(entry.value).to eq( - rspec: { name: :rspec, - script: %w[rspec], - ignore: false, - stage: 'test', - only: { refs: %w[branches tags] }, - variables: {} }, - spinach: { name: :spinach, - script: %w[spinach], - ignore: false, - stage: 'test', - only: { refs: %w[branches tags] }, - variables: {} }) + my_trigger: { + ignore: false, + name: :my_trigger, + only: { refs: %w[branches tags] }, + stage: 'test', + trigger: { project: 'my/project' } + }, + regular_job: { + ignore: false, + name: :regular_job, + only: { refs: %w[branches tags] }, + script: ['something'], + stage: 'test', + variables: {} + }) end end describe '#descendants' do it 'creates valid descendant nodes' do - expect(entry.descendants.count).to eq 3 - expect(entry.descendants.first(2)) - .to all(be_an_instance_of(Gitlab::Ci::Config::Entry::Job)) - expect(entry.descendants.last) - .to be_an_instance_of(Gitlab::Ci::Config::Entry::Hidden) + expect(entry.descendants.map(&:class)).to eq [ + Gitlab::Ci::Config::Entry::Hidden, + Gitlab::Ci::Config::Entry::Hidden, + Gitlab::Ci::Config::Entry::Job, + Gitlab::Ci::Config::Entry::Bridge + ] end end describe '#value' do it 'returns value of visible jobs only' do - expect(entry.value.keys).to eq [:rspec, :spinach] + expect(entry.value.keys).to eq [:regular_job, :my_trigger] end end end diff --git a/spec/lib/gitlab/ci/config/entry/trigger_spec.rb b/spec/lib/gitlab/ci/config/entry/trigger_spec.rb new file mode 100644 index 00000000000..752c3f59a95 --- /dev/null +++ b/spec/lib/gitlab/ci/config/entry/trigger_spec.rb @@ -0,0 +1,164 @@ +# frozen_string_literal: true + +require 'spec_helper' + +describe Gitlab::Ci::Config::Entry::Trigger do + subject { described_class.new(config) } + + context 'when trigger config is a non-empty string' do + let(:config) { 'some/project' } + + describe '#valid?' do + it { is_expected.to be_valid } + end + + describe '#value' do + it 'returns a trigger configuration hash' do + expect(subject.value).to eq(project: 'some/project') + end + end + end + + context 'when trigger config an empty string' do + let(:config) { '' } + + describe '#valid?' do + it { is_expected.not_to be_valid } + end + + describe '#errors' do + it 'returns an error about an empty config' do + expect(subject.errors.first) + .to match /config can't be blank/ + end + end + end + + context 'when trigger is a hash' do + context 'when branch is provided' do + let(:config) { { project: 'some/project', branch: 'feature' } } + + describe '#valid?' do + it { is_expected.to be_valid } + end + + describe '#value' do + it 'returns a trigger configuration hash' do + expect(subject.value) + .to eq(project: 'some/project', branch: 'feature') + end + end + end + + context 'when strategy is provided' do + context 'when strategy is depend' do + let(:config) { { project: 'some/project', strategy: 'depend' } } + + describe '#valid?' do + it { is_expected.to be_valid } + end + + describe '#value' do + it 'returns a trigger configuration hash' do + expect(subject.value) + .to eq(project: 'some/project', strategy: 'depend') + end + end + end + + context 'when strategy is invalid' do + let(:config) { { project: 'some/project', strategy: 'notdepend' } } + + describe '#valid?' do + it { is_expected.not_to be_valid } + end + + describe '#errors' do + it 'returns an error about unknown config key' do + expect(subject.errors.first) + .to match /trigger strategy should be depend/ + end + end + end + end + + describe '#include' do + context 'with simple include' do + let(:config) { { include: 'path/to/config.yml' } } + + it { is_expected.to be_valid } + + it 'returns a trigger configuration hash' do + expect(subject.value).to eq(include: 'path/to/config.yml' ) + end + end + + context 'with project' do + let(:config) { { project: 'some/project', include: 'path/to/config.yml' } } + + it { is_expected.not_to be_valid } + + it 'returns an error' do + expect(subject.errors.first) + .to match /config contains unknown keys: project/ + end + end + + context 'with branch' do + let(:config) { { branch: 'feature', include: 'path/to/config.yml' } } + + it { is_expected.not_to be_valid } + + it 'returns an error' do + expect(subject.errors.first) + .to match /config contains unknown keys: branch/ + end + end + + context 'when feature flag is off' do + before do + stub_feature_flags(ci_parent_child_pipeline: false) + end + + let(:config) { { include: 'path/to/config.yml' } } + + it 'is returns an error if include is used' do + expect(subject.errors.first) + .to match /config must specify project/ + end + end + end + + context 'when config contains unknown keys' do + let(:config) { { project: 'some/project', unknown: 123 } } + + describe '#valid?' do + it { is_expected.not_to be_valid } + end + + describe '#errors' do + it 'returns an error about unknown config key' do + expect(subject.errors.first) + .to match /config contains unknown keys: unknown/ + end + end + end + end + + context 'when trigger configuration is not valid' do + context 'when branch is not provided' do + let(:config) { 123 } + + describe '#valid?' do + it { is_expected.not_to be_valid } + end + + describe '#errors' do + it 'returns an error message' do + expect(subject.errors.first) + .to match /has to be either a string or a hash/ + end + end + end + end +end diff --git a/spec/models/ci/bridge_spec.rb b/spec/models/ci/bridge_spec.rb index 1a97dd60c0e..c9d6687f0ea 100644 --- a/spec/models/ci/bridge_spec.rb +++ b/spec/models/ci/bridge_spec.rb @@ -63,6 +63,26 @@ describe Ci::Bridge do end end + describe 'state machine transitions' do + context 'when bridge points towards downstream' do + it 'schedules downstream pipeline creation' do + expect(bridge).to receive(:schedule_downstream_pipeline!) + + bridge.enqueue! + end + end + end + + describe 'state machine transitions' do + context 'when bridge points towards downstream' do + it 'schedules downstream pipeline creation' do + expect(bridge).to receive(:schedule_downstream_pipeline!) + + bridge.enqueue! + end + end + end + describe '#inherit_status_from_downstream!' do let(:downstream_pipeline) { build(:ci_pipeline, status: downstream_status) } diff --git a/spec/models/clusters/applications/prometheus_spec.rb b/spec/models/clusters/applications/prometheus_spec.rb index cf33d2b4273..ba344a234b8 100644 --- a/spec/models/clusters/applications/prometheus_spec.rb +++ b/spec/models/clusters/applications/prometheus_spec.rb @@ -4,6 +4,7 @@ require 'spec_helper' describe Clusters::Applications::Prometheus do include KubernetesHelpers + include StubRequests include_examples 'cluster application core specs', :clusters_applications_prometheus include_examples 'cluster application status specs', :clusters_applications_prometheus @@ -320,6 +321,16 @@ describe Clusters::Applications::Prometheus do it { is_expected.to be_falsey } end + + context 'when the kubernetes URL is blocked' do + before do + blocked_ip = '127.0.0.1' # localhost addresses are blocked by default + + stub_all_dns(cluster.platform.api_url, ip_address: blocked_ip) + end + + it { is_expected.to be_falsey } + end end context 'when a kubenetes client is not present' do diff --git a/spec/services/ci/create_cross_project_pipeline_service_spec.rb b/spec/services/ci/create_cross_project_pipeline_service_spec.rb index f90cdb55a7a..8a1763b30ed 100644 --- a/spec/services/ci/create_cross_project_pipeline_service_spec.rb +++ b/spec/services/ci/create_cross_project_pipeline_service_spec.rb @@ -342,9 +342,7 @@ describe Ci::CreateCrossProjectPipelineService, '#execute' do let(:service) { described_class.new(upstream_project, upstream_project.owner) } context 'that include the bridge job' do - # TODO: this is skipped because `trigger` keyword does not exist yet. - # enabling it in the next MR: https://gitlab.com/gitlab-org/gitlab/-/merge_requests/24393 - xit 'creates the downstream pipeline' do + it 'creates the downstream pipeline' do expect { service.execute(bridge) } .to change(downstream_project.ci_pipelines, :count).by(1) end diff --git a/spec/services/ci/create_pipeline_service/parent_child_pipeline_spec.rb b/spec/services/ci/create_pipeline_service/parent_child_pipeline_spec.rb new file mode 100644 index 00000000000..cfb68ffa585 --- /dev/null +++ b/spec/services/ci/create_pipeline_service/parent_child_pipeline_spec.rb @@ -0,0 +1,133 @@ +# frozen_string_literal: true + +require 'spec_helper' + +describe Ci::CreatePipelineService, '#execute' do + set(:project) { create(:project, :repository) } + set(:user) { create(:user) } + let(:ref_name) { 'master' } + + let(:service) do + params = { ref: ref_name, + before: '00000000', + after: project.commit.id, + commits: [{ message: 'some commit' }] } + + described_class.new(project, user, params) + end + + before do + project.add_developer(user) + stub_ci_pipeline_to_return_yaml_file + end + + describe 'child pipeline triggers' do + before do + stub_ci_pipeline_yaml_file <<~YAML + test: + script: rspec + + deploy: + variables: + CROSS: downstream + stage: deploy + trigger: + include: + - local: path/to/child.yml + YAML + end + + it 'creates bridge jobs correctly' do + pipeline = create_pipeline! + + test = pipeline.statuses.find_by(name: 'test') + bridge = pipeline.statuses.find_by(name: 'deploy') + + expect(pipeline).to be_persisted + expect(test).to be_a Ci::Build + expect(bridge).to be_a Ci::Bridge + expect(bridge.stage).to eq 'deploy' + expect(pipeline.statuses).to match_array [test, bridge] + expect(bridge.options).to eq( + 'trigger' => { 'include' => [{ 'local' => 'path/to/child.yml' }] } + ) + expect(bridge.yaml_variables) + .to include(key: 'CROSS', value: 'downstream', public: true) + end + end + + describe 'child pipeline triggers' do + context 'when YAML is valid' do + before do + stub_ci_pipeline_yaml_file <<~YAML + test: + script: rspec + + deploy: + variables: + CROSS: downstream + stage: deploy + trigger: + include: + - local: path/to/child.yml + YAML + end + + it 'creates bridge jobs correctly' do + pipeline = create_pipeline! + + test = pipeline.statuses.find_by(name: 'test') + bridge = pipeline.statuses.find_by(name: 'deploy') + + expect(pipeline).to be_persisted + expect(test).to be_a Ci::Build + expect(bridge).to be_a Ci::Bridge + expect(bridge.stage).to eq 'deploy' + expect(pipeline.statuses).to match_array [test, bridge] + expect(bridge.options).to eq( + 'trigger' => { 'include' => [{ 'local' => 'path/to/child.yml' }] } + ) + expect(bridge.yaml_variables) + .to include(key: 'CROSS', value: 'downstream', public: true) + end + end + + context 'when YAML is invalid' do + let(:config) do + { + test: { script: 'rspec' }, + deploy: { + trigger: { include: included_files } + } + } + end + + let(:included_files) do + Array.new(include_max_size + 1) do |index| + { local: "file#{index}.yml" } + end + end + + let(:include_max_size) do + Gitlab::Ci::Config::Entry::Trigger::ComplexTrigger::SameProjectTrigger::INCLUDE_MAX_SIZE + end + + before do + stub_ci_pipeline_yaml_file(YAML.dump(config)) + end + + it 'returns errors' do + pipeline = create_pipeline! + + expect(pipeline.errors.full_messages.first).to match(/trigger:include config is too long/) + expect(pipeline.failure_reason).to eq 'config_error' + expect(pipeline).to be_persisted + expect(pipeline.status).to eq 'failed' + end + end + end + + def create_pipeline! + service.execute(:push) + end +end diff --git a/spec/support/shared_examples/tasks/gitlab/import_export/import_measurement_shared_examples.rb b/spec/support/shared_examples/tasks/gitlab/import_export/import_measurement_shared_examples.rb new file mode 100644 index 00000000000..e232f237df9 --- /dev/null +++ b/spec/support/shared_examples/tasks/gitlab/import_export/import_measurement_shared_examples.rb @@ -0,0 +1,31 @@ +# frozen_string_literal: true + +RSpec.shared_examples 'import measurement' do + context 'when measurement is enabled' do + let(:measurement_enabled) { true } + + it 'prints measurement results' do + expect { subject }.to output(including('Measuring enabled...', 'Number of sql calls:', 'Total GC count:', 'Total GC count:')).to_stdout + end + end + + context 'when measurement is not enabled' do + let(:measurement_enabled) { false } + + it 'does not output measurement results' do + expect { subject }.not_to output(/Measuring enabled.../).to_stdout + end + end + + context 'when measurement is not provided' do + let(:task_params) { [username, namespace_path, project_name, archive_path] } + + it 'does not output measurement results' do + expect { subject }.not_to output(/Measuring enabled.../).to_stdout + end + + it 'does not raise any exception' do + expect { subject }.not_to raise_error + end + end +end diff --git a/spec/tasks/gitlab/import_export/import_rake_spec.rb b/spec/tasks/gitlab/import_export/import_rake_spec.rb index 14e53273a5b..3a819d23299 100644 --- a/spec/tasks/gitlab/import_export/import_rake_spec.rb +++ b/spec/tasks/gitlab/import_export/import_rake_spec.rb @@ -6,7 +6,8 @@ describe 'gitlab:import_export:import rake task' do let(:username) { 'root' } let(:namespace_path) { username } let!(:user) { create(:user, username: username) } - let(:task_params) { [username, namespace_path, project_name, archive_path] } + let(:measurement_enabled) { false } + let(:task_params) { [username, namespace_path, project_name, archive_path, measurement_enabled] } let(:project) { Project.find_by_full_path("#{namespace_path}/#{project_name}") } before do @@ -68,6 +69,8 @@ describe 'gitlab:import_export:import rake task' do subject end + + it_behaves_like 'import measurement' end context 'when project import is invalid' do |