diff options
| -rw-r--r-- | doc/administration/troubleshooting/gitlab_rails_cheat_sheet.md | 17 | ||||
| -rw-r--r-- | doc/development/documentation/index.md | 6 | ||||
| -rw-r--r-- | doc/development/documentation/styleguide.md | 4 | ||||
| -rw-r--r-- | doc/development/elasticsearch.md | 2 | ||||
| -rw-r--r-- | doc/development/event_tracking/frontend.md | 2 | ||||
| -rw-r--r-- | doc/development/feature_flags/controls.md | 2 | ||||
| -rw-r--r-- | doc/development/gitaly.md | 2 | ||||
| -rw-r--r-- | qa/qa.rb | 8 | ||||
| -rw-r--r-- | qa/qa/resource/runner.rb | 4 | ||||
| -rw-r--r-- | qa/qa/service/docker_run/base.rb | 43 | ||||
| -rw-r--r-- | qa/qa/service/docker_run/gitlab_runner.rb | 58 | ||||
| -rw-r--r-- | qa/qa/service/docker_run/ldap.rb | 41 | ||||
| -rw-r--r-- | qa/qa/service/ldap.rb | 64 | ||||
| -rw-r--r-- | qa/qa/service/runner.rb | 73 | ||||
| -rw-r--r-- | qa/qa/specs/features/browser_ui/4_verify/pipeline/create_and_process_pipeline_spec.rb | 2 | ||||
| -rw-r--r-- | qa/qa/specs/features/browser_ui/4_verify/runner/register_runner_spec.rb | 2 | ||||
| -rw-r--r-- | qa/qa/specs/features/browser_ui/6_release/deploy_key/clone_using_deploy_key_spec.rb | 2 |
17 files changed, 179 insertions, 153 deletions
diff --git a/doc/administration/troubleshooting/gitlab_rails_cheat_sheet.md b/doc/administration/troubleshooting/gitlab_rails_cheat_sheet.md index a3b346f2981..34a5acbe7b7 100644 --- a/doc/administration/troubleshooting/gitlab_rails_cheat_sheet.md +++ b/doc/administration/troubleshooting/gitlab_rails_cheat_sheet.md @@ -892,6 +892,23 @@ queue = Sidekiq::Queue.new('repository_import') queue.each { |job| job.delete if <condition>} ``` +`<condition>` probably includes references to job arguments, which depend on the type of job in question. + +| queue | worker | job args | +| ----- | ------ | -------- | +| repository_import | RepositoryImportWorker | project_id | +| update_merge_requests | UpdateMergeRequestsWorker | project_id, user_id, oldrev, newrev, ref | + +**Example:** Delete all UpdateMergeRequestsWorker jobs associated with a merge request on project_id 125, +merging branch `ref/heads/my_branch`. + +```ruby +queue = Sidekiq::Queue.new('update_merge_requests') +queue.each { |job| job.delete if job.args[0]==125 and job.args[4]=='ref/heads/my_branch'} +``` + +**Note:** Running jobs will not be killed. Stop sidekiq before doing this, to get all matching jobs. + ### Enable debug logging of Sidekiq ```ruby diff --git a/doc/development/documentation/index.md b/doc/development/documentation/index.md index 4ac33fddfd2..ce75f6ee0ad 100644 --- a/doc/development/documentation/index.md +++ b/doc/development/documentation/index.md @@ -13,8 +13,8 @@ In addition to this page, the following resources can help you craft and contrib - [Workflows](workflow.md) - A landing page for our key workflows: - [Documentation process for feature changes](feature-change-workflow.md) - Adding required documentation when developing a GitLab feature. - [Documentation improvement workflow](improvement-workflow.md) - New content not associated with a new feature. -- [Markdown Guide](../../user/markdown.md) - A reference for all markdown syntax supported by GitLab. -- [Site architecture](site_architecture/index.md) - How <http://docs.gitlab.com> is built. +- [Markdown Guide](../../user/markdown.md) - A reference for all Markdown syntax supported by GitLab. +- [Site architecture](site_architecture/index.md) - How <https://docs.gitlab.com> is built. ## Source files and rendered web locations @@ -195,7 +195,7 @@ available online on 2018-09-15, but, as the feature freeze date has passed, if the MR does not have a "pick into 11.3" label, the milestone has to be changed to 11.4 and it will be shipped with all GitLab packages only on 2018-10-22, with GitLab 11.4. Meaning, it will only be available under `/help` from GitLab -11.4 onwards, but available on <https://docs.gitlab.com/archives/> on the same day it was merged. +11.4 onwards, but available on <https://docs.gitlab.com/> on the same day it was merged. ### Linking to `/help` diff --git a/doc/development/documentation/styleguide.md b/doc/development/documentation/styleguide.md index c3bbf86492f..a348de58fe3 100644 --- a/doc/development/documentation/styleguide.md +++ b/doc/development/documentation/styleguide.md @@ -598,7 +598,7 @@ You can link any up-to-date video that is useful to the GitLab user. > [Introduced](https://gitlab.com/gitlab-org/gitlab-docs/merge_requests/472) in GitLab 12.1. -The [GitLab docs site](https://docs.gitlab.com) supports embedded videos. +The [GitLab Docs site](https://docs.gitlab.com) supports embedded videos. You can only embed videos from [GitLab's official YouTube account](https://www.youtube.com/channel/UCnMGQ8QHMAnVIsI3xJrihhg). @@ -693,7 +693,7 @@ use the following markup for highlighting. _Note that the alert boxes only work for one paragraph only. Multiple paragraphs, lists, headers, etc will not render correctly. For multiple lines, use blockquotes instead._ -Alert boxes only render properly on the GitLab Docs site (<http://docs.gitlab.com>). +Alert boxes only render on the GitLab Docs site (<https://docs.gitlab.com>). Within GitLab itself, they will appear as plain markdown text (like the examples above the rendered versions, below). diff --git a/doc/development/elasticsearch.md b/doc/development/elasticsearch.md index a63712dac20..1375bd6d56d 100644 --- a/doc/development/elasticsearch.md +++ b/doc/development/elasticsearch.md @@ -3,7 +3,7 @@ This area is to maintain a compendium of useful information when working with Elasticsearch. Information on how to enable Elasticsearch and perform the initial indexing is in -the [Elasticsearch integration documentation](../integration/elasticsearch.md#enabling-elasticsearch) +the [Elasticsearch integration documentation](../integration/elasticsearch.md#enabling-elasticsearch). ## Deep Dive diff --git a/doc/development/event_tracking/frontend.md b/doc/development/event_tracking/frontend.md index a3a7b4ca152..c767efc65b2 100644 --- a/doc/development/event_tracking/frontend.md +++ b/doc/development/event_tracking/frontend.md @@ -1,6 +1,6 @@ # Frontend tracking guide -GitLab provides `Tracking`, an interface that wraps the [Snowplow JavaScript Tracker](https://github.com/snowplow/snowplow/wiki/javascript-tracker) for tracking custom events. There are a few ways to utilizing tracking, but each generally requires at minimum, a `category` and an `action`. Additional data can be provided that adheres to our [Feature instrumentation taxonomy](https://about.gitlab.com/handbook/product/feature-instrumentation/#taxonomy). +GitLab provides `Tracking`, an interface that wraps the [Snowplow JavaScript Tracker](https://github.com/snowplow/snowplow/wiki/javascript-tracker) for tracking custom events. There are a few ways to utilize tracking, but each generally requires at minimum, a `category` and an `action`. Additional data can be provided that adheres to our [Feature instrumentation taxonomy](https://about.gitlab.com/handbook/product/feature-instrumentation/#taxonomy). | field | type | default value | description | |:-----------|:-------|:---------------------------|:------------| diff --git a/doc/development/feature_flags/controls.md b/doc/development/feature_flags/controls.md index 39f8f1bfea6..f22c3bb1e37 100644 --- a/doc/development/feature_flags/controls.md +++ b/doc/development/feature_flags/controls.md @@ -54,7 +54,7 @@ If you get an error "Whoops! This action is not allowed. This incident will be reported." that means your Slack account is not allowed to change feature flags or you do not [have access](#access). -### Enabling feature for internal testing +### Enabling feature for preproduction testing As a first step in a feature rollout, you should enable the feature on <https://staging.gitlab.com> and <https://dev.gitlab.org>. diff --git a/doc/development/gitaly.md b/doc/development/gitaly.md index 6194274660c..64f283f69d9 100644 --- a/doc/development/gitaly.md +++ b/doc/development/gitaly.md @@ -17,7 +17,7 @@ In May 2019, Bob Van Landuyt hosted a [Deep Dive] on GitLab's [Gitaly project] a Start by reading the Gitaly repository's [Beginner's guide to Gitaly contributions](https://gitlab.com/gitlab-org/gitaly/blob/master/doc/beginners_guide.md). -It describes how to setup Gitaly, the various components of Gitaly and what they do, and how to run its test suites. +It describes how to set up Gitaly, the various components of Gitaly and what they do, and how to run its test suites. ## Developing new Git features @@ -396,8 +396,6 @@ module QA autoload :Shellout, 'qa/service/shellout' autoload :KubernetesCluster, 'qa/service/kubernetes_cluster' autoload :Omnibus, 'qa/service/omnibus' - autoload :Runner, 'qa/service/runner' - autoload :LDAP, 'qa/service/ldap' module ClusterProvider autoload :Base, 'qa/service/cluster_provider/base' @@ -405,6 +403,12 @@ module QA autoload :Minikube, 'qa/service/cluster_provider/minikube' autoload :K3d, 'qa/service/cluster_provider/k3d' end + + module DockerRun + autoload :Base, 'qa/service/docker_run/base' + autoload :LDAP, 'qa/service/docker_run/ldap' + autoload :GitlabRunner, 'qa/service/docker_run/gitlab_runner' + end end ## diff --git a/qa/qa/resource/runner.rb b/qa/qa/resource/runner.rb index 3f0eed7458a..1be2429bc04 100644 --- a/qa/qa/resource/runner.rb +++ b/qa/qa/resource/runner.rb @@ -29,7 +29,7 @@ module QA end def fabricate_via_api! - Service::Runner.new(name).tap do |runner| + Service::DockerRun::GitlabRunner.new(name).tap do |runner| runner.pull runner.token = @token ||= project.runners_token runner.address = Runtime::Scenario.gitlab_address @@ -46,7 +46,7 @@ module QA super - Service::Runner.new(name).remove! + Service::DockerRun::GitlabRunner.new(name).remove! end def api_delete_path diff --git a/qa/qa/service/docker_run/base.rb b/qa/qa/service/docker_run/base.rb new file mode 100644 index 00000000000..3f42c09ad2c --- /dev/null +++ b/qa/qa/service/docker_run/base.rb @@ -0,0 +1,43 @@ +# frozen_string_literal: true + +module QA + module Service + module DockerRun + class Base + include Service::Shellout + + def initialize + @network = Runtime::Scenario.attributes[:network] || 'test' + end + + def network + shell "docker network inspect #{@network}" + rescue CommandError + 'bridge' + else + @network + end + + def pull + shell "docker pull #{@image}" + end + + def host_name + "#{@name}.#{network}" + end + + def register! + raise NotImplementedError + end + + def remove! + shell "docker rm -f #{@name}" if running? + end + + def running? + `docker ps -f name=#{@name}`.include?(@name) + end + end + end + end +end diff --git a/qa/qa/service/docker_run/gitlab_runner.rb b/qa/qa/service/docker_run/gitlab_runner.rb new file mode 100644 index 00000000000..6856a5a8399 --- /dev/null +++ b/qa/qa/service/docker_run/gitlab_runner.rb @@ -0,0 +1,58 @@ +# frozen_string_literal: true + +require 'securerandom' + +module QA + module Service + module DockerRun + class GitlabRunner < Base + attr_accessor :token, :address, :tags, :image, :run_untagged + attr_writer :config + + def initialize(name) + @image = 'gitlab/gitlab-runner:alpine' + @name = name || "qa-runner-#{SecureRandom.hex(4)}" + @tags = %w[qa test] + @run_untagged = false + + super() + end + + def config + @config ||= <<~END + concurrent = 1 + check_interval = 0 + + [session_server] + session_timeout = 1800 + END + end + + def register! + shell <<~CMD.tr("\n", ' ') + docker run -d --rm --entrypoint=/bin/sh + --network #{network} --name #{@name} + -p 8093:8093 + -e CI_SERVER_URL=#{@address} + -e REGISTER_NON_INTERACTIVE=true + -e REGISTRATION_TOKEN=#{@token} + -e RUNNER_EXECUTOR=shell + -e RUNNER_TAG_LIST=#{@tags.join(',')} + -e RUNNER_NAME=#{@name} + #{@image} -c "#{register_command}" + CMD + end + + private + + def register_command + <<~CMD + printf '#{config.chomp.gsub(/\n/, "\\n").gsub('"', '\"')}' > /etc/gitlab-runner/config.toml && + gitlab-runner register --run-untagged=#{@run_untagged} && + gitlab-runner run + CMD + end + end + end + end +end diff --git a/qa/qa/service/docker_run/ldap.rb b/qa/qa/service/docker_run/ldap.rb new file mode 100644 index 00000000000..c33d75ff640 --- /dev/null +++ b/qa/qa/service/docker_run/ldap.rb @@ -0,0 +1,41 @@ +# frozen_string_literal: true + +module QA + module Service + module DockerRun + class LDAP < Base + def initialize(volume) + @image = 'osixia/openldap:latest' + @name = 'ldap-server' + @volume = volume + + super() + end + + def register! + shell <<~CMD.tr("\n", ' ') + docker run -d --rm + --network #{network} + --hostname #{host_name} + --name #{@name} + -p 389:389 + --volume #{volume_or_fixture(@volume)}:/container/service/slapd/assets/config/bootstrap/ldif/custom + #{@image} --copy-service + CMD + end + + def volume_or_fixture(volume_name) + if volume_exists?(volume_name) + volume_name + else + File.expand_path("../fixtures/ldap/#{volume_name}", __dir__) + end + end + + def volume_exists?(volume_name) + `docker volume ls -q -f name=#{volume_name}`.include?(volume_name) + end + end + end + end +end diff --git a/qa/qa/service/ldap.rb b/qa/qa/service/ldap.rb deleted file mode 100644 index 7a02cebeb8f..00000000000 --- a/qa/qa/service/ldap.rb +++ /dev/null @@ -1,64 +0,0 @@ -# frozen_string_literal: true - -module QA - module Service - class LDAP - include Service::Shellout - - def initialize(volume) - @image = 'osixia/openldap:latest' - @name = 'ldap-server' - @network = Runtime::Scenario.attributes[:network] || 'test' - @volume = volume - end - - def network - shell "docker network inspect #{@network}" - rescue CommandError - 'bridge' - else - @network - end - - def pull - shell "docker pull #{@image}" - end - - def host_name - "#{@name}.#{network}" - end - - def register! - shell <<~CMD.tr("\n", ' ') - docker run -d --rm - --network #{network} - --hostname #{host_name} - --name #{@name} - -p 389:389 - --volume #{volume_or_fixture(@volume)}:/container/service/slapd/assets/config/bootstrap/ldif/custom - #{@image} --copy-service - CMD - end - - def remove! - shell "docker rm -f #{@name}" if running? - end - - def running? - `docker ps -f name=#{@name}`.include?(@name) - end - - def volume_or_fixture(volume_name) - if volume_exists?(volume_name) - volume_name - else - File.expand_path("../fixtures/ldap/#{volume_name}", __dir__) - end - end - - def volume_exists?(volume_name) - `docker volume ls -q -f name=#{volume_name}`.include?(volume_name) - end - end - end -end diff --git a/qa/qa/service/runner.rb b/qa/qa/service/runner.rb deleted file mode 100644 index 6fc5984b12a..00000000000 --- a/qa/qa/service/runner.rb +++ /dev/null @@ -1,73 +0,0 @@ -# frozen_string_literal: true - -require 'securerandom' - -module QA - module Service - class Runner - include Service::Shellout - - attr_accessor :token, :address, :tags, :image, :run_untagged - attr_writer :config - - def initialize(name) - @image = 'gitlab/gitlab-runner:alpine' - @name = name || "qa-runner-#{SecureRandom.hex(4)}" - @network = Runtime::Scenario.attributes[:network] || 'test' - @tags = %w[qa test] - @run_untagged = false - end - - def config - @config ||= <<~END - concurrent = 1 - check_interval = 0 - - [session_server] - session_timeout = 1800 - END - end - - def network - shell "docker network inspect #{@network}" - rescue CommandError - 'bridge' - else - @network - end - - def pull - shell "docker pull #{@image}" - end - - def register! - shell <<~CMD.tr("\n", ' ') - docker run -d --rm --entrypoint=/bin/sh - --network #{network} --name #{@name} - -p 8093:8093 - -e CI_SERVER_URL=#{@address} - -e REGISTER_NON_INTERACTIVE=true - -e REGISTRATION_TOKEN=#{@token} - -e RUNNER_EXECUTOR=shell - -e RUNNER_TAG_LIST=#{@tags.join(',')} - -e RUNNER_NAME=#{@name} - #{@image} -c "#{register_command}" - CMD - end - - def remove! - shell "docker rm -f #{@name}" - end - - private - - def register_command - <<~CMD - printf '#{config.chomp.gsub(/\n/, "\\n").gsub('"', '\"')}' > /etc/gitlab-runner/config.toml && - gitlab-runner register --run-untagged=#{@run_untagged} && - gitlab-runner run - CMD - end - end - end -end diff --git a/qa/qa/specs/features/browser_ui/4_verify/pipeline/create_and_process_pipeline_spec.rb b/qa/qa/specs/features/browser_ui/4_verify/pipeline/create_and_process_pipeline_spec.rb index 2952a54ff5d..5d91b70082c 100644 --- a/qa/qa/specs/features/browser_ui/4_verify/pipeline/create_and_process_pipeline_spec.rb +++ b/qa/qa/specs/features/browser_ui/4_verify/pipeline/create_and_process_pipeline_spec.rb @@ -6,7 +6,7 @@ module QA let(:executor) { "qa-runner-#{Time.now.to_i}" } after do - Service::Runner.new(executor).remove! + Service::DockerRun::GitlabRunner.new(executor).remove! end it 'users creates a pipeline which gets processed' do diff --git a/qa/qa/specs/features/browser_ui/4_verify/runner/register_runner_spec.rb b/qa/qa/specs/features/browser_ui/4_verify/runner/register_runner_spec.rb index 900ddcb7f59..58f129b846d 100644 --- a/qa/qa/specs/features/browser_ui/4_verify/runner/register_runner_spec.rb +++ b/qa/qa/specs/features/browser_ui/4_verify/runner/register_runner_spec.rb @@ -6,7 +6,7 @@ module QA let(:executor) { "qa-runner-#{Time.now.to_i}" } after do - Service::Runner.new(executor).remove! + Service::DockerRun::GitlabRunner.new(executor).remove! end it 'user registers a new specific runner' do diff --git a/qa/qa/specs/features/browser_ui/6_release/deploy_key/clone_using_deploy_key_spec.rb b/qa/qa/specs/features/browser_ui/6_release/deploy_key/clone_using_deploy_key_spec.rb index 141166f6971..e45ce438fc2 100644 --- a/qa/qa/specs/features/browser_ui/6_release/deploy_key/clone_using_deploy_key_spec.rb +++ b/qa/qa/specs/features/browser_ui/6_release/deploy_key/clone_using_deploy_key_spec.rb @@ -26,7 +26,7 @@ module QA end after do - Service::Runner.new(@runner_name).remove! + Service::DockerRun::GitlabRunner.new(@runner_name).remove! end keys = [ |