Merge branch 'docs-debug-harishsr' into 'master'

Port 3 more debug docs to the docs project

Closes #63035

See merge request gitlab-org/gitlab-ce!31368

4 files changed, 1531 insertions, 10 deletions

diff --git a/doc/administration/index.md b/doc/administration/index.md
index 711eebcd61e..d557068e6c8 100644
--- a/doc/administration/index.md
+++ b/doc/administration/index.md
@@ -187,13 +187,29 @@ Learn how to install, configure, update, and maintain your GitLab instance.
 - [Debugging tips](troubleshooting/debug.md): Tips to debug problems when things go wrong
 - [Log system](logs.md): Where to look for logs.
 - [Sidekiq Troubleshooting](troubleshooting/sidekiq.md): Debug when Sidekiq appears hung and is not processing jobs.
-- Useful [diagnostics tools](troubleshooting/diagnostics_tools.md) that are sometimes used by the GitLab
-  Support team.
-- [Troubleshooting ElasticSearch](troubleshooting/elasticsearch.md): Tips to troubleshoot ElasticSearch.
-- [Kubernetes troubleshooting](troubleshooting/kubernetes_cheat_sheet.md): Commands and tips useful
-  for troubleshooting Kubernetes-related issues.
-- Useful links from the Support Team:
-  - [GitLab Developer Docs](https://docs.gitlab.com/ee/development/README.html).
-  - [Repairing and recovering broken Git repositories](https://git.seveas.net/repairing-and-recovering-broken-git-repositories.html).
-  - [Testing with OpenSSL](https://www.feistyduck.com/library/openssl-cookbook/online/ch-testing-with-openssl.html).
-  - [Strace zine](https://wizardzines.com/zines/strace/).
+- [Troubleshooting ElasticSearch](troubleshooting/elasticsearch.md)
+
+### Support Team Docs
+
+The GitLab Support Team has collected a lot of information about troubleshooting GitLab
+instances. These documents are normally used by the Support Team itself, or by customers
+with direct guidance from a Support Team member. GitLab administrators may find the
+information useful for troubleshooting, but if you are experiencing trouble with your
+GitLab instance, you should check your [support options](https://about.gitlab.com/support/)
+before referring to these documents.
+
+CAUTION: **Warning:**
+Using the commands listed in the documentation below could result in data loss or
+other damage to a GitLab instance, and should only be used by experienced administrators
+who are aware of the risks.
+
+- [Useful diagnostics tools](troubleshooting/diagnostics_tools.md)
+- [Useful Linux commands](troubleshooting/linux_cheat_sheet.md)
+- [Troubleshooting Kubernetes](troubleshooting/kubernetes_cheat_sheet.md)
+- [Guide to test environments](troubleshooting/test_environments.md) (for Support Engineers)
+- [GitLab rails console commands](troubleshooting/gitlab_rails_cheat_sheet.md) (for Support Engineers)
+- Useful links:
+  - [GitLab Developer Docs](../development/README.md)
+  - [Repairing and recovering broken Git repositories](https://git.seveas.net/repairing-and-recovering-broken-git-repositories.html)
+  - [Testing with OpenSSL](https://www.feistyduck.com/library/openssl-cookbook/online/ch-testing-with-openssl.html)
+  - [Strace zine](https://wizardzines.com/zines/strace/)
diff --git a/doc/administration/troubleshooting/gitlab_rails_cheat_sheet.md b/doc/administration/troubleshooting/gitlab_rails_cheat_sheet.md
new file mode 100644
index 00000000000..0c5611aa6cd
--- /dev/null
+++ b/doc/administration/troubleshooting/gitlab_rails_cheat_sheet.md
@@ -0,0 +1,1040 @@
+---
+type: reference
+---
+
+# GitLab Rails Console Cheat Sheet
+
+This is the GitLab Support Team's collection of information regarding the GitLab rails
+console, for use while troubleshooting. It is listed here for transparency,
+and it may be useful for users with experience with these tools. If you are currently
+having an issue with GitLab, it is highly recommended that you check your
+[support options](https://about.gitlab.com/support/) first, before attempting to use
+this information.
+
+CAUTION: **CAUTION:**
+Please note that some of these scripts could be damaging if not run correctly,
+or under the right conditions. We highly recommend running them under the
+guidance of a Support Engineer, or running them in a test environment with a
+backup of the instance ready to be restored, just in case.
+
+CAUTION: **CAUTION:**
+Please also note that as GitLab changes, changes to the code are inevitable,
+and so some scripts may not work as they once used to. These are not kept
+up-to-date as these scripts/commands were added as they were found/needed. As
+mentioned above, we recommend running these scripts under the supervision of a
+Support Engineer, who can also verify that they will continue to work as they
+should and, if needed, update the script for the latest version of GitLab.
+
+## Use the Rails Runner
+
+If the script you want to run is short, you can use the Rails Runner to avoid
+entering the rails console in the first place. Here's an example of its use:
+
+```bash
+gitlab-rails runner "RAILS_COMMAND"
+
+# Example with a 2-line script
+gitlab-rails runner "user = User.first; puts user.username"
+```
+
+## Enable debug logging on rails console
+
+```ruby
+Rails.logger.level = 0
+```
+
+## Enable debug logging for ActiveRecord (db issues)
+
+```ruby
+ActiveRecord::Base.logger = Logger.new(STDOUT)
+```
+
+## Temporarily Disable Timeout
+
+```ruby
+ActiveRecord::Base.connection.execute('SET statement_timeout TO 0')
+```
+
+## Find specific methods for an object
+
+```ruby
+Array.methods.select { |m| m.to_s.include? "sing" }
+Array.methods.grep(/sing/)
+```
+
+## Find method source
+
+Works for [non-instrumented methods](https://docs.gitlab.com/ce/development/instrumentation.html#checking-instrumented-methods):
+
+```ruby
+instance_of_object.method(:foo).source_location
+
+# Example for when we would call project.private?
+project.method(:private?).source_location
+```
+
+## Query an object
+
+```ruby
+o = Object.where('attribute like ?', 'ex')
+```
+
+## View all keys in cache
+
+```ruby
+Rails.cache.instance_variable_get(:@data).keys
+```
+
+## Rails console history
+
+```ruby
+puts Readline::HISTORY.to_a
+```
+
+## Profile a page
+
+```ruby
+# Before 11.6.0
+logger = Logger.new(STDOUT)
+admin_token = User.find_by_username('ADMIN_USERNAME').personal_access_tokens.first.token
+app.get("URL/?private_token=#{admin_token}")
+
+# From 11.6.0
+admin = User.find_by_username('ADMIN_USERNAME')
+url = "/url/goes/here"
+Gitlab::Profiler.with_user(admin) { app.get(url) }
+```
+
+## Using the GitLab profiler inside console (used as of 10.5)
+
+```ruby
+logger = Logger.new(STDOUT)
+admin = User.find_by_username('ADMIN_USERNAME')
+Gitlab::Profiler.profile('URL', logger: logger, user: admin)
+```
+
+## Time an operation
+
+```ruby
+# A single operation
+Benchmark.measure { <operation> }
+
+# A breakdown of multiple operations
+Benchmark.bm do |x|
+  x.report(:label1) { <operation_1> }
+  x.report(:label2) { <operation_2> }
+end
+```
+
+## Command Line
+
+### Check the GitLab version fast
+
+```bash
+grep -m 1 gitlab /opt/gitlab/version-manifest.txt
+```
+
+### Debugging SSH
+
+```bash
+GIT_SSH_COMMAND="ssh -vvv" git clone <repository>
+```
+
+### Debugging over HTTPS
+
+```bash
+GIT_CURL_VERBOSE=1 GIT_TRACE=1 git clone <repository>
+```
+
+## Projects
+
+### Find projects
+
+```ruby
+# A single project
+project = Project.find_by_full_path('PROJECT_PATH')
+
+# All projects in a particular namespace.  Can be a username, a group
+# ('gitlab-org'), or even include subgroups ('gitlab-org/distribution')
+namespace = Namespace.find_by_full_path('NAMESPACE_PATH')
+projects = namespace.all_projects
+```
+
+### Clear a project's cache
+
+```ruby
+ProjectCacheWorker.perform_async(project.id)
+```
+
+### Expire the .exists? cache
+
+```ruby
+project.repository.expire_exists_cache
+```
+
+### Make all projects private
+
+```ruby
+Project.update_all(visibility_level: 0)
+```
+
+### Find & remove projects that are pending deletion
+
+```ruby
+#
+# This section will list all the projects which are pending deletion
+#
+projects = Project.where(pending_delete: true)
+projects.each do |p|
+  puts "Project name: #{p.id}"
+  puts "Project name: #{p.name}"
+  puts "Repository path: #{p.repository.storage_path}"
+end
+
+#
+# Assign a user (the root user will do)
+#
+user = User.find_by_username('root')
+
+#
+# For each project listed repeat these two commands
+#
+
+# Find the project, update the xxx-changeme values from above
+project = Project.find_by_full_path('group-changeme/project-changeme')
+
+# Delete the project
+::Projects::DestroyService.new(project, user, {}).execute
+```
+
+Next, run `sudo gitlab-rake gitlab:cleanup:repos` on the command line to finish.
+
+### Destroy a project
+
+```ruby
+project = Project.find_by_full_path('')
+user = User.find_by_username('')
+ProjectDestroyWorker.perform_async(project.id, user.id, {})
+# or ProjectDestroyWorker.new.perform(project.id, user.id, {})
+# or Projects::DestroyService.new(project, user).execute
+```
+
+### Remove fork relationship manually
+
+```ruby
+p = Project.find_by_full_path('')
+u = User.find_by_username('')
+::Projects::UnlinkForkService.new(p, u).execute
+```
+
+### Make a project read-only (can only be done in the console)
+
+```ruby
+# Make a project read-only
+project.repository_read_only = true; project.save
+
+# OR
+project.update!(repository_read_only: true)
+```
+
+### Bulk update service integration password for _all_ projects
+
+For example, change the Jira user's password for all projects that have the Jira
+integration active:
+
+```ruby
+p = Project.find_by_sql("SELECT p.id FROM projects p LEFT JOIN services s ON p.id = s.project_id WHERE s.type = 'JiraService' AND s.active = true")
+
+p.each do |project|
+  project.jira_service.update_attribute(:password, '<your-new-password>')
+end
+```
+
+### Identify un-indexed projects
+
+```ruby
+Project.find_each do |project|
+  puts "id #{project.id}: #{project.namespace.name.to_s}/#{project.name.to_s}" if project.index_status.nil?
+end
+```
+
+## Imports / Exports
+
+```ruby
+# Find the project and get the error
+p = Project.find_by_full_path('<username-or-group>/<project-name>')
+
+p.import_error
+
+# To finish the import on GitLab running version before 11.6
+p.import_finish
+
+# To finish the import on GitLab running version 11.6 or after
+p.import_state.mark_as_failed("Failed manually through console.")
+```
+
+### Rename imported repository
+
+In a specific situation, an imported repository needed to be renamed. The Support
+Team was informed of a backup restore that failed on a single repository, which created
+the project with an empty repository. The project was successfully restored to a dev
+instance, then exported, and imported into a new project under a different name.
+
+The Support Team was able to transfer the incorrectly named imported project into the
+correctly named empty project using the steps below.
+
+Move the new repository to the empty repository:
+
+```bash
+mv /var/opt/gitlab/git-data/repositories/<group>/<new-project> /var/opt/gitlab/git-data/repositories/<group>/<empty-project>
+```
+
+Make sure the permissions are correct:
+
+```bash
+chown -R git:git <path-to-directory>.git
+```
+
+Clear the cache:
+
+```bash
+sudo gitlab-rake cache:clear
+```
+
+## Repository
+
+### Search sequence of pushes to a repository
+
+If it seems that a commit has gone "missing", search the sequence of pushes to a repository.
+[This StackOverflow article](https://stackoverflow.com/questions/13468027/the-mystery-of-the-missing-commit-across-merges)
+describes how you can end up in this state without a force push.
+
+If you look at the output from the sample code below for the target branch, you will
+see a discontinuity in the from/to commits as you step through the output. Each new
+push should be "from" the "to" SHA of the previous push. When this discontinuity happens,
+you will see two pushes with the same "from" SHA:
+
+```ruby
+p = Project.find_with_namespace('u/p')
+p.events.code_push.last(100).each do |e|
+  printf "%-20.20s %8s...%8s (%s)\n", e.data[:ref], e.data[:before], e.data[:after], e.author.try(:username)
+end
+```
+
+GitLab 9.5 and above:
+
+```ruby
+p = Project.find_by_full_path('u/p')
+p.events.code_push.last(100).each do |e|
+  printf "%-20.20s %8s...%8s (%s)\n", e.push_event_payload[:ref], e.push_event_payload[:commit_from], e.push_event_payload[:commit_to], e.author.try(:username)
+end
+```
+
+## Mirrors
+
+### Find mirrors with "bad decrypt" errors
+
+```ruby
+total = 0
+bad = []
+ProjectImportData.find_each do |data|
+  begin
+    total += 1
+    data.credentials
+  rescue => e
+    bad << data
+  end
+end
+
+puts "Bad count: #{bad.count} / #{total}"
+bad.each do |repo|
+  puts Project.find(repo.project_id).full_path
+end; bad.count
+```
+
+### Transfer mirror users and tokens to a single service account
+
+Use case: If you have multiple users using their own GitHub credentials to set up
+repository mirroring, mirroring breaks when people leave the company. Use this
+script to migrate disparate mirroring users and tokens into a single service account:
+
+```ruby
+svc_user = User.find_by(username: 'ourServiceUser')
+token = 'githubAccessToken'
+
+Project.where(mirror: true).each do |project|
+  import_url = project.import_url
+
+  # The url we want is https://token@project/path.git
+  repo_url = if import_url.include?('@')
+               # Case 1: The url is something like https://23423432@project/path.git
+               import_url.split('@').last
+             elsif import_url.include?('//')
+               # Case 2: The url is something like https://project/path.git
+               import_url.split('//').last
+             end
+
+  next unless repo_url
+
+  final_url = "https://#{token}@#{repo_url}"
+
+  project.mirror_user = svc_user
+  project.import_url = final_url
+  project.username_only_import_url = final_url
+  project.save
+end
+```
+
+## Users
+
+### Finding users
+
+```ruby
+# By username
+user = User.find_by(username: '')
+
+# By primary email
+user = User.find_by(email: '')
+
+# By any email (primary or secondary)
+user = User.find_by_any_email('')
+
+# Admins
+User.admins
+admin = User.admins.first
+```
+
+### Block
+
+```ruby
+User.find_by_username().block!
+```
+
+### Unblock
+
+```ruby
+User.find_by_username().active
+```
+
+### Skip reconfirmation
+
+```ruby
+user = User.find_by_username ''
+user.skip_reconfirmation!
+```
+
+### Get an admin token
+
+```ruby
+# Get the first admin's first access token (no longer works on 11.9+. see: https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/22743)
+User.where(admin:true).first.personal_access_tokens.first.token
+
+# Get the first admin's private token (no longer works on 10.2+)
+User.where(admin:true).private_token
+```
+
+### Create personal access token
+
+```ruby
+personal_access_token = User.find(123).personal_access_tokens.create(
+  name: 'apitoken',
+  impersonation: false,
+  scopes: [:api]
+)
+
+personal_access_token.token
+```
+
+You might also want to manually set the token string:
+
+```ruby
+User.find(123).personal_access_tokens.create(
+  name: 'apitoken',
+  token_digest: Gitlab::CryptoHelper.sha256('some-token-string-here'),
+  impersonation: false,
+  scopes: [:api]
+)
+```
+
+### Disable 2FA on a user
+
+```ruby
+user = User.find_by_username('username')
+user.disable_two_factor!
+```
+
+### Active users & Historical users
+
+```ruby
+# Active users on the instance, now
+User.active.count
+
+# The historical max on the instance as of the past year
+::HistoricalData.max_historical_user_count
+```
+
+```bash
+# Using curl and jq (up to a max 100, see [pagination](https://docs.gitlab.com/ee/api/#pagination)
+curl --silent --header "Private-Token: ********************" "https://gitlab.example.com/api/v4/users?per_page=100&active" | jq --compact-output '.[] | [.id,.name,.username]'
+```
+
+### Block or Delete Users that have no projects or groups
+
+```ruby
+users = User.where('id NOT IN (select distinct(user_id) from project_authorizations)')
+
+# How many users will be removed?
+users.count
+
+# If that count looks sane:
+
+# You can either block the users:
+users.each { |user| user.block! }
+
+# Or you can delete them:
+  # need 'current user' (your user) for auditing purposes
+current_user = User.find_by(username: '<your username>')
+
+users.each do |user|
+  DeleteUserWorker.perform_async(current_user.id, user.id)
+end
+```
+
+### Block Users that have no recent activity
+
+```ruby
+days_inactive = 60
+inactive_users = User.active.where("last_activity_on <= ?", days_inactive.days.ago)
+
+inactive_users.each do |user|
+    puts "user '#{user.username}': #{user.last_activity_on}"
+    user.block!
+end
+```
+
+### Find Max permissions for project/group
+
+```ruby
+user = User.find_by_username 'username'
+project = Project.find_by_full_path 'group/project'
+user.max_member_access_for_project project.id
+```
+
+```ruby
+user = User.find_by_username 'username'
+group = Group.find_by_full_path 'group'
+user.max_member_access_for_group group.id
+```
+
+## Groups
+
+### Count unique users in a group and sub-groups
+
+```ruby
+group = Group.find_by_path_or_name("groupname")
+members = []
+for member in group.members_with_descendants
+   members.push(member.user_name)
+end
+
+members.uniq.length
+```
+
+```ruby
+group = Group.find_by_path_or_name("groupname")
+
+# Count users from subgroup and up (inherited)
+group.members_with_parents.count
+
+# Count users from parent group and down (specific grants)
+parent.members_with_descendants.count
+```
+
+### Delete a group
+
+```ruby
+GroupDestroyWorker.perform_async(group_id, user_id)
+```
+
+## LDAP
+
+### LDAP commands in the rails console
+
+TIP: **TIP:**
+Use the rails runner to avoid entering the rails console in the first place.
+This is great when only a single command (such as a UserSync or GroupSync)
+is needed.
+
+```ruby
+# Get debug output
+Rails.logger.level = Logger::DEBUG
+
+# Run a UserSync (normally performed once a day)
+LdapSyncWorker.new.perform
+
+# Run a GroupSync for all groups (9.3-)
+LdapGroupSyncWorker.new.perform
+
+# Run a GroupSync for all groups (9.3+)
+LdapAllGroupsSyncWorker.new.perform
+
+# Run a GroupSync for a single group (10.6-)
+group = Group.find_by(name: 'my_gitlab_group')
+EE::Gitlab::LDAP::Sync::Group.execute_all_providers(group)
+
+# Run a GroupSync for a single group (10.6+)
+group = Group.find_by(name: 'my_gitlab_group')
+EE::Gitlab::Auth::LDAP::Sync::Group.execute_all_providers(group)
+
+# Query an LDAP group directly (10.6-)
+adapter = Gitlab::LDAP::Adapter.new('ldapmain') # If `main` is the LDAP provider
+ldap_group = EE::Gitlab::LDAP::Group.find_by_cn('group_cn_here', adapter)
+ldap_group.member_dns
+ldap_group.member_uids
+
+# Query an LDAP group directly (10.6+)
+adapter = Gitlab::Auth::LDAP::Adapter.new('ldapmain') # If `main` is the LDAP provider
+ldap_group = EE::Gitlab::Auth::LDAP::Group.find_by_cn('group_cn_here', adapter)
+ldap_group.member_dns
+ldap_group.member_uids
+
+# Lookup a particular user (10.6+)
+# This could expose potential errors connecting to and/or querying LDAP that may seem to
+# fail silently in the GitLab UI
+adapter = Gitlab::Auth::LDAP::Adapter.new('ldapmain') # If `main` is the LDAP provider
+user = Gitlab::Auth::LDAP::Person.find_by_uid('<username>',adapter)
+
+# Query the LDAP server directly (10.6+)
+## For an example, see https://gitlab.com/gitlab-org/gitlab-ee/blob/master/ee/lib/ee/gitlab/auth/ldap/adapter.rb
+adapter = Gitlab::Auth::LDAP::Adapter.new('ldapmain')
+options = {
+    # the :base is required
+    # use adapter.config.base for the base or .group_base for the group_base
+    base: adapter.config.group_base,
+
+    # :filter is optional
+    # 'cn' looks for all "cn"s under :base
+    # '*' is the search string - here, it's a wildcard
+    filter: Net::LDAP::Filter.eq('cn', '*'),
+
+    # :attributes is optional
+    # the attributes we want to get returned
+    attributes: %w(dn cn memberuid member submember uniquemember memberof)
+}
+adapter.ldap_search(options)
+```
+
+### Update user accounts when the `dn` and email change
+
+The following will require that any accounts with the new email address are removed.
+Emails have to be unique in GitLab. This is expected to work but unverified as of yet:
+
+```ruby
+# Here's an example with a couple users.
+# Each entry will have to include the old username and the new email
+emails = {
+  'ORIGINAL_USERNAME' => 'NEW_EMAIL_ADDRESS',
+  ...
+}
+
+emails.each do |username, email|
+  user = User.find_by_username(username)
+  user.email = email
+  user.skip_reconfirmation!
+  user.save!
+end
+
+# Run the UserSync to update the above users' data
+LdapSyncWorker.new.perform
+```
+
+## Routes
+
+### Remove redirecting routes
+
+See <https://gitlab.com/gitlab-org/gitlab-ce/issues/41758#note_54828133>.
+
+```ruby
+path = 'foo'
+conflicting_permanent_redirects = RedirectRoute.matching_path_and_descendants(path)
+
+# Check that conflicting_permanent_redirects is as expected
+conflicting_permanent_redirects.destroy_all
+```
+
+## Merge Requests
+
+### Find Merge Request
+
+```ruby
+m = project.merge_requests.find_by(iid: <IID>)
+m = MergeRequest.find_by_title('NEEDS UNIQUE TITLE!!!')
+```
+
+### Close a merge request properly (if merged but still marked as open)
+
+```ruby
+p = Project.find_by_full_path('')
+m = project.merge_requests.find_by(iid: )
+u = User.find_by_username('')
+MergeRequests::PostMergeService.new(p, u).execute(m)
+```
+
+### Rebase manually
+
+```ruby
+p = Project.find_by_full_path('')
+m = project.merge_requests.find_by(iid: )
+u = User.find_by_username('')
+MergeRequests::RebaseService.new(m.target_project, u).execute(m)
+```
+
+## CI
+
+### Cancel stuck pending pipelines
+
+See <https://gitlab.com/gitlab-com/support-forum/issues/2449#note_41929707>.
+
+```ruby
+Ci::Pipeline.where(project_id: p.id).where(status: 'pending').count
+Ci::Pipeline.where(project_id: p.id).where(status: 'pending').each {|p| p.cancel}
+Ci::Pipeline.where(project_id: p.id).where(status: 'pending').count
+```
+
+### Manually modify runner minutes
+
+```ruby
+Namespace.find_by_full_path("user/proj").namespace_statistics.update(shared_runners_seconds: 27360)
+```
+
+### Remove artifacts more than a week old
+
+```ruby
+### SELECTING THE BUILDS TO CLEAR
+# For a single project:
+project = Project.find_by_full_path('')
+builds_with_artifacts =  project.builds.with_artifacts_archive
+
+# Prior to 10.6 the above line would be:
+# builds_with_artifacts = project.builds.with_artifacts
+
+# Instance-wide:
+builds_with_artifacts = Ci::Build.with_artifacts
+
+### CLEAR THEM OUT
+builds_to_clear = builds_with_artifacts.where("finished_at < ?", 1.week.ago)
+builds_to_clear.each do |build|
+  build.artifacts_expire_at = Time.now
+  build.erase_erasable_artifacts!
+end
+```
+
+### Find reason failure (for when build trace is empty) (Introduced in 10.3.0)
+
+See <https://gitlab.com/gitlab-org/gitlab-ce/issues/41111>.
+
+```ruby
+build = Ci::Build.find(78420)
+
+build.failure_reason
+
+build.dependencies.each do |d| { puts "status: #{d.status}, finished at: #{d.finished_at},
+  completed: #{d.complete?}, artifacts_expired: #{d.artifacts_expired?}, erased: #{d.erased?}" }
+```
+
+### Disable strict artifact checking (Introduced in GitLab 10.3.0)
+
+See <https://docs.gitlab.com/ee/administration/job_artifacts.html#validation-for-dependencies>.
+
+```ruby
+Feature.enable('ci_disable_validates_dependencies')
+```
+
+### Remove CI traces older than 6 months
+
+```ruby
+current_user = User.find_by_email('cindy@gitlap.com')
+Ci::Build.where("finished_at < ?", 6.months.ago.to_date).each {|b| puts b.id; b.erase(erased_by: current_user) if b.erasable?};nil
+```
+
+### Try CI service
+
+```ruby
+p = Project.find_by_full_path('')
+m = project.merge_requests.find_by(iid: )
+m.project.try(:ci_service)
+```
+
+### Disable AutoDevOps on Existing Projects
+
+```ruby
+Project.all.each do |p|
+  p.auto_devops_attributes={"enabled"=>"0"}
+  p.save
+end
+```
+
+## License
+
+### See license plan name (since v9.3.0-ee)
+
+```ruby
+License.current.plan
+```
+
+### Check if a project feature is available on the instance
+
+Features listed in <https://gitlab.com/gitlab-org/gitlab-ee/blob/master/ee/app/models/license.rb>.
+
+```ruby
+License.current.feature_available?(:jira_dev_panel_integration)
+```
+
+### Check if a project feature is available in a project
+
+Features listed in <https://gitlab.com/gitlab-org/gitlab-ee/blob/master/ee/app/models/license.rb>.
+
+```ruby
+p = Project.find_by_full_path('<group>/<project>')
+p.feature_available?(:jira_dev_panel_integration)
+```
+
+### Add a license through the console
+
+```ruby
+key = "<key>"
+license = License.new(data: key)
+license.save
+License.current # check to make sure it applied
+```
+
+## Unicorn
+
+From [Zendesk ticket #91083](https://gitlab.zendesk.com/agent/tickets/91083) (internal)
+
+### Poll unicorn requests by seconds
+
+```ruby
+require 'rubygems'
+require 'unicorn'
+
+# Usage for this program
+def usage
+  puts "ruby unicorn_status.rb <path to unix socket> <poll interval in seconds>"
+  puts "Polls the given Unix socket every interval in seconds. Will not allow you to drop below 3 second poll intervals."
+  puts "Example: /opt/gitlab/embedded/bin/ruby poll_unicorn.rb /var/opt/gitlab/gitlab-rails/sockets/gitlab.socket 10"
+end
+
+# Look for required args. Throw usage and exit if they don't exist.
+if ARGV.count < 2
+  usage
+  exit 1
+end
+
+# Get the socket and threshold values.
+socket = ARGV[0]
+threshold = (ARGV[1]).to_i
+
+# Check threshold - is it less than 3? If so, set to 3 seconds. Safety first!
+if threshold.to_i < 3
+  threshold = 3
+end
+
+# Check - does that socket exist?
+unless File.exist?(socket)
+  puts "Socket file not found: #{socket}"
+  exit 1
+end
+
+# Poll the given socket every THRESHOLD seconds as specified above.
+puts "Running infinite loop. Use CTRL+C to exit."
+puts "------------------------------------------"
+loop do
+  Raindrops::Linux.unix_listener_stats([socket]).each do |addr, stats|
+    puts DateTime.now.to_s + " Active: " + stats.active.to_s + " Queued: " + stats.queued.to_s
+  end
+  sleep threshold
+end
+```
+
+## Sidekiq
+
+### Size of a queue
+
+```ruby
+Sidekiq::Queue.new('background_migration').size
+```
+
+### Kill a worker's Sidekiq jobs
+
+```ruby
+queue = Sidekiq::Queue.new('repository_import')
+queue.each { |job| job.delete if <condition>}
+```
+
+### Enable debug logging of Sidekiq
+
+```ruby
+gitlab_rails['env'] = {
+  'SIDEKIQ_LOG_ARGUMENTS' => "1"
+}
+```
+
+Then `gitlab-ctl reconfigure; gitlab-ctl restart sidekiq`.  The Sidekiq logs will now include additional data for troubleshooting.
+
+### Sidekiq kill signals
+
+See <https://github.com/mperham/sidekiq/wiki/Signals#ttin>.
+
+## Redis
+
+### Connect to redis (omnibus)
+
+```sh
+/opt/gitlab/embedded/bin/redis-cli -s /var/opt/gitlab/redis/redis.socket
+```
+
+### Connect to redis (HA)
+
+```sh
+/opt/gitlab/embedded/bin/redis-cli -h <host ip> -a <password>
+```
+
+## LFS
+
+### Get info about LFS objects and associated project
+
+```ruby
+o=LfsObject.find_by(oid: "<oid>")
+p=Project.find(LfsObjectsProject.find_by_lfs_object_id(o.id).project_id)
+```
+
+You can then delete these records from the database with:
+
+```ruby
+LfsObjectsProject.find_by_lfs_object_id(o.id).destroy
+o.destroy
+```
+
+You would also want to combine this with deleting the LFS file in the LFS storage
+area on disk. It remains to be seen exactly how or whether the deletion is useful, however.
+
+## Decryption Problems
+
+### Bad Decrypt Script (for encrypted variables)
+
+See <https://gitlab.com/snippets/1730735/raw>.
+
+This script will go through all the encrypted variables and count how many are not able
+to be decrypted. Might be helpful to run on multiple nodes to see which `gitlab-secrets.json`
+file is most up to date:
+
+```bash
+wget -O /tmp/bad-decrypt.rb https://gitlab.com/snippets/1730735/raw
+gitlab-rails runner /tmp/bad-decrypt.rb
+```
+
+If `ProjectImportData Bad count:` is detected and the decision is made to delete the
+encrypted credentials to allow manual reentry:
+
+```ruby
+  # Find the ids of the corrupt ProjectImportData objects
+  total = 0
+  bad = []
+  ProjectImportData.find_each do |data|
+    begin
+      total += 1
+      data.credentials
+    rescue => e
+      bad << data.id
+    end
+  end
+
+  puts "Bad count: #{bad.count} / #{total}"
+
+  # See the bad ProjectImportData ids
+  bad
+
+  # Remove the corrupted credentials
+  import_data = ProjectImportData.where(id: bad)
+  import_data.each do |data|
+    data.update_columns({ encrypted_credentials: nil, encrypted_credentials_iv: nil, encrypted_credentials_salt: nil})
+  end
+```
+
+If `User OTP Secret Bad count:` is detected. For each user listed disable/enable
+two-factor authentication.
+
+### Decrypt Script for encrypted tokens
+
+This script will search for all encrypted tokens that are causing decryption errors,
+and update or reset as needed:
+
+```bash
+wget -O /tmp/encrypted-tokens.rb https://gitlab.com/snippets/1876342/raw
+gitlab-rails runner /tmp/encrypted-tokens.rb
+```
+
+## Geo
+
+### Artifacts
+
+#### Find failed artifacts
+
+```ruby
+Geo::JobArtifactRegistry.failed
+```
+
+#### Download artifact
+
+```ruby
+Gitlab::Geo::JobArtifactDownloader.new(:job_artifact, <artifact_id>).execute
+```
+
+#### Get a count of the synced artifacts
+
+```ruby
+Geo::JobArtifactRegistry.synced.count
+```
+
+#### Find `ID` of synced artifacts that are missing on primary
+
+```ruby
+Geo::JobArtifactRegistry.synced.missing_on_primary.pluck(:artifact_id)
+```
+
+### Repository verification failures
+
+#### Get the number of verification failed repositories
+
+```ruby
+Geo::ProjectRegistryFinder.new.count_verification_failed_repositories
+```
+
+#### Find the verification failed repositories
+
+```ruby
+Geo::ProjectRegistry.verification_failed_repos
+```
+
+### Find repositories that failed to sync
+
+```ruby
+Geo::ProjectRegistryFinder.new.find_failed_project_registries('repository')
+```
+
+### Resync repositories
+
+#### Queue up all repositories for resync. Sidekiq will handle each sync
+
+```ruby
+Geo::ProjectRegistry.update_all(resync_repository: true, resync_wiki: true)
+```
+
+#### Sync individual repository now
+
+```ruby
+project = Project.find_by_full_path('<group/project>')
+
+Geo::RepositorySyncService.new(project).execute
+```
diff --git a/doc/administration/troubleshooting/linux_cheat_sheet.md b/doc/administration/troubleshooting/linux_cheat_sheet.md
new file mode 100644
index 00000000000..2bbb498f020
--- /dev/null
+++ b/doc/administration/troubleshooting/linux_cheat_sheet.md
@@ -0,0 +1,339 @@
+---
+type: reference
+---
+
+# Linux Cheat Sheet
+
+This is the GitLab Support Team's collection of information regarding Linux, that they
+sometimes use while troubleshooting. It is listed here for transparency,
+and it may be useful for users with experience with Linux. If you are currently
+having an issue with GitLab, you may want to check your [support options](https://about.gitlab.com/support/)
+first, before attempting to use this information.
+
+CAUTION: **CAUTION:**
+If you are administering GitLab you are expected to know these commands for your distribution
+of choice. If you are a GitLab Support Engineer, consider this a cross-reference to
+translate `yum` -> `apt-get` and the like.
+
+Note: **Note:**
+Most of the commands below have not been labeled as to which distribution they work
+on. Contributions are welcome to help add them.
+
+## System Commands
+
+### Distro Information
+
+```bash
+# Debian/Ubuntu
+uname -a
+lsb_release -a
+
+# CentOS/RedHat
+cat /etc/centos-release
+cat /etc/redhat-release
+
+# This will provide a lot more information
+cat /etc/os-release
+```
+
+### Shut down or Reboot
+
+```bash
+shutdown -h now
+reboot
+```
+
+### Permissions
+
+```bash
+# change the user:group ownership of a file/dir
+chown root:git <file_or_dir>
+
+# make a file executable
+chmod u+x <file>
+```
+
+### Files & Dirs
+
+```bash
+# create a new directory and all subdirectories
+mkdir -p dir/dir2/dir3
+
+# Send a command's output to file.txt, no STDOUT
+ls > file.txt
+
+# Send a command's output to file.txt AND see it in STDOUT
+ls | tee /tmp/file.txt
+
+# Search and Replace within a file
+sed -i 's/original-text/new-text/g' <filename>
+```
+
+### See all set environment variables
+
+```bash
+env
+```
+
+## Searching
+
+### File names
+
+```bash
+# search for a file in a filesystem
+find . -name 'filename.rb' -print
+
+# locate a file
+locate <filename>
+
+# see command history
+history
+
+# search CLI history
+<ctrl>-R
+```
+
+### File contents
+
+```bash
+# -B/A = show 2 lines before/after search_term
+grep -B 2 -A 2 search_term <filename>
+
+# -<number> shows both before and after
+grep -2 search_term <filename>
+
+# Search on all files in directory (recursively)
+grep -r search_term <directory>
+
+# search through *.gz files is the same except with zgrep
+zgrep search_term <filename>
+
+# Fast grep printing lines containing a string pattern
+fgrep -R string_pattern <filename or directory>
+```
+
+### CLI
+
+```bash
+# View command history
+history
+
+# Run last command that started with 'his' (3 letters min)
+!his
+
+# Search through command history
+<ctrl>-R
+
+# Execute last command with sudo
+sudo !!
+```
+
+## Managing resources
+
+### Memory, Disk, & CPU usage
+
+```bash
+# disk space info.  The '-h' gives the data in human-readable values
+df -h
+
+# size of each file/dir and its contents in the current dir
+du -hd 1
+
+# or alternative
+du -h --max-depth=1
+
+# find files greater than certain size(k, M, G) and list them in order
+# get rid of the + for exact, - for less than
+find / -type f -size +100M -print0 | xargs -0 du -hs | sort -h
+
+# Find free memory on a system
+free -m
+
+# Find what processes are using memory/CPU and organize by it
+# Load average is 1/CPU for 1, 5, and 15 minutes
+top -o %MEM
+top -o %CPU
+```
+
+### Strace
+
+```bash
+# strace a process
+strace -tt -T -f -y -s 1024 -p <pid>
+
+# -tt   print timestamps with microsecond accuracy
+
+# -T    print the time spent in each syscall
+
+# -f    also trace any child processes that forked
+
+# -y    print the path associated with file handles
+
+# -s    max string length to print for an event
+
+# -o    output file
+
+# run strace on all unicorn processes
+ps auwx | grep unicorn | awk '{ print " -p " $2}' | xargs strace -tt -T -f -y -s 1024 -o /tmp/unicorn.txt
+```
+
+See the [strace zine](https://wizardzines.com/zines/strace/) for a quick walkthrough.
+
+Brendan Gregg has a more detailed explanation of [how to use strace](http://www.brendangregg.com/blog/2014-05-11/strace-wow-much-syscall.html).
+
+Be aware that strace can have major impacts to system performance when it is running.
+
+### The Strace Parser tool
+
+Our [strace-parser tool](https://gitlab.com/wchandler/strace-parser) can be used to
+provide a high level summary of the `strace` output.  It is similar to `strace -C`,
+but provides much more detailed statistics.
+
+MacOS and Linux binaries [are available](https://gitlab.com/gitlab-com/support/toolbox/strace-parser/-/tags),
+or you can build it from source if you have the Rust compiler.
+
+#### How to use the tool
+
+First run the tool with no arguments other than the strace output file name to get
+a summary of the top processes sorted by time spent actively performing tasks. You
+can also sort based on total time, # of syscalls made, PID #, and # of child processes
+using the `-S` or `--sort` flag. The number of results defaults to 25 processes, but
+can be changed using the `-c`/`--count` option.  See `--help` for full details.
+
+```sh
+$ ./strace-parser strace.txt
+
+Top 25 PIDs
+-----------
+
+  pid           active (ms)  wait (ms)  total (ms)   % active    syscalls
+  ----------    ----------  ---------   ---------   ---------   ---------
+  8795             689.072   45773.832   46462.902     16.89%       23018
+  13408            679.432   55910.891   56590.320     16.65%       28593
+  6423             554.822   13175.485   13730.308     13.60%       13735
+...
+```
+
+Based on the summary, you can then view the details of syscalls made by one or more
+procsses using the `-p`/`--pid` for a specific process, or `-s`/`--stats` flags for
+a sorted list. `--stats` takes the same sorting and count options as summary.
+
+```sh
+$ ./strace-parse strace.text -p 6423
+
+PID 6423
+13735 syscalls, active time: 554.822ms, total time: 13730.308ms
+
+  syscall              count         total         max         avg         min  errors
+                                      (ms)        (ms)        (ms)        (ms)
+  ---------------   --------    ----------  ----------  ----------  ----------  --------
+  epoll_wait             628     13175.485      21.259      20.980       0.020
+  clock_gettime         7326       199.500       0.249       0.027       0.013
+  stat                  2101       110.768      19.056       0.053       0.017  ENOENT: 2076
+  ...
+  ---------------
+
+  Parent PID: 495
+  Child PIDs:  8383, 8418, 8419, 8420, 8421
+
+  Slowest file access times for PID 6423:
+
+     open (ms)        timestamp              error     file name
+  -----------   ---------------    ---------------     ----------
+      29.818    10:53:11.528954                        /srv/gitlab-data/builds/2018_08/6174/954448.log
+      12.309    10:53:46.708274                        /srv/gitlab-data/builds/2018_08/5342/954186.log
+      0.039     10:53:49.222110                        /opt/gitlab/embedded/service/gitlab-rails/app/views/events/event/_note.html.haml
+      0.035     10:53:49.125115                        /opt/gitlab/embedded/service/gitlab-rails/app/views/events/event/_push.html.haml
+  ...
+```
+
+In the example above, we can see that file opening times on `/srv/gitlab-data` are
+extremely slow, about 100X slower than `/opt/gitlab`.
+
+When nothing stands out in the results, a good way to get more context is to run `strace`
+on your own GitLab instance while performing the action performed by the customer,
+then compare summaries of both results and dive into the differences.
+
+#### Stats for the open syscall
+
+Rough numbers for calls to `open` and `openat` (used to access files) on various configurations.
+Slow storage can cause the dreaded `DeadlineExceeded` error in Gitaly.
+
+Also [see this entry](https://docs.gitlab.com/ee/administration/operations/filesystem_benchmarking.html)
+in the handbook for quick tests customers can perform to check their filesystem performance.
+
+Keep in mind that timing information from `strace` is often somewhat inaccurate, so
+small differences should not be considered significant.
+
+|Setup          | access times  |
+|:--------------|:--------------|
+| EFS           | 10 - 30ms     |
+| Local Storage | 0.01 - 1ms    |
+
+## Networking
+
+### Ports
+
+```bash
+# Find the programs that are listening on ports
+netstat -plnt
+ss -plnt
+lsof -i -P | grep <port>
+```
+
+### Internet/DNS
+
+```bash
+# Show domain IP address
+dig +short example.com
+nslookup example.com
+
+# Check DNS using specific nameserver
+# 8.8.8.8 = google, 1.1.1.1 = cloudflare, 208.67.222.222 = opendns
+dig @8.8.8.8 example.com
+nslookup example.com 1.1.1.1
+
+# Find host provider
+whois <ip_address> | grep -i "orgname\|netname"
+
+# Curl headers with redirect
+curl --head --location https://example.com
+```
+
+## Package Management
+
+```bash
+# Debian/Ubuntu
+
+# List packages
+dpkg -l
+apt list --installed
+
+# Find an installed package
+dpkg -l | grep <package>
+apt list --installed | grep <package>
+
+# Install a package
+dpkg -i <package_name>.deb
+apt-get install <package>
+apt install <package>
+
+# CentOS/RedHat
+
+# Install a package
+yum install <package>
+dnf install <package> # RHEL/CentOS 8+
+
+rpm -ivh <package_name>.rpm
+
+# Find an installed package
+rpm -qa | grep <package>
+```
+
+## Logs
+
+```bash
+# Print last lines in log file where 'n'
+# is the number of lines to print
+tail -n /path/to/log/file
+```
diff --git a/doc/administration/troubleshooting/test_environments.md b/doc/administration/troubleshooting/test_environments.md
new file mode 100644
index 00000000000..075effc5dc3
--- /dev/null
+++ b/doc/administration/troubleshooting/test_environments.md
@@ -0,0 +1,126 @@
+---
+type: reference
+---
+
+# Apps for a Testing Environment
+
+This is the GitLab Support Team's collection of information regarding testing environments,
+for use while troubleshooting. It is listed here for transparency, and it may be useful
+for users with experience with these tools. If you are currently having an issue with
+GitLab, you may want to check your [support options](https://about.gitlab.com/support/)
+first, before attempting to use this information.
+
+NOTE: **Note:**
+This page was initially written for Support Engineers, so some of the links
+are only available internally at GitLab.
+
+## Docker
+
+The following were tested on docker containers running in the cloud. Support Engineers,
+please see [these docs](https://gitlab.com/gitlab-com/dev-resources/tree/master/dev-resources#running-docker-containers)
+on how to run Docker containers on `dev-resources`. Other setups haven't been tested,
+but contributions are welcome.
+
+### GitLab
+
+Please see [our Docker test environment docs](https://docs.gitlab.com/ee/install/digitaloceandocker.html#create-new-gitlab-container)
+for how to run GitLab on Docker. When spinning this up with `docker-machine`, ensure
+you change a few things:
+
+1. Update the name of the `docker-machine` host. You can see a list of hosts
+   with `docker-machine ls`.
+1. Expose the necessary ports using the `-p` flag. Docker normally doesn't
+   allow access to any ports it uses outside of the container, so they must be
+   explicitly exposed.
+1. Add any necessary `gitlab.rb` configuration to the
+   `GITLAB_OMNIBUS_CONFIG` variable.
+
+For example, when the `docker-machine` host we want to use is `do-docker`:
+
+```sh
+docker run --detach --name gitlab \
+--env GITLAB_OMNIBUS_CONFIG="external_url 'http://$(docker-machine ip do-docker)'; gitlab_rails['gitlab_shell_ssh_port'] = 2222;" \
+--hostname $(docker-machine ip do-docker) \
+-p 80:80 -p 2222:22 \
+gitlab/gitlab-ee:11.5.3-ee.0
+```
+
+### SAML
+
+#### SAML for Authentication
+
+We can use the [test-saml-idp Docker image](https://hub.docker.com/r/jamedjo/test-saml-idp)
+to do the work for us:
+
+```sh
+docker run --name gitlab_saml -p 8080:8080 -p 8443:8443 \
+-e SIMPLESAMLPHP_SP_ENTITY_ID=<GITLAB_IP_OR_DOMAIN> \
+-e SIMPLESAMLPHP_SP_ASSERTION_CONSUMER_SERVICE=<GITLAB_IP_OR_DOMAIN>/users/auth/saml/callback \
+-d jamedjo/test-saml-idp
+```
+
+The following will also need to go in your `/etc/gitlab/gitlab.rb`. See [our SAML docs](https://docs.gitlab.com/ee/integration/saml.html)
+for more, as well as the list of [default usernames, passwords, and emails](https://hub.docker.com/r/jamedjo/test-saml-idp/#usage).
+
+```ruby
+gitlab_rails['omniauth_enabled'] = true
+gitlab_rails['omniauth_allow_single_sign_on'] = ['saml']
+gitlab_rails['omniauth_sync_email_from_provider'] = 'saml'
+gitlab_rails['omniauth_sync_profile_from_provider'] = ['saml']
+gitlab_rails['omniauth_sync_profile_attributes'] = ['email']
+gitlab_rails['omniauth_auto_sign_in_with_provider'] = 'saml'
+gitlab_rails['omniauth_block_auto_created_users'] = false
+gitlab_rails['omniauth_auto_link_ldap_user'] = false
+gitlab_rails['omniauth_auto_link_saml_user'] = true
+gitlab_rails['omniauth_providers'] = [
+  {
+    "name" => "saml",
+    "label" => "SAML",
+    "args" => {
+      assertion_consumer_service_url: '<GITLAB_IP_OR_DOMAIN>/users/auth/saml/callback',
+      idp_cert_fingerprint: '119b9e027959cdb7c662cfd075d9e2ef384e445f',
+      idp_sso_target_url: '<SAML_IP_OR_DOMAIN>:8080/simplesaml/saml2/idp/SSOService.php',
+      issuer: '<GITLAB_IP_OR_DOMAIN>',
+      name_identifier_format: 'urn:oasis:names:tc:SAML:2.0:nameid-format:persistent'
+    }
+  }
+]
+```
+
+#### GroupSAML for GitLab.com
+
+See [the GDK SAML documentation](https://gitlab.com/gitlab-org/gitlab-development-kit/blob/master/doc/howto/saml.md).
+
+### ElasticSearch
+
+```sh
+docker run -d --name elasticsearch \
+-p 9200:9200 -p 9300:9300 \
+-e "discovery.type=single-node" \
+docker.elastic.co/elasticsearch/elasticsearch:5.5.1
+```
+
+Then confirm it works in the browser at `curl http://<IP_ADDRESS>:9200/_cat/health`.
+ElasticSearch's default username is `elastic` and password is `changeme`.
+
+### PlantUML
+
+See [our PlantUML docs](../integration/plantuml.md#docker)
+on running PlantUML in Docker.
+
+### Jira
+
+```sh
+docker run -d -p 8081:8080 cptactionhank/atlassian-jira:latest
+```
+
+Then go to `<IP_ADDRESS>:8081` in the browser to set it up. This requires a
+Jira license.
+
+### Grafana
+
+```sh
+docker run -d --name grafana -e "GF_SECURITY_ADMIN_PASSWORD=gitlab" -p 3000:3000 grafana/grafana
+```
+
+Access it at `<IP_ADDRESS>:3000`.