Move 3 references from debug project to docs

Move rails and linux cheat sheets, and test_environment
reference docs to the docs project

3 files changed, 1460 insertions, 0 deletions

diff --git a/doc/administration/troubleshooting/gitlab_rails_cheat_sheet.md b/doc/administration/troubleshooting/gitlab_rails_cheat_sheet.md
new file mode 100644
index 00000000000..f7e96a70edc
--- /dev/null
+++ b/doc/administration/troubleshooting/gitlab_rails_cheat_sheet.md
@@ -0,0 +1,1022 @@
+---
+is_hidden: true
+comments: false
+---
+
+# GitLab Rails Console Cheat Sheet
+
+CAUTION: **CAUTION:**
+Please note that some of these scripts could be damaging if not run correctly,
+or under the right conditions.  We highly recommend running them under the
+guidance of a Support Engineer, or running them in a test environment with a
+backup of the instance ready to be restored, just in case.
+
+CAUTION: **CAUTION:**
+Please also note that as GitLab changes, changes to the code are inevitable
+and, thus, some scripts may not work as they once used to.  These are not kept
+up-to-date as these scripts/commands were added as they were found/needed.  As
+mentioned above, we recommend running these scripts under the supervision of a
+Support Engineer, who can also verify that they will continue to work as they
+should and, if needed, update the script for the latest version of GitLab.
+
+## Use the Rails Runner
+
+If the script you want to run is short, you can use the Rails Runner to avoid
+entering the rails console in the first place.  Here's an example of its use:
+
+```bash
+gitlab-rails runner "RAILS_COMMAND"
+
+# Example with a 2-line script
+gitlab-rails runner "user = User.first; puts user.username"
+```
+
+## Enable debug logging on rails console
+
+```ruby
+Rails.logger.level = 0
+```
+
+## Enable debug logging for ActiveRecord (db issues)
+
+```ruby
+ActiveRecord::Base.logger = Logger.new(STDOUT)
+```
+
+## Temporarily Disable Timeout
+
+```
+ActiveRecord::Base.connection.execute('SET statement_timeout TO 0')
+```
+
+## Find specific methods for an object
+
+```ruby
+Array.methods.select { |m| m.to_s.include? "sing" }
+Array.methods.grep(/sing/)
+```
+
+## Find method source
+
+Works for [non-instrumented methods](https://docs.gitlab.com/ce/development/instrumentation.html#checking-instrumented-methods)
+
+```ruby
+instance_of_object.method(:foo).source_location
+
+# Example for when we would call project.private?
+project.method(:private?).source_location
+```
+
+## Query an object
+
+```ruby
+o = Object.where('attribute like ?', 'ex')
+```
+
+## View all keys in cache
+
+```ruby
+Rails.cache.instance_variable_get(:@data).keys
+```
+
+
+## Rails console history
+
+```ruby
+puts Readline::HISTORY.to_a
+```
+
+## Profile a page
+
+```ruby
+# Before 11.6.0
+logger = Logger.new(STDOUT)
+admin_token = User.find_by_username('ADMIN_USERNAME').personal_access_tokens.first.token
+app.get("URL/?private_token=#{admin_token}")
+
+# From 11.6.0
+admin = User.find_by_username('ADMIN_USERNAME')
+url = "/url/goes/here"
+Gitlab::Profiler.with_user(admin) { app.get(url) }
+```
+
+##  Using the GitLab profiler inside console (used as of 10.5)
+
+```ruby
+logger = Logger.new(STDOUT)
+admin = User.find_by_username('ADMIN_USERNAME')
+Gitlab::Profiler.profile('URL', logger: logger, user: admin)
+```
+
+## Time an operation
+
+```ruby
+# A single operation
+Benchmark.measure { <operation> }
+
+# A breakdown of multiple operations
+Benchmark.bm do |x|
+  x.report(:label1) { <operation_1> }
+  x.report(:label2) { <operation_2> }
+end
+```
+
+## Command Line
+
+### Check the GitLab version fast
+
+```bash
+grep -m 1 gitlab /opt/gitlab/version-manifest.txt
+```
+
+### Debugging SSH
+
+```bash
+GIT_SSH_COMMAND="ssh -vvv" git clone <repository>
+```
+
+### Debugging over HTTPS
+
+```bash
+GIT_CURL_VERBOSE=1 GIT_TRACE=1 git clone <repository>
+```
+
+## Projects
+
+### Find projects
+
+```ruby
+# A single project
+project = Project.find_by_full_path('PROJECT_PATH')
+
+# All projects in a particular namespace.  Can be a username, a group
+# ('gitlab-org'), or even include subgroups ('gitlab-org/distribution')
+namespace = Namespace.find_by_full_path('NAMESPACE_PATH')
+projects = namespace.all_projects
+```
+
+### Clear a project's cache
+
+```ruby
+ProjectCacheWorker.perform_async(project.id)
+```
+
+### Expire the .exists? cache
+
+```ruby
+project.repository.expire_exists_cache
+```
+
+### Make all projects private
+
+```ruby
+Project.update_all(visibility_level: 0)
+```
+
+### Find & remove projects that are pending deletion
+
+```ruby
+#
+# This section will list all the projects which are pending deletion
+#
+projects = Project.where(pending_delete: true)
+projects.each do |p|
+  puts "Project name: #{p.id}"
+  puts "Project name: #{p.name}"
+  puts "Repository path: #{p.repository.storage_path}"
+end
+
+#
+# Assign a user (the root user will do)
+#
+user = User.find_by_username('root')
+
+#
+# For each project listed repeat these two commands
+#
+
+# Find the project, update the xxx-changeme values from above
+project = Project.find_by_full_path('group-changeme/project-changeme')
+
+# Delete the project
+::Projects::DestroyService.new(project, user, {}).execute
+```
+
+Next, run `sudo gitlab-rake gitlab:cleanup:repos` on the command line to finish.
+
+### Destroy a project
+
+```ruby
+project = Project.find_by_full_path('')
+user = User.find_by_username('')
+ProjectDestroyWorker.perform_async(project.id, user.id, {})
+# or ProjectDestroyWorker.new.perform(project.id, user.id, {})
+# or Projects::DestroyService.new(project, user).execute
+```
+
+### Remove fork relationship manually
+
+```ruby
+p = Project.find_by_full_path('')
+u = User.find_by_username('')
+::Projects::UnlinkForkService.new(p, u).execute
+```
+
+### Make a project read-only (can only be done in the console)
+
+```ruby
+Make a project read-only
+project.repository_read_only = true; project.save
+
+# OR
+project.update!(repository_read_only: true)
+```
+
+### Bulk update service integration password for _all_ projects
+
+For example, change the JIRA user's password for all projects that have the JIRA
+integration active:
+
+```ruby
+p = Project.find_by_sql("SELECT p.id FROM projects p LEFT JOIN services s ON p.id = s.project_id WHERE s.type = 'JiraService' AND s.active = true")
+
+p.each do |project|
+  project.jira_service.update_attribute(:password, '<your-new-password>')
+end
+```
+
+### Identify un-indexed projects
+
+```
+Project.find_each do |project|
+  puts "id #{project.id}: #{project.namespace.name.to_s}/#{project.name.to_s}" if project.index_status.nil?
+end
+```
+
+## Imports / Exports
+
+### Fail an import
+
+```ruby
+Fail an import
+# Find the project and get the error
+p = Project.find_by_full_path('<username-or-group>/<project-name>')
+
+p.import_error
+
+# To finish the import on GitLab running version before 11.6
+p.import_finish
+
+# To finish the import on GitLab running version 11.6 or after
+p.import_state.mark_as_failed("Failed manually through console.")
+```
+
+### Rename imported repository
+
+A customer had a backup restore that failed on a single repository. It created the project with an empty repository. They were able to successfully restore it to a dev instance. From there, they created an export and imported it into a new project. They did not want the project under the new name. We were able to transfer the newly imported project to the empty project.
+
+Move the new repository to the empty repository
+
+```bash
+mv /var/opt/gitlab/git-data/repositories/<group>/<new-project> /var/opt/gitlab/git-data/repositories/<group>/<empty-project>
+```
+
+Make sure the permissions are correct
+
+```bash
+chown -R git:git <path-to-directory>.git
+```
+
+Clear the cache
+
+```bash
+sudo gitlab-rake cache:clear
+```
+
+## Repository
+
+### Search sequence of pushes on a repository
+
+<https://gitlab.slack.com/archives/C4Y5DRKLK/p1523984649000229>
+
+```ruby
+p = Project.find_with_namespace('u/p')
+p.events.code_push.last(100).each do |e|
+  printf "%-20.20s %8s...%8s (%s)\n", e.data[:ref], e.data[:before], e.data[:after], e.author.try(:username)
+end
+```
+
+GitLab 9.5 and above:
+
+```ruby
+p = Project.find_by_full_path('u/p')
+p.events.code_push.last(100).each do |e|
+  printf "%-20.20s %8s...%8s (%s)\n", e.push_event_payload[:ref], e.push_event_payload[:commit_from], e.push_event_payload[:commit_to], e.author.try(:username)
+end
+```
+
+If you're here, chances are you have a customer saying "GitLab lost my commit!". [This StackOverflow article](https://stackoverflow.com/questions/13468027/the-mystery-of-the-missing-commit-across-merges) describes how you can end up in this state without a force push. If you look at the above output for the target branch, you will see a discontinuity in the from/to commits as you step through the output. Each new push should be "from" the "to" sha of the previous push. When this discontinuity happens, you will see two pushes with the same "from" sha.
+
+## Mirrors
+
+### Find mirrors with "bad decrypt" errors
+
+```ruby
+total = 0
+bad = []
+ProjectImportData.find_each do |data|
+  begin
+    total += 1
+    data.credentials
+  rescue => e
+    bad << data
+  end
+end
+
+puts "Bad count: #{bad.count} / #{total}"
+bad.each do |repo|
+  puts Project.find(repo.project_id).full_path
+end; bad.count
+```
+
+### Transfer mirror users and tokens to a single service account
+
+Use case: If you have multiple users using their own GitHub credentials to set up
+repository mirroring, mirroring breaks when people leave the company. Use this
+script to migrate disparate mirroring users and tokens into a single service account.
+
+```ruby
+svc_user = User.find_by(username: 'ourServiceUser')
+token = 'githubAccessToken'
+
+Project.where(mirror: true).each do |project|
+  import_url = project.import_url
+
+  # The url we want is https://token@project/path.git
+  repo_url = if import_url.include?('@')
+               # Case 1: The url is something like https://23423432@project/path.git
+               import_url.split('@').last
+             elsif import_url.include?('//')
+               # Case 2: The url is something like https://project/path.git
+               import_url.split('//').last
+             end
+
+  next unless repo_url
+
+  final_url = "https://#{token}@#{repo_url}"
+
+  project.mirror_user = svc_user
+  project.import_url = final_url
+  project.username_only_import_url = final_url
+  project.save
+end
+```
+
+## Users
+
+### Finding users
+
+```ruby
+# By username
+user = User.find_by(username: '')
+
+# By primary email
+user = User.find_by(email: '')
+
+# By any email (primary or secondary)
+user = User.find_by_any_email('')
+
+# Admins
+User.admins
+admin = User.admins.first
+```
+
+### Block
+
+```ruby
+User.find_by_username().block!
+```
+
+### Unblock
+
+```ruby
+User.find_by_username().active
+```
+
+### Skip reconfirmation
+
+```ruby
+user = User.find_by_username ''
+user.skip_reconfirmation!
+```
+
+### Get an admin token
+
+```ruby
+# Get the first admin's first access token (no longer works on 11.9+. see: https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/22743)
+User.where(admin:true).first.personal_access_tokens.first.token
+
+# Get the first admin's private token (no longer works on 10.2+)
+User.where(admin:true).private_token
+```
+
+### Create personal access token
+
+```ruby
+personal_access_token = User.find(123).personal_access_tokens.create(
+  name: 'apitoken',
+  impersonation: false,
+  scopes: [:api]
+)
+
+personal_access_token.token
+```
+
+You might also want to manually set the token string:
+
+```ruby
+User.find(123).personal_access_tokens.create(
+  name: 'apitoken',
+  token_digest: Gitlab::CryptoHelper.sha256('some-token-string-here'),
+  impersonation: false,
+  scopes: [:api]
+)
+```
+
+### Disable 2FA on a user
+
+```ruby
+user = User.find_by_username('username')
+user.disable_two_factor!
+```
+
+### Active users & Historical users
+
+```ruby
+# Active users on the instance, now
+User.active.count
+
+# The historical max on the instance as of the past year
+::HistoricalData.max_historical_user_count
+```
+
+```bash
+# Using curl and jq (up to a max 100, see [pagination](https://docs.gitlab.com/ee/api/#pagination)
+curl --silent --header "Private-Token: ********************" "https://gitlab.example.com/api/v4/users?per_page=100&active" | jq --compact-output '.[] | [.id,.name,.username]'
+```
+
+### Block or Delete Users that have no projects or groups
+
+```ruby
+users = User.where('id NOT IN (select distinct(user_id) from project_authorizations)')
+
+# How many users will be removed?
+users.count
+
+# If that count looks sane:
+
+# You can either block the users:
+users.each { |user| user.block! }
+
+# Or you can delete them:
+  # need 'current user' (your user) for auditing purposes
+current_user = User.find_by(username: '<your username>')
+
+users.each do |user|
+  DeleteUserWorker.perform_async(current_user.id, user.id)
+end
+```
+
+### Block Users that have no recent activity
+
+```ruby
+days_inactive = 60
+inactive_users = User.active.where("last_activity_on <= ?", days_inactive.days.ago)
+
+inactive_users.each do |user|
+    puts "user '#{user.username}': #{user.last_activity_on}"
+    user.block!
+end
+```
+
+### Find Max permissions for project/group
+
+```
+user = User.find_by_username 'username'
+project = Project.find_by_full_path 'group/project'
+user.max_member_access_for_project project.id
+```
+
+```
+user = User.find_by_username 'username'
+group = Group.find_by_full_path 'group'
+user.max_member_access_for_group group.id
+
+```
+
+## Groups
+
+### Count unique users in a group and sub-groups
+
+```ruby
+group = Group.find_by_path_or_name("groupname")
+members = []
+for member in group.members_with_descendants
+   members.push(member.user_name)
+end
+
+members.uniq.length
+```
+
+```ruby
+group = Group.find_by_path_or_name("groupname")
+
+# Count users from subgroup and up (inherited)
+group.members_with_parents.count
+
+# Count users from parent group and down (specific grants)
+parent.members_with_descendants.count
+```
+
+### Delete a group
+
+```ruby
+GroupDestroyWorker.perform_async(group_id, user_id)
+```
+
+## LDAP
+
+### LDAP commands in the rails console
+
+TIP: **TIP:**
+Use the rails runner to avoid entering the rails console in the first place.
+This is great when only a single command (such as a UserSync or GroupSync)
+is needed.
+
+```ruby
+# Get debug output
+Rails.logger.level = Logger::DEBUG
+
+# Run a UserSync (normally performed once a day)
+LdapSyncWorker.new.perform
+
+# Run a GroupSync for all groups (9.3-)
+LdapGroupSyncWorker.new.perform
+
+# Run a GroupSync for all groups (9.3+)
+LdapAllGroupsSyncWorker.new.perform
+
+# Run a GroupSync for a single group (10.6-)
+group = Group.find_by(name: 'my_gitlab_group')
+EE::Gitlab::LDAP::Sync::Group.execute_all_providers(group)
+
+# Run a GroupSync for a single group (10.6+)
+group = Group.find_by(name: 'my_gitlab_group')
+EE::Gitlab::Auth::LDAP::Sync::Group.execute_all_providers(group)
+
+# Query an LDAP group directly (10.6-)
+adapter = Gitlab::LDAP::Adapter.new('ldapmain') # If `main` is the LDAP provider
+ldap_group = EE::Gitlab::LDAP::Group.find_by_cn('group_cn_here', adapter)
+ldap_group.member_dns
+ldap_group.member_uids
+
+# Query an LDAP group directly (10.6+)
+adapter = Gitlab::Auth::LDAP::Adapter.new('ldapmain') # If `main` is the LDAP provider
+ldap_group = EE::Gitlab::Auth::LDAP::Group.find_by_cn('group_cn_here', adapter)
+ldap_group.member_dns
+ldap_group.member_uids
+
+# Lookup a particular user (10.6+)
+# This could expose potential errors connecting to and/or querying LDAP that may seem to
+# fail silently in the GitLab UI
+adapter = Gitlab::Auth::LDAP::Adapter.new('ldapmain') # If `main` is the LDAP provider
+user = Gitlab::Auth::LDAP::Person.find_by_uid('<username>',adapter)
+
+# Query the LDAP server directly (10.6+)
+## For an example, see https://gitlab.com/gitlab-org/gitlab-ee/blob/master/ee/lib/ee/gitlab/auth/ldap/adapter.rb
+adapter = Gitlab::Auth::LDAP::Adapter.new('ldapmain')
+options = {
+    # the :base is required
+    # use adapter.config.base for the base or .group_base for the group_base
+    base: adapter.config.group_base,
+
+    # :filter is optional
+    # 'cn' looks for all "cn"s under :base
+    # '*' is the search string - here, it's a wildcard
+    filter: Net::LDAP::Filter.eq('cn', '*'),
+
+    # :attributes is optional
+    # the attributes we want to get returned
+    attributes: %w(dn cn memberuid member submember uniquemember memberof)
+}
+adapter.ldap_search(options)
+```
+
+### Update user accounts when the `dn` and email change
+
+The following will require that any accounts with the new email address are removed.  Emails have to be unique in GitLab.  This is expected to work but unverified as of yet.
+
+```ruby
+# Here's an example with a couple users.
+# Each entry will have to include the old username and the new email
+emails = {
+  'ORIGINAL_USERNAME' => 'NEW_EMAIL_ADDRESS',
+  ...
+}
+
+emails.each do |username, email|
+  user = User.find_by_username(username)
+  user.email = email
+  user.skip_reconfirmation!
+  user.save!
+end
+
+# Run the UserSync to grab update the above users' data
+LdapSyncWorker.new.perform
+```
+
+## Routes
+
+### Remove redirecting routes
+
+<https://gitlab.com/gitlab-org/gitlab-ce/issues/41758#note_54828133>
+
+```ruby
+path = 'foo'
+conflicting_permanent_redirects = RedirectRoute.matching_path_and_descendants(path)
+
+# Check that conflicting_permanent_redirects is as expected
+conflicting_permanent_redirects.destroy_all
+```
+
+## MRs
+
+### Find MR
+
+```ruby
+m = project.merge_requests.find_by(iid: <IID>)
+m = MergeRequest.find_by_title('NEEDS UNIQUE TITLE!!!')
+```
+
+### Close a merge request properly (if merged but still marked as open)
+
+```ruby
+p = Project.find_by_full_path('')
+m = project.merge_requests.find_by(iid: )
+u = User.find_by_username('')
+MergeRequests::PostMergeService.new(p, u).execute(m)
+```
+
+### Rebase manually
+
+```ruby
+p = Project.find_by_full_path('')
+m = project.merge_requests.find_by(iid: )
+u = User.find_by_username('')
+MergeRequests::RebaseService.new(m.target_project, u).execute(m)
+```
+
+## CI
+
+### Cancel stuck pending pipelines
+
+<https://gitlab.com/gitlab-com/support-forum/issues/2449#note_41929707>
+
+```ruby
+Ci::Pipeline.where(project_id: p.id).where(status: 'pending').count
+Ci::Pipeline.where(project_id: p.id).where(status: 'pending').each {|p| p.cancel}
+Ci::Pipeline.where(project_id: p.id).where(status: 'pending').count
+```
+
+### Manually modify runner minutes
+
+```ruby
+Namespace.find_by_full_path("user/proj").namespace_statistics.update(shared_runners_seconds: 27360)
+```
+
+### Remove artifacts more than a week old
+
+```ruby
+### SELECTING THE BUILDS TO CLEAR
+# For a single project:
+project = Project.find_by_full_path('')
+builds_with_artifacts =  project.builds.with_artifacts_archive
+
+# Prior to 10.6 the above line would be:
+# builds_with_artifacts = project.builds.with_artifacts
+
+# Instance-wide:
+builds_with_artifacts = Ci::Build.with_artifacts
+
+### CLEAR THEM OUT
+builds_to_clear = builds_with_artifacts.where("finished_at < ?", 1.week.ago)
+builds_to_clear.each do |build|
+  build.artifacts_expire_at = Time.now
+  build.erase_erasable_artifacts!
+end
+```
+
+### Find reason failure (for when build trace is empty) (Introduced in 10.3.0)
+
+<https://gitlab.com/gitlab-org/gitlab-ce/issues/41111>
+
+```ruby
+build = Ci::Build.find(78420)
+
+build.failure_reason
+
+build.dependencies.each do |d| { puts "status: #{d.status}, finished at: #{d.finished_at},
+completed: #{d.complete?}, artifacts_expired: #{d.artifacts_expired?}, erased: #{d.erased?}" }
+```
+
+### Disable strict artifact checking (Introduced in GitLab 10.3.0)
+
+<https://docs.gitlab.com/ee/administration/job_artifacts.html#validation-for-dependencies>
+
+```ruby
+Feature.enable('ci_disable_validates_dependencies')
+```
+
+### Remove CI traces older than 6 months
+
+```ruby
+current_user = User.find_by_email('cindy@gitlap.com')
+Ci::Build.where("finished_at < ?", 6.months.ago.to_date).each {|b| puts b.id; b.erase(erased_by: current_user) if b.erasable?};nil
+```
+
+### Try CI service
+
+```ruby
+p = Project.find_by_full_path('')
+m = project.merge_requests.find_by(iid: )
+m.project.try(:ci_service)
+```
+
+### Disable AutoDevOps on Existing Projects
+
+```ruby
+Project.all.each do |p|
+  p.auto_devops_attributes={"enabled"=>"0"}
+  p.save
+end
+```
+
+## License
+
+### See license plan name (since v9.3.0-ee)
+
+```ruby
+License.current.plan
+```
+
+### Check if a project feature is available on the instance
+
+Features listed in https://gitlab.com/gitlab-org/gitlab-ee/blob/master/ee/app/models/license.rb
+
+```ruby
+License.current.feature_available?(:jira_dev_panel_integration)
+```
+
+### Check if a project feature is available in a project
+
+Features listed in https://gitlab.com/gitlab-org/gitlab-ee/blob/master/ee/app/models/license.rb
+
+```ruby
+p = Project.find_by_full_path('<group>/<project>')
+p.feature_available?(:jira_dev_panel_integration)
+```
+
+### Add a license through the console
+
+```ruby
+key = "<key>"
+license = License.new(data: key)
+license.save
+License.current # check to make sure it applied
+```
+
+## Unicorn
+
+From #91083
+
+### Poll unicorn requests by seconds
+
+```ruby
+require 'rubygems'
+require 'unicorn'
+
+# Usage for this program
+def usage
+  puts "ruby unicorn_status.rb <path to unix socket> <poll interval in seconds>"
+  puts "Polls the given Unix socket every interval in seconds. Will not allow you to drop below 3 second poll intervals."
+  puts "Example: /opt/gitlab/embedded/bin/ruby poll_unicorn.rb /var/opt/gitlab/gitlab-rails/sockets/gitlab.socket 10"
+end
+
+# Look for required args. Throw usage and exit if they don't exist.
+if ARGV.count < 2
+  usage
+  exit 1
+end
+
+# Get the socket and threshold values.
+socket = ARGV[0]
+threshold = (ARGV[1]).to_i
+
+# Check threshold - is it less than 3? If so, set to 3 seconds. Safety first!
+if threshold.to_i < 3
+  threshold = 3
+end
+
+# Check - does that socket exist?
+unless File.exist?(socket)
+  puts "Socket file not found: #{socket}"
+  exit 1
+end
+
+# Poll the given socket every THRESHOLD seconds as specified above.
+puts "Running infinite loop. Use CTRL+C to exit."
+puts "------------------------------------------"
+loop do
+  Raindrops::Linux.unix_listener_stats([socket]).each do |addr, stats|
+    puts DateTime.now.to_s + " Active: " + stats.active.to_s + " Queued: " + stats.queued.to_s
+  end
+  sleep threshold
+end
+```
+
+## Sidekiq
+
+### Size of a queue
+
+```ruby
+# Run this in the rails console
+Sidekiq::Queue.new('background_migration').size
+```
+
+### Kill a worker's Sidekiq jobs
+
+```ruby
+queue = Sidekiq::Queue.new('repository_import')
+queue.each { |job| job.delete if <condition>}
+```
+
+### Enable debug logging of Sidekiq
+
+```ruby
+gitlab_rails['env'] = {
+  'SIDEKIQ_LOG_ARGUMENTS' => "1"
+}
+```
+
+Then `gitlab-ctl reconfigure; gitlab-ctl restart sidekiq`.  The Sidekiq logs will now include additional data for troubleshooting.
+
+### Sidekiq kill signals
+
+https://github.com/mperham/sidekiq/wiki/Signals#ttin
+
+## Redis
+
+### Connect to redis (omnibus)
+
+```shell
+/opt/gitlab/embedded/bin/redis-cli -s /var/opt/gitlab/redis/redis.socket
+```
+
+### Connect to redis (HA)
+
+```shell
+/opt/gitlab/embedded/bin/redis-cli -h <host ip> -a <password>
+```
+
+## LFS
+
+### Get info about LFS objects and associated project
+
+```ruby
+o=LfsObject.find_by(oid: "<oid>")
+p=Project.find(LfsObjectsProject.find_by_lfs_object_id(o.id).project_id)
+```
+
+You can then delete these records from the database with:
+
+```ruby
+LfsObjectsProject.find_by_lfs_object_id(o.id).destroy
+o.destroy
+```
+
+which you would also want to combine with deleting the LFS file in the LFS storage area on disk. It remains to be seen exactly how or whether the deletion is useful, however.
+
+## Decryption Problems
+
+### Bad Decrypt Script (for encrypted variables)
+
+<https://gitlab.com/snippets/1730735/raw>
+
+This script will go through all the encrypted variables and count how many are not able to be decrypted. Might be helpful to run on multiple nodes to see which `gitlab-secrets.json` file is most up to date. To run:
+
+```bash
+wget -O /tmp/bad-decrypt.rb https://gitlab.com/snippets/1730735/raw
+gitlab-rails runner /tmp/bad-decrypt.rb
+```
+
+If `ProjectImportData Bad count:` is detected and the decision is made to delete the encrypted credentials to allow manual reentry.  This can be done with
+
+```ruby
+  # Find the ids of the corrupt ProjectImportData objects
+  total = 0
+  bad = []
+  ProjectImportData.find_each do |data|
+    begin
+      total += 1
+      data.credentials
+    rescue => e
+      bad << data.id
+    end
+  end
+
+  puts "Bad count: #{bad.count} / #{total}"
+
+  # See the bad ProjectImportData ids
+  bad
+
+  # Remove the corrupted credentials
+  import_data = ProjectImportData.where(id: bad)
+  import_data.each do |data|
+    data.update_columns({ encrypted_credentials: nil, encrypted_credentials_iv: nil, encrypted_credentials_salt: nil})
+  end
+```
+
+If `User OTP Secret Bad count:` is detected.  For each user listed disable/enable two-factor authentication.
+
+### Decrypt Script for encrypted tokens
+
+This script will search for all encrypted tokens that are causing decryption errors, and update or reset as needed 
+
+```bash
+wget -O /tmp/encrypted-tokens.rb https://gitlab.com/snippets/1876342/raw
+gitlab-rails runner /tmp/encrypted-tokens.rb
+```
+
+## Geo
+
+### Artifacts
+
+#### Find failed artifacts
+
+```ruby
+Geo::JobArtifactRegistry.failed
+```
+
+#### Download artifact
+
+```ruby
+Gitlab::Geo::JobArtifactDownloader.new(:job_artifact, <artifact_id>).execute
+```
+
+#### Get a count of the synced artifacts
+
+```ruby
+Geo::JobArtifactRegistry.synced.count
+```
+
+#### Find `ID` of synced artifacts that are missing on primary
+
+```ruby
+Geo::JobArtifactRegistry.synced.missing_on_primary.pluck(:artifact_id)
+```
+
+### Repository verification failures
+
+#### Get the number of verification failed repositories
+
+```ruby
+Geo::ProjectRegistryFinder.new.count_verification_failed_repositories
+```
+
+#### Find the verification failed repositories
+
+```ruby
+Geo::ProjectRegistry.verification_failed_repos
+```
+
+### Find repositories that failed to sync
+
+```ruby
+Geo::ProjectRegistryFinder.new.find_failed_project_registries('repository')
+```
+
+### Resync repositories
+
+#### Queue up all repositories for resync. Sidekiq will handle each sync
+
+```ruby
+Geo::ProjectRegistry.update_all(resync_repository: true, resync_wiki: true)
+```
+
+#### Sync individual repository now
+
+```ruby
+project = Project.find_by_full_path('<group/project>')
+
+Geo::RepositorySyncService.new(project).execute
+```
\ No newline at end of file
diff --git a/doc/administration/troubleshooting/linux_cheat_sheet.md b/doc/administration/troubleshooting/linux_cheat_sheet.md
new file mode 100644
index 00000000000..8b57fdb9119
--- /dev/null
+++ b/doc/administration/troubleshooting/linux_cheat_sheet.md
@@ -0,0 +1,317 @@
+---
+is_hidden: true
+comments: false
+---
+
+# Linux Cheat Sheet
+
+CAUTION: **CAUTION:**
+If you are administering GitLab you are expected to know these commands for your distro of choice. If you are a GitLab Support Engineer, consider this a cross-reference to translate `yum` -> `apt-get` and the like.
+
+CAUTION: **CAUTION:**
+Most of the commands below have not been labeled as to which distro they work
+on. Contributions are welcome to help add them.
+
+## System Commands
+
+### Distro Information
+
+```bash
+# Debian/Ubuntu
+uname -a
+lsb_release -a
+
+# CentOS/RedHat
+cat /etc/centos-release
+cat /etc/redhat-release
+
+# This will provide a lot more information
+cat /etc/os-release
+```
+
+### Shut down or Reboot
+
+```bash
+shutdown -h now
+reboot
+```
+
+### Permissions
+
+```bash
+# change the user:group ownership of a file/dir
+chown root:git <file_or_dir>
+
+# make a file executable
+chmod u+x <file>
+```
+
+### Files & Dirs
+
+```bash
+# create a new directory and all subdirectories
+mkdir -p dir/dir2/dir3
+
+# Send a command's output to file.txt, no STDOUT
+ls > file.txt
+
+# Send a command's output to file.txt AND see it in STDOUT
+ls | tee /tmp/file.txt
+
+# Search and Replace within a file 
+sed -i 's/original-text/new-text/g' <filename>
+```
+
+### See all set environment variables
+
+```bash
+env
+```
+
+## Searching
+
+### File names
+
+```bash
+# search for a file in a filesystem
+find . -name 'filename.rb' -print
+
+# locate a file
+locate <filename>
+
+# see command history
+history
+
+# search CLI history
+<ctrl>-R
+```
+
+### File contents
+
+```bash
+# -B/A = show 2 lines before/after search_term
+grep -B 2 -A 2 search_term <filename>
+
+# -<number> shows both before and after
+grep -2 search_term <filename>
+
+# Search on all files in directory (recursively)
+grep -r search_term <directory>
+
+# search through *.gz files is the same except with zgrep
+zgrep search_term <filename>
+
+# Fast grep printing lines containing a string pattern
+fgrep -R string_pattern <filename or directory>
+```
+
+### CLI
+
+```bash
+# View command history
+history
+
+# Run last command that started with 'his' (3 letters min)
+!his
+
+# Search through command history
+<ctrl>-R
+
+# Execute last command with sudo
+sudo !!
+```
+
+## Managing resources
+
+### Memory, Disk, & CPU usage
+
+```bash
+# disk space info.  The '-h' gives the data in human-readable values
+df -h
+
+# size of each file/dir and its contents in the current dir
+du -hd 1
+
+# or alternative
+du -h --max-depth=1
+
+# find files greater than certain size(k, M, G) and list them in order
+# get rid of the + for exact, - for less than
+find / -type f -size +100M -print0 | xargs -0 du -hs | sort -h
+
+# Find free memory on a system
+free -m
+
+# Find what processes are using memory/CPU and organize by it
+# Load average is 1/CPU for 1, 5, and 15 minutes
+top -o %MEM
+top -o %CPU
+```
+
+### Strace
+
+```bash
+# strace a process
+strace -tt -T -f -y -s 1024 -p <pid>
+
+# -tt   print timestamps with microsecond accuracy
+
+# -T    print the time spent in each syscall
+
+# -f    also trace any child processes that forked
+
+# -y    print the path associated with file handles
+
+# -s    max string length to print for an event
+
+# -o    output file
+
+# run strace on all unicorn processes
+ps auwx | grep unicorn | awk '{ print " -p " $2}' | xargs strace -tt -T -f -y -s 1024 -o /tmp/unicorn.txt
+```
+
+See the [strace zine](https://wizardzines.com/zines/strace/) linked on the resources page for a quick walkthrough.
+
+Brendan Gregg has [a more detailed explanation of how to use strace](http://www.brendangregg.com/blog/2014-05-11/strace-wow-much-syscall.html).
+
+Be aware that strace can have major impacts to system performance when it is running.
+
+### The Strace Parser tool
+Our [strace-parser tool](https://gitlab.com/wchandler/strace-parser) can be used to provide a high level summary of the `strace` output.  It is similar to `strace -C`, but provides much more detailed statistics.
+
+MacOS and Linux binaries can be downloaded [here](https://gitlab.com/gitlab-com/support/toolbox/strace-parser/-/tags), or you can build it from source if you have the Rust compiler.
+
+#### How to use the tool:
+First run the tool with no arguments other than the strace output file name to get a summary of the top processes sorted by time spent actively performing tasks.  You can also sort based on total time, # of syscalls made, PID #, and # of child processes using the `-S` or `--sort` flag.  The number of results defaults to 25 processes, but can be changed using the `-c`/`--count` option.  See `--help` for full details.
+
+
+   ``` bash
+   $ ./strace-parser strace.txt
+
+   Top 25 PIDs
+-----------
+
+  pid       	active (ms)	 wait (ms)	total (ms)	 % active	 syscalls
+  ----------	----------	---------	---------	---------	---------
+  8795      	   689.072	 45773.832	 46462.902	   16.89%	    23018
+  13408     	   679.432	 55910.891	 56590.320	   16.65%	    28593
+  6423      	   554.822	 13175.485	 13730.308	   13.60%	    13735
+   ...
+   ```
+
+Based on the summary, you can then view the details of syscalls made by one or more procsses using the `-p`/`--pid` for a specific process, or `-s`/`--stats` flags for a sorted list.  `--stats` takes the same sorting and count options as summary.
+
+```
+$ ./strace-parse strace.text -p 6423
+
+PID 6423
+13735 syscalls, active time: 554.822ms, total time: 13730.308ms
+
+  syscall        	   count	     total	       max	       avg	       min	errors
+                 	        	      (ms)	      (ms)	      (ms)	      (ms)
+  ---------------	--------	----------	----------	----------	----------	--------
+  epoll_wait     	     628	 13175.485	    21.259	    20.980	     0.020
+  clock_gettime  	    7326	   199.500	     0.249	     0.027	     0.013
+  stat           	    2101	   110.768	    19.056	     0.053	     0.017	ENOENT: 2076
+  ...
+  ---------------
+
+  Parent PID: 495
+  Child PIDs:  8383, 8418, 8419, 8420, 8421
+
+  Slowest file access times for PID 6423:
+
+     open (ms)	      timestamp	             error	   file name
+  -----------	---------------	   ---------------	   ----------
+      29.818	10:53:11.528954	                  	   /srv/gitlab-data/builds/2018_08/6174/954448.log
+      12.309	10:53:46.708274	                  	   /srv/gitlab-data/builds/2018_08/5342/954186.log
+      0.039	10:53:49.222110	                  	   /opt/gitlab/embedded/service/gitlab-rails/app/views/events/event/_note.html.haml
+      0.035	10:53:49.125115	                  	   /opt/gitlab/embedded/service/gitlab-rails/app/views/events/event/_push.html.haml
+  ...
+```
+
+In the example above, we can see that file opening times on `/srv/gitlab-data` are extremely slow, about 100X slower than `/opt/gitlab`.
+
+When nothing stands out in the results, a good way to get more context is to run `strace` on your own GitLab instance while performing the action performed by the customer, then compare summaries of both results and dive into the differences.
+
+#### Stats for the open syscall
+
+Rough numbers for calls to `open` and `openat` (used to access files) on various configurations.  Slow storage can cause the dreaded DeadlineExceeded error in Gitaly.
+
+Also [see this entry](https://docs.gitlab.com/ee/administration/operations/filesystem_benchmarking.html) in the handbook for quick tests customers can perform to check their filesystem performance.
+
+Keep in mind that timing information from `strace` is often somewhat inaccurate, so small differences should not be considered significant.
+
+|Setup          | access times  |
+|:--------------|:--------------|
+| EFS           | 10 - 30ms     |
+| Local Storage | 0.01 - 1ms     |
+
+
+## Networking
+
+### Ports
+
+```bash
+# Find the programs that are listening on ports
+netstat -plnt
+ss -plnt
+lsof -i -P | grep <port>
+```
+
+### Internet/DNS
+
+```bash
+# Show domain IP address
+dig +short example.com
+nslookup example.com
+
+# Check DNS using specific nameserver
+# 8.8.8.8 = google, 1.1.1.1 = cloudflare, 208.67.222.222 = opendns
+dig @8.8.8.8 example.com
+nslookup example.com 1.1.1.1
+
+# Find host provider
+whois <ip_address> | grep -i "orgname\|netname"
+
+# Curl headers with redirect
+curl -IL https://example.com
+```
+
+## Package Management
+
+```bash
+# Debian/Ubuntu
+
+# List packages
+dpkg -l
+apt list --installed
+
+# Find an installed package
+dpkg -l | grep <package>
+apt list --installed | grep <package>
+
+# Install a package 
+dpkg -i <package_name>.deb
+apt-get install <package>
+apt install <package>
+
+# CentOS/RedHat
+
+# Install a package
+yum install <package>
+dnf install <package> # RHEL/CentOS 8+
+
+rpm -ivh <package_name>.rpm
+
+# Find an installed package
+rpm -qa | grep <package>
+```
+
+## Logs
+
+```bash
+# Print last lines in log file where 'n'
+# is the number of lines to print
+tail -n /path/to/log/file
+```
\ No newline at end of file
diff --git a/doc/administration/troubleshooting/test_environments.md b/doc/administration/troubleshooting/test_environments.md
new file mode 100644
index 00000000000..548b17c2cfd
--- /dev/null
+++ b/doc/administration/troubleshooting/test_environments.md
@@ -0,0 +1,121 @@
+---
+is_hidden: true
+comments: false
+---
+
+# Apps for a Testing Environment
+
+## Docker
+
+NOTE: **Note:**
+This page was initially written for Support Engineers, so some of the links
+are only available internally at GitLab.
+
+NOTE: **Note:**
+The following were tested on docker containers running in the cloud.
+ Support Engineers, please see [these docs](https://gitlab.com/gitlab-com/dev-resources/tree/master/dev-resources#running-docker-containers)
+on how to run Docker containers on `dev-resources`. Other setups haven't been tested, but contributions are
+welcome.
+
+### GitLab
+
+Please see [our Docker test environment docs](https://docs.gitlab.com/ee/install/digitaloceandocker.html#create-new-gitlab-container) for how to run GitLab with docker-machine. When spinning this up in an instance running docker-machine, ensure you change a few things:
+
+   1. Update the name of the `docker-machine` host. You can see a list of hosts
+      with `docker-machine ls`.
+   1. Expose the necessary ports using the `-p` flag. Docker normally doesn't
+      allow access to any ports it uses outside of the container so they must be
+      explicitly exposed.
+   1. Add any necessary `gitlab.rb` configuration to the
+      `GITLAB_OMNIBUS_CONFIG` variable.
+
+For example, when the `docker-machine` host we want to use is `do-docker`:
+
+   ```sh
+   docker run --detach --name gitlab \
+   --env GITLAB_OMNIBUS_CONFIG="external_url 'http://$(docker-machine ip do-docker)'; gitlab_rails['gitlab_shell_ssh_port'] = 2222;" \
+   --hostname $(docker-machine ip do-docker) \
+   -p 80:80 -p 2222:22 \
+   gitlab/gitlab-ee:11.5.3-ee.0
+   ```
+
+### SAML
+
+##### SAML for Authentication
+
+We can use the [test-saml-idp Docker image](https://hub.docker.com/r/jamedjo/test-saml-idp) to do the work for us:
+
+   ```sh
+   docker run --name gitlab_saml -p 8080:8080 -p 8443:8443 \
+   -e SIMPLESAMLPHP_SP_ENTITY_ID=<GITLAB_IP_OR_DOMAIN> \
+   -e SIMPLESAMLPHP_SP_ASSERTION_CONSUMER_SERVICE=<GITLAB_IP_OR_DOMAIN>/users/auth/saml/callback \
+   -d jamedjo/test-saml-idp
+   ```
+
+The following will also need to go in your `/etc/gitlab/gitlab.rb`. See [our docs](https://docs.gitlab.com/ee/integration/saml.html) for more. [See the default usernames, passwords, and emails](https://hub.docker.com/r/jamedjo/test-saml-idp/#usage)
+
+   ```ruby
+   gitlab_rails['omniauth_enabled'] = true
+   gitlab_rails['omniauth_allow_single_sign_on'] = ['saml']
+   gitlab_rails['omniauth_sync_email_from_provider'] = 'saml'
+   gitlab_rails['omniauth_sync_profile_from_provider'] = ['saml']
+   gitlab_rails['omniauth_sync_profile_attributes'] = ['email']
+   gitlab_rails['omniauth_auto_sign_in_with_provider'] = 'saml'
+   gitlab_rails['omniauth_block_auto_created_users'] = false
+   gitlab_rails['omniauth_auto_link_ldap_user'] = false
+   gitlab_rails['omniauth_auto_link_saml_user'] = true
+   gitlab_rails['omniauth_providers'] = [
+     {
+       "name" => "saml",
+       "label" => "SAML",
+       "args" => {
+         assertion_consumer_service_url: '<GITLAB_IP_OR_DOMAIN>/users/auth/saml/callback',
+         idp_cert_fingerprint: '119b9e027959cdb7c662cfd075d9e2ef384e445f',
+         idp_sso_target_url: '<SAML_IP_OR_DOMAIN>:8080/simplesaml/saml2/idp/SSOService.php',
+         issuer: '<GITLAB_IP_OR_DOMAIN>',
+         name_identifier_format: 'urn:oasis:names:tc:SAML:2.0:nameid-format:persistent'
+       }
+     }
+   ]
+   ```
+
+##### GroupSAML for GitLab.com
+
+See [these GDK docs](https://gitlab.com/gitlab-org/gitlab-development-kit/blob/master/doc/howto/saml.md)
+
+### LDAP
+
+Coming soon
+
+### ElasticSearch
+
+   ```sh
+   docker run -d --name elasticsearch \
+   -p 9200:9200 -p 9300:9300 \
+   -e "discovery.type=single-node" \
+   docker.elastic.co/elasticsearch/elasticsearch:5.5.1
+   ```
+
+Then confirm it works in the browser at `curl http://<IP_ADDRESS>:9200/_cat/health`.
+  ElasticSearch's default username is `elastic` and password is `changeme`.
+
+### PlantUML
+
+See [our docs](https://docs.gitlab.com/ee/administration/integration/plantuml.html#docker) on running PlantUML in Docker.
+
+### Jira
+
+```sh
+docker run -d -p 8081:8080 cptactionhank/atlassian-jira:latest
+```
+
+Then go to `<IP_ADDRESS>:8081` in the browser to set it up.  This requires a
+Jira license.
+
+### Grafana
+
+```sh
+docker run -d --name grafana -e "GF_SECURITY_ADMIN_PASSWORD=gitlab" -p 3000:3000 grafana/grafana
+```
+
+Access it at `<IP_ADDRESS>:3000`.
\ No newline at end of file