diff options
Diffstat (limited to 'doc/administration/troubleshooting')
9 files changed, 92 insertions, 53 deletions
diff --git a/doc/administration/troubleshooting/debug.md b/doc/administration/troubleshooting/debug.md index a1b4df9b94e..ef95bdc8602 100644 --- a/doc/administration/troubleshooting/debug.md +++ b/doc/administration/troubleshooting/debug.md @@ -5,27 +5,15 @@ in production. ## Starting a Rails console session -Troubleshooting and debugging your GitLab instance often requires a -[Rails console](https://guides.rubyonrails.org/command_line.html#rails-console). +Troubleshooting and debugging your GitLab instance often requires a Rails console. + +Your type of GitLab installation determines how +[to start a rails console](../operations/rails_console.md). See also: - [GitLab Rails Console Cheat Sheet](gitlab_rails_cheat_sheet.md). - [Navigating GitLab via Rails console](navigating_gitlab_via_rails_console.md). -**For Omnibus installations** - -```shell -sudo gitlab-rails console -``` - -**For installations from source** - -```shell -sudo -u git -H bundle exec rails console -e production -``` - -Kubernetes: the console is in the task-runner pod, refer to our [Kubernetes cheat sheet](kubernetes_cheat_sheet.md#gitlab-specific-kubernetes-information) for details. - ### Enabling Active Record logging You can enable output of Active Record debug logging in the Rails console @@ -98,7 +86,7 @@ sudo -u git -H bundle exec rails runner -e production /path/to/script.rb A common problem is that mails are not being sent for some reason. Suppose you configured an SMTP server, but you're not seeing mail delivered. Here's how to check the settings: -1. Run a [Rails console](#starting-a-rails-console-session). +1. Run a [Rails console](../operations/rails_console.md#starting-a-rails-console-session). 1. Look at the ActionMailer `delivery_method` to make sure it matches what you intended. If you configured SMTP, it should say `:smtp`. If you're using @@ -238,7 +226,7 @@ separate Rails process to debug the issue: 1. Log in to your GitLab account. 1. Copy the URL that is causing problems (e.g. `https://gitlab.com/ABC`). 1. Create a Personal Access Token for your user (Profile Settings -> Access Tokens). -1. Bring up the [GitLab Rails console.](#starting-a-rails-console-session) +1. Bring up the [GitLab Rails console.](../operations/rails_console.md#starting-a-rails-console-session) 1. At the Rails console, run: ```ruby diff --git a/doc/administration/troubleshooting/elasticsearch.md b/doc/administration/troubleshooting/elasticsearch.md index e13261e3074..6de5bb71d75 100644 --- a/doc/administration/troubleshooting/elasticsearch.md +++ b/doc/administration/troubleshooting/elasticsearch.md @@ -164,7 +164,7 @@ Troubleshooting search result issues is rather straight forward on Elasticsearch The first step is to confirm GitLab is using Elasticsearch for the search function. To do this: -1. Confirm the integration is enabled in **Admin Area > Settings > Integrations**. +1. Confirm the integration is enabled in **Admin Area > Settings > General**. 1. Confirm searches utilize Elasticsearch by accessing the rails console (`sudo gitlab-rails console`) and running the following commands: @@ -206,7 +206,7 @@ The best place to start is to determine if the issue is with creating an empty i If it is, check on the Elasticsearch side to determine if the `gitlab-production` (the name for the GitLab index) exists. If it exists, manually delete it on the Elasticsearch side and attempt to recreate it from the -[`recreate_index`](../../integration/elasticsearch.md#gitlab-elasticsearch-rake-tasks) +[`recreate_index`](../../integration/elasticsearch.md#gitlab-advanced-search-rake-tasks) Rake task. If you still encounter issues, try creating an index manually on the Elasticsearch @@ -225,8 +225,8 @@ during the indexing of projects. If errors do occur, they will either stem from If the indexing process does not present errors, you will want to check the status of the indexed projects. You can do this via the following Rake tasks: -- [`sudo gitlab-rake gitlab:elastic:index_projects_status`](../../integration/elasticsearch.md#gitlab-elasticsearch-rake-tasks) (shows the overall status) -- [`sudo gitlab-rake gitlab:elastic:projects_not_indexed`](../../integration/elasticsearch.md#gitlab-elasticsearch-rake-tasks) (shows specific projects that are not indexed) +- [`sudo gitlab-rake gitlab:elastic:index_projects_status`](../../integration/elasticsearch.md#gitlab-advanced-search-rake-tasks) (shows the overall status) +- [`sudo gitlab-rake gitlab:elastic:projects_not_indexed`](../../integration/elasticsearch.md#gitlab-advanced-search-rake-tasks) (shows specific projects that are not indexed) If: diff --git a/doc/administration/troubleshooting/gitlab_rails_cheat_sheet.md b/doc/administration/troubleshooting/gitlab_rails_cheat_sheet.md index 9a23a115765..dbda889d370 100644 --- a/doc/administration/troubleshooting/gitlab_rails_cheat_sheet.md +++ b/doc/administration/troubleshooting/gitlab_rails_cheat_sheet.md @@ -222,11 +222,12 @@ namespace = Namespace.find_by_full_path("") ```ruby # ID will be the webhook_id -WebHookLog.where(web_hook_id: ID).each_slice(ID) do |slice| - slice.each(&:destroy) -end +hook=WebHook.find(ID) + +WebHooks::DestroyService.new(current_user).execute(hook) -WebHook.find(ID).destroy +#In case the service gets timeout consider removing webhook_logs +hook.web_hook_logs.limit(BATCH_SIZE).delete_all ``` ### Bulk update service integration password for _all_ projects @@ -285,6 +286,16 @@ GitlabShellWorker.perform_in(0, :remove_repository, p.repository_storage, p.wiki p.create_wiki ### creates the wiki project on the filesystem ``` +## Issue boards + +### In case of issue boards not loading properly and it's getting time out. We need to call the Issue Rebalancing service to fix this + +```ruby +p=Project.find_by_full_path('PROJECT PATH') + +IssueRebalancingService.new(p.issues.take).execute +``` + ## Imports / Exports ```ruby @@ -411,6 +422,9 @@ user.skip_reconfirmation! # Active users on the instance, now User.active.count +# Users taking a seat on the instance +License.current.current_active_users_count + # The historical max on the instance as of the past year ::HistoricalData.max_historical_user_count ``` @@ -506,6 +520,54 @@ group = Group.find_by_path_or_name('group-name') group.project_creation_level=0 ``` +## SCIM + +### Fixing bad SCIM identities + +```ruby +def delete_bad_scim(email, group_path) + output = "" + u = User.find_by_email(email) + uid = u.id + g = Group.find_by_full_path(group_path) + saml_prov_id = SamlProvider.find_by(group_id: g.id).id + saml = Identity.where(user_id: uid, saml_provider_id: saml_prov_id) + scim = ScimIdentity.where(user_id: uid , group_id: g.id) + if saml[0] + saml_eid = saml[0].extern_uid + output += "%s," % [email] + output += "SAML: %s," % [saml_eid] + if scim[0] + scim_eid = scim[0].extern_uid + output += "SCIM: %s" % [scim_eid] + if saml_eid == scim_eid + output += " Identities matched, not deleted \n" + else + scim[0].destroy + output += " Deleted \n" + end + else + output = "ERROR No SCIM identify found for: [%s]\n" % [email] + puts output + return 1 + end + else + output = "ERROR No SAML identify found for: [%s]\n" % [email] + puts output + return 1 + end + puts output + return 0 +end + +# In case of multiple emails +emails = [email1, email2] + +emails.each do |e| + delete_bad_scim(e,'GROUPPATH') +end +``` + ## Routes ### Remove redirecting routes @@ -525,8 +587,8 @@ conflicting_permanent_redirects.destroy_all ### Close a merge request properly (if merged but still marked as open) ```ruby -p = Project.find_by_full_path('') -m = project.merge_requests.find_by(iid: ) +p = Project.find_by_full_path('<full/path/to/project>') +m = p.merge_requests.find_by(iid: <iid>) u = User.find_by_username('') MergeRequests::PostMergeService.new(p, u).execute(m) ``` diff --git a/doc/administration/troubleshooting/kubernetes_cheat_sheet.md b/doc/administration/troubleshooting/kubernetes_cheat_sheet.md index 01532032b49..7c5a9e0d79f 100644 --- a/doc/administration/troubleshooting/kubernetes_cheat_sheet.md +++ b/doc/administration/troubleshooting/kubernetes_cheat_sheet.md @@ -84,8 +84,7 @@ and they will assist you with any issues you are having. ## GitLab-specific Kubernetes information -- Minimal config that can be used to test a Kubernetes Helm chart can be found - [here](https://gitlab.com/gitlab-org/charts/gitlab/-/issues/620). +- Minimal configuration that can be used to [test a Kubernetes Helm chart](https://gitlab.com/gitlab-org/charts/gitlab/-/issues/620). - Tailing logs of a separate pod. An example for a Webservice pod: @@ -190,7 +189,7 @@ and they will assist you with any issues you are having. be possible to use [Updating GitLab using the Helm Chart](https://docs.gitlab.com/charts/index.html#updating-gitlab-using-the-helm-chart) for upgrades. -- How to apply changes to GitLab config: +- How to apply changes to GitLab configuration: - Modify the `gitlab.yaml` file. - Run the following command to apply changes: @@ -255,7 +254,7 @@ to those documents for details. helm install gitlab -f <path-to-yaml-file> gitlab/gitlab ``` - If you want to modify some GitLab settings, you can use the above-mentioned config + If you want to modify some GitLab settings, you can use the above-mentioned configuration as a base and create your own YAML file. - Monitor the installation progress via `helm status gitlab` and `minikube dashboard`. diff --git a/doc/administration/troubleshooting/log_parsing.md b/doc/administration/troubleshooting/log_parsing.md index dcd1df2f423..7914628a756 100644 --- a/doc/administration/troubleshooting/log_parsing.md +++ b/doc/administration/troubleshooting/log_parsing.md @@ -3,11 +3,11 @@ We recommend using log aggregation and search tools like Kibana and Splunk whenever possible, but if they are not available you can still quickly parse [GitLab logs](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/26311) in JSON format -(the default since GitLab 12.0) using [`jq`](https://stedolan.github.io/jq/). +(the default in GitLab 12.0 and later) using [`jq`](https://stedolan.github.io/jq/). ## What is JQ? -As noted in its [manual](https://stedolan.github.io/jq/manual/), jq is a command-line JSON processor. The following examples +As noted in its [manual](https://stedolan.github.io/jq/manual/), `jq` is a command-line JSON processor. The following examples include use cases targeted for parsing GitLab log files. ## Parsing Logs diff --git a/doc/administration/troubleshooting/navigating_gitlab_via_rails_console.md b/doc/administration/troubleshooting/navigating_gitlab_via_rails_console.md index 571973c12d9..a1485249b0e 100644 --- a/doc/administration/troubleshooting/navigating_gitlab_via_rails_console.md +++ b/doc/administration/troubleshooting/navigating_gitlab_via_rails_console.md @@ -3,7 +3,7 @@ At the heart of GitLab is a web application [built using the Ruby on Rails framework](https://about.gitlab.com/blog/2018/10/29/why-we-use-rails-to-build-gitlab/). Thanks to this, we also get access to the amazing tools built right into Rails. -In this guide, we'll introduce the [Rails console](debug.md#starting-a-rails-console-session) +In this guide, we'll introduce the [Rails console](../operations/rails_console.md#starting-a-rails-console-session) and the basics of interacting with your GitLab instance from the command line. CAUTION: **Caution:** @@ -20,20 +20,10 @@ Rails experience is helpful to have but not a must. ## Starting a Rails console session -Omnibus GitLab comes with a convenient wrapper command which automatically loads -the production GitLab environment: +Your type of GitLab installation determines how +[to start a rails console](../operations/rails_console.md). -```shell -sudo gitlab-rails console -``` - -For source installations, you'll have to instead run: - -```shell -sudo -u git -H bundle exec rails console -e production -``` - -Further code examples will all take place inside the Rails console and also +The following code examples will all take place inside the Rails console and also assume an Omnibus GitLab installation. ## Active Record objects diff --git a/doc/administration/troubleshooting/sidekiq.md b/doc/administration/troubleshooting/sidekiq.md index 404e806c5d9..b7762f8ac3e 100644 --- a/doc/administration/troubleshooting/sidekiq.md +++ b/doc/administration/troubleshooting/sidekiq.md @@ -7,7 +7,7 @@ may be filling up. Users will notice when this happens because new branches may not show up and merge requests may not be updated. The following are some troubleshooting steps that will help you diagnose the bottleneck. -NOTE **Note:** +NOTE: **Note:** GitLab administrators/users should consider working through these debug steps with GitLab Support so the backtraces can be analyzed by our team. It may reveal a bug or necessary improvement in GitLab. diff --git a/doc/administration/troubleshooting/ssl.md b/doc/administration/troubleshooting/ssl.md index e6c081e1eea..4d4b9755fa9 100644 --- a/doc/administration/troubleshooting/ssl.md +++ b/doc/administration/troubleshooting/ssl.md @@ -23,7 +23,7 @@ After configuring a GitLab instance with an internal CA certificate, you might n More details here: https://curl.haxx.se/docs/sslcerts.html ``` -- Testing via the [rails console](debug.md#starting-a-rails-console-session) also fails: +- Testing via the [rails console](../operations/rails_console.md#starting-a-rails-console-session) also fails: ```ruby uri = URI.parse("https://gitlab.domain.tld") @@ -205,6 +205,6 @@ Some of these errors come from the Excon Ruby gem, and could be generated in cir where GitLab is configured to initiate an HTTPS session to a remote server that is serving just HTTP. -One scenario is that you're using [object storage](../high_availability/object_storage.md) +One scenario is that you're using [object storage](../object_storage.md) which is not served under HTTPS. GitLab is misconfigured and attempts a TLS handshake, but the object storage will respond with plain HTTP. diff --git a/doc/administration/troubleshooting/tracing_correlation_id.md b/doc/administration/troubleshooting/tracing_correlation_id.md index 03c342595a3..ae9ebd90951 100644 --- a/doc/administration/troubleshooting/tracing_correlation_id.md +++ b/doc/administration/troubleshooting/tracing_correlation_id.md @@ -4,7 +4,7 @@ type: reference # Finding relevant log entries with a correlation ID -Since GitLab 11.6, a unique request tracking ID, known as the "correlation ID" has been +In GitLab 11.6 and later, a unique request tracking ID, known as the "correlation ID" has been logged by the GitLab instance for most requests. Each individual request to GitLab gets its own correlation ID, which then gets logged in each GitLab component's logs for that request. This makes it easier to trace behavior in a @@ -122,5 +122,5 @@ If you have done some horizontal scaling in your GitLab infrastructure, then you will need to search across _all_ of your GitLab nodes. You can do this with some sort of log aggregation software like Loki, ELK, Splunk, or others. -You can use a tool like Ansible or PSSH (parellel SSH) that can execute identical commands across your servers in +You can use a tool like Ansible or PSSH (parallel SSH) that can execute identical commands across your servers in parallel, or craft your own solution. |