diff options
Diffstat (limited to 'doc/administration/troubleshooting/debug.md')
| -rw-r--r-- | doc/administration/troubleshooting/debug.md | 292 |
1 files changed, 7 insertions, 285 deletions
diff --git a/doc/administration/troubleshooting/debug.md b/doc/administration/troubleshooting/debug.md index 81ca1bda5d0..d5019f1aa4a 100644 --- a/doc/administration/troubleshooting/debug.md +++ b/doc/administration/troubleshooting/debug.md @@ -1,289 +1,11 @@ --- -stage: Systems -group: Distribution -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +redirect_to: '../reference_architectures/troubleshooting.md' +remove_date: '2022-10-19' --- -# Debugging tips **(FREE SELF)** +This document was moved to [another location](../reference_architectures/troubleshooting.md). -Sometimes things don't work the way they should. Here are some tips on debugging issues out -in production. - -## Starting a Rails console session - -Troubleshooting and debugging your GitLab instance often requires a Rails console. - -Your type of GitLab installation determines how -[to start a rails console](../operations/rails_console.md). -See also: - -- [GitLab Rails Console Cheat Sheet](gitlab_rails_cheat_sheet.md). -- [Navigating GitLab via Rails console](navigating_gitlab_via_rails_console.md). - -### Enabling Active Record logging - -You can enable output of Active Record debug logging in the Rails console -session by running: - -```ruby -ActiveRecord::Base.logger = Logger.new($stdout) -``` - -This will show information about database queries triggered by any Ruby code -you may run in the console. To turn off logging again, run: - -```ruby -ActiveRecord::Base.logger = nil -``` - -### Disabling database statement timeout - -You can disable the PostgreSQL statement timeout for the current Rails console -session by running: - -```ruby -ActiveRecord::Base.connection.execute('SET statement_timeout TO 0') -``` - -This change only affects the current Rails console session and will -not be persisted in the GitLab production environment or in the next Rails -console session. - -### Output Rails console session history - -If you'd like to output your Rails console command history in a format that's -easy to copy and save for future reference, you can run: - -```ruby -puts Readline::HISTORY.to_a -``` - -## Using the Rails runner - -If you need to run some Ruby code in the context of your GitLab production -environment, you can do so using the [Rails runner](https://guides.rubyonrails.org/command_line.html#rails-runner). When executing a script file, the script must be accessible by the `git` user. - -**For Omnibus installations** - -```shell -sudo gitlab-rails runner "RAILS_COMMAND" - -# Example with a two-line Ruby script -sudo gitlab-rails runner "user = User.first; puts user.username" - -# Example with a ruby script file (make sure to use the full path) -sudo gitlab-rails runner /path/to/script.rb -``` - -**For installations from source** - -```shell -sudo -u git -H bundle exec rails runner -e production "RAILS_COMMAND" - -# Example with a two-line Ruby script -sudo -u git -H bundle exec rails runner -e production "user = User.first; puts user.username" - -# Example with a ruby script file (make sure to use the full path) -sudo -u git -H bundle exec rails runner -e production /path/to/script.rb -``` - -## Mail not working - -A common problem is that mails are not being sent for some reason. Suppose you configured -an SMTP server, but you're not seeing mail delivered. Here's how to check the settings: - -1. Run a [Rails console](../operations/rails_console.md#starting-a-rails-console-session). - -1. Look at the ActionMailer `delivery_method` to make sure it matches what you - intended. If you configured SMTP, it should say `:smtp`. If you're using - Sendmail, it should say `:sendmail`: - - ```ruby - irb(main):001:0> ActionMailer::Base.delivery_method - => :smtp - ``` - -1. If you're using SMTP, check the mail settings: - - ```ruby - irb(main):002:0> ActionMailer::Base.smtp_settings - => {:address=>"localhost", :port=>25, :domain=>"localhost.localdomain", :user_name=>nil, :password=>nil, :authentication=>nil, :enable_starttls_auto=>true} - ``` - - In the example above, the SMTP server is configured for the local machine. If this is intended, you may need to check your local mail - logs (for example, `/var/log/mail.log`) for more details. - -1. Send a test message via the console. - - ```ruby - irb(main):003:0> Notify.test_email('youremail@email.com', 'Hello World', 'This is a test message').deliver_now - ``` - - If you do not receive an email and/or see an error message, then check - your mail server settings. - -## Advanced Issues - -For more advanced issues, `gdb` is a must-have tool for debugging issues. - -### The GNU Project Debugger (GDB) - -To install on Ubuntu/Debian: - -```shell -sudo apt-get install gdb -``` - -On CentOS: - -```shell -sudo yum install gdb -``` - -<!-- vale gitlab.Spelling = NO --> - -### rbtrace - -<!-- vale gitlab.Spelling = YES --> - -GitLab 11.2 ships with [`rbtrace`](https://github.com/tmm1/rbtrace), which -allows you to trace Ruby code, view all running threads, take memory dumps, -and more. However, this is not enabled by default. To enable it, define the -`ENABLE_RBTRACE` variable to the environment. For example, in Omnibus: - -```ruby -gitlab_rails['env'] = {"ENABLE_RBTRACE" => "1"} -``` - -Then reconfigure the system and restart Puma and Sidekiq. To run this -in Omnibus, run as root: - -```ruby -/opt/gitlab/embedded/bin/ruby /opt/gitlab/embedded/bin/rbtrace -``` - -## Common Problems - -Many of the tips to diagnose issues below apply to many different situations. We'll use one -concrete example to illustrate what you can do to learn what is going wrong. - -### 502 Gateway Timeout after Unicorn spins at 100% CPU - -This error occurs when the Web server times out (default: 60 s) after not -hearing back from the Unicorn worker. If the CPU spins to 100% while this in -progress, there may be something taking longer than it should. - -To fix this issue, we first need to figure out what is happening. The -following tips are only recommended if you do not mind users being affected by -downtime. Otherwise skip to the next section. - -1. Load the problematic URL -1. Run `sudo gdb -p <PID>` to attach to the Puma process. -1. In the GDB window, type: - - ```plaintext - call (void) rb_backtrace() - ``` - -1. This forces the process to generate a Ruby backtrace. Check - `/var/log/gitlab/puma/puma_stderr.log` for the backtrace. For example, you may see: - - ```plaintext - from /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/metrics/sampler.rb:33:in `block in start' - from /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/metrics/sampler.rb:33:in `loop' - from /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/metrics/sampler.rb:36:in `block (2 levels) in start' - from /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/metrics/sampler.rb:44:in `sample' - from /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/metrics/sampler.rb:68:in `sample_objects' - from /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/metrics/sampler.rb:68:in `each_with_object' - from /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/metrics/sampler.rb:68:in `each' - from /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/metrics/sampler.rb:69:in `block in sample_objects' - from /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/metrics/sampler.rb:69:in `name' - ``` - -1. To see the current threads, run: - - ```plaintext - thread apply all bt - ``` - -1. Once you're done debugging with `gdb`, be sure to detach from the process and exit: - - ```plaintext - detach - exit - ``` - -If the Puma process terminates before you are able to run these -commands, GDB will report an error. To buy more time, you can always raise the -Puma worker timeout. For omnibus users, you can edit `/etc/gitlab/gitlab.rb` and -increase it from 60 seconds to 600: - -```ruby -gitlab_rails['env'] = { - 'GITLAB_RAILS_RACK_TIMEOUT' => 600 -} -``` - -For source installations, set the environment variable. -Refer to [Puma Worker timeout](../operations/puma.md#change-the-worker-timeout). - -[Reconfigure](../restart_gitlab.md#omnibus-gitlab-reconfigure) GitLab for the changes to take effect. - -#### Troubleshooting without affecting other users - -The previous section attached to a running Unicorn process, and this may have -undesirable effects for users trying to access GitLab during this time. If you -are concerned about affecting others during a production system, you can run a -separate Rails process to debug the issue: - -1. Log in to your GitLab account. -1. Copy the URL that is causing problems (for example, `https://gitlab.com/ABC`). -1. Create a Personal Access Token for your user (User Settings -> Access Tokens). -1. Bring up the [GitLab Rails console.](../operations/rails_console.md#starting-a-rails-console-session) -1. At the Rails console, run: - - ```ruby - app.get '<URL FROM STEP 2>/?private_token=<TOKEN FROM STEP 3>' - ``` - - For example: - - ```ruby - app.get 'https://gitlab.com/gitlab-org/gitlab-foss/-/issues/1?private_token=123456' - ``` - -1. In a new window, run `top`. It should show this Ruby process using 100% CPU. Write down the PID. -1. Follow step 2 from the previous section on using GDB. - -### GitLab: API is not accessible - -This often occurs when GitLab Shell attempts to request authorization via the -[internal API](../../development/internal_api/index.md) (for example, `http://localhost:8080/api/v4/internal/allowed`), and -something in the check fails. There are many reasons why this may happen: - -1. Timeout connecting to a database (for example, PostgreSQL or Redis) -1. Error in Git hooks or push rules -1. Error accessing the repository (for example, stale NFS handles) - -To diagnose this problem, try to reproduce the problem and then see if there -is a Unicorn worker that is spinning via `top`. Try to use the `gdb` -techniques above. In addition, using `strace` may help isolate issues: - -```shell -strace -ttTfyyy -s 1024 -p <PID of puma worker> -o /tmp/puma.txt -``` - -If you cannot isolate which Unicorn worker is the issue, try to run `strace` -on all the Unicorn workers to see where the -[`/internal/allowed`](../../development/internal_api/index.md) endpoint gets stuck: - -```shell -ps auwx | grep puma | awk '{ print " -p " $2}' | xargs strace -ttTfyyy -s 1024 -o /tmp/puma.txt -``` - -The output in `/tmp/puma.txt` may help diagnose the root cause. - -## More information - -- [Debugging Stuck Ruby Processes](https://newrelic.com/blog/best-practices/debugging-stuck-ruby-processes-what-to-do-before-you-kill-9) -- [Cheat sheet of using GDB and Ruby processes](gdb-stuck-ruby.txt) +<!-- This redirect file can be deleted after 2022-10-19. --> +<!-- Redirects that point to other docs in the same project expire in three months. --> +<!-- Redirects that point to docs in a different project or site (for example, link is not relative and starts with `https:`) expire in one year. --> +<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/redirects.html --> |