diff options
author | GitLab Bot <gitlab-bot@gitlab.com> | 2022-11-17 11:33:21 +0000 |
---|---|---|
committer | GitLab Bot <gitlab-bot@gitlab.com> | 2022-11-17 11:33:21 +0000 |
commit | 7021455bd1ed7b125c55eb1b33c5a01f2bc55ee0 (patch) | |
tree | 5bdc2229f5198d516781f8d24eace62fc7e589e9 /doc/administration/troubleshooting | |
parent | 185b095e93520f96e9cfc31d9c3e69b498cdab7c (diff) | |
download | gitlab-ce-7021455bd1ed7b125c55eb1b33c5a01f2bc55ee0.tar.gz |
Add latest changes from gitlab-org/gitlab@15-6-stable-eev15.6.0-rc42
Diffstat (limited to 'doc/administration/troubleshooting')
12 files changed, 116 insertions, 1041 deletions
diff --git a/doc/administration/troubleshooting/debug.md b/doc/administration/troubleshooting/debug.md deleted file mode 100644 index d5019f1aa4a..00000000000 --- a/doc/administration/troubleshooting/debug.md +++ /dev/null @@ -1,11 +0,0 @@ ---- -redirect_to: '../reference_architectures/troubleshooting.md' -remove_date: '2022-10-19' ---- - -This document was moved to [another location](../reference_architectures/troubleshooting.md). - -<!-- This redirect file can be deleted after 2022-10-19. --> -<!-- Redirects that point to other docs in the same project expire in three months. --> -<!-- Redirects that point to docs in a different project or site (for example, link is not relative and starts with `https:`) expire in one year. --> -<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/redirects.html --> diff --git a/doc/administration/troubleshooting/defcon.md b/doc/administration/troubleshooting/defcon.md deleted file mode 100644 index f2554f523f0..00000000000 --- a/doc/administration/troubleshooting/defcon.md +++ /dev/null @@ -1,11 +0,0 @@ ---- -redirect_to: '../../ci/troubleshooting.md#disaster-recovery' -remove_date: '2022-10-04' ---- - -This document was moved to [another location](../../ci/troubleshooting.md#disaster-recovery). - -<!-- This redirect file can be deleted after <2022-10-04>. --> -<!-- Redirects that point to other docs in the same project expire in three months. --> -<!-- Redirects that point to docs in a different project or site (for example, link is not relative and starts with `https:`) expire in one year. --> -<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/redirects.html --> diff --git a/doc/administration/troubleshooting/gitlab_rails_cheat_sheet.md b/doc/administration/troubleshooting/gitlab_rails_cheat_sheet.md index 20ce52a9094..6993d48b450 100644 --- a/doc/administration/troubleshooting/gitlab_rails_cheat_sheet.md +++ b/doc/administration/troubleshooting/gitlab_rails_cheat_sheet.md @@ -1,660 +1,11 @@ --- -stage: Systems -group: Distribution -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments +redirect_to: 'index.md' +remove_date: '2023-02-01' --- -# GitLab Rails Console Cheat Sheet **(FREE SELF)** +This document was moved to [another location](index.md). -This is the GitLab Support Team's collection of information regarding the GitLab Rails -console, for use while troubleshooting. It is listed here for transparency, -and for users with experience with these tools. If you are currently -having an issue with GitLab, it is highly recommended that you first check -our guide on [our Rails console](../operations/rails_console.md), -and your [support options](https://about.gitlab.com/support/), before attempting to use -this information. - -WARNING: -Some of these scripts could be damaging if not run correctly, -or under the right conditions. We highly recommend running them under the -guidance of a Support Engineer, or running them in a test environment with a -backup of the instance ready to be restored, just in case. - -WARNING: -As GitLab changes, changes to the code are inevitable, -and so some scripts may not work as they once used to. These are not kept -up-to-date as these scripts/commands were added as they were found/needed. As -mentioned above, we recommend running these scripts under the supervision of a -Support Engineer, who can also verify that they continue to work as they -should and, if needed, update the script for the latest version of GitLab. - -## Attributes - -View available attributes, formatted using pretty print (`pp`). - -For example, determine what attributes contain users' names and email addresses: - -```ruby -u = User.find_by_username('someuser') -pp u.attributes -``` - -Partial output: - -```plaintext -{"id"=>1234, - "email"=>"someuser@example.com", - "sign_in_count"=>99, - "name"=>"S User", - "username"=>"someuser", - "first_name"=>nil, - "last_name"=>nil, - "bot_type"=>nil} -``` - -Then make use of the attributes, [testing SMTP, for example](https://docs.gitlab.com/omnibus/settings/smtp.html#testing-the-smtp-configuration): - -```ruby -e = u.email -n = u.name -Notify.test_email(e, "Test email for #{n}", 'Test email').deliver_now -# -Notify.test_email(u.email, "Test email for #{u.name}", 'Test email').deliver_now -``` - -## Imports and exports - -### Import a project - -```ruby -# Find the project and get the error -p = Project.find_by_full_path('<username-or-group>/<project-name>') - -p.import_error - -# To finish the import on GitLab running version before 11.6 -p.import_finish - -# To finish the import on GitLab running version 11.6 or after -p.import_state.mark_as_failed("Failed manually through console.") -``` - -### Rename imported repository - -In a specific situation, an imported repository needed to be renamed. The Support -Team was informed of a backup restore that failed on a single repository, which created -the project with an empty repository. The project was successfully restored to a development -instance, then exported, and imported into a new project under a different name. - -The Support Team was able to transfer the incorrectly named imported project into the -correctly named empty project using the steps below. - -Move the new repository to the empty repository: - -```shell -mv /var/opt/gitlab/git-data/repositories/<group>/<new-project> /var/opt/gitlab/git-data/repositories/<group>/<empty-project> -``` - -Make sure the permissions are correct: - -```shell -chown -R git:git <path-to-directory>.git -``` - -Clear the cache: - -```shell -sudo gitlab-rake cache:clear -``` - -### Export a project - -It's typically recommended to export a project through [the web interface](../../user/project/settings/import_export.md#export-a-project-and-its-data) or through [the API](../../api/project_import_export.md). In situations where this is not working as expected, it may be preferable to export a project directly via the Rails console: - -```ruby -user = User.find_by_username('<username>') -# Sufficient permissions needed -# Read https://docs.gitlab.com/ee/user/permissions.html#project-members-permissions - -project = Project.find_by_full_path('<username-or-group>/<project-name') -Projects::ImportExport::ExportService.new(project, user).execute -``` - -If this all runs successfully, you see an output like the following before being returned to the Rails console prompt: - -```ruby -=> nil -``` - -The exported project is located in a `.tar.gz` file in `/var/opt/gitlab/gitlab-rails/uploads/-/system/import_export_upload/export_file/`. - -If this fails, [enable verbose logging](../operations/rails_console.md#looking-up-database-persisted-objects), -repeat the above procedure after, -and report the output to -[GitLab Support](https://about.gitlab.com/support/). - -## Mirrors - -### Find mirrors with "bad decrypt" errors - -This content has been converted to a Rake task, see [verify database values can be decrypted using the current secrets](../raketasks/check.md#verify-database-values-can-be-decrypted-using-the-current-secrets). - -### Transfer mirror users and tokens to a single service account - -This content has been moved to [Troubleshooting Repository mirroring](../../user/project/repository/mirror/index.md#transfer-mirror-users-and-tokens-to-a-single-service-account-in-rails-console). - -## Users - -### Create new user - -```ruby -u = User.new(username: 'test_user', email: 'test@example.com', name: 'Test User', password: 'password', password_confirmation: 'password') -u.skip_confirmation! # Use it only if you wish user to be automatically confirmed. If skipped, user receives confirmation e-mail -u.save! -``` - -### Skip reconfirmation - -```ruby -user = User.find_by_username('<username>') -user.skip_reconfirmation! -``` - -### Disable 2fa for single user - -**In GitLab 13.5 and later:** - -Use the code under [Disable 2FA | For a single user](../../security/two_factor_authentication.md#for-a-single-user) so that the target user -is notified that 2FA has been disabled. - -**In GitLab 13.4 and earlier:** - -```ruby -user = User.find_by_username('<username>') -user.disable_two_factor! -``` - -### Active users & Historical users - -```ruby -# Active users on the instance, now -User.active.count - -# Users taking a seat on the instance -User.billable.count - -# The historical max on the instance as of the past year -::HistoricalData.max_historical_user_count(from: 1.year.ago.beginning_of_day, to: Time.current.end_of_day) -``` - -Using cURL and jq (up to a max 100, see [Pagination](../../api/index.md#pagination)): - -```shell -curl --silent --header "Private-Token: ********************" \ - "https://gitlab.example.com/api/v4/users?per_page=100&active" | jq --compact-output '.[] | [.id,.name,.username]' -``` - -### Update Daily Billable & Historical users - -```ruby -# Forces recount of historical (max) users -::HistoricalDataWorker.new.perform - -# Forces recount of daily billable users -identifier = Analytics::UsageTrends::Measurement.identifiers[:billable_users] -::Analytics::UsageTrends::CounterJobWorker.new.perform(identifier, User.minimum(:id), User.maximum(:id), Time.zone.now) -``` - -### Block or Delete Users that have no projects or groups - -```ruby -users = User.where('id NOT IN (select distinct(user_id) from project_authorizations)') - -# How many users are removed? -users.count - -# If that count looks sane: - -# You can either block the users: -users.each { |user| user.blocked? ? nil : user.block! } - -# Or you can delete them: - # need 'current user' (your user) for auditing purposes -current_user = User.find_by(username: '<your username>') - -users.each do |user| - DeleteUserWorker.perform_async(current_user.id, user.id) -end -``` - -### Deactivate Users that have no recent activity - -```ruby -days_inactive = 90 -inactive_users = User.active.where("last_activity_on <= ?", days_inactive.days.ago) - -inactive_users.each do |user| - puts "user '#{user.username}': #{user.last_activity_on}" - user.deactivate! -end -``` - -### Block Users that have no recent activity - -```ruby -days_inactive = 90 -inactive_users = User.active.where("last_activity_on <= ?", days_inactive.days.ago) - -inactive_users.each do |user| - puts "user '#{user.username}': #{user.last_activity_on}" - user.block! -end -``` - -### Find a user's max permissions for project/group - -```ruby -user = User.find_by_username 'username' -project = Project.find_by_full_path 'group/project' -user.max_member_access_for_project project.id -``` - -```ruby -user = User.find_by_username 'username' -group = Group.find_by_full_path 'group' -user.max_member_access_for_group group.id -``` - -## Merge requests - -### Close a merge request - -```ruby -u = User.find_by_username('<username>') -p = Project.find_by_full_path('<namespace/project>') -m = p.merge_requests.find_by(iid: <iid>) -MergeRequests::CloseService.new(project: p, current_user: u).execute(m) -``` - -### Delete a merge request - -```ruby -u = User.find_by_username('<username>') -p = Project.find_by_full_path('<namespace/project>') -m = p.merge_requests.find_by(iid: <iid>) -Issuable::DestroyService.new(project: m.project, current_user: u).execute(m) -``` - -### Rebase manually - -```ruby -u = User.find_by_username('<username>') -p = Project.find_by_full_path('<namespace/project>') -m = p.merge_requests.find_by(iid: <iid>) -MergeRequests::RebaseService.new(project: m.target_project, current_user: u).execute(m) -``` - -### Set a merge request as merged - -Use when a merge request was accepted and the changes merged into the Git repository, -but the merge request still shows as open. - -If the changes are not merged yet, this action causes the merge request to -incorrectly show `merged into <branch-name>`. - -```ruby -u = User.find_by_username('<username>') -p = Project.find_by_full_path('<namespace/project>') -m = p.merge_requests.find_by(iid: <iid>) -MergeRequests::PostMergeService.new(project: p, current_user: u).execute(m) -``` - -## CI - -### Cancel stuck pending pipelines - -For more information, see the [confidential issue](../../user/project/issues/confidential_issues.md) -`https://gitlab.com/gitlab-com/support-forum/issues/2449#note_41929707`. - -```ruby -Ci::Pipeline.where(project_id: p.id).where(status: 'pending').count -Ci::Pipeline.where(project_id: p.id).where(status: 'pending').each {|p| p.cancel if p.stuck?} -Ci::Pipeline.where(project_id: p.id).where(status: 'pending').count -``` - -### Remove artifacts more than a week old - -This section has been moved to the [job artifacts troubleshooting documentation](../job_artifacts.md#delete-job-artifacts-from-jobs-completed-before-a-specific-date). - -### Find reason failure (for when build trace is empty) (Introduced in 10.3.0) - -See <https://gitlab.com/gitlab-org/gitlab-foss/-/issues/41111>. - -```ruby -build = Ci::Build.find(78420) - -build.failure_reason - -build.dependencies.each do |d| { puts "status: #{d.status}, finished at: #{d.finished_at}, - completed: #{d.complete?}, artifacts_expired: #{d.artifacts_expired?}, erased: #{d.erased?}" } -``` - -### Try CI integration - -```ruby -p = Project.find_by_full_path('<project_path>') -m = project.merge_requests.find_by(iid: ) -m.project.try(:ci_integration) -``` - -### Validate the `.gitlab-ci.yml` - -```ruby -project = Project.find_by_full_path 'group/project' -content = project.repository.gitlab_ci_yml_for(project.repository.root_ref_sha) -Gitlab::Ci::Lint.new(project: project, current_user: User.first).validate(content) -``` - -### Disable AutoDevOps on Existing Projects - -```ruby -Project.all.each do |p| - p.auto_devops_attributes={"enabled"=>"0"} - p.save -end -``` - -### Obtain runners registration token - -```ruby -Gitlab::CurrentSettings.current_application_settings.runners_registration_token -``` - -### Seed runners registration token - -```ruby -appSetting = Gitlab::CurrentSettings.current_application_settings -appSetting.set_runners_registration_token('<new-runners-registration-token>') -appSetting.save! -``` - -### Run pipeline schedules manually - -You can run pipeline schedules manually through the Rails console to reveal any errors that are usually not visible. - -```ruby -# schedule_id can be obtained from Edit Pipeline Schedule page -schedule = Ci::PipelineSchedule.find_by(id: <schedule_id>) - -# Select the user that you want to run the schedule for -user = User.find_by_username('<username>') - -# Run the schedule -ps = Ci::CreatePipelineService.new(schedule.project, user, ref: schedule.ref).execute!(:schedule, ignore_skip_ci: true, save_on_errors: false, schedule: schedule) -``` - -## License - -### See current license information - -```ruby -# License information (name, company, email address) -License.current.licensee - -# Plan: -License.current.plan - -# Uploaded: -License.current.created_at - -# Started: -License.current.starts_at - -# Expires at: -License.current.expires_at - -# Is this a trial license? -License.current.trial? - -# License ID for lookup on CustomersDot -License.current.license_id - -# License data in Base64-encoded ASCII format -License.current.data -``` - -### Check if a project feature is available on the instance - -Features listed in <https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/app/models/license.rb>. - -```ruby -License.current.feature_available?(:jira_dev_panel_integration) -``` - -### Check if a project feature is available in a project - -Features listed in [`license.rb`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/app/models/license.rb). - -```ruby -p = Project.find_by_full_path('<group>/<project>') -p.feature_available?(:jira_dev_panel_integration) -``` - -### Add a license through the console - -```ruby -key = "<key>" -license = License.new(data: key) -license.save -License.current # check to make sure it applied -``` - -This is needed for example in a known edge-case with -[expired license and multiple LDAP servers](../auth/ldap/ldap-troubleshooting.md#expired-license-causes-errors-with-multiple-ldap-servers). - -### Remove licenses - -To clean up the [License History table](../../user/admin_area/license_file.md#view-license-details-and-history): - -```ruby -TYPE = :trial? -# or :expired? - -License.select(&TYPE).each(&:destroy!) - -# or even License.all.each(&:destroy!) -``` - -## Registry - -### Registry Disk Space Usage by Project - -Find this content in the [Container Registry troubleshooting documentation](../packages/container_registry.md#registry-disk-space-usage-by-project). - -### Run the Cleanup policy now - -Find this content in the [Container Registry troubleshooting documentation](../packages/container_registry.md#run-the-cleanup-policy-now). - -## Sidekiq - -This content has been moved to [Troubleshooting Sidekiq](sidekiq.md). - -## LFS - -### Get information about LFS objects and associated project - -```ruby -o = LfsObject.find_by(oid: "<oid>") -p = Project.find(LfsObjectsProject.find_by_lfs_object_id(o.id).project_id) -``` - -You can then delete these records from the database with: - -```ruby -LfsObjectsProject.find_by_lfs_object_id(o.id).destroy -o.destroy -``` - -You would also want to combine this with deleting the LFS file in the LFS storage -area on disk. It remains to be seen exactly how or whether the deletion is useful, however. - -## Decryption Problems - -### Bad Decrypt Script (for encrypted variables) - -This content has been converted to a Rake task, see [verify database values can be decrypted using the current secrets](../raketasks/check.md#verify-database-values-can-be-decrypted-using-the-current-secrets). - -As an example of repairing, if `ProjectImportData Bad count:` is detected and the decision is made to delete the -encrypted credentials to allow manual reentry: - -```ruby - # Find the ids of the corrupt ProjectImportData objects - total = 0 - bad = [] - ProjectImportData.find_each do |data| - begin - total += 1 - data.credentials - rescue => e - bad << data.id - end - end - - puts "Bad count: #{bad.count} / #{total}" - - # See the bad ProjectImportData ids - bad - - # Remove the corrupted credentials - import_data = ProjectImportData.where(id: bad) - import_data.each do |data| - data.update_columns({ encrypted_credentials: nil, encrypted_credentials_iv: nil, encrypted_credentials_salt: nil}) - end -``` - -If `User OTP Secret Bad count:` is detected. For each user listed disable/enable -two-factor authentication. - -The following script searches in some of the tables for encrypted tokens that are -causing decryption errors, and update or reset as needed: - -```shell -wget -O /tmp/encrypted-tokens.rb https://gitlab.com/snippets/1876342/raw -gitlab-rails runner /tmp/encrypted-tokens.rb -``` - -### Decrypt Script for encrypted tokens - -This content has been converted to a Rake task, see [verify database values can be decrypted using the current secrets](../raketasks/check.md#verify-database-values-can-be-decrypted-using-the-current-secrets). - -## Geo - -### Reverify all uploads (or any SSF data type which is verified) - -1. SSH into a GitLab Rails node in the primary Geo site. -1. Open [Rails console](../operations/rails_console.md). -1. Mark all uploads as "pending verification": - - ```ruby - Upload.verification_state_table_class.each_batch do |relation| - relation.update_all(verification_state: 0) - end - ``` - -1. This will cause the primary to start checksumming all Uploads. -1. When a primary successfully checksums a record, then all secondaries rechecksum as well, and they compare the values. - -A similar thing can be done for all Models handled by the [Geo Self-Service Framework](../../development/geo/framework.md) which have implemented verification: - -- `LfsObject` -- `MergeRequestDiff` -- `Packages::PackageFile` -- `Terraform::StateVersion` -- `SnippetRepository` -- `Ci::PipelineArtifact` -- `PagesDeployment` -- `Upload` -- `Ci::JobArtifact` -- `Ci::SecureFile` - -NOTE: -`GroupWikiRepository` is not in the previous list since verification is not implemented. -There is an [issue to implement this functionality in the Admin UI](https://gitlab.com/gitlab-org/gitlab/-/issues/364729). - -### Artifacts - -Moved to [Geo replication troubleshooting](../geo/replication/troubleshooting.md#find-failed-artifacts). - -### Repository verification failures - -Moved to [Geo replication troubleshooting](../geo/replication/troubleshooting.md#repository-verification-failures). - -### Resync repositories - -Moved to [Geo replication troubleshooting](../geo/replication/troubleshooting.md#resync-repositories). - -### Blob types - -Moved to [Geo replication troubleshooting](../geo/replication/troubleshooting.md#blob-types). - -## Generate Service Ping - -The [Service Ping Guide](../../development/service_ping/index.md) in our developer documentation -has more information about Service Ping. - -### Generate or get the cached Service Ping - -```ruby -Gitlab::Usage::ServicePingReport.for(output: :all_metrics_values, cached: true) -``` - -### Generate a fresh new Service Ping - -This also refreshes the cached Service Ping displayed in the Admin Area - -```ruby -Gitlab::Usage::ServicePingReport.for(output: :all_metrics_values) -``` - -### Generate and print - -Generates Service Ping data in JSON format. - -```shell -rake gitlab:usage_data:generate -``` - -Generates Service Ping data in YAML format: - -```shell -rake gitlab:usage_data:dump_sql_in_yaml -``` - -### Generate and send Service Ping - -Prints the metrics saved in `conversational_development_index_metrics`. - -```shell -rake gitlab:usage_data:generate_and_send -``` - -## GraphQL - -Call a [GraphQL](../../api/graphql/getting_started.md) endpoint through the Rails console: - -```ruby -query = <<~EOQ -query securityGetProjects($search: String!) { - projects(search: $search) { - nodes { - path - } - } -} -EOQ - -variables = { "search": "gitlab" } - -result = GitlabSchema.execute(query, variables: variables, context: { current_user: current_user }) -result.to_h -``` +<!-- This redirect file can be deleted after 2023-02-01. --> +<!-- Redirects that point to other docs in the same project expire in three months. --> +<!-- Redirects that point to docs in a different project or site (for example, link is not relative and starts with `https:`) expire in one year. --> +<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/redirects.html --> diff --git a/doc/administration/troubleshooting/group_saml_scim.md b/doc/administration/troubleshooting/group_saml_scim.md deleted file mode 100644 index b5187504231..00000000000 --- a/doc/administration/troubleshooting/group_saml_scim.md +++ /dev/null @@ -1,11 +0,0 @@ ---- -redirect_to: '../../user/group/saml_sso/example_saml_config.md' -remove_date: '2022-10-29' ---- - -This document was moved to [another location](../../user/group/saml_sso/example_saml_config.md). - -<!-- This redirect file can be deleted after <2022-10-29>. --> -<!-- Redirects that point to other docs in the same project expire in three months. --> -<!-- Redirects that point to docs in a different project or site (link is not relative and starts with `https:`) expire in one year. --> -<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/redirects.html --> diff --git a/doc/administration/troubleshooting/index.md b/doc/administration/troubleshooting/index.md index ce548f9e100..db572f202ea 100644 --- a/doc/administration/troubleshooting/index.md +++ b/doc/administration/troubleshooting/index.md @@ -9,19 +9,21 @@ info: To determine the technical writer assigned to the Stage/Group associated w This page documents a collection of resources to help you troubleshoot a GitLab installation. +This list is not necessarily comprehensive. If you don't find what you're looking +for in this list, you should search the documentation. + ## Troubleshooting guides - [SSL](ssl.md) - [Geo](../geo/replication/troubleshooting.md) -- [GitLab Rails console cheat sheet](gitlab_rails_cheat_sheet.md) -- [Example group SAML and SCIM configurations](../../user/group/saml_sso/example_saml_config.md) +- [SAML](../../user/group/saml_sso/troubleshooting.md) - [Kubernetes cheat sheet](https://docs.gitlab.com/charts/troubleshooting/kubernetes_cheat_sheet.html) - [Linux cheat sheet](linux_cheat_sheet.md) - [Parsing GitLab logs with `jq`](../logs/log_parsing.md) - [Diagnostics tools](diagnostics_tools.md) Some feature documentation pages also have a troubleshooting section at the end -that you can check for feature-specific help. +that you can check for feature-specific help, including helpful Rails commands. If you need a testing environment to troubleshoot, see the [apps for a testing environment](test_environments.md). diff --git a/doc/administration/troubleshooting/kubernetes_cheat_sheet.md b/doc/administration/troubleshooting/kubernetes_cheat_sheet.md deleted file mode 100644 index 15ec8d5940b..00000000000 --- a/doc/administration/troubleshooting/kubernetes_cheat_sheet.md +++ /dev/null @@ -1,11 +0,0 @@ ---- -redirect_to: 'https://docs.gitlab.com/charts/troubleshooting/kubernetes_cheat_sheet.html' -remove_date: '2022-10-05' ---- - -This document was moved to [another location](https://docs.gitlab.com/charts/troubleshooting/kubernetes_cheat_sheet.html). - -<!-- This redirect file can be deleted after 2022-10-05. --> -<!-- Redirects that point to other docs in the same project expire in three months. --> -<!-- Redirects that point to docs in a different project or site (for example, link is not relative and starts with `https:`) expire in one year. --> -<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/redirects.html --> diff --git a/doc/administration/troubleshooting/linux_cheat_sheet.md b/doc/administration/troubleshooting/linux_cheat_sheet.md index 0c6fcd31d1d..90cd1e24c79 100644 --- a/doc/administration/troubleshooting/linux_cheat_sheet.md +++ b/doc/administration/troubleshooting/linux_cheat_sheet.md @@ -210,7 +210,7 @@ using the `-s` or `--sort` flag. The number of results defaults to 25 processes, can be changed using the `-c`/`--count` option. See `--help` for full details. ```shell -$ ./strace-parser sidekiq_trace.txt summary -c15 -s=pid +$ ./strace-parser sidekiq_trace.txt summary -c15 -s=pid Top 15 PIDs by PID # ----------- @@ -244,7 +244,7 @@ processes using the `-p`/`--pid` for a specific process, or `-s`/`--stats` flags a sorted list. `--stats` takes the same sorting and count options as summary. ```shell -./strace-parser sidekiq_trace.txt p 16815 +./strace-parser sidekiq_trace.txt p 16815 PID 16815 diff --git a/doc/administration/troubleshooting/log_parsing.md b/doc/administration/troubleshooting/log_parsing.md deleted file mode 100644 index 929a49494be..00000000000 --- a/doc/administration/troubleshooting/log_parsing.md +++ /dev/null @@ -1,11 +0,0 @@ ---- -redirect_to: '../logs/log_parsing.md' -remove_date: '2022-10-24' ---- - -This document was moved to [another location](../logs/log_parsing.md). - -<!-- This redirect file can be deleted after <2022-10-24>. --> -<!-- Redirects that point to other docs in the same project expire in three months. --> -<!-- Redirects that point to docs in a different project or site (link is not relative and starts with `https:`) expire in one year. --> -<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/redirects.html --> diff --git a/doc/administration/troubleshooting/navigating_gitlab_via_rails_console.md b/doc/administration/troubleshooting/navigating_gitlab_via_rails_console.md deleted file mode 100644 index 09a5cb8d185..00000000000 --- a/doc/administration/troubleshooting/navigating_gitlab_via_rails_console.md +++ /dev/null @@ -1,11 +0,0 @@ ---- -redirect_to: '../operations/rails_console.md' -remove_date: '2022-10-05' ---- - -This document was moved to [another location](../operations/rails_console.md). - -<!-- This redirect file can be deleted after <2022-10-05>. --> -<!-- Redirects that point to other docs in the same project expire in three months. --> -<!-- Redirects that point to docs in a different project or site (link is not relative and starts with `https:`) expire in one year. --> -<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/redirects.html --> diff --git a/doc/administration/troubleshooting/postgresql.md b/doc/administration/troubleshooting/postgresql.md index 8a76d4f88ab..829fed38060 100644 --- a/doc/administration/troubleshooting/postgresql.md +++ b/doc/administration/troubleshooting/postgresql.md @@ -58,59 +58,6 @@ This section is for links to information elsewhere in the GitLab documentation. some of which is absolutely not for production use. Including: - Understanding EXPLAIN plans. -### Troubleshooting/Fixes - -- [GitLab database requirements](../../install/requirements.md#database), - including - - Support for MySQL was removed in GitLab 12.1; [migrate to PostgreSQL](../../update/mysql_to_postgresql.md). - - Required extension: `pg_trgm` - - Required extension: `btree_gist` - -- Errors like this in the `production/sidekiq` log; see: - [Set `default_transaction_isolation` into read committed](https://docs.gitlab.com/omnibus/settings/database.html#set-default_transaction_isolation-into-read-committed): - - ```plaintext - ActiveRecord::StatementInvalid PG::TRSerializationFailure: ERROR: could not serialize access due to concurrent update - ``` - -- PostgreSQL HA [replication slot errors](https://docs.gitlab.com/omnibus/settings/database.html#troubleshooting-upgrades-in-an-ha-cluster): - - ```plaintext - pg_basebackup: could not create temporary replication slot "pg_basebackup_12345": ERROR: all replication slots are in use - HINT: Free one or increase max_replication_slots. - ``` - -- Geo [replication errors](../geo/replication/troubleshooting.md#fixing-replication-errors) including: - - ```plaintext - ERROR: replication slots can only be used if max_replication_slots > 0 - - FATAL: could not start WAL streaming: ERROR: replication slot "geo_secondary_my_domain_com" does not exist - - Command exceeded allowed execution time - - PANIC: could not write to file 'pg_xlog/xlogtemp.123': No space left on device - ``` - -- [Checking Geo configuration](../geo/replication/troubleshooting.md), including: - - Reconfiguring hosts/ports. - - Checking and fixing user/password mappings. - -- [Common Geo errors](../geo/replication/troubleshooting.md#fixing-common-errors). - -- Mismatch in `pg_dump` and `psql` versions: - - ```plaintext - Dumping PostgreSQL database gitlabhq_production ... pg_dump: error: server version: 13.3; pg_dump version: 14.2 - pg_dump: error: aborting because of server version mismatch - ``` - - To fix this, see [Backup and restore a non-packaged PostgreSQL database](https://docs.gitlab.com/omnibus/settings/database.html#backup-and-restore-a-non-packaged-postgresql-database). - -- Deploying PostgreSQL on Azure Database for PostgreSQL - Flexible Server may result in an error stating `extension "btree_gist" is not allow-listed for "azure_pg_admin" users in Azure Database for PostgreSQL` - - To resolve the above error, [allow-list the extension](https://learn.microsoft.com/en-us/azure/postgresql/flexible-server/concepts-extensions#how-to-use-postgresql-extensions) prior to install. - ## Support topics ### Database deadlocks @@ -236,3 +183,96 @@ To temporarily change the statement timeout: 1. Perform the action for which you need a different timeout (for example the backup or the Rails command). 1. Revert the edit in `/var/opt/gitlab/gitlab-rails/etc/database.yml`. + +## Troubleshooting + +### Database is not accepting commands to avoid wraparound data loss + +This error likely means that AUTOVACUUM is failing to complete its run: + +```plaintext +ERROR: database is not accepting commands to avoid wraparound data loss in database "gitlabhq_production" +``` + +To resolve the error, run `VACUUM` manually: + +1. Stop GitLab with the command `gitlab-ctl stop`. +1. Place the database in single-user mode with the command: + + ```shell + /opt/gitlab/embedded/bin/postgres --single -D /var/opt/gitlab/postgresql/data gitlabhq_production + ``` + +1. In the `backend>` prompt, run `VACUUM;`. This command can take several minutes to complete. +1. Wait for the command to complete, then press <kbd>Control</kbd> + <kbd>D</kbd> to exit. +1. Start GitLab with the command `gitlab-ctl start`. + +### GitLab database requirements + +The [database requirements](../../install/requirements.md#database) for GitLab include: + +- Support for MySQL was removed in GitLab 12.1; [migrate to PostgreSQL](../../update/mysql_to_postgresql.md). +- Review and install the [required extension list](../../install/postgresql_extensions.md). + +### Serialization errors in the `production/sidekiq` log + +If you receive errors like this example in your `production/sidekiq` log, read +about [setting `default_transaction_isolation` into read committed](https://docs.gitlab.com/omnibus/settings/database.html#set-default_transaction_isolation-into-read-committed) to fix the problem: + +```plaintext +ActiveRecord::StatementInvalid PG::TRSerializationFailure: ERROR: could not serialize access due to concurrent update +``` + +### PostgreSQL replication slot errors + +If you receive errors like this example, read about how to resolve PostgreSQL HA +[replication slot errors](https://docs.gitlab.com/omnibus/settings/database.html#troubleshooting-upgrades-in-an-ha-cluster): + +```plaintext +pg_basebackup: could not create temporary replication slot "pg_basebackup_12345": ERROR: all replication slots are in use +HINT: Free one or increase max_replication_slots. +``` + +### Geo replication errors + +If you receive errors like this example, read about how to resolve +[Geo replication errors](../geo/replication/troubleshooting.md#fixing-postgresql-database-replication-errors): + +```plaintext +ERROR: replication slots can only be used if max_replication_slots > 0 + +FATAL: could not start WAL streaming: ERROR: replication slot "geo_secondary_my_domain_com" does not exist + +Command exceeded allowed execution time + +PANIC: could not write to file 'pg_xlog/xlogtemp.123': No space left on device +``` + +### Review Geo configuration and common errors + +When troubleshooting problems with Geo, you should: + +- Review [common Geo errors](../geo/replication/troubleshooting.md#fixing-common-errors). +- [Review your Geo configuration](../geo/replication/troubleshooting.md), including: + - Reconfiguring hosts and ports. + - Reviewing and fixing the user and password mappings. + +### Mismatch in `pg_dump` and `psql` versions + +If you receive errors like this example, read about how to +[back up and restore a non-packaged PostgreSQL database](https://docs.gitlab.com/omnibus/settings/database.html#backup-and-restore-a-non-packaged-postgresql-database): + +```plaintext +Dumping PostgreSQL database gitlabhq_production ... pg_dump: error: server version: 13.3; pg_dump version: 14.2 +pg_dump: error: aborting because of server version mismatch +``` + +### Extension `btree_gist` is not allow-listed + +Deploying PostgreSQL on an Azure Database for PostgreSQL - Flexible Server may result in this error: + +```plaintext +extension "btree_gist" is not allow-listed for "azure_pg_admin" users in Azure Database for PostgreSQL +``` + +To resolve this error, [allow-list the extension](https://learn.microsoft.com/en-us/azure/postgresql/flexible-server/concepts-extensions#how-to-use-postgresql-extensions) prior to install. diff --git a/doc/administration/troubleshooting/ssl.md b/doc/administration/troubleshooting/ssl.md index 245ff9f4982..bf8e6b45fde 100644 --- a/doc/administration/troubleshooting/ssl.md +++ b/doc/administration/troubleshooting/ssl.md @@ -1,263 +1,11 @@ --- -stage: Systems -group: Distribution -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments -type: reference +redirect_to: 'https://docs.gitlab.com/omnibus/settings/ssl/ssl_troubleshooting.html' +remove_date: '2023-10-27' --- -# Troubleshooting SSL **(FREE SELF)** +This document was moved to [another location](https://docs.gitlab.com/omnibus/settings/ssl/ssl_troubleshooting.html). -This page contains a list of common SSL-related errors and scenarios that you -may encounter while working with GitLab. It should serve as an addition to the -main SSL documentation: - -- [Omnibus SSL Configuration](https://docs.gitlab.com/omnibus/settings/ssl.html). -- [Self-signed certificates or custom Certification Authorities for GitLab Runner](https://docs.gitlab.com/runner/configuration/tls-self-signed.html). -- [Configure HTTPS manually](https://docs.gitlab.com/omnibus/settings/ssl.html#configure-https-manually). - -## Using an internal CA certificate with GitLab - -After configuring a GitLab instance with an internal CA certificate, you might -not be able to access it by using various CLI tools. You may experience the -following issues: - -- `curl` fails: - - ```shell - curl "https://gitlab.domain.tld" - curl: (60) SSL certificate problem: unable to get local issuer certificate - More details here: https://curl.haxx.se/docs/sslcerts.html - ``` - -- Testing by using the [rails console](../operations/rails_console.md#starting-a-rails-console-session) - also fails: - - ```ruby - uri = URI.parse("https://gitlab.domain.tld") - http = Net::HTTP.new(uri.host, uri.port) - http.use_ssl = true - http.verify_mode = 1 - response = http.request(Net::HTTP::Get.new(uri.request_uri)) - ... - Traceback (most recent call last): - 1: from (irb):5 - OpenSSL::SSL::SSLError (SSL_connect returned=1 errno=0 state=error: certificate verify failed (unable to get local issuer certificate)) - ``` - -- The error `SSL certificate problem: unable to get local issuer certificate` - is displayed when setting up a [mirror](../../user/project/repository/mirror/index.md) - from this GitLab instance. -- `openssl` works when specifying the path to the certificate: - - ```shell - /opt/gitlab/embedded/bin/openssl s_client -CAfile /root/my-cert.crt -connect gitlab.domain.tld:443 -servername gitlab.domain.tld - ``` - -If you have the previously described issues, add your certificate to -`/etc/gitlab/trusted-certs`, and then run `sudo gitlab-ctl reconfigure`. - -## X.509 key values mismatch error - -After configuring your instance with a certificate bundle, NGINX may display -the following error message: - -`SSL: error:0B080074:x509 certificate routines:X509_check_private_key:key values mismatch` - -This error message means that the server certificate and key you have provided -don't match. You can confirm this by running the following command and then -comparing the output: - -```shell -openssl rsa -noout -modulus -in path/to/your/.key | openssl md5 -openssl x509 -noout -modulus -in path/to/your/.crt | openssl md5 -``` - -The following is an example of an md5 output between a matching key and -certificate. Note the matching md5 hashes: - -```shell -$ openssl rsa -noout -modulus -in private.key | openssl md5 -4f49b61b25225abeb7542b29ae20e98c -$ openssl x509 -noout -modulus -in public.crt | openssl md5 -4f49b61b25225abeb7542b29ae20e98c -``` - -This is an opposing output with a non-matching key and certificate which shows -different md5 hashes: - -```shell -$ openssl rsa -noout -modulus -in private.key | openssl md5 -d418865077299af27707b1d1fa83cd99 -$ openssl x509 -noout -modulus -in public.crt | openssl md5 -4f49b61b25225abeb7542b29ae20e98c -``` - -If the two outputs differ like the previous example, there's a mismatch between -the certificate and key. Contact the provider of the SSL certificate for -further support. - -## Using GitLab Runner with a GitLab instance configured with internal CA certificate or self-signed certificate - -Besides getting the errors mentioned in -[Using an internal CA certificate with GitLab](ssl.md#using-an-internal-ca-certificate-with-gitlab), -your CI pipelines may get stuck in `Pending` status. In the runner logs you may -see the following error message: - -```shell -Dec 6 02:43:17 runner-host01 gitlab-runner[15131]: #033[0;33mWARNING: Checking for jobs... failed -#033[0;m #033[0;33mrunner#033[0;m=Bfkz1fyb #033[0;33mstatus#033[0;m=couldn't execute POST against -https://gitlab.domain.tld/api/v4/jobs/request: Post https://gitlab.domain.tld/api/v4/jobs/request: -x509: certificate signed by unknown authority -``` - -Follow the details in [Self-signed certificates or custom Certification Authorities for GitLab Runner](https://docs.gitlab.com/runner/configuration/tls-self-signed.html). - -## Mirroring a remote GitLab repository that uses a self-signed SSL certificate - -When configuring a local GitLab instance to [mirror a repository](../../user/project/repository/mirror/index.md) -from a remote GitLab instance that uses a self-signed certificate, you may see -the `SSL certificate problem: self signed certificate` error message in the -user interface. - -The cause of the issue can be confirmed by checking if: - -- `curl` fails: - - ```shell - $ curl "https://gitlab.domain.tld" - curl: (60) SSL certificate problem: self signed certificate - More details here: https://curl.haxx.se/docs/sslcerts.html - ``` - -- Testing by using the Rails console also fails: - - ```ruby - uri = URI.parse("https://gitlab.domain.tld") - http = Net::HTTP.new(uri.host, uri.port) - http.use_ssl = true - http.verify_mode = 1 - response = http.request(Net::HTTP::Get.new(uri.request_uri)) - ... - Traceback (most recent call last): - 1: from (irb):5 - OpenSSL::SSL::SSLError (SSL_connect returned=1 errno=0 state=error: certificate verify failed (unable to get local issuer certificate)) - ``` - -To fix this problem: - -- Add the self-signed certificate from the remote GitLab instance to the - `/etc/gitlab/trusted-certs` directory on the local GitLab instance, and then - run `sudo gitlab-ctl reconfigure` as per the instructions for - [installing custom public certificates](https://docs.gitlab.com/omnibus/settings/ssl.html#install-custom-public-certificates). -- If your local GitLab instance was installed using the Helm Charts, you can - [add your self-signed certificate to your GitLab instance](https://docs.gitlab.com/runner/install/kubernetes.html#providing-a-custom-certificate-for-accessing-gitlab). - -You may also get another error message when trying to mirror a repository from -a remote GitLab instance that uses a self-signed certificate: - -```shell -2:Fetching remote upstream failed: fatal: unable to access &#39;https://gitlab.domain.tld/root/test-repo/&#39;: -SSL: unable to obtain common name from peer certificate -``` - -In this case, the problem can be related to the certificate itself: - -1. Validate that your self-signed certificate isn't missing a common name. If it - is, regenerate a valid certificate -1. Add the certificate to `/etc/gitlab/trusted-certs`. -1. Run `sudo gitlab-ctl reconfigure`. - -## Unable to perform Git operations due to an internal or self-signed certificate - -If your GitLab instance is using a self-signed certificate, or if the -certificate is signed by an internal certificate authority (CA), you might -experience the following errors when attempting to perform Git operations: - -```shell -$ git clone https://gitlab.domain.tld/group/project.git -Cloning into 'project'... -fatal: unable to access 'https://gitlab.domain.tld/group/project.git/': SSL certificate problem: self signed certificate -``` - -```shell -$ git clone https://gitlab.domain.tld/group/project.git -Cloning into 'project'... -fatal: unable to access 'https://gitlab.domain.tld/group/project.git/': server certificate verification failed. CAfile: /etc/ssl/certs/ca-certificates.crt CRLfile: none -``` - -To fix this problem: - -- If possible, use SSH remotes for all Git operations. This is considered more - secure and convenient to use. -- If you must use HTTPS remotes, you can try the following: - - Copy the self-signed certificate or the internal root CA certificate to a - local directory (for example, `~/.ssl`) and configure Git to trust your - certificate: - - ```shell - git config --global http.sslCAInfo ~/.ssl/gitlab.domain.tld.crt - ``` - - - Disable SSL verification in your Git client. This is intended as a - temporary measure, as it could be considered a security risk. - - ```shell - git config --global http.sslVerify false - ``` - -## SSL_connect wrong version number - -A misconfiguration may result in: - -- `gitlab-rails/exceptions_json.log` entries containing: - - ```plaintext - "exception.class":"Excon::Error::Socket","exception.message":"SSL_connect returned=1 errno=0 state=error: wrong version number (OpenSSL::SSL::SSLError)", - "exception.class":"Excon::Error::Socket","exception.message":"SSL_connect returned=1 errno=0 state=error: wrong version number (OpenSSL::SSL::SSLError)", - ``` - -- `gitlab-workhorse/current` containing: - - ```plaintext - http: server gave HTTP response to HTTPS client - http: server gave HTTP response to HTTPS client - ``` - -- `gitlab-rails/sidekiq.log` or `sidekiq/current` containing: - - ```plaintext - message: SSL_connect returned=1 errno=0 state=error: wrong version number (OpenSSL::SSL::SSLError) - message: SSL_connect returned=1 errno=0 state=error: wrong version number (OpenSSL::SSL::SSLError) - ``` - -Some of these errors come from the Excon Ruby gem, and could be generated in -circumstances where GitLab is configured to initiate an HTTPS session to a -remote server that is serving only HTTP. - -One scenario is that you're using [object storage](../object_storage.md), which -isn't served under HTTPS. GitLab is misconfigured and attempts a TLS handshake, -but the object storage responds with plain HTTP. - -## `schannel: SEC_E_UNTRUSTED_ROOT` - -If you're on Windows and get the following error: - -```plaintext -Fatal: unable to access 'https://gitlab.domain.tld/group/project.git': schannel: SEC_E_UNTRUSTED_ROOT (0x80090325) - The certificate chain was issued by an authority that is not trusted." -``` - -You must specify that Git should use OpenSSL: - -```shell -git config --system http.sslbackend openssl -``` - -Alternatively, you can ignore SSL verification by running: - -WARNING: -Proceed with caution when [ignoring SSL](https://git-scm.com/docs/git-config#Documentation/git-config.txt-httpsslVerify) -due to the potential security issues associated with disabling this option at global level. Use this option _only_ when troubleshooting, and reinstate SSL verification immediately after. - -```shell -git config --global http.sslVerify false -``` +<!-- This redirect file can be deleted after <2023-01-25>. --> +<!-- Redirects that point to other docs in the same project expire in three months. --> +<!-- Redirects that point to docs in a different project or site (for example, link is not relative and starts with `https:`) expire in one year. --> +<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/redirects.html --> diff --git a/doc/administration/troubleshooting/test_environments.md b/doc/administration/troubleshooting/test_environments.md index a249d5bd133..c4d191f8c0f 100644 --- a/doc/administration/troubleshooting/test_environments.md +++ b/doc/administration/troubleshooting/test_environments.md @@ -20,13 +20,13 @@ are only available internally at GitLab. ## Docker The following were tested on Docker containers running in the cloud. Support Engineers, -please see [these docs](https://gitlab.com/gitlab-com/dev-resources/tree/master/dev-resources#running-docker-containers) +see [these docs](https://gitlab.com/gitlab-com/dev-resources/tree/master/dev-resources#running-docker-containers) on how to run Docker containers on `dev-resources`. Other setups haven't been tested, but contributions are welcome. ### GitLab -Please see [our official Docker installation method](../../install/docker.md) +See [our official Docker installation method](../../install/docker.md) for how to run GitLab on Docker. ### SAML |