diff options
Diffstat (limited to 'doc/administration/raketasks')
| -rw-r--r-- | doc/administration/raketasks/github_import.md | 8 | ||||
| -rw-r--r-- | doc/administration/raketasks/maintenance.md | 6 | ||||
| -rw-r--r-- | doc/administration/raketasks/project_import_export.md | 144 | ||||
| -rw-r--r-- | doc/administration/raketasks/service_desk_email.md | 2 | ||||
| -rw-r--r-- | doc/administration/raketasks/uploads/sanitize.md | 2 |
5 files changed, 113 insertions, 49 deletions
diff --git a/doc/administration/raketasks/github_import.md b/doc/administration/raketasks/github_import.md index d089682f78e..1babafc902e 100644 --- a/doc/administration/raketasks/github_import.md +++ b/doc/administration/raketasks/github_import.md @@ -6,7 +6,8 @@ info: To determine the technical writer assigned to the Stage/Group associated w # GitHub import Rake task **(FREE SELF)** -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/390690) in GitLab 15.9, Rake task no longer automatically creates namespaces or groups that don't exist. +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/390690) in GitLab 15.9, Rake task no longer automatically creates namespaces or groups that don't exist. +> - Requirement for Maintainer role instead of Developer role introduced in GitLab 16.0 and backported to GitLab 15.11.1 and GitLab 15.10.5. To retrieve and import GitHub repositories, you need a [GitHub personal access token](https://github.com/settings/tokens). A username should be passed as the second argument to the Rake task, @@ -19,10 +20,9 @@ before/after the brackets. Also, some shells (for example, Zsh) can interpret th Prerequisite: -- At least the Maintainer role on the destination group to import to. Using the Developer role for this purpose was - [deprecated](https://gitlab.com/gitlab-org/gitlab/-/issues/387891) in GitLab 15.8 and will be removed in GitLab 16.0. +- At least the Maintainer role on the destination group to import to. -## Caveats +## Rate limit If the GitHub [rate limit](https://docs.github.com/en/rest/rate-limit) is reached while importing, the importing process waits (`sleep()`) until it can continue importing. diff --git a/doc/administration/raketasks/maintenance.md b/doc/administration/raketasks/maintenance.md index 5c258d73fdb..06f7203f695 100644 --- a/doc/administration/raketasks/maintenance.md +++ b/doc/administration/raketasks/maintenance.md @@ -11,7 +11,7 @@ GitLab provides Rake tasks for general maintenance. ## Gather GitLab and system information This command gathers information about your GitLab installation and the system it runs on. -These may be useful when asking for help or reporting issues. +These may be useful when asking for help or reporting issues. In a multi-node environment, run this command on nodes running GitLab Rails to avoid PostgreSQL socket errors. **Omnibus Installation** @@ -117,7 +117,7 @@ If you're running Geo, see also the [Geo Health check Rake task](../geo/replicat You may also have a look at our troubleshooting guides for: -- [GitLab](../index.md#troubleshooting) +- [GitLab](../troubleshooting/index.md) - [Omnibus GitLab](https://docs.gitlab.com/omnibus/index.html#troubleshooting) Additionally you should also [verify database values can be decrypted using the current secrets](check.md#verify-database-values-can-be-decrypted-using-the-current-secrets). @@ -386,7 +386,7 @@ You can also [enable reindexing as a regular cron job](https://docs.gitlab.com/o Sometimes you may need to re-import the common metrics that power the Metrics dashboards. -This could be as a result of [updating existing metrics](../../development/prometheus_metrics.md#update-existing-metrics), or as a [troubleshooting measure](../../operations/metrics/dashboards/index.md#troubleshooting). +This could be as a result of [updating existing metrics](../../development/prometheus_metrics.md#update-existing-metrics). To re-import the metrics you can run: diff --git a/doc/administration/raketasks/project_import_export.md b/doc/administration/raketasks/project_import_export.md index 4694af18af2..17a0eb46a30 100644 --- a/doc/administration/raketasks/project_import_export.md +++ b/doc/administration/raketasks/project_import_export.md @@ -4,61 +4,42 @@ group: Distribution info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- -# Project import/export administration **(FREE SELF)** +# Project import and export Rake tasks **(FREE SELF)** -GitLab provides Rake tasks relating to project import and export. For more information, see: +GitLab provides Rake tasks for [project import and export](../../user/project/settings/import_export.md). -- [Project import/export documentation](../../user/project/settings/import_export.md). -- [Project import/export API](../../api/project_import_export.md). -- [Developer documentation: project import/export](../../development/import_export.md) +You can only import from a [compatible](../../user/project/settings/import_export.md#compatibility) GitLab instance. -## Project import status +## Import large projects -You can query an import through the [Project import/export API](../../api/project_import_export.md#import-status). -As described in the API documentation, the query may return an import error or exceptions. +> The [Rake task](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/tasks/gitlab/import_export/import.rake) was [introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/20724) in GitLab 12.6, replacing a GitLab.com Ruby script. -## Import/export Rake tasks +This script was introduced in GitLab 12.6 for importing large GitLab project exports. -The GitLab import/export version can be checked by using the following command: +As part of this script, we also disable direct upload. This avoids uploading a huge archive to GCS, which can cause idle transaction timeouts. -```shell -# Omnibus installations -sudo gitlab-rake gitlab:import_export:version +We can run this script from the terminal: -# Installations from source -bundle exec rake gitlab:import_export:version RAILS_ENV=production -``` +Parameters: -The current list of DB tables to export can be listed by using the following command: +| Attribute | Type | Required | Description | +| --------- | ---- | -------- | ----------- | +| `username` | string | yes | User name | +| `namespace_path` | string | yes | Namespace path | +| `project_path` | string | yes | Project path | +| `archive_path` | string | yes | Path to the exported project tarball you want to import | ```shell -# Omnibus installations -sudo gitlab-rake gitlab:import_export:data - -# Installations from source -bundle exec rake gitlab:import_export:data RAILS_ENV=production +bundle exec rake "gitlab:import_export:import[root, group/subgroup, testingprojectimport, /path/to/file.tar.gz]" ``` -Note the following: - -- Importing is only possible if the version of the import and export GitLab instances are - [compatible](../../user/project/settings/import_export.md#compatibility). -- The project import option must be enabled: - - 1. On the top bar, select **Main menu > Admin**. - 1. On the left sidebar, select **Settings > General**. - 1. Expand **Visibility and access controls**. - 1. Under **Import sources**, check the "Project export enabled" option. - 1. Select **Save changes**. - -- The exports are stored in a temporary directory and are deleted every - 24 hours by a specific worker. - -### Import large projects using a Rake task +If you're running Omnibus, run the following Rake task: -If you have a larger project, consider using a Rake task as described in our [developer documentation](../../development/import_project.md#importing-via-a-rake-task). +```shell +gitlab-rake "gitlab:import_export:import[root, group/subgroup, testingprojectimport, /path/to/file.tar.gz]" +``` -### Export using a Rake task +## Export large projects > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/25598) in GitLab 12.9. @@ -88,3 +69,86 @@ IMPORT_DEBUG=true gitlab-rake "gitlab:import_export:import[root, group/subgroup, # Export EXPORT_DEBUG=true gitlab-rake "gitlab:import_export:export[root, group/subgroup, projectnametoexport, /tmp/export_file.tar.gz]" ``` + +Check the common errors listed below, what they mean, and how to fix them. + +### `Exception: undefined method 'name' for nil:NilClass` + +The `username` is not valid. + +### `Exception: undefined method 'full_path' for nil:NilClass` + +The `namespace_path` does not exist. +For example, one of the groups or subgroups is mistyped or missing, +or you've specified the project name in the path. + +The task only creates the project. +If you want to import it to a new group or subgroup, create it first. + +### `Exception: No such file or directory @ rb_sysopen - (filename)` + +The specified project export file in `archive_path` is missing. + +### `Exception: Permission denied @ rb_sysopen - (filename)` + +The specified project export file cannot be accessed by the `git` user. + +To fix the issue: + +1. Set the file owner to `git:git`. +1. Change the file permissions to `0400`. +1. Move the file to a public folder (for example `/tmp/`). + +### `Name can contain only letters, digits, emojis ...` + +```plaintext +Name can contain only letters, digits, emojis, '_', '.', '+', dashes, or spaces. It must start with a letter, +digit, emoji, or '_', and Path can contain only letters, digits, '_', '-', or '.'. It cannot start +with '-', end in '.git', or end in '.atom'. +``` + +The project name specified in `project_path` is not valid for one of the specified reasons. + +Only put the project name in `project_path`. For example, if you provide a path of subgroups +it fails with this error as `/` is not a valid character in a project name. + +### `Name has already been taken and Path has already been taken` + +A project with that name already exists. + +### `Exception: Error importing repository into (namespace) - No space left on device` + +The disk has insufficient space to complete the import. + +During import, the tarball is cached in your configured `shared_path` directory. Verify the +disk has enough free space to accommodate both the cached tarball and the unpacked +project files on disk. + +### Import is successful, but with a `Total number of not imported relations: XX` message, and issues are not created during the import + +If you receive a `Total number of not imported relations: XX` message, and issues +aren't created during the import, check [exceptions_json.log](../logs/index.md#exceptions_jsonlog). +You might see an error like `N is out of range for ActiveModel::Type::Integer with limit 4 bytes`, +where `N` is the integer exceeding the 4-byte integer limit. If that's the case, you +are likely hitting the issue with rebalancing of `relative_position` field of the issues. + +```ruby +# Check the current maximum value of relative_position +Issue.where(project_id: Project.find(ID).root_namespace.all_projects).maximum(:relative_position) + +# Run the rebalancing process and check if the maximum value of relative_position has changed +Issues::RelativePositionRebalancingService.new(Project.find(ID).root_namespace.all_projects).execute +Issue.where(project_id: Project.find(ID).root_namespace.all_projects).maximum(:relative_position) +``` + +Repeat the import attempt and check if the issues are imported successfully. + +### Gitaly calls error when importing + +If you're attempting to import a large project into a development environment, Gitaly might throw an error about too many calls or invocations. For example: + +```plaintext +Error importing repository into qa-perf-testing/gitlabhq - GitalyClient#call called 31 times from single request. Potential n+1? +``` + +This error is due to a [n+1 calls limit for development setups](../../development/gitaly.md#toomanyinvocationserror-errors). To resolve this error, set `GITALY_DISABLE_REQUEST_LIMITS=1` as an environment variable. Then restart your development environment and import again. diff --git a/doc/administration/raketasks/service_desk_email.md b/doc/administration/raketasks/service_desk_email.md index 10de379b1cd..1cbdec35171 100644 --- a/doc/administration/raketasks/service_desk_email.md +++ b/doc/administration/raketasks/service_desk_email.md @@ -12,7 +12,7 @@ The following are Service Desk email-related Rake tasks. ## Secrets -GitLab can use [Service Desk email](../../user/project/service_desk.md#configuring-a-custom-mailbox) secrets read from an encrypted file instead of storing them in plaintext in the file system. The following Rake tasks are provided for updating the contents of the encrypted file. +GitLab can use [Service Desk email](../../user/project/service_desk.md#configure-a-custom-mailbox) secrets read from an encrypted file instead of storing them in plaintext in the file system. The following Rake tasks are provided for updating the contents of the encrypted file. ### Show secret diff --git a/doc/administration/raketasks/uploads/sanitize.md b/doc/administration/raketasks/uploads/sanitize.md index 831abee9739..567a20a37f3 100644 --- a/doc/administration/raketasks/uploads/sanitize.md +++ b/doc/administration/raketasks/uploads/sanitize.md @@ -11,7 +11,7 @@ In GitLab 11.9 and later, EXIF data is automatically stripped from JPG or TIFF i EXIF data may contain sensitive information (for example, GPS location), so you can remove EXIF data from existing images that were uploaded to an earlier version of GitLab. -## Requirements +## Prerequisite To run this Rake task, you need `exiftool` installed on your system. If you installed GitLab: |