diff options
Diffstat (limited to 'doc/administration/raketasks/project_import_export.md')
-rw-r--r-- | doc/administration/raketasks/project_import_export.md | 144 |
1 files changed, 104 insertions, 40 deletions
diff --git a/doc/administration/raketasks/project_import_export.md b/doc/administration/raketasks/project_import_export.md index 4694af18af2..17a0eb46a30 100644 --- a/doc/administration/raketasks/project_import_export.md +++ b/doc/administration/raketasks/project_import_export.md @@ -4,61 +4,42 @@ group: Distribution info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- -# Project import/export administration **(FREE SELF)** +# Project import and export Rake tasks **(FREE SELF)** -GitLab provides Rake tasks relating to project import and export. For more information, see: +GitLab provides Rake tasks for [project import and export](../../user/project/settings/import_export.md). -- [Project import/export documentation](../../user/project/settings/import_export.md). -- [Project import/export API](../../api/project_import_export.md). -- [Developer documentation: project import/export](../../development/import_export.md) +You can only import from a [compatible](../../user/project/settings/import_export.md#compatibility) GitLab instance. -## Project import status +## Import large projects -You can query an import through the [Project import/export API](../../api/project_import_export.md#import-status). -As described in the API documentation, the query may return an import error or exceptions. +> The [Rake task](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/tasks/gitlab/import_export/import.rake) was [introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/20724) in GitLab 12.6, replacing a GitLab.com Ruby script. -## Import/export Rake tasks +This script was introduced in GitLab 12.6 for importing large GitLab project exports. -The GitLab import/export version can be checked by using the following command: +As part of this script, we also disable direct upload. This avoids uploading a huge archive to GCS, which can cause idle transaction timeouts. -```shell -# Omnibus installations -sudo gitlab-rake gitlab:import_export:version +We can run this script from the terminal: -# Installations from source -bundle exec rake gitlab:import_export:version RAILS_ENV=production -``` +Parameters: -The current list of DB tables to export can be listed by using the following command: +| Attribute | Type | Required | Description | +| --------- | ---- | -------- | ----------- | +| `username` | string | yes | User name | +| `namespace_path` | string | yes | Namespace path | +| `project_path` | string | yes | Project path | +| `archive_path` | string | yes | Path to the exported project tarball you want to import | ```shell -# Omnibus installations -sudo gitlab-rake gitlab:import_export:data - -# Installations from source -bundle exec rake gitlab:import_export:data RAILS_ENV=production +bundle exec rake "gitlab:import_export:import[root, group/subgroup, testingprojectimport, /path/to/file.tar.gz]" ``` -Note the following: - -- Importing is only possible if the version of the import and export GitLab instances are - [compatible](../../user/project/settings/import_export.md#compatibility). -- The project import option must be enabled: - - 1. On the top bar, select **Main menu > Admin**. - 1. On the left sidebar, select **Settings > General**. - 1. Expand **Visibility and access controls**. - 1. Under **Import sources**, check the "Project export enabled" option. - 1. Select **Save changes**. - -- The exports are stored in a temporary directory and are deleted every - 24 hours by a specific worker. - -### Import large projects using a Rake task +If you're running Omnibus, run the following Rake task: -If you have a larger project, consider using a Rake task as described in our [developer documentation](../../development/import_project.md#importing-via-a-rake-task). +```shell +gitlab-rake "gitlab:import_export:import[root, group/subgroup, testingprojectimport, /path/to/file.tar.gz]" +``` -### Export using a Rake task +## Export large projects > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/25598) in GitLab 12.9. @@ -88,3 +69,86 @@ IMPORT_DEBUG=true gitlab-rake "gitlab:import_export:import[root, group/subgroup, # Export EXPORT_DEBUG=true gitlab-rake "gitlab:import_export:export[root, group/subgroup, projectnametoexport, /tmp/export_file.tar.gz]" ``` + +Check the common errors listed below, what they mean, and how to fix them. + +### `Exception: undefined method 'name' for nil:NilClass` + +The `username` is not valid. + +### `Exception: undefined method 'full_path' for nil:NilClass` + +The `namespace_path` does not exist. +For example, one of the groups or subgroups is mistyped or missing, +or you've specified the project name in the path. + +The task only creates the project. +If you want to import it to a new group or subgroup, create it first. + +### `Exception: No such file or directory @ rb_sysopen - (filename)` + +The specified project export file in `archive_path` is missing. + +### `Exception: Permission denied @ rb_sysopen - (filename)` + +The specified project export file cannot be accessed by the `git` user. + +To fix the issue: + +1. Set the file owner to `git:git`. +1. Change the file permissions to `0400`. +1. Move the file to a public folder (for example `/tmp/`). + +### `Name can contain only letters, digits, emojis ...` + +```plaintext +Name can contain only letters, digits, emojis, '_', '.', '+', dashes, or spaces. It must start with a letter, +digit, emoji, or '_', and Path can contain only letters, digits, '_', '-', or '.'. It cannot start +with '-', end in '.git', or end in '.atom'. +``` + +The project name specified in `project_path` is not valid for one of the specified reasons. + +Only put the project name in `project_path`. For example, if you provide a path of subgroups +it fails with this error as `/` is not a valid character in a project name. + +### `Name has already been taken and Path has already been taken` + +A project with that name already exists. + +### `Exception: Error importing repository into (namespace) - No space left on device` + +The disk has insufficient space to complete the import. + +During import, the tarball is cached in your configured `shared_path` directory. Verify the +disk has enough free space to accommodate both the cached tarball and the unpacked +project files on disk. + +### Import is successful, but with a `Total number of not imported relations: XX` message, and issues are not created during the import + +If you receive a `Total number of not imported relations: XX` message, and issues +aren't created during the import, check [exceptions_json.log](../logs/index.md#exceptions_jsonlog). +You might see an error like `N is out of range for ActiveModel::Type::Integer with limit 4 bytes`, +where `N` is the integer exceeding the 4-byte integer limit. If that's the case, you +are likely hitting the issue with rebalancing of `relative_position` field of the issues. + +```ruby +# Check the current maximum value of relative_position +Issue.where(project_id: Project.find(ID).root_namespace.all_projects).maximum(:relative_position) + +# Run the rebalancing process and check if the maximum value of relative_position has changed +Issues::RelativePositionRebalancingService.new(Project.find(ID).root_namespace.all_projects).execute +Issue.where(project_id: Project.find(ID).root_namespace.all_projects).maximum(:relative_position) +``` + +Repeat the import attempt and check if the issues are imported successfully. + +### Gitaly calls error when importing + +If you're attempting to import a large project into a development environment, Gitaly might throw an error about too many calls or invocations. For example: + +```plaintext +Error importing repository into qa-perf-testing/gitlabhq - GitalyClient#call called 31 times from single request. Potential n+1? +``` + +This error is due to a [n+1 calls limit for development setups](../../development/gitaly.md#toomanyinvocationserror-errors). To resolve this error, set `GITALY_DISABLE_REQUEST_LIMITS=1` as an environment variable. Then restart your development environment and import again. |