diff options
Diffstat (limited to 'doc/raketasks')
| -rw-r--r-- | doc/raketasks/README.md | 46 | ||||
| -rw-r--r-- | doc/raketasks/backup_hrz.png | bin | 11441 -> 0 bytes | |||
| -rw-r--r-- | doc/raketasks/backup_restore.md | 118 | ||||
| -rw-r--r-- | doc/raketasks/cleanup.md | 13 | ||||
| -rw-r--r-- | doc/raketasks/features.md | 18 | ||||
| -rw-r--r-- | doc/raketasks/generate_sample_prometheus_data.md | 20 | ||||
| -rw-r--r-- | doc/raketasks/import.md | 90 | ||||
| -rw-r--r-- | doc/raketasks/list_repos.md | 21 | ||||
| -rw-r--r-- | doc/raketasks/migrate_snippets.md | 96 | ||||
| -rw-r--r-- | doc/raketasks/user_management.md | 117 | ||||
| -rw-r--r-- | doc/raketasks/web_hooks.md | 52 | ||||
| -rw-r--r-- | doc/raketasks/x509_signatures.md | 21 |
12 files changed, 378 insertions, 234 deletions
diff --git a/doc/raketasks/README.md b/doc/raketasks/README.md index 1670849016c..4bba4f2bc5a 100644 --- a/doc/raketasks/README.md +++ b/doc/raketasks/README.md @@ -15,25 +15,27 @@ GitLab Rake tasks are performed using: The following are available Rake tasks: -| Tasks | Description | -|:-------------------------------------------------------------------------------------------|:------------------------------------------------------------------------------------------| -| [Back up and restore](backup_restore.md) | Back up, restore, and migrate GitLab instances between servers. | -| [Clean up](cleanup.md) | Clean up unneeded items GitLab instances. | -| [Development](../development/rake_tasks.md) | Tasks for GitLab contributors. | -| [Elasticsearch](../integration/elasticsearch.md#gitlab-elasticsearch-rake-tasks) | Maintain Elasticsearch in a GitLab instance. | -| [Enable namespaces](features.md) | Enable usernames and namespaces for user projects. | -| [General maintenance](../administration/raketasks/maintenance.md) | General maintenance and self-check tasks. | -| [Geo maintenance](../administration/raketasks/geo.md) **(PREMIUM ONLY)** | [Geo](../administration/geo/replication/index.md)-related maintenance. | -| [GitHub import](../administration/raketasks/github_import.md) | Retrieve and import repositories from GitHub. | -| [Import repositories](import.md) | Import bare repositories into your GitLab instance. | -| [Import large project exports](../development/import_project.md#importing-via-a-rake-task) | Import large GitLab [project exports](../user/project/settings/import_export.md). | -| [Integrity checks](../administration/raketasks/check.md) | Check the integrity of repositories, files, and LDAP. | -| [LDAP maintenance](../administration/raketasks/ldap.md) | [LDAP](../administration/auth/ldap.md)-related tasks. | -| [List repositories](list_repos.md) | List of all GitLab-managed Git repositories on disk. | -| [Project import/export](../administration/raketasks/project_import_export.md) | Prepare for [project exports and imports](../user/project/settings/import_export.md). | -| [Repository storage](../administration/raketasks/storage.md) | List and migrate existing projects and attachments from legacy storage to hashed storage. | -| [Uploads migrate](../administration/raketasks/uploads/migrate.md) | Migrate uploads between storage local and object storage. | -| [Uploads sanitize](../administration/raketasks/uploads/sanitize.md) | Remove EXIF data from images uploaded to earlier versions of GitLab. | -| [User management](user_management.md) | Perform user management tasks. | -| [Webhooks administration](web_hooks.md) | Maintain project Webhooks. | -| [X509 signatures](x509_signatures.md) | Update x509 commit signatures, useful if certificate store has changed. | +| Tasks | Description | +|:----------------------------------------------------------------------------------------------------|:------------------------------------------------------------------------------------------| +| [Back up and restore](backup_restore.md) | Back up, restore, and migrate GitLab instances between servers. | +| [Clean up](cleanup.md) | Clean up unneeded items from GitLab instances. | +| [Development](../development/rake_tasks.md) | Tasks for GitLab contributors. | +| [Elasticsearch](../integration/elasticsearch.md#gitlab-elasticsearch-rake-tasks) **(STARTER ONLY)** | Maintain Elasticsearch in a GitLab instance. | +| [Enable namespaces](features.md) | Enable usernames and namespaces for user projects. | +| [General maintenance](../administration/raketasks/maintenance.md) | General maintenance and self-check tasks. | +| [Geo maintenance](../administration/raketasks/geo.md) **(PREMIUM ONLY)** | [Geo](../administration/geo/replication/index.md)-related maintenance. | +| [GitHub import](../administration/raketasks/github_import.md) | Retrieve and import repositories from GitHub. | +| [Import repositories](import.md) | Import bare repositories into your GitLab instance. | +| [Import large project exports](../development/import_project.md#importing-via-a-rake-task) | Import large GitLab [project exports](../user/project/settings/import_export.md). | +| [Integrity checks](../administration/raketasks/check.md) | Check the integrity of repositories, files, and LDAP. | +| [LDAP maintenance](../administration/raketasks/ldap.md) | [LDAP](../administration/auth/ldap.md)-related tasks. | +| [List repositories](list_repos.md) | List of all GitLab-managed Git repositories on disk. | +| [Project import/export](../administration/raketasks/project_import_export.md) | Prepare for [project exports and imports](../user/project/settings/import_export.md). | +| [Sample Prometheus data](generate_sample_prometheus_data.md) | Generate sample Prometheus data. | +| [Repository storage](../administration/raketasks/storage.md) | List and migrate existing projects and attachments from legacy storage to hashed storage. | +| [Uploads migrate](../administration/raketasks/uploads/migrate.md) | Migrate uploads between storage local and object storage. | +| [Uploads sanitize](../administration/raketasks/uploads/sanitize.md) | Remove EXIF data from images uploaded to earlier versions of GitLab. | +| [User management](user_management.md) | Perform user management tasks. | +| [Webhooks administration](web_hooks.md) | Maintain project Webhooks. | +| [X.509 signatures](x509_signatures.md) | Update X.509 commit signatures, useful if certificate store has changed. | +| [Migrate Snippets to Git](migrate_snippets.md) | Migrate GitLab Snippets to Git repositories and show migration status | diff --git a/doc/raketasks/backup_hrz.png b/doc/raketasks/backup_hrz.png Binary files differdeleted file mode 100644 index 32690b2904c..00000000000 --- a/doc/raketasks/backup_hrz.png +++ /dev/null diff --git a/doc/raketasks/backup_restore.md b/doc/raketasks/backup_restore.md index 24e4ed43543..3c81bab644e 100644 --- a/doc/raketasks/backup_restore.md +++ b/doc/raketasks/backup_restore.md @@ -1,6 +1,6 @@ -# Backing up and restoring GitLab +# Backing up and restoring GitLab **(CORE ONLY)** - +GitLab provides Rake tasks for backing up and restoring GitLab instances. An application data backup creates an archive file that contains the database, all repositories and all attachments. @@ -14,31 +14,26 @@ from one server to another is through backup restore. In order to be able to backup and restore, you need two essential tools installed on your system. -### Rsync +- **Rsync**: If you installed GitLab: + - Using the Omnibus package, you're all set. + - From source, make sure `rsync` is installed. For example: -If you installed GitLab: + ```shell + # Debian/Ubuntu + sudo apt-get install rsync -- Using the Omnibus package, you're all set. -- From source, make sure `rsync` is installed: + # RHEL/CentOS + sudo yum install rsync + ``` - ```shell - # Debian/Ubuntu - sudo apt-get install rsync +- **Tar**: Backup and restore tasks use `tar` under the hood to create and extract + archives. Ensure you have version 1.30 or above of `tar` available in your + system. To check the version, run: - # RHEL/CentOS - sudo yum install rsync + ```shell + tar --version ``` -### Tar - -Backup and restore tasks use `tar` under the hood to create and extract -archives. Ensure you have version 1.30 or above of `tar` available in your -system. To check the version, run: - -```shell -tar --version -``` - ## Backup timestamp NOTE: **Note:** @@ -56,9 +51,9 @@ available. For example, if the backup name is `1493107454_2018_04_25_10.6.4-ce_gitlab_backup.tar`, then the timestamp is `1493107454_2018_04_25_10.6.4-ce`. -## Creating a backup of the GitLab system +## Back up GitLab -GitLab provides a simple command line interface to backup your whole instance. +GitLab provides a simple command line interface to back up your whole instance. It backs up your: - Database @@ -142,12 +137,12 @@ Deleting tmp directories...[DONE] Deleting old backups... [SKIPPING] ``` -## Storing configuration files +### Storing configuration files -A backup performed by the [raketask GitLab provides](#creating-a-backup-of-the-gitlab-system) +The [backup Rake task](#back-up-gitlab) GitLab provides does **not** store your configuration files. The primary reason for this is that your database contains encrypted information for two-factor authentication, the CI/CD -'secure variables', etc. Storing encrypted information along with its key in the +'secure variables', and so on. Storing encrypted information along with its key in the same place defeats the purpose of using encryption in the first place. CAUTION: **Warning:** @@ -182,11 +177,11 @@ If you use Omnibus GitLab, see some additional information In the unlikely event that the secrets file is lost, see the [troubleshooting section](#when-the-secrets-file-is-lost). -## Backup options +### Backup options The command line tool GitLab provides to backup your instance can take more options. -### Backup strategy option +#### Backup strategy option > [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/8728) in GitLab 8.17. @@ -214,7 +209,7 @@ sudo gitlab-backup create STRATEGY=copy NOTE: **Note** For GitLab 12.1 and earlier, use `gitlab-rake gitlab:backup:create`. -### Backup filename +#### Backup filename CAUTION: **Warning:** If you use a custom backup filename, you will not be able to @@ -231,7 +226,7 @@ For GitLab 12.1 and earlier, use `gitlab-rake gitlab:backup:create`. The resulting file will then be `dump_gitlab_backup.tar`. This is useful for systems that make use of rsync and incremental backups, and will result in considerably faster transfer speeds. -### Rsyncable +#### Rsyncable To make sure the generated archive is intelligently transferable by rsync, the `GZIP_RSYNCABLE=yes` option can be set. This will set the `--rsyncable` option to `gzip`. This is only useful in combination with setting [the Backup filename option](#backup-filename). @@ -244,7 +239,7 @@ sudo gitlab-backup create BACKUP=dump GZIP_RSYNCABLE=yes NOTE: **Note** For GitLab 12.1 and earlier, use `gitlab-rake gitlab:backup:create`. -### Excluding specific directories from the backup +#### Excluding specific directories from the backup You can choose what should be exempt from the backup up by adding the environment variable `SKIP`. The available options are: @@ -278,7 +273,7 @@ For installations from source: sudo -u git -H bundle exec rake gitlab:backup:create SKIP=db,uploads RAILS_ENV=production ``` -### Skipping tar creation +#### Skipping tar creation The last part of creating a backup is generation of a `.tar` file containing all the parts. In some cases (for example, if the backup is picked up by other @@ -303,19 +298,19 @@ For installations from source: sudo -u git -H bundle exec rake gitlab:backup:create SKIP=tar RAILS_ENV=production ``` -### Uploading backups to a remote (cloud) storage +#### Uploading backups to a remote (cloud) storage -Starting with GitLab 7.4 you can let the backup script upload the '.tar' file it creates. +Starting with GitLab 7.4 you can let the backup script upload the `.tar` file it creates. It uses the [Fog library](http://fog.io/) to perform the upload. In the example below we use Amazon S3 for storage, but Fog also lets you use [other storage providers](http://fog.io/storage/). GitLab [imports cloud drivers](https://gitlab.com/gitlab-org/gitlab/blob/30f5b9a5b711b46f1065baf755e413ceced5646b/Gemfile#L88) -for AWS, Google, OpenStack Swift, Rackspace and Aliyun as well. A local driver is +for AWS, Google, OpenStack Swift, Rackspace, and Aliyun as well. A local driver is [also available](#uploading-to-locally-mounted-shares). [Read more about using object storage with GitLab](../administration/object_storage.md). -#### Using Amazon S3 +##### Using Amazon S3 For Omnibus GitLab packages: @@ -333,9 +328,9 @@ For Omnibus GitLab packages: gitlab_rails['backup_upload_remote_directory'] = 'my.s3.bucket' ``` -1. [Reconfigure GitLab] for the changes to take effect +1. [Reconfigure GitLab](../administration/restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect -#### Digital Ocean Spaces +##### Digital Ocean Spaces This example can be used for a bucket in Amsterdam (AMS3). @@ -352,7 +347,7 @@ This example can be used for a bucket in Amsterdam (AMS3). gitlab_rails['backup_upload_remote_directory'] = 'my.s3.bucket' ``` -1. [Reconfigure GitLab] for the changes to take effect +1. [Reconfigure GitLab](../administration/restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect NOTE: **Note:** If you see `400 Bad Request` by using Digital Ocean Spaces, the cause may be the @@ -360,7 +355,7 @@ usage of backup encryption. Remove or comment the line that contains `gitlab_rails['backup_encryption']` since Digital Ocean Spaces doesn't support encryption. -#### Other S3 Providers +##### Other S3 Providers Not all S3 providers are fully-compatible with the Fog library. For example, if you see `411 Length Required` errors after attempting to upload, you may @@ -397,7 +392,7 @@ For installations from source: # storage_class: 'STANDARD' ``` -1. [Restart GitLab] for the changes to take effect +1. [Restart GitLab](../administration/restart_gitlab.md#installations-from-source) for the changes to take effect If you are uploading your backups to S3 you will probably want to create a new IAM user with restricted access rights. To give the upload user access only for @@ -450,7 +445,7 @@ with the name of your bucket: } ``` -#### Using Google Cloud Storage +##### Using Google Cloud Storage If you want to use Google Cloud Storage to save backups, you'll have to create an access key from the Google console first: @@ -476,7 +471,7 @@ For Omnibus GitLab packages: gitlab_rails['backup_upload_remote_directory'] = 'my.google.bucket' ``` -1. [Reconfigure GitLab] for the changes to take effect +1. [Reconfigure GitLab](../administration/restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect For installations from source: @@ -492,9 +487,9 @@ For installations from source: remote_directory: 'my.google.bucket' ``` -1. [Restart GitLab] for the changes to take effect +1. [Restart GitLab](../administration/restart_gitlab.md#installations-from-source) for the changes to take effect -#### Specifying a custom directory for backups +##### Specifying a custom directory for backups Note: This option only works for remote storage. If you want to group your backups you can pass a `DIRECTORY` environment variable: @@ -507,9 +502,9 @@ sudo gitlab-backup create DIRECTORY=weekly NOTE: **Note** For GitLab 12.1 and earlier, use `gitlab-rake gitlab:backup:create`. -### Uploading to locally mounted shares +#### Uploading to locally mounted shares -You may also send backups to a mounted share (`NFS` / `CIFS` / `SMB` / etc.) by +You may also send backups to a mounted share (for example, `NFS`,`CIFS`, or `SMB`) by using the Fog [`Local`](https://github.com/fog/fog-local#usage) storage provider. The directory pointed to by the `local_root` key **must** be owned by the `git` user **when mounted** (mounting with the `uid=` of the `git` user for `CIFS` and @@ -539,7 +534,7 @@ For Omnibus GitLab packages: gitlab_rails['backup_upload_remote_directory'] = 'gitlab_backups' ``` -1. [Reconfigure GitLab] for the changes to take effect. +1. [Reconfigure GitLab](../administration/restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. For installations from source: @@ -557,9 +552,9 @@ For installations from source: remote_directory: 'gitlab_backups' ``` -1. [Restart GitLab] for the changes to take effect. +1. [Restart GitLab](../administration/restart_gitlab.md#installations-from-source) for the changes to take effect. -### Backup archive permissions +#### Backup archive permissions The backup archives created by GitLab (`1393513186_2014_02_27_gitlab_backup.tar`) will have owner/group `git`/`git` and 0600 permissions by default. @@ -574,7 +569,7 @@ For Omnibus GitLab packages: gitlab_rails['backup_archive_permissions'] = 0644 # Makes the backup archives world-readable ``` -1. [Reconfigure GitLab] for the changes to take effect. +1. [Reconfigure GitLab](../administration/restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. For installations from source: @@ -585,9 +580,9 @@ For installations from source: archive_permissions: 0644 # Makes the backup archives world-readable ``` -1. [Restart GitLab] for the changes to take effect. +1. [Restart GitLab](../administration/restart_gitlab.md#installations-from-source) for the changes to take effect. -### Configuring cron to make daily backups +#### Configuring cron to make daily backups CAUTION: **Warning:** The following cron jobs do not [backup your GitLab configuration files](#storing-configuration-files) @@ -669,9 +664,9 @@ For installations from source: keep_time: 604800 ``` -1. [Restart GitLab] for the changes to take effect. +1. [Restart GitLab](../administration/restart_gitlab.md#installations-from-source) for the changes to take effect. -## Restore +## Restore GitLab GitLab provides a simple command line interface to restore your whole installation, and is flexible enough to fit your needs. @@ -692,7 +687,7 @@ before restoring the backup. You need to have a working GitLab installation before you can perform a restore. This is mainly because the system user performing the restore actions (`git`) is usually not allowed to create or delete -the SQL database it needs to import data into ('gitlabhq_production'). +the SQL database it needs to import data into (`gitlabhq_production`). All existing data will be either erased (SQL) or moved to a separate directory (repositories, uploads). @@ -718,7 +713,7 @@ more of the following options: Read what the [backup timestamp is about](#backup-timestamp). - `force=yes` - Does not ask if the authorized_keys file should get regenerated and assumes 'yes' for warning that database tables will be removed, enabling the "Write to authorized_keys file" setting, and updating LDAP providers. -If you are restoring into directories that are mountpoints you will need to make +If you are restoring into directories that are mount points, you will need to make sure these directories are empty before attempting a restore. Otherwise GitLab will attempt to move these directories before restoring the new data and this would cause an error. @@ -902,6 +897,8 @@ will have all your repositories, but not any other data. ## Troubleshooting +The following are possible problems you might encounter with possible solutions. + ### Restoring database backup using Omnibus packages outputs warnings If you are using backup restore procedures you might encounter the following warnings: @@ -1079,12 +1076,9 @@ If you have changed the default filesystem location for the registry, you will want to run the `chown` against your custom location instead of `/var/opt/gitlab/gitlab-rails/shared/registry/docker`. -[reconfigure GitLab]: ../administration/restart_gitlab.md#omnibus-gitlab-reconfigure -[restart GitLab]: ../administration/restart_gitlab.md#installations-from-source - ### Backup fails to complete with Gzip error -While running the backup, you may receive a gzip error: +While running the backup, you may receive a Gzip error: ```shell sudo /opt/gitlab/bin/gitlab-backup create @@ -1098,5 +1092,5 @@ Backup failed If this happens, check the following: -1. Confirm there is sufficient disk space for the gzip operation. +1. Confirm there is sufficient disk space for the Gzip operation. 1. If NFS is being used, check if the mount option `timeout` is set. The default is `600`, and changing this to smaller values have resulted in this error. diff --git a/doc/raketasks/cleanup.md b/doc/raketasks/cleanup.md index ef69b1cce15..7908af8da84 100644 --- a/doc/raketasks/cleanup.md +++ b/doc/raketasks/cleanup.md @@ -1,4 +1,4 @@ -# Clean up +# Clean up **(CORE ONLY)** GitLab provides Rake tasks for cleaning up GitLab instances. @@ -7,9 +7,11 @@ GitLab provides Rake tasks for cleaning up GitLab instances. > [Introduced](https://gitlab.com/gitlab-org/gitlab/issues/36628) in GitLab 12.10. DANGER: **Danger:** -Do not run this within 12 hours of a GitLab upgrade. This is to ensure that all background migrations have finished, which otherwise may lead to data loss. +Do not run this within 12 hours of a GitLab upgrade. This is to ensure that all background migrations +have finished, which otherwise may lead to data loss. -When you remove LFS files from a repository's history, they become orphaned and continue to consume disk space. With this Rake task, you can remove invalid references from the database, which +When you remove LFS files from a repository's history, they become orphaned and continue to consume +disk space. With this Rake task, you can remove invalid references from the database, which will allow garbage collection of LFS files. For example: @@ -177,3 +179,8 @@ sudo gitlab-rake gitlab:cleanup:sessions:active_sessions_lookup_keys # installation from source bundle exec rake gitlab:cleanup:sessions:active_sessions_lookup_keys RAILS_ENV=production ``` + +## Container Registry garbage collection + +Container Registry can use considerable amounts of disk space. To clear up +unused layers, the registry includes a [garbage collect command](../administration/packages/container_registry.md#container-registry-garbage-collection). diff --git a/doc/raketasks/features.md b/doc/raketasks/features.md index 5425a0a0667..1d68671a545 100644 --- a/doc/raketasks/features.md +++ b/doc/raketasks/features.md @@ -1,19 +1,21 @@ -# Namespaces +# Namespaces **(CORE ONLY)** + +This Rake task enables [namespaces](../user/group/index.md#namespaces) for projects. ## Enable usernames and namespaces for user projects -This command will enable the namespaces feature introduced in v4.0. It will move every project in its namespace folder. +This command will enable the namespaces feature introduced in GitLab 4.0. It will move every project in its namespace folder. Note: -- Because the **repository location will change**, you will need to **update all your Git URLs** to point to the new location. -- Username can be changed at **Profile ➔ Account**. - -**Example:** +- The **repository location will change**, so you will need to **update all your Git URLs** to + point to the new location. +- The username can be changed at **Profile > Account**. -Old path: `git@example.org:myrepo.git` +For example: -New path: `git@example.org:username/myrepo.git` or `git@example.org:groupname/myrepo.git` +- Old path: `git@example.org:myrepo.git`. +- New path: `git@example.org:username/myrepo.git` or `git@example.org:groupname/myrepo.git`. ```shell bundle exec rake gitlab:enable_namespaces RAILS_ENV=production diff --git a/doc/raketasks/generate_sample_prometheus_data.md b/doc/raketasks/generate_sample_prometheus_data.md index 78e8ef188bf..902be6862c8 100644 --- a/doc/raketasks/generate_sample_prometheus_data.md +++ b/doc/raketasks/generate_sample_prometheus_data.md @@ -1,15 +1,25 @@ -# Generate Sample Prometheus Data +# Generate sample Prometheus data **(CORE ONLY)** This command will run Prometheus queries for each of the metrics of a specific environment -for a series of time intervals: 30 minutes, 3 hours, 8 hours, 24 hours, 72 hours, and 7 days -to now. The results of each of query are stored under a `sample_metrics` directory as a yaml +for a series of time intervals to now: + +- 30 minutes +- 3 hours +- 8 hours +- 24 hours +- 72 hours +- 7 days + +The results of each of query are stored under a `sample_metrics` directory as a YAML file named by the metric's `identifier`. When the environmental variable `USE_SAMPLE_METRICS` is set, the Prometheus API query is re-routed to `Projects::Environments::SampleMetricsController` which loads the appropriate data set if it is present within the `sample_metrics` directory. -- This command requires an id from an Environment with an available Prometheus installation. +This command requires an ID from an environment with an available Prometheus installation. + +## Example -**Example:** +The following example demonstrates how to run the Rake task: ```shell bundle exec rake gitlab:generate_sample_prometheus_data[21] diff --git a/doc/raketasks/import.md b/doc/raketasks/import.md index cda742b6077..5229ce2ab08 100644 --- a/doc/raketasks/import.md +++ b/doc/raketasks/import.md @@ -1,61 +1,63 @@ -# Import bare repositories into your GitLab instance +# Import bare repositories **(CORE ONLY)** -## Notes +Rake tasks are available to import bare repositories into a GitLab instance. -- The owner of the project will be the first admin -- The groups will be created as needed, including subgroups -- The owner of the group will be the first admin -- Existing projects will be skipped -- Projects in hashed storage may be skipped (see [Importing bare repositories from hashed storage](#importing-bare-repositories-from-hashed-storage)) -- The existing Git repos will be moved from disk (removed from the original path) +Note that: -## How to use +- The owner of the project will be the first administrator. +- The groups will be created as needed, including subgroups. +- The owner of the group will be the first administrator. +- Existing projects will be skipped. +- Projects in hashed storage may be skipped. For more information, see + [Importing bare repositories from hashed storage](#importing-bare-repositories-from-hashed-storage). +- The existing Git repositories will be moved from disk (removed from the original path). -### Create a new folder to import your Git repositories from +To import bare repositories into a GitLab instance: -The new folder needs to have Git user ownership and read/write/execute access for Git user and its group: +1. Create a new folder to import your Git repositories from. The new folder needs to have Git user + ownership and read/write/execute access for Git user and its group: -```shell -sudo -u git mkdir -p /var/opt/gitlab/git-data/repository-import-<date>/new_group -``` - -### Copy your bare repositories inside this newly created folder + ```shell + sudo -u git mkdir -p /var/opt/gitlab/git-data/repository-import-<date>/new_group + ``` -- Any `.git` repositories found on any of the subfolders will be imported as projects -- Groups will be created as needed, these could be nested folders. Example: +1. Copy your bare repositories inside this newly created folder. Note: -If we copy the repos to `/var/opt/gitlab/git-data/repository-import-<date>`, and repo A needs to be under the groups G1 and G2, it will -have to be created under those folders: `/var/opt/gitlab/git-data/repository-import-<date>/G1/G2/A.git`. + - Any `.git` repositories found on any of the subfolders will be imported as projects. + - Groups will be created as needed, these could be nested folders. -```shell -sudo cp -r /old/git/foo.git /var/opt/gitlab/git-data/repository-import-<date>/new_group/ + For example, if we copy the repositories to `/var/opt/gitlab/git-data/repository-import-<date>`, + and repository `A` needs to be under the groups `G1` and `G2`, it must be created under those folders: + `/var/opt/gitlab/git-data/repository-import-<date>/G1/G2/A.git`. -# Do this once when you are done copying git repositories -sudo chown -R git:git /var/opt/gitlab/git-data/repository-import-<date> -``` + ```shell + sudo cp -r /old/git/foo.git /var/opt/gitlab/git-data/repository-import-<date>/new_group/ -`foo.git` needs to be owned by the `git` user and `git` users group. + # Do this once when you are done copying git repositories + sudo chown -R git:git /var/opt/gitlab/git-data/repository-import-<date> + ``` -If you are using an installation from source, replace `/var/opt/gitlab/` with `/home/git`. + `foo.git` needs to be owned by the `git` user and `git` users group. -### Run the command below depending on your type of installation + If you are using an installation from source, replace `/var/opt/gitlab/` with `/home/git`. -#### Omnibus Installation +1. Run the following command depending on your type of installation: -```shell -sudo gitlab-rake gitlab:import:repos['/var/opt/gitlab/git-data/repository-import-<date>'] -``` + - Omnibus Installation -#### Installation from source + ```shell + sudo gitlab-rake gitlab:import:repos['/var/opt/gitlab/git-data/repository-import-<date>'] + ``` -Before running this command you need to change the directory to where your GitLab installation is located: + - Installation from source. Before running this command you need to change to the directory where + your GitLab installation is located: -```shell -cd /home/git/gitlab -sudo -u git -H bundle exec rake gitlab:import:repos['/var/opt/gitlab/git-data/repository-import-<date>'] RAILS_ENV=production -``` + ```shell + cd /home/git/gitlab + sudo -u git -H bundle exec rake gitlab:import:repos['/var/opt/gitlab/git-data/repository-import-<date>'] RAILS_ENV=production + ``` -#### Example output +## Example output ```plaintext Processing /var/opt/gitlab/git-data/repository-import-1/a/b/c/blah.git @@ -73,8 +75,6 @@ Processing /var/opt/gitlab/git-data/repository-import-1/group/xyz.git ## Importing bare repositories from hashed storage -### Background - Projects in legacy storage have a directory structure that mirrors their full project path in GitLab, including their namespace structure. This information is leveraged by the bare repository importer to import projects into their proper @@ -86,17 +86,17 @@ improved performance and data integrity. See [Repository Storage Types](../administration/repository_storage_types.md) for more details. -### Which repositories are importable? +The repositories that are importable depends on the version of GitLab. -#### GitLab 10.3 or earlier +### GitLab 10.3 or earlier Importing bare repositories from hashed storage is unsupported. -#### GitLab 10.4 and later +### GitLab 10.4 and later To support importing bare repositories from hashed storage, GitLab 10.4 and later stores the full project path with each repository, in a special section of -the Git repository's config file. This section is formatted as follows: +the Git repository's configuration file. This section is formatted as follows: ```ini [gitlab] diff --git a/doc/raketasks/list_repos.md b/doc/raketasks/list_repos.md index 10e6cb04bfa..411de52e379 100644 --- a/doc/raketasks/list_repos.md +++ b/doc/raketasks/list_repos.md @@ -1,7 +1,8 @@ -# Listing repository directories +# Listing repository directories **(CORE ONLY)** -You can print a list of all Git repositories on disk managed by -GitLab with the following command: +You can print a list of all Git repositories on disk managed by GitLab. + +To print a list, run the following command: ```shell # Omnibus @@ -12,10 +13,13 @@ cd /home/git/gitlab sudo -u git -H bundle exec rake gitlab:list_repos RAILS_ENV=production ``` -If you only want to list projects with recent activity you can pass -a date with the 'SINCE' environment variable. The time you specify -is parsed by the Rails [TimeZone#parse -function](https://api.rubyonrails.org/classes/ActiveSupport/TimeZone.html#method-i-parse). +NOTE: **Note:** +The results use the default ordering of the GitLab Rails application. + +## Limit search results + +To list only projects with recent activity, pass a date with the `SINCE` environment variable. The +time you specify is parsed by the Rails [TimeZone#parse function](https://api.rubyonrails.org/classes/ActiveSupport/TimeZone.html#method-i-parse). ```shell # Omnibus @@ -25,6 +29,3 @@ sudo gitlab-rake gitlab:list_repos SINCE='Sep 1 2015' cd /home/git/gitlab sudo -u git -H bundle exec rake gitlab:list_repos RAILS_ENV=production SINCE='Sep 1 2015' ``` - -Note that the projects listed are NOT sorted by activity; they use -the default ordering of the GitLab Rails application. diff --git a/doc/raketasks/migrate_snippets.md b/doc/raketasks/migrate_snippets.md new file mode 100644 index 00000000000..fce91ab703e --- /dev/null +++ b/doc/raketasks/migrate_snippets.md @@ -0,0 +1,96 @@ +# Migration to Versioned Snippets **(CORE ONLY)** + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/215861) in GitLab 13.0. + +In GitLab 13.0, [GitLab Snippets are backed by Git repositories](../user/snippets.md#versioned-snippets). +This means that the snippet content will be stored in the repository +and users can update it directly through Git. + +Nevertheless, existing GitLab Snippets have to be migrated to this new functionality. +For each snippet, a new repository is created and the snippet content is committed +to the repository inside a file whose name is the file name used in the snippet +as well. + +GitLab performs this migration through a [Background Migration](../development/background_migrations.md) +automatically when the GitLab instance is upgrade to 13.0 or a higher version. +However, if the migration fails for any of the snippets, they still need +to be migrated individually. + +The following Rake tasks will help with that process. + +## Migrate specific snippets to Git + +In case you want to migrate a range of snippets, run the tasks as described below. + +For Omnibus installations, run: + +```shell +sudo gitlab-rake gitlab:snippets:migrate SNIPPET_IDS=1,2,3,4 +``` + +For installations from source code, run: + +```shell +bundle exec rake gitlab:snippets:migrate SNIPPET_IDS=1,2,3,4 +``` + +There is a default limit (100) to the number of ids supported in the migration +process. You can modify this limit by using the env variable `LIMIT`. + +```shell +sudo gitlab-rake gitlab:snippets:migrate SNIPPET_IDS=1,2,3,4 LIMIT=50 +``` + +For installations from source code, run: + +```shell +bundle exec rake gitlab:snippets:migrate SNIPPET_IDS=1,2,3,4 LIMIT=50 +``` + +## Show whether the snippet background migration is running + +In case you want to check the status of the snippet background migration, +whether it is running or not, you can use the following task. + +For Omnibus installations, run: + +```shell +sudo gitlab-rake gitlab:snippets:migration_status +``` + +For installations from source code, run: + +```shell +bundle exec rake gitlab:snippets:migration_status RAILS_ENV=production +``` + +## List non-migrated snippets + +With the following task, you can get the ids of all of the snippets +that haven't been migrated yet or failed to migrate. + +For Omnibus installations, run: + +```shell +sudo gitlab-rake gitlab:snippets:list_non_migrated +``` + +For installations from source code, run: + +```shell +bundle exec rake gitlab:snippets:list_non_migrated RAILS_ENV=production +``` + +As the number of non-migrated snippets can be large, we limit +by default the size of the number of ids returned to 100. You can +modify this limit by using the env variable `LIMIT`. + +```shell +sudo gitlab-rake gitlab:snippets:list_non_migrated LIMIT=200 +``` + +For installations from source code, run: + +```shell +bundle exec rake gitlab:snippets:list_non_migrated RAILS_ENV=production LIMIT=200 +``` diff --git a/doc/raketasks/user_management.md b/doc/raketasks/user_management.md index ffb1baa0b6e..61e30f9ddb1 100644 --- a/doc/raketasks/user_management.md +++ b/doc/raketasks/user_management.md @@ -1,7 +1,11 @@ -# User management +# User management **(CORE ONLY)** + +GitLab provides Rake tasks for user management. ## Add user as a developer to all projects +To add a user as a developer to all projects, run: + ```shell # omnibus-gitlab sudo gitlab-rake gitlab:import:user_to_projects[username@domain.tld] @@ -12,9 +16,7 @@ bundle exec rake gitlab:import:user_to_projects[username@domain.tld] RAILS_ENV=p ## Add all users to all projects -Notes: - -- admin users are added as maintainers +To add all users to all projects, run: ```shell # omnibus-gitlab @@ -24,8 +26,13 @@ sudo gitlab-rake gitlab:import:all_users_to_all_projects bundle exec rake gitlab:import:all_users_to_all_projects RAILS_ENV=production ``` +NOTE: **Note:** +Admin users are added as maintainers. + ## Add user as a developer to all groups +To add a user as a developer to all groups, run: + ```shell # omnibus-gitlab sudo gitlab-rake gitlab:import:user_to_groups[username@domain.tld] @@ -36,9 +43,7 @@ bundle exec rake gitlab:import:user_to_groups[username@domain.tld] RAILS_ENV=pro ## Add all users to all groups -Notes: - -- admin users are added as owners so they can add additional users to the group +To add all users to all groups, run: ```shell # omnibus-gitlab @@ -48,19 +53,25 @@ sudo gitlab-rake gitlab:import:all_users_to_all_groups bundle exec rake gitlab:import:all_users_to_all_groups RAILS_ENV=production ``` -## Maintain tight control over the number of active users on your GitLab installation +NOTE: **Note:** +Admin users are added as owners so they can add additional users to the group. + +## Control the number of active users -- Enable this setting to keep new users blocked until they have been cleared by the admin (default: false). +Enable this setting to keep new users blocked until they have been cleared by the administrator. +Defaults to `false`: ```plaintext block_auto_created_users: false ``` -## Disable Two-factor Authentication (2FA) for all users +## Disable two-factor authentication for all users -This task will disable 2FA for all users that have it enabled. This can be +This task disables two-factor authentication (2FA) for all users that have it enabled. This can be useful if GitLab's `config/secrets.yml` file has been lost and users are unable -to login, for example. +to log in, for example. + +To disable two-factor authentication for all users, run: ```shell # omnibus-gitlab @@ -70,65 +81,65 @@ sudo gitlab-rake gitlab:two_factor:disable_for_all_users bundle exec rake gitlab:two_factor:disable_for_all_users RAILS_ENV=production ``` -## Rotate Two-factor Authentication (2FA) encryption key +## Rotate two-factor authentication encryption key -GitLab stores the secret data enabling 2FA to work in an encrypted database -column. The encryption key for this data is known as `otp_key_base`, and is +GitLab stores the secret data required for two-factor authentication (2FA) in an encrypted +database column. The encryption key for this data is known as `otp_key_base`, and is stored in `config/secrets.yml`. If that file is leaked, but the individual 2FA secrets have not, it's possible to re-encrypt those secrets with a new encryption key. This allows you to change the leaked key without forcing all users to change their 2FA details. -First, look up the old key. This is in the `config/secrets.yml` file, but -**make sure you're working with the production section**. The line you're -interested in will look like this: +To rotate the two-factor authentication encryption key: -```yaml -production: - otp_key_base: ffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff -``` +1. Look up the old key. This is in the `config/secrets.yml` file, but **make sure you're working + with the production section**. The line you're interested in will look like this: -Next, generate a new secret: + ```yaml + production: + otp_key_base: ffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff + ``` -```shell -# omnibus-gitlab -sudo gitlab-rake secret +1. Generate a new secret: -# installation from source -bundle exec rake secret RAILS_ENV=production -``` + ```shell + # omnibus-gitlab + sudo gitlab-rake secret -Now you need to stop the GitLab server, back up the existing secrets file and -update the database: + # installation from source + bundle exec rake secret RAILS_ENV=production + ``` -```shell -# omnibus-gitlab -sudo gitlab-ctl stop -sudo cp config/secrets.yml config/secrets.yml.bak -sudo gitlab-rake gitlab:two_factor:rotate_key:apply filename=backup.csv old_key=<old key> new_key=<new key> +1. Stop the GitLab server, back up the existing secrets file, and update the database: -# installation from source -sudo /etc/init.d/gitlab stop -cp config/secrets.yml config/secrets.yml.bak -bundle exec rake gitlab:two_factor:rotate_key:apply filename=backup.csv old_key=<old key> new_key=<new key> RAILS_ENV=production -``` + ```shell + # omnibus-gitlab + sudo gitlab-ctl stop + sudo cp config/secrets.yml config/secrets.yml.bak + sudo gitlab-rake gitlab:two_factor:rotate_key:apply filename=backup.csv old_key=<old key> new_key=<new key> -The `<old key>` value can be read from `config/secrets.yml`; `<new key>` was -generated earlier. The **encrypted** values for the user 2FA secrets will be -written to the specified `filename` - you can use this to rollback in case of -error. + # installation from source + sudo /etc/init.d/gitlab stop + cp config/secrets.yml config/secrets.yml.bak + bundle exec rake gitlab:two_factor:rotate_key:apply filename=backup.csv old_key=<old key> new_key=<new key> RAILS_ENV=production + ``` -Finally, change `config/secrets.yml` to set `otp_key_base` to `<new key>` and -restart. Again, make sure you're operating in the **production** section. + The `<old key>` value can be read from `config/secrets.yml` (`<new key>` was + generated earlier). The **encrypted** values for the user 2FA secrets will be + written to the specified `filename`. You can use this to rollback in case of + error. -```shell -# omnibus-gitlab -sudo gitlab-ctl start +1. Change `config/secrets.yml` to set `otp_key_base` to `<new key>` and restart. Again, make sure + you're operating in the **production** section. -# installation from source -sudo /etc/init.d/gitlab start -``` + ```shell + # omnibus-gitlab + sudo gitlab-ctl start + + # installation from source + sudo /etc/init.d/gitlab start + ``` If there are any problems (perhaps using the wrong value for `old_key`), you can restore your backup of `config/secrets.yml` and rollback the changes: diff --git a/doc/raketasks/web_hooks.md b/doc/raketasks/web_hooks.md index 22084c862ba..7bd2ed311d2 100644 --- a/doc/raketasks/web_hooks.md +++ b/doc/raketasks/web_hooks.md @@ -1,60 +1,78 @@ # Webhooks administration **(CORE ONLY)** -## Add a webhook for **ALL** projects +GitLab provides Rake tasks for webhooks management. + +Requests to the [local network by webhooks](../security/webhooks.md) can be allowed or blocked by an +administrator. + +## Add a webhook to all projects + +To add a webhook to all projects, run: ```shell # omnibus-gitlab sudo gitlab-rake gitlab:web_hook:add URL="http://example.com/hook" + # source installations bundle exec rake gitlab:web_hook:add URL="http://example.com/hook" RAILS_ENV=production ``` -## Add a webhook for projects in a given **NAMESPACE** +## Add a webhook to projects in a namespace + +To add a webhook to all projects in a specific namespace, run: ```shell # omnibus-gitlab -sudo gitlab-rake gitlab:web_hook:add URL="http://example.com/hook" NAMESPACE=acme +sudo gitlab-rake gitlab:web_hook:add URL="http://example.com/hook" NAMESPACE=<namespace> + # source installations -bundle exec rake gitlab:web_hook:add URL="http://example.com/hook" NAMESPACE=acme RAILS_ENV=production +bundle exec rake gitlab:web_hook:add URL="http://example.com/hook" NAMESPACE=<namespace> RAILS_ENV=production ``` -## Remove a webhook from **ALL** projects using +## Remove a webhook from projects + +To remove a webhook from all projects, run: ```shell # omnibus-gitlab sudo gitlab-rake gitlab:web_hook:rm URL="http://example.com/hook" + # source installations bundle exec rake gitlab:web_hook:rm URL="http://example.com/hook" RAILS_ENV=production ``` -## Remove a webhook from projects in a given **NAMESPACE** +## Remove a webhook from projects in a namespace + +To remove a webhook from projects in a specific namespace, run: ```shell # omnibus-gitlab -sudo gitlab-rake gitlab:web_hook:rm URL="http://example.com/hook" NAMESPACE=acme +sudo gitlab-rake gitlab:web_hook:rm URL="http://example.com/hook" NAMESPACE=<namespace> + # source installations -bundle exec rake gitlab:web_hook:rm URL="http://example.com/hook" NAMESPACE=acme RAILS_ENV=production +bundle exec rake gitlab:web_hook:rm URL="http://example.com/hook" NAMESPACE=<namespace> RAILS_ENV=production ``` -## List **ALL** webhooks +## List all webhooks + +To list all webhooks, run: ```shell # omnibus-gitlab sudo gitlab-rake gitlab:web_hook:list + # source installations bundle exec rake gitlab:web_hook:list RAILS_ENV=production ``` -## List the webhooks from projects in a given **NAMESPACE** +## List webhooks for projects in a namespace + +To list all webhook for projects in a specified namespace, run: ```shell # omnibus-gitlab -sudo gitlab-rake gitlab:web_hook:list NAMESPACE=acme +sudo gitlab-rake gitlab:web_hook:list NAMESPACE=<namespace> + # source installations -bundle exec rake gitlab:web_hook:list NAMESPACE=acme RAILS_ENV=production +bundle exec rake gitlab:web_hook:list NAMESPACE=<namespace> RAILS_ENV=production ``` - -## Local requests in webhooks - -[Requests to local network by webhooks](../security/webhooks.md) can be allowed -or blocked by an administrator. diff --git a/doc/raketasks/x509_signatures.md b/doc/raketasks/x509_signatures.md index dd518997f7b..f7c47794690 100644 --- a/doc/raketasks/x509_signatures.md +++ b/doc/raketasks/x509_signatures.md @@ -1,21 +1,24 @@ -# X509 signatures +# X.509 signatures **(CORE ONLY)** -When [signing commits with x509](../user/project/repository/x509_signed_commits/index.md) -the trust anchor might change and the signatures stored within the database have -to be updated. +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/122159) in GitLab 12.10. -## Update all x509 signatures +When [signing commits with X.509](../user/project/repository/x509_signed_commits/index.md), +the trust anchor might change and the signatures stored within the database must be updated. -This task loops through all X509 signed commits and updates their verification -based on current certificate store. +## Update all X.509 signatures -**Omnibus Installation** +This task loops through all X.509 signed commits and updates their verification based on current +certificate store. + +To update all X.509 signatures, run: + +**Omnibus Installations:** ```shell sudo gitlab-rake gitlab:x509:update_signatures ``` -**Source Installation** +**Source Installations:** ```shell sudo -u git -H bundle exec rake gitlab:x509:update_signatures RAILS_ENV=production |
