summaryrefslogtreecommitdiff
path: root/doc/raketasks
diff options
context:
space:
mode:
Diffstat (limited to 'doc/raketasks')
-rw-r--r--doc/raketasks/README.md46
-rw-r--r--doc/raketasks/backup_hrz.pngbin11441 -> 0 bytes
-rw-r--r--doc/raketasks/backup_restore.md118
-rw-r--r--doc/raketasks/cleanup.md13
-rw-r--r--doc/raketasks/features.md18
-rw-r--r--doc/raketasks/generate_sample_prometheus_data.md20
-rw-r--r--doc/raketasks/import.md90
-rw-r--r--doc/raketasks/list_repos.md21
-rw-r--r--doc/raketasks/migrate_snippets.md96
-rw-r--r--doc/raketasks/user_management.md117
-rw-r--r--doc/raketasks/web_hooks.md52
-rw-r--r--doc/raketasks/x509_signatures.md21
12 files changed, 378 insertions, 234 deletions
diff --git a/doc/raketasks/README.md b/doc/raketasks/README.md
index 1670849016c..4bba4f2bc5a 100644
--- a/doc/raketasks/README.md
+++ b/doc/raketasks/README.md
@@ -15,25 +15,27 @@ GitLab Rake tasks are performed using:
The following are available Rake tasks:
-| Tasks | Description |
-|:-------------------------------------------------------------------------------------------|:------------------------------------------------------------------------------------------|
-| [Back up and restore](backup_restore.md) | Back up, restore, and migrate GitLab instances between servers. |
-| [Clean up](cleanup.md) | Clean up unneeded items GitLab instances. |
-| [Development](../development/rake_tasks.md) | Tasks for GitLab contributors. |
-| [Elasticsearch](../integration/elasticsearch.md#gitlab-elasticsearch-rake-tasks) | Maintain Elasticsearch in a GitLab instance. |
-| [Enable namespaces](features.md) | Enable usernames and namespaces for user projects. |
-| [General maintenance](../administration/raketasks/maintenance.md) | General maintenance and self-check tasks. |
-| [Geo maintenance](../administration/raketasks/geo.md) **(PREMIUM ONLY)** | [Geo](../administration/geo/replication/index.md)-related maintenance. |
-| [GitHub import](../administration/raketasks/github_import.md) | Retrieve and import repositories from GitHub. |
-| [Import repositories](import.md) | Import bare repositories into your GitLab instance. |
-| [Import large project exports](../development/import_project.md#importing-via-a-rake-task) | Import large GitLab [project exports](../user/project/settings/import_export.md). |
-| [Integrity checks](../administration/raketasks/check.md) | Check the integrity of repositories, files, and LDAP. |
-| [LDAP maintenance](../administration/raketasks/ldap.md) | [LDAP](../administration/auth/ldap.md)-related tasks. |
-| [List repositories](list_repos.md) | List of all GitLab-managed Git repositories on disk. |
-| [Project import/export](../administration/raketasks/project_import_export.md) | Prepare for [project exports and imports](../user/project/settings/import_export.md). |
-| [Repository storage](../administration/raketasks/storage.md) | List and migrate existing projects and attachments from legacy storage to hashed storage. |
-| [Uploads migrate](../administration/raketasks/uploads/migrate.md) | Migrate uploads between storage local and object storage. |
-| [Uploads sanitize](../administration/raketasks/uploads/sanitize.md) | Remove EXIF data from images uploaded to earlier versions of GitLab. |
-| [User management](user_management.md) | Perform user management tasks. |
-| [Webhooks administration](web_hooks.md) | Maintain project Webhooks. |
-| [X509 signatures](x509_signatures.md) | Update x509 commit signatures, useful if certificate store has changed. |
+| Tasks | Description |
+|:----------------------------------------------------------------------------------------------------|:------------------------------------------------------------------------------------------|
+| [Back up and restore](backup_restore.md) | Back up, restore, and migrate GitLab instances between servers. |
+| [Clean up](cleanup.md) | Clean up unneeded items from GitLab instances. |
+| [Development](../development/rake_tasks.md) | Tasks for GitLab contributors. |
+| [Elasticsearch](../integration/elasticsearch.md#gitlab-elasticsearch-rake-tasks) **(STARTER ONLY)** | Maintain Elasticsearch in a GitLab instance. |
+| [Enable namespaces](features.md) | Enable usernames and namespaces for user projects. |
+| [General maintenance](../administration/raketasks/maintenance.md) | General maintenance and self-check tasks. |
+| [Geo maintenance](../administration/raketasks/geo.md) **(PREMIUM ONLY)** | [Geo](../administration/geo/replication/index.md)-related maintenance. |
+| [GitHub import](../administration/raketasks/github_import.md) | Retrieve and import repositories from GitHub. |
+| [Import repositories](import.md) | Import bare repositories into your GitLab instance. |
+| [Import large project exports](../development/import_project.md#importing-via-a-rake-task) | Import large GitLab [project exports](../user/project/settings/import_export.md). |
+| [Integrity checks](../administration/raketasks/check.md) | Check the integrity of repositories, files, and LDAP. |
+| [LDAP maintenance](../administration/raketasks/ldap.md) | [LDAP](../administration/auth/ldap.md)-related tasks. |
+| [List repositories](list_repos.md) | List of all GitLab-managed Git repositories on disk. |
+| [Project import/export](../administration/raketasks/project_import_export.md) | Prepare for [project exports and imports](../user/project/settings/import_export.md). |
+| [Sample Prometheus data](generate_sample_prometheus_data.md) | Generate sample Prometheus data. |
+| [Repository storage](../administration/raketasks/storage.md) | List and migrate existing projects and attachments from legacy storage to hashed storage. |
+| [Uploads migrate](../administration/raketasks/uploads/migrate.md) | Migrate uploads between storage local and object storage. |
+| [Uploads sanitize](../administration/raketasks/uploads/sanitize.md) | Remove EXIF data from images uploaded to earlier versions of GitLab. |
+| [User management](user_management.md) | Perform user management tasks. |
+| [Webhooks administration](web_hooks.md) | Maintain project Webhooks. |
+| [X.509 signatures](x509_signatures.md) | Update X.509 commit signatures, useful if certificate store has changed. |
+| [Migrate Snippets to Git](migrate_snippets.md) | Migrate GitLab Snippets to Git repositories and show migration status |
diff --git a/doc/raketasks/backup_hrz.png b/doc/raketasks/backup_hrz.png
deleted file mode 100644
index 32690b2904c..00000000000
--- a/doc/raketasks/backup_hrz.png
+++ /dev/null
Binary files differ
diff --git a/doc/raketasks/backup_restore.md b/doc/raketasks/backup_restore.md
index 24e4ed43543..3c81bab644e 100644
--- a/doc/raketasks/backup_restore.md
+++ b/doc/raketasks/backup_restore.md
@@ -1,6 +1,6 @@
-# Backing up and restoring GitLab
+# Backing up and restoring GitLab **(CORE ONLY)**
-![backup banner](backup_hrz.png)
+GitLab provides Rake tasks for backing up and restoring GitLab instances.
An application data backup creates an archive file that contains the database,
all repositories and all attachments.
@@ -14,31 +14,26 @@ from one server to another is through backup restore.
In order to be able to backup and restore, you need two essential tools
installed on your system.
-### Rsync
+- **Rsync**: If you installed GitLab:
+ - Using the Omnibus package, you're all set.
+ - From source, make sure `rsync` is installed. For example:
-If you installed GitLab:
+ ```shell
+ # Debian/Ubuntu
+ sudo apt-get install rsync
-- Using the Omnibus package, you're all set.
-- From source, make sure `rsync` is installed:
+ # RHEL/CentOS
+ sudo yum install rsync
+ ```
- ```shell
- # Debian/Ubuntu
- sudo apt-get install rsync
+- **Tar**: Backup and restore tasks use `tar` under the hood to create and extract
+ archives. Ensure you have version 1.30 or above of `tar` available in your
+ system. To check the version, run:
- # RHEL/CentOS
- sudo yum install rsync
+ ```shell
+ tar --version
```
-### Tar
-
-Backup and restore tasks use `tar` under the hood to create and extract
-archives. Ensure you have version 1.30 or above of `tar` available in your
-system. To check the version, run:
-
-```shell
-tar --version
-```
-
## Backup timestamp
NOTE: **Note:**
@@ -56,9 +51,9 @@ available.
For example, if the backup name is `1493107454_2018_04_25_10.6.4-ce_gitlab_backup.tar`,
then the timestamp is `1493107454_2018_04_25_10.6.4-ce`.
-## Creating a backup of the GitLab system
+## Back up GitLab
-GitLab provides a simple command line interface to backup your whole instance.
+GitLab provides a simple command line interface to back up your whole instance.
It backs up your:
- Database
@@ -142,12 +137,12 @@ Deleting tmp directories...[DONE]
Deleting old backups... [SKIPPING]
```
-## Storing configuration files
+### Storing configuration files
-A backup performed by the [raketask GitLab provides](#creating-a-backup-of-the-gitlab-system)
+The [backup Rake task](#back-up-gitlab) GitLab provides
does **not** store your configuration files. The primary reason for this is that your
database contains encrypted information for two-factor authentication, the CI/CD
-'secure variables', etc. Storing encrypted information along with its key in the
+'secure variables', and so on. Storing encrypted information along with its key in the
same place defeats the purpose of using encryption in the first place.
CAUTION: **Warning:**
@@ -182,11 +177,11 @@ If you use Omnibus GitLab, see some additional information
In the unlikely event that the secrets file is lost, see the
[troubleshooting section](#when-the-secrets-file-is-lost).
-## Backup options
+### Backup options
The command line tool GitLab provides to backup your instance can take more options.
-### Backup strategy option
+#### Backup strategy option
> [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/8728) in GitLab 8.17.
@@ -214,7 +209,7 @@ sudo gitlab-backup create STRATEGY=copy
NOTE: **Note**
For GitLab 12.1 and earlier, use `gitlab-rake gitlab:backup:create`.
-### Backup filename
+#### Backup filename
CAUTION: **Warning:**
If you use a custom backup filename, you will not be able to
@@ -231,7 +226,7 @@ For GitLab 12.1 and earlier, use `gitlab-rake gitlab:backup:create`.
The resulting file will then be `dump_gitlab_backup.tar`. This is useful for systems that make use of rsync and incremental backups, and will result in considerably faster transfer speeds.
-### Rsyncable
+#### Rsyncable
To make sure the generated archive is intelligently transferable by rsync, the `GZIP_RSYNCABLE=yes` option can be set. This will set the `--rsyncable` option to `gzip`. This is only useful in combination with setting [the Backup filename option](#backup-filename).
@@ -244,7 +239,7 @@ sudo gitlab-backup create BACKUP=dump GZIP_RSYNCABLE=yes
NOTE: **Note**
For GitLab 12.1 and earlier, use `gitlab-rake gitlab:backup:create`.
-### Excluding specific directories from the backup
+#### Excluding specific directories from the backup
You can choose what should be exempt from the backup up by adding the environment variable `SKIP`.
The available options are:
@@ -278,7 +273,7 @@ For installations from source:
sudo -u git -H bundle exec rake gitlab:backup:create SKIP=db,uploads RAILS_ENV=production
```
-### Skipping tar creation
+#### Skipping tar creation
The last part of creating a backup is generation of a `.tar` file containing
all the parts. In some cases (for example, if the backup is picked up by other
@@ -303,19 +298,19 @@ For installations from source:
sudo -u git -H bundle exec rake gitlab:backup:create SKIP=tar RAILS_ENV=production
```
-### Uploading backups to a remote (cloud) storage
+#### Uploading backups to a remote (cloud) storage
-Starting with GitLab 7.4 you can let the backup script upload the '.tar' file it creates.
+Starting with GitLab 7.4 you can let the backup script upload the `.tar` file it creates.
It uses the [Fog library](http://fog.io/) to perform the upload.
In the example below we use Amazon S3 for storage, but Fog also lets you use
[other storage providers](http://fog.io/storage/). GitLab
[imports cloud drivers](https://gitlab.com/gitlab-org/gitlab/blob/30f5b9a5b711b46f1065baf755e413ceced5646b/Gemfile#L88)
-for AWS, Google, OpenStack Swift, Rackspace and Aliyun as well. A local driver is
+for AWS, Google, OpenStack Swift, Rackspace, and Aliyun as well. A local driver is
[also available](#uploading-to-locally-mounted-shares).
[Read more about using object storage with GitLab](../administration/object_storage.md).
-#### Using Amazon S3
+##### Using Amazon S3
For Omnibus GitLab packages:
@@ -333,9 +328,9 @@ For Omnibus GitLab packages:
gitlab_rails['backup_upload_remote_directory'] = 'my.s3.bucket'
```
-1. [Reconfigure GitLab] for the changes to take effect
+1. [Reconfigure GitLab](../administration/restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect
-#### Digital Ocean Spaces
+##### Digital Ocean Spaces
This example can be used for a bucket in Amsterdam (AMS3).
@@ -352,7 +347,7 @@ This example can be used for a bucket in Amsterdam (AMS3).
gitlab_rails['backup_upload_remote_directory'] = 'my.s3.bucket'
```
-1. [Reconfigure GitLab] for the changes to take effect
+1. [Reconfigure GitLab](../administration/restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect
NOTE: **Note:**
If you see `400 Bad Request` by using Digital Ocean Spaces, the cause may be the
@@ -360,7 +355,7 @@ usage of backup encryption. Remove or comment the line that
contains `gitlab_rails['backup_encryption']` since Digital Ocean Spaces
doesn't support encryption.
-#### Other S3 Providers
+##### Other S3 Providers
Not all S3 providers are fully-compatible with the Fog library. For example,
if you see `411 Length Required` errors after attempting to upload, you may
@@ -397,7 +392,7 @@ For installations from source:
# storage_class: 'STANDARD'
```
-1. [Restart GitLab] for the changes to take effect
+1. [Restart GitLab](../administration/restart_gitlab.md#installations-from-source) for the changes to take effect
If you are uploading your backups to S3 you will probably want to create a new
IAM user with restricted access rights. To give the upload user access only for
@@ -450,7 +445,7 @@ with the name of your bucket:
}
```
-#### Using Google Cloud Storage
+##### Using Google Cloud Storage
If you want to use Google Cloud Storage to save backups, you'll have to create
an access key from the Google console first:
@@ -476,7 +471,7 @@ For Omnibus GitLab packages:
gitlab_rails['backup_upload_remote_directory'] = 'my.google.bucket'
```
-1. [Reconfigure GitLab] for the changes to take effect
+1. [Reconfigure GitLab](../administration/restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect
For installations from source:
@@ -492,9 +487,9 @@ For installations from source:
remote_directory: 'my.google.bucket'
```
-1. [Restart GitLab] for the changes to take effect
+1. [Restart GitLab](../administration/restart_gitlab.md#installations-from-source) for the changes to take effect
-#### Specifying a custom directory for backups
+##### Specifying a custom directory for backups
Note: This option only works for remote storage. If you want to group your backups
you can pass a `DIRECTORY` environment variable:
@@ -507,9 +502,9 @@ sudo gitlab-backup create DIRECTORY=weekly
NOTE: **Note**
For GitLab 12.1 and earlier, use `gitlab-rake gitlab:backup:create`.
-### Uploading to locally mounted shares
+#### Uploading to locally mounted shares
-You may also send backups to a mounted share (`NFS` / `CIFS` / `SMB` / etc.) by
+You may also send backups to a mounted share (for example, `NFS`,`CIFS`, or `SMB`) by
using the Fog [`Local`](https://github.com/fog/fog-local#usage) storage provider.
The directory pointed to by the `local_root` key **must** be owned by the `git`
user **when mounted** (mounting with the `uid=` of the `git` user for `CIFS` and
@@ -539,7 +534,7 @@ For Omnibus GitLab packages:
gitlab_rails['backup_upload_remote_directory'] = 'gitlab_backups'
```
-1. [Reconfigure GitLab] for the changes to take effect.
+1. [Reconfigure GitLab](../administration/restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect.
For installations from source:
@@ -557,9 +552,9 @@ For installations from source:
remote_directory: 'gitlab_backups'
```
-1. [Restart GitLab] for the changes to take effect.
+1. [Restart GitLab](../administration/restart_gitlab.md#installations-from-source) for the changes to take effect.
-### Backup archive permissions
+#### Backup archive permissions
The backup archives created by GitLab (`1393513186_2014_02_27_gitlab_backup.tar`)
will have owner/group `git`/`git` and 0600 permissions by default.
@@ -574,7 +569,7 @@ For Omnibus GitLab packages:
gitlab_rails['backup_archive_permissions'] = 0644 # Makes the backup archives world-readable
```
-1. [Reconfigure GitLab] for the changes to take effect.
+1. [Reconfigure GitLab](../administration/restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect.
For installations from source:
@@ -585,9 +580,9 @@ For installations from source:
archive_permissions: 0644 # Makes the backup archives world-readable
```
-1. [Restart GitLab] for the changes to take effect.
+1. [Restart GitLab](../administration/restart_gitlab.md#installations-from-source) for the changes to take effect.
-### Configuring cron to make daily backups
+#### Configuring cron to make daily backups
CAUTION: **Warning:**
The following cron jobs do not [backup your GitLab configuration files](#storing-configuration-files)
@@ -669,9 +664,9 @@ For installations from source:
keep_time: 604800
```
-1. [Restart GitLab] for the changes to take effect.
+1. [Restart GitLab](../administration/restart_gitlab.md#installations-from-source) for the changes to take effect.
-## Restore
+## Restore GitLab
GitLab provides a simple command line interface to restore your whole installation,
and is flexible enough to fit your needs.
@@ -692,7 +687,7 @@ before restoring the backup.
You need to have a working GitLab installation before you can perform
a restore. This is mainly because the system user performing the
restore actions (`git`) is usually not allowed to create or delete
-the SQL database it needs to import data into ('gitlabhq_production').
+the SQL database it needs to import data into (`gitlabhq_production`).
All existing data will be either erased (SQL) or moved to a separate
directory (repositories, uploads).
@@ -718,7 +713,7 @@ more of the following options:
Read what the [backup timestamp is about](#backup-timestamp).
- `force=yes` - Does not ask if the authorized_keys file should get regenerated and assumes 'yes' for warning that database tables will be removed, enabling the "Write to authorized_keys file" setting, and updating LDAP providers.
-If you are restoring into directories that are mountpoints you will need to make
+If you are restoring into directories that are mount points, you will need to make
sure these directories are empty before attempting a restore. Otherwise GitLab
will attempt to move these directories before restoring the new data and this
would cause an error.
@@ -902,6 +897,8 @@ will have all your repositories, but not any other data.
## Troubleshooting
+The following are possible problems you might encounter with possible solutions.
+
### Restoring database backup using Omnibus packages outputs warnings
If you are using backup restore procedures you might encounter the following warnings:
@@ -1079,12 +1076,9 @@ If you have changed the default filesystem location for the registry, you will
want to run the `chown` against your custom location instead of
`/var/opt/gitlab/gitlab-rails/shared/registry/docker`.
-[reconfigure GitLab]: ../administration/restart_gitlab.md#omnibus-gitlab-reconfigure
-[restart GitLab]: ../administration/restart_gitlab.md#installations-from-source
-
### Backup fails to complete with Gzip error
-While running the backup, you may receive a gzip error:
+While running the backup, you may receive a Gzip error:
```shell
sudo /opt/gitlab/bin/gitlab-backup create
@@ -1098,5 +1092,5 @@ Backup failed
If this happens, check the following:
-1. Confirm there is sufficient disk space for the gzip operation.
+1. Confirm there is sufficient disk space for the Gzip operation.
1. If NFS is being used, check if the mount option `timeout` is set. The default is `600`, and changing this to smaller values have resulted in this error.
diff --git a/doc/raketasks/cleanup.md b/doc/raketasks/cleanup.md
index ef69b1cce15..7908af8da84 100644
--- a/doc/raketasks/cleanup.md
+++ b/doc/raketasks/cleanup.md
@@ -1,4 +1,4 @@
-# Clean up
+# Clean up **(CORE ONLY)**
GitLab provides Rake tasks for cleaning up GitLab instances.
@@ -7,9 +7,11 @@ GitLab provides Rake tasks for cleaning up GitLab instances.
> [Introduced](https://gitlab.com/gitlab-org/gitlab/issues/36628) in GitLab 12.10.
DANGER: **Danger:**
-Do not run this within 12 hours of a GitLab upgrade. This is to ensure that all background migrations have finished, which otherwise may lead to data loss.
+Do not run this within 12 hours of a GitLab upgrade. This is to ensure that all background migrations
+have finished, which otherwise may lead to data loss.
-When you remove LFS files from a repository's history, they become orphaned and continue to consume disk space. With this Rake task, you can remove invalid references from the database, which
+When you remove LFS files from a repository's history, they become orphaned and continue to consume
+disk space. With this Rake task, you can remove invalid references from the database, which
will allow garbage collection of LFS files.
For example:
@@ -177,3 +179,8 @@ sudo gitlab-rake gitlab:cleanup:sessions:active_sessions_lookup_keys
# installation from source
bundle exec rake gitlab:cleanup:sessions:active_sessions_lookup_keys RAILS_ENV=production
```
+
+## Container Registry garbage collection
+
+Container Registry can use considerable amounts of disk space. To clear up
+unused layers, the registry includes a [garbage collect command](../administration/packages/container_registry.md#container-registry-garbage-collection).
diff --git a/doc/raketasks/features.md b/doc/raketasks/features.md
index 5425a0a0667..1d68671a545 100644
--- a/doc/raketasks/features.md
+++ b/doc/raketasks/features.md
@@ -1,19 +1,21 @@
-# Namespaces
+# Namespaces **(CORE ONLY)**
+
+This Rake task enables [namespaces](../user/group/index.md#namespaces) for projects.
## Enable usernames and namespaces for user projects
-This command will enable the namespaces feature introduced in v4.0. It will move every project in its namespace folder.
+This command will enable the namespaces feature introduced in GitLab 4.0. It will move every project in its namespace folder.
Note:
-- Because the **repository location will change**, you will need to **update all your Git URLs** to point to the new location.
-- Username can be changed at **Profile ➔ Account**.
-
-**Example:**
+- The **repository location will change**, so you will need to **update all your Git URLs** to
+ point to the new location.
+- The username can be changed at **Profile > Account**.
-Old path: `git@example.org:myrepo.git`
+For example:
-New path: `git@example.org:username/myrepo.git` or `git@example.org:groupname/myrepo.git`
+- Old path: `git@example.org:myrepo.git`.
+- New path: `git@example.org:username/myrepo.git` or `git@example.org:groupname/myrepo.git`.
```shell
bundle exec rake gitlab:enable_namespaces RAILS_ENV=production
diff --git a/doc/raketasks/generate_sample_prometheus_data.md b/doc/raketasks/generate_sample_prometheus_data.md
index 78e8ef188bf..902be6862c8 100644
--- a/doc/raketasks/generate_sample_prometheus_data.md
+++ b/doc/raketasks/generate_sample_prometheus_data.md
@@ -1,15 +1,25 @@
-# Generate Sample Prometheus Data
+# Generate sample Prometheus data **(CORE ONLY)**
This command will run Prometheus queries for each of the metrics of a specific environment
-for a series of time intervals: 30 minutes, 3 hours, 8 hours, 24 hours, 72 hours, and 7 days
-to now. The results of each of query are stored under a `sample_metrics` directory as a yaml
+for a series of time intervals to now:
+
+- 30 minutes
+- 3 hours
+- 8 hours
+- 24 hours
+- 72 hours
+- 7 days
+
+The results of each of query are stored under a `sample_metrics` directory as a YAML
file named by the metric's `identifier`. When the environmental variable `USE_SAMPLE_METRICS`
is set, the Prometheus API query is re-routed to `Projects::Environments::SampleMetricsController`
which loads the appropriate data set if it is present within the `sample_metrics` directory.
-- This command requires an id from an Environment with an available Prometheus installation.
+This command requires an ID from an environment with an available Prometheus installation.
+
+## Example
-**Example:**
+The following example demonstrates how to run the Rake task:
```shell
bundle exec rake gitlab:generate_sample_prometheus_data[21]
diff --git a/doc/raketasks/import.md b/doc/raketasks/import.md
index cda742b6077..5229ce2ab08 100644
--- a/doc/raketasks/import.md
+++ b/doc/raketasks/import.md
@@ -1,61 +1,63 @@
-# Import bare repositories into your GitLab instance
+# Import bare repositories **(CORE ONLY)**
-## Notes
+Rake tasks are available to import bare repositories into a GitLab instance.
-- The owner of the project will be the first admin
-- The groups will be created as needed, including subgroups
-- The owner of the group will be the first admin
-- Existing projects will be skipped
-- Projects in hashed storage may be skipped (see [Importing bare repositories from hashed storage](#importing-bare-repositories-from-hashed-storage))
-- The existing Git repos will be moved from disk (removed from the original path)
+Note that:
-## How to use
+- The owner of the project will be the first administrator.
+- The groups will be created as needed, including subgroups.
+- The owner of the group will be the first administrator.
+- Existing projects will be skipped.
+- Projects in hashed storage may be skipped. For more information, see
+ [Importing bare repositories from hashed storage](#importing-bare-repositories-from-hashed-storage).
+- The existing Git repositories will be moved from disk (removed from the original path).
-### Create a new folder to import your Git repositories from
+To import bare repositories into a GitLab instance:
-The new folder needs to have Git user ownership and read/write/execute access for Git user and its group:
+1. Create a new folder to import your Git repositories from. The new folder needs to have Git user
+ ownership and read/write/execute access for Git user and its group:
-```shell
-sudo -u git mkdir -p /var/opt/gitlab/git-data/repository-import-<date>/new_group
-```
-
-### Copy your bare repositories inside this newly created folder
+ ```shell
+ sudo -u git mkdir -p /var/opt/gitlab/git-data/repository-import-<date>/new_group
+ ```
-- Any `.git` repositories found on any of the subfolders will be imported as projects
-- Groups will be created as needed, these could be nested folders. Example:
+1. Copy your bare repositories inside this newly created folder. Note:
-If we copy the repos to `/var/opt/gitlab/git-data/repository-import-<date>`, and repo A needs to be under the groups G1 and G2, it will
-have to be created under those folders: `/var/opt/gitlab/git-data/repository-import-<date>/G1/G2/A.git`.
+ - Any `.git` repositories found on any of the subfolders will be imported as projects.
+ - Groups will be created as needed, these could be nested folders.
-```shell
-sudo cp -r /old/git/foo.git /var/opt/gitlab/git-data/repository-import-<date>/new_group/
+ For example, if we copy the repositories to `/var/opt/gitlab/git-data/repository-import-<date>`,
+ and repository `A` needs to be under the groups `G1` and `G2`, it must be created under those folders:
+ `/var/opt/gitlab/git-data/repository-import-<date>/G1/G2/A.git`.
-# Do this once when you are done copying git repositories
-sudo chown -R git:git /var/opt/gitlab/git-data/repository-import-<date>
-```
+ ```shell
+ sudo cp -r /old/git/foo.git /var/opt/gitlab/git-data/repository-import-<date>/new_group/
-`foo.git` needs to be owned by the `git` user and `git` users group.
+ # Do this once when you are done copying git repositories
+ sudo chown -R git:git /var/opt/gitlab/git-data/repository-import-<date>
+ ```
-If you are using an installation from source, replace `/var/opt/gitlab/` with `/home/git`.
+ `foo.git` needs to be owned by the `git` user and `git` users group.
-### Run the command below depending on your type of installation
+ If you are using an installation from source, replace `/var/opt/gitlab/` with `/home/git`.
-#### Omnibus Installation
+1. Run the following command depending on your type of installation:
-```shell
-sudo gitlab-rake gitlab:import:repos['/var/opt/gitlab/git-data/repository-import-<date>']
-```
+ - Omnibus Installation
-#### Installation from source
+ ```shell
+ sudo gitlab-rake gitlab:import:repos['/var/opt/gitlab/git-data/repository-import-<date>']
+ ```
-Before running this command you need to change the directory to where your GitLab installation is located:
+ - Installation from source. Before running this command you need to change to the directory where
+ your GitLab installation is located:
-```shell
-cd /home/git/gitlab
-sudo -u git -H bundle exec rake gitlab:import:repos['/var/opt/gitlab/git-data/repository-import-<date>'] RAILS_ENV=production
-```
+ ```shell
+ cd /home/git/gitlab
+ sudo -u git -H bundle exec rake gitlab:import:repos['/var/opt/gitlab/git-data/repository-import-<date>'] RAILS_ENV=production
+ ```
-#### Example output
+## Example output
```plaintext
Processing /var/opt/gitlab/git-data/repository-import-1/a/b/c/blah.git
@@ -73,8 +75,6 @@ Processing /var/opt/gitlab/git-data/repository-import-1/group/xyz.git
## Importing bare repositories from hashed storage
-### Background
-
Projects in legacy storage have a directory structure that mirrors their full
project path in GitLab, including their namespace structure. This information is
leveraged by the bare repository importer to import projects into their proper
@@ -86,17 +86,17 @@ improved performance and data integrity. See
[Repository Storage Types](../administration/repository_storage_types.md) for
more details.
-### Which repositories are importable?
+The repositories that are importable depends on the version of GitLab.
-#### GitLab 10.3 or earlier
+### GitLab 10.3 or earlier
Importing bare repositories from hashed storage is unsupported.
-#### GitLab 10.4 and later
+### GitLab 10.4 and later
To support importing bare repositories from hashed storage, GitLab 10.4 and
later stores the full project path with each repository, in a special section of
-the Git repository's config file. This section is formatted as follows:
+the Git repository's configuration file. This section is formatted as follows:
```ini
[gitlab]
diff --git a/doc/raketasks/list_repos.md b/doc/raketasks/list_repos.md
index 10e6cb04bfa..411de52e379 100644
--- a/doc/raketasks/list_repos.md
+++ b/doc/raketasks/list_repos.md
@@ -1,7 +1,8 @@
-# Listing repository directories
+# Listing repository directories **(CORE ONLY)**
-You can print a list of all Git repositories on disk managed by
-GitLab with the following command:
+You can print a list of all Git repositories on disk managed by GitLab.
+
+To print a list, run the following command:
```shell
# Omnibus
@@ -12,10 +13,13 @@ cd /home/git/gitlab
sudo -u git -H bundle exec rake gitlab:list_repos RAILS_ENV=production
```
-If you only want to list projects with recent activity you can pass
-a date with the 'SINCE' environment variable. The time you specify
-is parsed by the Rails [TimeZone#parse
-function](https://api.rubyonrails.org/classes/ActiveSupport/TimeZone.html#method-i-parse).
+NOTE: **Note:**
+The results use the default ordering of the GitLab Rails application.
+
+## Limit search results
+
+To list only projects with recent activity, pass a date with the `SINCE` environment variable. The
+time you specify is parsed by the Rails [TimeZone#parse function](https://api.rubyonrails.org/classes/ActiveSupport/TimeZone.html#method-i-parse).
```shell
# Omnibus
@@ -25,6 +29,3 @@ sudo gitlab-rake gitlab:list_repos SINCE='Sep 1 2015'
cd /home/git/gitlab
sudo -u git -H bundle exec rake gitlab:list_repos RAILS_ENV=production SINCE='Sep 1 2015'
```
-
-Note that the projects listed are NOT sorted by activity; they use
-the default ordering of the GitLab Rails application.
diff --git a/doc/raketasks/migrate_snippets.md b/doc/raketasks/migrate_snippets.md
new file mode 100644
index 00000000000..fce91ab703e
--- /dev/null
+++ b/doc/raketasks/migrate_snippets.md
@@ -0,0 +1,96 @@
+# Migration to Versioned Snippets **(CORE ONLY)**
+
+> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/215861) in GitLab 13.0.
+
+In GitLab 13.0, [GitLab Snippets are backed by Git repositories](../user/snippets.md#versioned-snippets).
+This means that the snippet content will be stored in the repository
+and users can update it directly through Git.
+
+Nevertheless, existing GitLab Snippets have to be migrated to this new functionality.
+For each snippet, a new repository is created and the snippet content is committed
+to the repository inside a file whose name is the file name used in the snippet
+as well.
+
+GitLab performs this migration through a [Background Migration](../development/background_migrations.md)
+automatically when the GitLab instance is upgrade to 13.0 or a higher version.
+However, if the migration fails for any of the snippets, they still need
+to be migrated individually.
+
+The following Rake tasks will help with that process.
+
+## Migrate specific snippets to Git
+
+In case you want to migrate a range of snippets, run the tasks as described below.
+
+For Omnibus installations, run:
+
+```shell
+sudo gitlab-rake gitlab:snippets:migrate SNIPPET_IDS=1,2,3,4
+```
+
+For installations from source code, run:
+
+```shell
+bundle exec rake gitlab:snippets:migrate SNIPPET_IDS=1,2,3,4
+```
+
+There is a default limit (100) to the number of ids supported in the migration
+process. You can modify this limit by using the env variable `LIMIT`.
+
+```shell
+sudo gitlab-rake gitlab:snippets:migrate SNIPPET_IDS=1,2,3,4 LIMIT=50
+```
+
+For installations from source code, run:
+
+```shell
+bundle exec rake gitlab:snippets:migrate SNIPPET_IDS=1,2,3,4 LIMIT=50
+```
+
+## Show whether the snippet background migration is running
+
+In case you want to check the status of the snippet background migration,
+whether it is running or not, you can use the following task.
+
+For Omnibus installations, run:
+
+```shell
+sudo gitlab-rake gitlab:snippets:migration_status
+```
+
+For installations from source code, run:
+
+```shell
+bundle exec rake gitlab:snippets:migration_status RAILS_ENV=production
+```
+
+## List non-migrated snippets
+
+With the following task, you can get the ids of all of the snippets
+that haven't been migrated yet or failed to migrate.
+
+For Omnibus installations, run:
+
+```shell
+sudo gitlab-rake gitlab:snippets:list_non_migrated
+```
+
+For installations from source code, run:
+
+```shell
+bundle exec rake gitlab:snippets:list_non_migrated RAILS_ENV=production
+```
+
+As the number of non-migrated snippets can be large, we limit
+by default the size of the number of ids returned to 100. You can
+modify this limit by using the env variable `LIMIT`.
+
+```shell
+sudo gitlab-rake gitlab:snippets:list_non_migrated LIMIT=200
+```
+
+For installations from source code, run:
+
+```shell
+bundle exec rake gitlab:snippets:list_non_migrated RAILS_ENV=production LIMIT=200
+```
diff --git a/doc/raketasks/user_management.md b/doc/raketasks/user_management.md
index ffb1baa0b6e..61e30f9ddb1 100644
--- a/doc/raketasks/user_management.md
+++ b/doc/raketasks/user_management.md
@@ -1,7 +1,11 @@
-# User management
+# User management **(CORE ONLY)**
+
+GitLab provides Rake tasks for user management.
## Add user as a developer to all projects
+To add a user as a developer to all projects, run:
+
```shell
# omnibus-gitlab
sudo gitlab-rake gitlab:import:user_to_projects[username@domain.tld]
@@ -12,9 +16,7 @@ bundle exec rake gitlab:import:user_to_projects[username@domain.tld] RAILS_ENV=p
## Add all users to all projects
-Notes:
-
-- admin users are added as maintainers
+To add all users to all projects, run:
```shell
# omnibus-gitlab
@@ -24,8 +26,13 @@ sudo gitlab-rake gitlab:import:all_users_to_all_projects
bundle exec rake gitlab:import:all_users_to_all_projects RAILS_ENV=production
```
+NOTE: **Note:**
+Admin users are added as maintainers.
+
## Add user as a developer to all groups
+To add a user as a developer to all groups, run:
+
```shell
# omnibus-gitlab
sudo gitlab-rake gitlab:import:user_to_groups[username@domain.tld]
@@ -36,9 +43,7 @@ bundle exec rake gitlab:import:user_to_groups[username@domain.tld] RAILS_ENV=pro
## Add all users to all groups
-Notes:
-
-- admin users are added as owners so they can add additional users to the group
+To add all users to all groups, run:
```shell
# omnibus-gitlab
@@ -48,19 +53,25 @@ sudo gitlab-rake gitlab:import:all_users_to_all_groups
bundle exec rake gitlab:import:all_users_to_all_groups RAILS_ENV=production
```
-## Maintain tight control over the number of active users on your GitLab installation
+NOTE: **Note:**
+Admin users are added as owners so they can add additional users to the group.
+
+## Control the number of active users
-- Enable this setting to keep new users blocked until they have been cleared by the admin (default: false).
+Enable this setting to keep new users blocked until they have been cleared by the administrator.
+Defaults to `false`:
```plaintext
block_auto_created_users: false
```
-## Disable Two-factor Authentication (2FA) for all users
+## Disable two-factor authentication for all users
-This task will disable 2FA for all users that have it enabled. This can be
+This task disables two-factor authentication (2FA) for all users that have it enabled. This can be
useful if GitLab's `config/secrets.yml` file has been lost and users are unable
-to login, for example.
+to log in, for example.
+
+To disable two-factor authentication for all users, run:
```shell
# omnibus-gitlab
@@ -70,65 +81,65 @@ sudo gitlab-rake gitlab:two_factor:disable_for_all_users
bundle exec rake gitlab:two_factor:disable_for_all_users RAILS_ENV=production
```
-## Rotate Two-factor Authentication (2FA) encryption key
+## Rotate two-factor authentication encryption key
-GitLab stores the secret data enabling 2FA to work in an encrypted database
-column. The encryption key for this data is known as `otp_key_base`, and is
+GitLab stores the secret data required for two-factor authentication (2FA) in an encrypted
+database column. The encryption key for this data is known as `otp_key_base`, and is
stored in `config/secrets.yml`.
If that file is leaked, but the individual 2FA secrets have not, it's possible
to re-encrypt those secrets with a new encryption key. This allows you to change
the leaked key without forcing all users to change their 2FA details.
-First, look up the old key. This is in the `config/secrets.yml` file, but
-**make sure you're working with the production section**. The line you're
-interested in will look like this:
+To rotate the two-factor authentication encryption key:
-```yaml
-production:
- otp_key_base: ffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff
-```
+1. Look up the old key. This is in the `config/secrets.yml` file, but **make sure you're working
+ with the production section**. The line you're interested in will look like this:
-Next, generate a new secret:
+ ```yaml
+ production:
+ otp_key_base: ffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff
+ ```
-```shell
-# omnibus-gitlab
-sudo gitlab-rake secret
+1. Generate a new secret:
-# installation from source
-bundle exec rake secret RAILS_ENV=production
-```
+ ```shell
+ # omnibus-gitlab
+ sudo gitlab-rake secret
-Now you need to stop the GitLab server, back up the existing secrets file and
-update the database:
+ # installation from source
+ bundle exec rake secret RAILS_ENV=production
+ ```
-```shell
-# omnibus-gitlab
-sudo gitlab-ctl stop
-sudo cp config/secrets.yml config/secrets.yml.bak
-sudo gitlab-rake gitlab:two_factor:rotate_key:apply filename=backup.csv old_key=<old key> new_key=<new key>
+1. Stop the GitLab server, back up the existing secrets file, and update the database:
-# installation from source
-sudo /etc/init.d/gitlab stop
-cp config/secrets.yml config/secrets.yml.bak
-bundle exec rake gitlab:two_factor:rotate_key:apply filename=backup.csv old_key=<old key> new_key=<new key> RAILS_ENV=production
-```
+ ```shell
+ # omnibus-gitlab
+ sudo gitlab-ctl stop
+ sudo cp config/secrets.yml config/secrets.yml.bak
+ sudo gitlab-rake gitlab:two_factor:rotate_key:apply filename=backup.csv old_key=<old key> new_key=<new key>
-The `<old key>` value can be read from `config/secrets.yml`; `<new key>` was
-generated earlier. The **encrypted** values for the user 2FA secrets will be
-written to the specified `filename` - you can use this to rollback in case of
-error.
+ # installation from source
+ sudo /etc/init.d/gitlab stop
+ cp config/secrets.yml config/secrets.yml.bak
+ bundle exec rake gitlab:two_factor:rotate_key:apply filename=backup.csv old_key=<old key> new_key=<new key> RAILS_ENV=production
+ ```
-Finally, change `config/secrets.yml` to set `otp_key_base` to `<new key>` and
-restart. Again, make sure you're operating in the **production** section.
+ The `<old key>` value can be read from `config/secrets.yml` (`<new key>` was
+ generated earlier). The **encrypted** values for the user 2FA secrets will be
+ written to the specified `filename`. You can use this to rollback in case of
+ error.
-```shell
-# omnibus-gitlab
-sudo gitlab-ctl start
+1. Change `config/secrets.yml` to set `otp_key_base` to `<new key>` and restart. Again, make sure
+ you're operating in the **production** section.
-# installation from source
-sudo /etc/init.d/gitlab start
-```
+ ```shell
+ # omnibus-gitlab
+ sudo gitlab-ctl start
+
+ # installation from source
+ sudo /etc/init.d/gitlab start
+ ```
If there are any problems (perhaps using the wrong value for `old_key`), you can
restore your backup of `config/secrets.yml` and rollback the changes:
diff --git a/doc/raketasks/web_hooks.md b/doc/raketasks/web_hooks.md
index 22084c862ba..7bd2ed311d2 100644
--- a/doc/raketasks/web_hooks.md
+++ b/doc/raketasks/web_hooks.md
@@ -1,60 +1,78 @@
# Webhooks administration **(CORE ONLY)**
-## Add a webhook for **ALL** projects
+GitLab provides Rake tasks for webhooks management.
+
+Requests to the [local network by webhooks](../security/webhooks.md) can be allowed or blocked by an
+administrator.
+
+## Add a webhook to all projects
+
+To add a webhook to all projects, run:
```shell
# omnibus-gitlab
sudo gitlab-rake gitlab:web_hook:add URL="http://example.com/hook"
+
# source installations
bundle exec rake gitlab:web_hook:add URL="http://example.com/hook" RAILS_ENV=production
```
-## Add a webhook for projects in a given **NAMESPACE**
+## Add a webhook to projects in a namespace
+
+To add a webhook to all projects in a specific namespace, run:
```shell
# omnibus-gitlab
-sudo gitlab-rake gitlab:web_hook:add URL="http://example.com/hook" NAMESPACE=acme
+sudo gitlab-rake gitlab:web_hook:add URL="http://example.com/hook" NAMESPACE=<namespace>
+
# source installations
-bundle exec rake gitlab:web_hook:add URL="http://example.com/hook" NAMESPACE=acme RAILS_ENV=production
+bundle exec rake gitlab:web_hook:add URL="http://example.com/hook" NAMESPACE=<namespace> RAILS_ENV=production
```
-## Remove a webhook from **ALL** projects using
+## Remove a webhook from projects
+
+To remove a webhook from all projects, run:
```shell
# omnibus-gitlab
sudo gitlab-rake gitlab:web_hook:rm URL="http://example.com/hook"
+
# source installations
bundle exec rake gitlab:web_hook:rm URL="http://example.com/hook" RAILS_ENV=production
```
-## Remove a webhook from projects in a given **NAMESPACE**
+## Remove a webhook from projects in a namespace
+
+To remove a webhook from projects in a specific namespace, run:
```shell
# omnibus-gitlab
-sudo gitlab-rake gitlab:web_hook:rm URL="http://example.com/hook" NAMESPACE=acme
+sudo gitlab-rake gitlab:web_hook:rm URL="http://example.com/hook" NAMESPACE=<namespace>
+
# source installations
-bundle exec rake gitlab:web_hook:rm URL="http://example.com/hook" NAMESPACE=acme RAILS_ENV=production
+bundle exec rake gitlab:web_hook:rm URL="http://example.com/hook" NAMESPACE=<namespace> RAILS_ENV=production
```
-## List **ALL** webhooks
+## List all webhooks
+
+To list all webhooks, run:
```shell
# omnibus-gitlab
sudo gitlab-rake gitlab:web_hook:list
+
# source installations
bundle exec rake gitlab:web_hook:list RAILS_ENV=production
```
-## List the webhooks from projects in a given **NAMESPACE**
+## List webhooks for projects in a namespace
+
+To list all webhook for projects in a specified namespace, run:
```shell
# omnibus-gitlab
-sudo gitlab-rake gitlab:web_hook:list NAMESPACE=acme
+sudo gitlab-rake gitlab:web_hook:list NAMESPACE=<namespace>
+
# source installations
-bundle exec rake gitlab:web_hook:list NAMESPACE=acme RAILS_ENV=production
+bundle exec rake gitlab:web_hook:list NAMESPACE=<namespace> RAILS_ENV=production
```
-
-## Local requests in webhooks
-
-[Requests to local network by webhooks](../security/webhooks.md) can be allowed
-or blocked by an administrator.
diff --git a/doc/raketasks/x509_signatures.md b/doc/raketasks/x509_signatures.md
index dd518997f7b..f7c47794690 100644
--- a/doc/raketasks/x509_signatures.md
+++ b/doc/raketasks/x509_signatures.md
@@ -1,21 +1,24 @@
-# X509 signatures
+# X.509 signatures **(CORE ONLY)**
-When [signing commits with x509](../user/project/repository/x509_signed_commits/index.md)
-the trust anchor might change and the signatures stored within the database have
-to be updated.
+> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/122159) in GitLab 12.10.
-## Update all x509 signatures
+When [signing commits with X.509](../user/project/repository/x509_signed_commits/index.md),
+the trust anchor might change and the signatures stored within the database must be updated.
-This task loops through all X509 signed commits and updates their verification
-based on current certificate store.
+## Update all X.509 signatures
-**Omnibus Installation**
+This task loops through all X.509 signed commits and updates their verification based on current
+certificate store.
+
+To update all X.509 signatures, run:
+
+**Omnibus Installations:**
```shell
sudo gitlab-rake gitlab:x509:update_signatures
```
-**Source Installation**
+**Source Installations:**
```shell
sudo -u git -H bundle exec rake gitlab:x509:update_signatures RAILS_ENV=production