Merge branch 'gl-version-backup-file' into 'master'

Add GitLab version to backup file name

See merge request !10901

1 files changed, 140 insertions, 114 deletions

diff --git a/doc/raketasks/backup_restore.md b/doc/raketasks/backup_restore.md
index e680a560888..5be6053b76e 100644
--- a/doc/raketasks/backup_restore.md
+++ b/doc/raketasks/backup_restore.md
@@ -1,41 +1,52 @@
-# Backup restore
+# Backing up and restoring GitLab
 
 ![backup banner](backup_hrz.png)
 
 An application data backup creates an archive file that contains the database,
 all repositories and all attachments.
-This archive will be saved in `backup_path`, which is specified in the
-`config/gitlab.yml` file.
-The filename will be `[TIMESTAMP]_gitlab_backup.tar`, where `TIMESTAMP`
-identifies the time at which each backup was created.
-
-> In GitLab 8.15 we changed the timestamp format from `EPOCH` (`1393513186`)
-> to `EPOCH_YYYY_MM_DD` (`1393513186_2014_02_27`)
 
-You can only restore a backup to exactly the same version of GitLab on which it
-was created.  The best way to migrate your repositories from one server to
+You can only restore a backup to **exactly the same version** of GitLab on which
+it was created. The best way to migrate your repositories from one server to
 another is through backup restore.
 
-To restore a backup, you will also need to restore `/etc/gitlab/gitlab-secrets.json`
-(for omnibus packages) or `/home/git/gitlab/.secret` (for installations
-from source). This file contains the database encryption key,
-[CI secret variables](../ci/variables/README.md#secret-variables), and
-secret variables used for [two-factor authentication](../security/two_factor_authentication.md).
-If you fail to restore this encryption key file along with the application data
-backup, users with two-factor authentication enabled and GitLab Runners will
-lose access to your GitLab server.
+## Backup
+
+GitLab provides a simple command line interface to backup your whole installation,
+and is flexible enough to fit your needs.
 
-## Create a backup of the GitLab system
+### Backup timestamp
+
+>**Note:**
+In GitLab 9.2 the timestamp format was changed from `EPOCH_YYYY_MM_DD` to
+`EPOCH_YYYY_MM_DD_GitLab version`, for example `1493107454_2017_04_25`
+would become `1493107454_2017_04_25_9.1.0`.
+
+The backup archive will be saved in `backup_path`, which is specified in the
+`config/gitlab.yml` file.
+The filename will be `[TIMESTAMP]_gitlab_backup.tar`, where `TIMESTAMP`
+identifies the time at which each backup was created, plus the GitLab version.
+The timestamp is needed if you need to restore GitLab and multiple backups are
+available.
+
+For example, if the backup name is `1493107454_2017_04_25_9.1.0_gitlab_backup.tar`,
+then the timestamp is `1493107454_2017_04_25_9.1.0`.
+
+### Creating a backup of the GitLab system
 
 Use this command if you've installed GitLab with the Omnibus package:
+
 ```
 sudo gitlab-rake gitlab:backup:create
 ```
+
 Use this if you've installed GitLab from source:
+
 ```
 sudo -u git -H bundle exec rake gitlab:backup:create RAILS_ENV=production
 ```
+
 If you are running GitLab within a Docker container, you can run the backup from the host:
+
 ```
 docker exec -t <container name> gitlab-rake gitlab:backup:create
 ```
@@ -69,9 +80,9 @@ Deleting tmp directories...[DONE]
 Deleting old backups... [SKIPPING]
 ```
 
-## Backup Strategy Option
+### Backup strategy option
 
-> **Note:** Introduced as an option in 8.17
+> **Note:** Introduced as an option in GitLab 8.17.
 
 The default backup strategy is to essentially stream data from the respective
 data locations to the backup using the Linux command `tar` and `gzip`. This works
@@ -91,7 +102,7 @@ To use the `copy` strategy instead of the default streaming strategy, specify
 `STRATEGY=copy` in the Rake task command. For example,
 `sudo gitlab-rake gitlab:backup:create STRATEGY=copy`.
 
-## Exclude specific directories from the backup
+### Excluding specific directories from the backup
 
 You can choose what should be backed up by adding the environment variable `SKIP`.
 The available options are:
@@ -115,7 +126,7 @@ sudo gitlab-rake gitlab:backup:create SKIP=db,uploads
 sudo -u git -H bundle exec rake gitlab:backup:create SKIP=db,uploads RAILS_ENV=production
 ```
 
-## Upload backups to remote (cloud) storage
+### Uploading backups to a remote (cloud) storage
 
 Starting with GitLab 7.4 you can let the backup script upload the '.tar' file it creates.
 It uses the [Fog library](http://fog.io/) to perform the upload.
@@ -259,7 +270,7 @@ For installations from source:
       remote_directory: 'gitlab_backups'
 ```
 
-## Backup archive permissions
+### Backup archive permissions
 
 The backup archives created by GitLab (`1393513186_2014_02_27_gitlab_backup.tar`)
 will have owner/group git:git and 0600 permissions by default.
@@ -277,11 +288,11 @@ gitlab_rails['backup_archive_permissions'] = 0644 # Makes the backup archives wo
     archive_permissions: 0644 # Makes the backup archives world-readable
 ```
 
-## Storing configuration files
+### Storing configuration files
 
 Please be informed that a backup does not store your configuration
-files.  One reason for this is that your database contains encrypted
-information for two-factor authentication.  Storing encrypted
+files. One reason for this is that your database contains encrypted
+information for two-factor authentication. Storing encrypted
 information along with its key in the same place defeats the purpose
 of using encryption in the first place!
 
@@ -294,11 +305,74 @@ At the very **minimum** you should backup `/etc/gitlab/gitlab.rb` and
 `/home/git/gitlab/config/secrets.yml` (source) to preserve your database
 encryption key.
 
-## Restore a previously created backup
+### Configuring cron to make daily backups
+
+>**Note:**
+The following cron jobs do not [backup your GitLab configuration files](#storing-configuration-files)
+or [SSH host keys](https://superuser.com/questions/532040/copy-ssh-keys-from-one-server-to-another-server/532079#532079).
+
+**For Omnibus installations**
+
+To schedule a cron job that backs up your repositories and GitLab metadata, use the root user:
+
+```
+sudo su -
+crontab -e
+```
+
+There, add the following line to schedule the backup for everyday at 2 AM:
+
+```
+0 2 * * * /opt/gitlab/bin/gitlab-rake gitlab:backup:create CRON=1
+```
+
+You may also want to set a limited lifetime for backups to prevent regular
+backups using all your disk space.  To do this add the following lines to
+`/etc/gitlab/gitlab.rb` and reconfigure:
 
-You can only restore a backup to exactly the same version of GitLab that you created it on, for example 7.2.1.
+```
+# limit backup lifetime to 7 days - 604800 seconds
+gitlab_rails['backup_keep_time'] = 604800
+```
 
-### Prerequisites
+Note that the `backup_keep_time` configuration option only manages local
+files. GitLab does not automatically prune old files stored in a third-party
+object storage (e.g., AWS S3) because the user may not have permission to list
+and delete files. We recommend that you configure the appropriate retention
+policy for your object storage. For example, you can configure [the S3 backup
+policy as described here](http://stackoverflow.com/questions/37553070/gitlab-omnibus-delete-backup-from-amazon-s3).
+
+**For installation from source**
+
+```
+cd /home/git/gitlab
+sudo -u git -H editor config/gitlab.yml # Enable keep_time in the backup section to automatically delete old backups
+sudo -u git crontab -e # Edit the crontab for the git user
+```
+
+Add the following lines at the bottom:
+
+```
+# Create a full backup of the GitLab repositories and SQL database every day at 4am
+0 4 * * * cd /home/git/gitlab && PATH=/usr/local/bin:/usr/bin:/bin bundle exec rake gitlab:backup:create RAILS_ENV=production CRON=1
+```
+
+The `CRON=1` environment setting tells the backup script to suppress all progress output if there are no errors.
+This is recommended to reduce cron spam.
+
+## Restore
+
+GitLab provides a simple command line interface to backup your whole installation,
+and is flexible enough to fit your needs.
+
+The [restore prerequisites section](#restore-prerequisites) includes crucial
+information. Make sure to read and test the whole restore process at least once
+before attempting to perform it in a production environment.
+
+You can only restore a backup to **exactly the same version** of GitLab that
+you created it on, for example 9.1.0.
+
+### Restore prerequisites
 
 You need to have a working GitLab installation before you can perform
 a restore. This is mainly because the system user performing the
@@ -307,13 +381,23 @@ the SQL database it needs to import data into ('gitlabhq_production').
 All existing data will be either erased (SQL) or moved to a separate
 directory (repositories, uploads).
 
-If some or all of your GitLab users are using two-factor authentication (2FA)
-then you must also make sure to restore `/etc/gitlab/gitlab.rb` and
-`/etc/gitlab/gitlab-secrets.json` (Omnibus), or
-`/home/git/gitlab/config/secrets.yml` (installations from source). Note that you
-need to run `gitlab-ctl reconfigure` after changing `gitlab-secrets.json`.
+To restore a backup, you will also need to restore `/etc/gitlab/gitlab-secrets.json`
+(for Omnibus packages) or `/home/git/gitlab/.secret` (for installations
+from source). This file contains the database encryption key,
+[CI secret variables](../ci/variables/README.md#secret-variables), and
+secret variables used for [two-factor authentication](../user/profile/account/two_factor_authentication.md).
+If you fail to restore this encryption key file along with the application data
+backup, users with two-factor authentication enabled and GitLab Runners will
+lose access to your GitLab server.
+
+Depending on your case, you might want to run the restore command with one or
+more of the following options:
+
+- `BACKUP=timestamp_of_backup` - Required if more than one backup exists.
+  Read what the [backup timestamp is about](#backup-timestamp).
+- `force=yes` - Do not ask if the authorized_keys file should get regenerated.
 
-### Installation from source
+### Restore for installation from source
 
 ```
 # Stop processes that are connected to the database
@@ -322,13 +406,6 @@ sudo service gitlab stop
 bundle exec rake gitlab:backup:restore RAILS_ENV=production
 ```
 
-Options:
-
-```
-BACKUP=timestamp_of_backup (required if more than one backup exists)
-force=yes (do not ask if the authorized_keys file should get regenerated)
-```
-
 Example output:
 
 ```
@@ -360,13 +437,13 @@ Restoring repositories:
 Deleting tmp directories...[DONE]
 ```
 
-### Omnibus installations
+### Restore for Omnibus installations
 
 This procedure assumes that:
 
-- You have installed the exact same version of GitLab Omnibus with which the
-  backup was created
-- You have run `sudo gitlab-ctl reconfigure` at least once
+- You have installed the **exact same version** of GitLab Omnibus with which the
+  backup was created.
+- You have run `sudo gitlab-ctl reconfigure` at least once.
 - GitLab is running.  If not, start it using `sudo gitlab-ctl start`.
 
 First make sure your backup tar file is in the backup directory described in the
@@ -374,7 +451,7 @@ First make sure your backup tar file is in the backup directory described in the
 `/var/opt/gitlab/backups`.
 
 ```shell
-sudo cp 1393513186_2014_02_27_gitlab_backup.tar /var/opt/gitlab/backups/
+sudo cp 1493107454_2017_04_25_9.1.0_gitlab_backup.tar /var/opt/gitlab/backups/
 ```
 
 Stop the processes that are connected to the database.  Leave the rest of GitLab
@@ -392,7 +469,7 @@ restore:
 
 ```shell
 # This command will overwrite the contents of your GitLab database!
-sudo gitlab-rake gitlab:backup:restore BACKUP=1393513186_2014_02_27
+sudo gitlab-rake gitlab:backup:restore BACKUP=1493107454_2017_04_25_9.1.0
 ```
 
 Restart and check GitLab:
@@ -404,59 +481,7 @@ sudo gitlab-rake gitlab:check SANITIZE=true
 
 If there is a GitLab version mismatch between your backup tar file and the installed
 version of GitLab, the restore command will abort with an error. Install the
-[correct GitLab version](https://about.gitlab.com/downloads/archives/) and try again.
-
-## Configure cron to make daily backups
-
-### For installation from source:
-```
-cd /home/git/gitlab
-sudo -u git -H editor config/gitlab.yml # Enable keep_time in the backup section to automatically delete old backups
-sudo -u git crontab -e # Edit the crontab for the git user
-```
-
-Add the following lines at the bottom:
-
-```
-# Create a full backup of the GitLab repositories and SQL database every day at 4am
-0 4 * * * cd /home/git/gitlab && PATH=/usr/local/bin:/usr/bin:/bin bundle exec rake gitlab:backup:create RAILS_ENV=production CRON=1
-```
-
-The `CRON=1` environment setting tells the backup script to suppress all progress output if there are no errors.
-This is recommended to reduce cron spam.
-
-### For omnibus installations
-
-To schedule a cron job that backs up your repositories and GitLab metadata, use the root user:
-
-```
-sudo su -
-crontab -e
-```
-
-There, add the following line to schedule the backup for everyday at 2 AM:
-
-```
-0 2 * * * /opt/gitlab/bin/gitlab-rake gitlab:backup:create CRON=1
-```
-
-You may also want to set a limited lifetime for backups to prevent regular
-backups using all your disk space.  To do this add the following lines to
-`/etc/gitlab/gitlab.rb` and reconfigure:
-
-```
-# limit backup lifetime to 7 days - 604800 seconds
-gitlab_rails['backup_keep_time'] = 604800
-```
-
-Note that the `backup_keep_time` configuration option only manages local
-files. GitLab does not automatically prune old files stored in a third-party
-object storage (e.g. AWS S3) because the user may not have permission to list
-and delete files. We recommend that you configure the appropriate retention
-policy for your object storage. For example, you can configure [the S3 backup
-policy here as described here](http://stackoverflow.com/questions/37553070/gitlab-omnibus-delete-backup-from-amazon-s3).
-
-NOTE: This cron job does not [backup your omnibus-gitlab configuration](#backup-and-restore-omnibus-gitlab-configuration) or [SSH host keys](https://superuser.com/questions/532040/copy-ssh-keys-from-one-server-to-another-server/532079#532079).
+[correct GitLab version](https://packages.gitlab.com/gitlab/) and try again.
 
 ## Alternative backup strategies
 
@@ -481,6 +506,19 @@ Example: LVM snapshots + rsync
 If you are running GitLab on a virtualized server you can possibly also create VM snapshots of the entire GitLab server.
 It is not uncommon however for a VM snapshot to require you to power down the server, so this approach is probably of limited practical use.
 
+## Additional notes
+
+This documentation is for GitLab Community and Enterprise Edition. We backup
+GitLab.com and make sure your data is secure, but you can't use these methods
+to export / backup your data yourself from GitLab.com.
+
+Issues are stored in the database. They can't be stored in Git itself.
+
+To migrate your repositories from one server to another with an up-to-date version of
+GitLab, you can use the [import rake task](import.md) to do a mass import of the
+repository. Note that if you do an import rake task, rather than a backup restore, you
+will have all your repositories, but not any other data.
+
 ## Troubleshooting
 
 ### Restoring database backup using omnibus packages outputs warnings
@@ -490,7 +528,6 @@ If you are using backup restore procedures you might encounter the following war
 psql:/var/opt/gitlab/backups/db/database.sql:22: ERROR:  must be owner of extension plpgsql
 psql:/var/opt/gitlab/backups/db/database.sql:2931: WARNING:  no privileges could be revoked for "public" (two occurrences)
 psql:/var/opt/gitlab/backups/db/database.sql:2933: WARNING:  no privileges were granted for "public" (two occurrences)
-
 ```
 
 Be advised that, backup is successfully restored in spite of these warnings.
@@ -499,14 +536,3 @@ The rake task runs this as the `gitlab` user which does not have the superuser a
 Those objects have no influence on the database backup/restore but they give this annoying warning.
 
 For more information see similar questions on postgresql issue tracker[here](http://www.postgresql.org/message-id/201110220712.30886.adrian.klaver@gmail.com) and [here](http://www.postgresql.org/message-id/2039.1177339749@sss.pgh.pa.us) as well as [stack overflow](http://stackoverflow.com/questions/4368789/error-must-be-owner-of-language-plpgsql).
-
-## Note
-This documentation is for GitLab CE.
-We backup GitLab.com and make sure your data is secure, but you can't use these methods to export / backup your data yourself from GitLab.com.
-
-Issues are stored in the database. They can't be stored in Git itself.
-
-To migrate your repositories from one server to another with an up-to-date version of
-GitLab, you can use the [import rake task](import.md) to do a mass import of the
-repository. Note that if you do an import rake task, rather than a backup restore, you
-will have all your repositories, but not any other data.