From 33aa02e7a38d8dfc5e470dd5d776c8d4ce5b2dd5 Mon Sep 17 00:00:00 2001 From: GitLab Bot Date: Wed, 1 Apr 2020 18:07:56 +0000 Subject: Add latest changes from gitlab-org/gitlab@master --- doc/topics/git/index.md | 8 +- doc/topics/git/lfs/img/git-annex-branches.png | Bin 0 -> 32164 bytes doc/topics/git/lfs/img/lfs-icon.png | Bin 0 -> 4317 bytes doc/topics/git/lfs/index.md | 273 +++++++++++++++++++++ .../git/lfs/migrate_from_git_annex_to_git_lfs.md | 253 +++++++++++++++++++ doc/topics/git/lfs/migrate_to_git_lfs.md | 174 +++++++++++++ doc/topics/git/migrate_to_git_lfs/index.md | 173 +------------ 7 files changed, 706 insertions(+), 175 deletions(-) create mode 100644 doc/topics/git/lfs/img/git-annex-branches.png create mode 100644 doc/topics/git/lfs/img/lfs-icon.png create mode 100644 doc/topics/git/lfs/index.md create mode 100644 doc/topics/git/lfs/migrate_from_git_annex_to_git_lfs.md create mode 100644 doc/topics/git/lfs/migrate_to_git_lfs.md (limited to 'doc/topics') diff --git a/doc/topics/git/index.md b/doc/topics/git/index.md index 4b76f5bdc58..9f9d502af60 100644 --- a/doc/topics/git/index.md +++ b/doc/topics/git/index.md @@ -84,9 +84,9 @@ Git-related queries from GitLab. The following relate to Git Large File Storage: - [Getting Started with Git LFS](https://about.gitlab.com/blog/2017/01/30/getting-started-with-git-lfs-tutorial/) -- [Migrate an existing Git repo with Git LFS](migrate_to_git_lfs/index.md) -- [Removing objects from LFS](../../administration/lfs/manage_large_binaries_with_git_lfs.md#removing-objects-from-lfs) -- [GitLab Git LFS user documentation](../../administration/lfs/manage_large_binaries_with_git_lfs.md) +- [Migrate an existing Git repo with Git LFS](lfs/migrate_to_git_lfs.md) +- [Removing objects from LFS](lfs/index.md#removing-objects-from-lfs) +- [GitLab Git LFS user documentation](lfs/index.md) - [GitLab Git LFS admin documentation](../../administration/lfs/lfs_administration.md) -- [git-annex to Git-LFS migration guide](../../administration/lfs/migrate_from_git_annex_to_git_lfs.md) +- [git-annex to Git-LFS migration guide](lfs/migrate_from_git_annex_to_git_lfs.md) - [Towards a production quality open source Git LFS server](https://about.gitlab.com/blog/2015/08/13/towards-a-production-quality-open-source-git-lfs-server/) diff --git a/doc/topics/git/lfs/img/git-annex-branches.png b/doc/topics/git/lfs/img/git-annex-branches.png new file mode 100644 index 00000000000..3d614f68177 Binary files /dev/null and b/doc/topics/git/lfs/img/git-annex-branches.png differ diff --git a/doc/topics/git/lfs/img/lfs-icon.png b/doc/topics/git/lfs/img/lfs-icon.png new file mode 100644 index 00000000000..eef9a14187a Binary files /dev/null and b/doc/topics/git/lfs/img/lfs-icon.png differ diff --git a/doc/topics/git/lfs/index.md b/doc/topics/git/lfs/index.md new file mode 100644 index 00000000000..dcd9706dce5 --- /dev/null +++ b/doc/topics/git/lfs/index.md @@ -0,0 +1,273 @@ +--- +disqus_identifier: 'https://docs.gitlab.com/ee/workflow/lfs/lfs/index.html' +--- + +# Git LFS + +Managing large files such as audio, video and graphics files has always been one +of the shortcomings of Git. The general recommendation is to not have Git repositories +larger than 1GB to preserve performance. + +![Git LFS tracking status](img/lfs-icon.png) + +An LFS icon is shown on files tracked by Git LFS to denote if a file is stored +as a blob or as an LFS pointer. + +## How it works + +Git LFS client talks with the GitLab server over HTTPS. It uses HTTP Basic Authentication +to authorize client requests. Once the request is authorized, Git LFS client receives +instructions from where to fetch or where to push the large file. + +## GitLab server configuration + +Documentation for GitLab instance administrators is under [LFS administration doc](../../../administration/lfs/lfs_administration.md). + +## Requirements + +- Git LFS is supported in GitLab starting with version 8.2 +- Git LFS must be enabled under project settings +- [Git LFS client](https://git-lfs.github.com) version 1.0.1 and up + +## Known limitations + +- Git LFS v1 original API is not supported since it was deprecated early in LFS + development +- When SSH is set as a remote, Git LFS objects still go through HTTPS +- Any Git LFS request will ask for HTTPS credentials to be provided so a good Git + credentials store is recommended +- Git LFS always assumes HTTPS so if you have GitLab server on HTTP you will have + to add the URL to Git config manually (see [troubleshooting](#troubleshooting)) + +NOTE: **Note:** +With 8.12 GitLab added LFS support to SSH. The Git LFS communication +still goes over HTTP, but now the SSH client passes the correct credentials +to the Git LFS client, so no action is required by the user. + +## Using Git LFS + +Lets take a look at the workflow when you need to check large files into your Git +repository with Git LFS. For example, if you want to upload a very large file and +check it into your Git repository: + +```shell +git clone git@gitlab.example.com:group/project.git +git lfs install # initialize the Git LFS project +git lfs track "*.iso" # select the file extensions that you want to treat as large files +``` + +Once a certain file extension is marked for tracking as a LFS object you can use +Git as usual without having to redo the command to track a file with the same extension: + +```shell +cp ~/tmp/debian.iso ./ # copy a large file into the current directory +git add . # add the large file to the project +git commit -am "Added Debian iso" # commit the file meta data +git push origin master # sync the git repo and large file to the GitLab server +``` + +**Make sure** that `.gitattributes` is tracked by Git. Otherwise Git +LFS will not be working properly for people cloning the project: + +```shell +git add .gitattributes +``` + +Cloning the repository works the same as before. Git automatically detects the +LFS-tracked files and clones them via HTTP. If you performed the `git clone` +command with a SSH URL, you have to enter your GitLab credentials for HTTP +authentication. + +```shell +git clone git@gitlab.example.com:group/project.git +``` + +If you already cloned the repository and you want to get the latest LFS object +that are on the remote repository, eg. for a branch from origin: + +```shell +git lfs fetch origin master +``` + +### Migrate an existing repo to Git LFS + +Read the documentation on how to [migrate an existing Git repo with Git LFS](migrate_to_git_lfs.md). + +### Removing objects from LFS + +To remove objects from LFS: + +1. Use [BFG-Cleaner](../../../user/project/repository/reducing_the_repo_size_using_git.md#using-the-bfg-repo-cleaner) or [filter-branch](../../../user/project/repository/reducing_the_repo_size_using_git.md#using-git-filter-branch) to remove the objects from the repository. +1. Delete the relevant LFS lines for the objects you have removed from your `.gitattributes` file and commit those changes. + +## File Locking + +> [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/issues/35856) in GitLab 10.5. + +The first thing to do before using File Locking is to tell Git LFS which +kind of files are lockable. The following command will store PNG files +in LFS and flag them as lockable: + +```shell +git lfs track "*.png" --lockable +``` + +After executing the above command a file named `.gitattributes` will be +created or updated with the following content: + +```shell +*.png filter=lfs diff=lfs merge=lfs -text lockable +``` + +You can also register a file type as lockable without using LFS +(In order to be able to lock/unlock a file you need a remote server that implements the LFS File Locking API), +in order to do that you can edit the `.gitattributes` file manually: + +```shell +*.pdf lockable +``` + +After a file type has been registered as lockable, Git LFS will make +them readonly on the file system automatically. This means you will +need to lock the file before editing it. + +### Managing Locked Files + +Once you're ready to edit your file you need to lock it first: + +```shell +git lfs lock images/banner.png +Locked images/banner.png +``` + +This will register the file as locked in your name on the server: + +```shell +git lfs locks +images/banner.png joe ID:123 +``` + +Once you have pushed your changes, you can unlock the file so others can +also edit it: + +```shell +git lfs unlock images/banner.png +``` + +You can also unlock by id: + +```shell +git lfs unlock --id=123 +``` + +If for some reason you need to unlock a file that was not locked by you, +you can use the `--force` flag as long as you have a `maintainer` access on +the project: + +```shell +git lfs unlock --id=123 --force +``` + +## Troubleshooting + +### error: Repository or object not found + +There are a couple of reasons why this error can occur: + +- You don't have permissions to access certain LFS object + +Check if you have permissions to push to the project or fetch from the project. + +- Project is not allowed to access the LFS object + +LFS object you are trying to push to the project or fetch from the project is not +available to the project anymore. Probably the object was removed from the server. + +- Local Git repository is using deprecated LFS API + +### Invalid status for `` : 501 + +Git LFS will log the failures into a log file. +To view this log file, while in project directory: + +```shell +git lfs logs last +``` + +If the status `error 501` is shown, it is because: + +- Git LFS is not enabled in project settings. Check your project settings and + enable Git LFS. + +- Git LFS support is not enabled on the GitLab server. Check with your GitLab + administrator why Git LFS is not enabled on the server. See + [LFS administration documentation](../../../administration/lfs/lfs_administration.md) for instructions + on how to enable LFS support. + +- Git LFS client version is not supported by GitLab server. Check your Git LFS + version with `git lfs version`. Check the Git config of the project for traces + of deprecated API with `git lfs -l`. If `batch = false` is set in the config, + remove the line and try to update your Git LFS client. Only version 1.0.1 and + newer are supported. + +### getsockopt: connection refused + +If you push a LFS object to a project and you receive an error similar to: +`Post /info/lfs/objects/batch: dial tcp IP: getsockopt: connection refused`, +the LFS client is trying to reach GitLab through HTTPS. However, your GitLab +instance is being served on HTTP. + +This behaviour is caused by Git LFS using HTTPS connections by default when a +`lfsurl` is not set in the Git config. + +To prevent this from happening, set the lfs url in project Git config: + +```shell +git config --add lfs.url "http://gitlab.example.com/group/project.git/info/lfs" +``` + +### Credentials are always required when pushing an object + +NOTE: **Note:** +With 8.12 GitLab added LFS support to SSH. The Git LFS communication +still goes over HTTP, but now the SSH client passes the correct credentials +to the Git LFS client, so no action is required by the user. + +Given that Git LFS uses HTTP Basic Authentication to authenticate the user pushing +the LFS object on every push for every object, user HTTPS credentials are required. + +By default, Git has support for remembering the credentials for each repository +you use. This is described in [Git credentials man pages](https://git-scm.com/docs/gitcredentials). + +For example, you can tell Git to remember the password for a period of time in +which you expect to push the objects: + +```shell +git config --global credential.helper 'cache --timeout=3600' +``` + +This will remember the credentials for an hour after which Git operations will +require re-authentication. + +If you are using OS X you can use `osxkeychain` to store and encrypt your credentials. +For Windows, you can use `wincred` or Microsoft's [Git Credential Manager for Windows](https://github.com/Microsoft/Git-Credential-Manager-for-Windows/releases). + +More details about various methods of storing the user credentials can be found +on [Git Credential Storage documentation](https://git-scm.com/book/en/v2/Git-Tools-Credential-Storage). + +### LFS objects are missing on push + +GitLab checks files to detect LFS pointers on push. If LFS pointers are detected, GitLab tries to verify that those files already exist in LFS on GitLab. + +Verify that LFS is installed locally and consider a manual push with `git lfs push --all`. + +If you are storing LFS files outside of GitLab you can disable LFS on the project by setting `lfs_enabled: false` with the [projects API](../../../api/projects.md#edit-project). + +### Hosting LFS objects externally + +It is possible to host LFS objects externally by setting a custom LFS url with `git config -f .lfsconfig lfs.url https://example.com/.git/info/lfs`. + +You might choose to do this if you are using an appliance like a Sonatype Nexus to store LFS data. If you choose to use an external LFS store, +GitLab will not be able to verify LFS objects which means that pushes will fail if you have GitLab LFS support enabled. + +To stop push failure, LFS support can be disabled in the [Project settings](../../../user/project/settings/index.md). This means you will lose GitLab LFS value-adds (Verifying LFS objects, UI integration for LFS). diff --git a/doc/topics/git/lfs/migrate_from_git_annex_to_git_lfs.md b/doc/topics/git/lfs/migrate_from_git_annex_to_git_lfs.md new file mode 100644 index 00000000000..d23199fa556 --- /dev/null +++ b/doc/topics/git/lfs/migrate_from_git_annex_to_git_lfs.md @@ -0,0 +1,253 @@ +# Migration guide from Git Annex to Git LFS + +>**Note:** +Git Annex support [has been removed][issue-remove-annex] in GitLab Enterprise +Edition 9.0 (2017/03/22). + +Both [Git Annex][] and [Git LFS][] are tools to manage large files in Git. + +## History + +Git Annex [was introduced in GitLab Enterprise Edition 7.8][post-3], at a time +where Git LFS didn't yet exist. A few months later, GitLab brought support for +Git LFS in [GitLab 8.2][post-2] and is available for both Community and +Enterprise editions. + +## Differences between Git Annex and Git LFS + +Some items below are general differences between the two protocols and some are +ones that GitLab developed. + +- Git Annex works only through SSH, whereas Git LFS works both with SSH and HTTPS + (SSH support was added in GitLab 8.12). +- Annex files are stored in a sub-directory of the normal repositories, whereas + LFS files are stored outside of the repositories in a place you can define. +- Git Annex requires a more complex setup, but has much more options than Git + LFS. You can compare the commands each one offers by running `man git-annex` + and `man git-lfs`. +- Annex files cannot be browsed directly in GitLab's interface, whereas LFS + files can. + +## Migration steps + +>**Note:** +Since Git Annex files are stored in a sub-directory of the normal repositories +(`.git/annex/objects`) and LFS files are stored outside of the repositories, +they are not compatible as they are using a different scheme. Therefore, the +migration has to be done manually per repository. + +There are basically two steps you need to take in order to migrate from Git +Annex to Git LFS. + +### TL; DR + +If you know what you are doing and want to skip the reading, this is what you +need to do (we assume you have [git-annex enabled](../../../administration/git_annex.md#using-gitlab-git-annex) in your +repository and that you have made backups in case something goes wrong). +Fire up a terminal, navigate to your Git repository and: + +1. Disable `git-annex`: + + ```shell + git annex sync --content + git annex direct + git annex uninit + git annex indirect + ``` + +1. Enable `git-lfs`: + + ```shell + git lfs install + git lfs track + git add . + git commit -m "commit message" + git push + ``` + +### Disabling Git Annex in your repo + +Before changing anything, make sure you have a backup of your repository first. +There are a couple of ways to do that, but you can simply clone it to another +local path and maybe push it to GitLab if you want a remote backup as well. +Here you'll find a guide on +[how to back up a **git-annex** repository to an external hard drive][bkp-ext-drive]. + +Since Annex files are stored as objects with symlinks and cannot be directly +modified, we need to first remove those symlinks. + +NOTE: **Note:** +Make sure the you read about the [`direct` mode][annex-direct] as it contains +useful information that may fit in your use case. Note that `annex direct` is +deprecated in Git Annex version 6, so you may need to upgrade your repository +if the server also has Git Annex 6 installed. Read more in the +[Git Annex troubleshooting tips](../../../administration/git_annex.md#troubleshooting-tips) section. + +1. Backup your repository + + ```shell + cd repository + git annex sync --content + cd .. + git clone repository repository-backup + cd repository-backup + git annex get + cd .. + ``` + +1. Use `annex direct`: + + ```shell + cd repository + git annex direct + ``` + + The output should be similar to this: + + ```shell + commit + On branch master + Your branch is up-to-date with 'origin/master'. + nothing to commit, working tree clean + ok + direct debian.iso ok + direct ok + ``` + +1. Disable Git Annex with [`annex uninit`][uninit]: + + ```shell + git annex uninit + ``` + + The output should be similar to this: + + ```shell + unannex debian.iso ok + Deleted branch git-annex (was 2534d2c). + ``` + + This will `unannex` every file in the repository, leaving the original files. + +1. Switch back to `indirect` mode: + + ```shell + git annex indirect + ``` + + The output should be similar to this: + + ```shell + (merging origin/git-annex into git-annex...) + (recording state in git...) + commit (recording state in git...) + + ok + (recording state in git...) + [master fac3194] commit before switching to indirect mode + 1 file changed, 1 deletion(-) + delete mode 120000 alpine-virt-3.4.4-x86_64.iso + ok + indirect ok + ok + ``` + +--- + +At this point, you have two options. Either add, commit and push the files +directly back to GitLab or switch to Git LFS. We will tackle the LFS switch in +the next section. + +### Enabling Git LFS in your repo + +Git LFS is enabled by default on all GitLab products (GitLab CE, GitLab EE, +GitLab.com), therefore, you don't need to do anything server-side. + +1. First, make sure you have `git-lfs` installed locally: + + ```shell + git lfs help + ``` + + If the terminal doesn't prompt you with a full response on `git-lfs` commands, + [install the Git LFS client][install-lfs] first. + +1. Inside the repo, run the following command to initiate LFS: + + ```shell + git lfs install + ``` + +1. Enable `git-lfs` for the group of files you want to track. You + can track specific files, all files containing the same extension, or an + entire directory: + + ```shell + git lfs track images/01.png # per file + git lfs track **/*.png # per extension + git lfs track images/ # per directory + ``` + + Once you do that, run `git status` and you'll see `.gitattributes` added + to your repo. It collects all file patterns that you chose to track via + `git-lfs`. + +1. Add the files, commit and push them to GitLab: + + ```shell + git add . + git commit -m "commit message" + git push + ``` + + If your remote is set up with HTTP, you will be asked to enter your login + credentials. If you have [2FA enabled](../../../user/profile/account/two_factor_authentication.md), make sure to use a + [personal access token](../../../user/profile/account/two_factor_authentication.md#personal-access-tokens) + instead of your password. + +## Removing the Git Annex branches + +After the migration finishes successfully, you can remove all `git-annex` +related branches from your repository. + +On GitLab, navigate to your project's **Repository ➔ Branches** and delete all +branches created by Git Annex: `git-annex`, and all under `synced/`. + +![repository branches](img/git-annex-branches.png) + +You can also do this on the command line with: + +```shell +git branch -d synced/master +git branch -d synced/git-annex +git push origin :synced/master +git push origin :synced/git-annex +git push origin :git-annex +git remote prune origin +``` + +If there are still some Annex objects inside your repository (`.git/annex/`) +or references inside `.git/config`, run `annex uninit` again: + +```shell +git annex uninit +``` + +## Further Reading + +- (Blog Post) [Getting Started with Git FLS][post-1] +- (Blog Post) [Announcing LFS Support in GitLab][post-2] +- (Blog Post) [GitLab Annex Solves the Problem of Versioning Large Binaries with Git][post-3] +- (GitLab Docs) [Git Annex](../../../administration/git_annex.md) +- (GitLab Docs) [Git LFS](index.md) + +[annex-direct]: https://git-annex.branchable.com/direct_mode/ +[bkp-ext-drive]: https://www.thomas-krenn.com/en/wiki/Git-annex_Repository_on_an_External_Hard_Drive +[Git Annex]: http://git-annex.branchable.com/ +[Git LFS]: https://git-lfs.github.com/ +[install-lfs]: https://git-lfs.github.com/ +[issue-remove-annex]: https://gitlab.com/gitlab-org/gitlab/issues/1648 +[post-1]: https://about.gitlab.com/blog/2017/01/30/getting-started-with-git-lfs-tutorial/ +[post-2]: https://about.gitlab.com/blog/2015/11/23/announcing-git-lfs-support-in-gitlab/ +[post-3]: https://about.gitlab.com/blog/2015/02/17/gitlab-annex-solves-the-problem-of-versioning-large-binaries-with-git/ +[uninit]: https://git-annex.branchable.com/git-annex-uninit/ diff --git a/doc/topics/git/lfs/migrate_to_git_lfs.md b/doc/topics/git/lfs/migrate_to_git_lfs.md new file mode 100644 index 00000000000..27ecf65b6cd --- /dev/null +++ b/doc/topics/git/lfs/migrate_to_git_lfs.md @@ -0,0 +1,174 @@ +--- +type: tutorial, concepts +description: "How to migrate an existing Git repository to Git LFS with BFG." +last_updated: 2019-07-11 +--- + +# Migrate a Git repo into Git LFS with BFG + +Using Git LFS can help you to reduce the size of your Git +repository and improve its performance. + +However, simply adding the +large files that are already in your repository to Git LFS, +will not actually reduce the size of your repository because +the files are still referenced by previous commits. + +Through the method described on this document, first migrate +to Git LFS with a tool such as the open source community-maintained [BFG](https://rtyley.github.io/bfg-repo-cleaner/) +through a mirror repo, then clean up the repository's history, +and lastly create LFS tracking rules to prevent new binary files +from being added. + +This tutorial was inspired by the guide +[Use BFG to migrate a repo to Git LFS](https://confluence.atlassian.com/bitbucket/use-bfg-to-migrate-a-repo-to-git-lfs-834233484.html). +For more information on Git LFS, see the [references](#references) +below. + +CAUTION: **Warning:** +The method described on this guide rewrites Git history. Make +sure to back up your repo before beginning and use it at your +own risk. + +## Requirements + +Before beginning, make sure: + +- You have enough LFS storage for the files you want to convert. + Storage is required for the entire history of all files. +- All the team members you share the repository with have pushed all changes. + Branches based on the repository before applying this method cannot be merged. + Branches based on the repo before applying this method cannot be merged. + +To follow this tutorial, you'll need: + +- Maintainer permissions to the existing Git repository + you'd like to migrate to LFS with access through the command line. +- [Git](https://git-scm.com/book/en/v2/Getting-Started-Installing-Git) + and [Java Runtime Environment](https://www.java.com/en/download/manual.jsp) + (Java 7 or above) installed locally. +- BFG installed locally: + + ```shell + brew install bfg + ``` + +- Git LFS installed locally: + + ```shell + brew install git-lfs + ``` + +NOTE: **Note:** +This guide was tested on macOS Mojave. + +## Steps + +Consider an example upstream project, `git@gitlab.com:gitlab-tests/test-git-lfs-repo-migration.git`. + +1. Back up your repository: + + Create a copy of your repository so that you can + recover it in case something goes wrong. + +1. Clone `--mirror` the repo: + + Cloning with the mirror flag will create a bare repository. + This ensures you get all the branches within the repo. + + It creates a directory called `.git` + (in our example, `test-git-lfs-repo-migration.git`), + mirroring the upstream project: + + ```shell + git clone --mirror git@gitlab.com:gitlab-tests/test-git-lfs-repo-migration.git + ``` + +1. Convert the Git history with BFG: + + ```shell + bfg --convert-to-git-lfs "*.{png,mp4,jpg,gif}" --no-blob-protection test-git-lfs-repo-migration.git + ``` + + It is scanning all the history, and looking for any files with + that extension, and then converting them to an LFS pointer. + +1. Clean up the repository: + + ```shell + # cd path/to/mirror/repo: + cd test-git-lfs-repo-migration.git + # clean up the repo: + git reflog expire --expire=now --all && git gc --prune=now --aggressive + ``` + + You can also take a look on how to further [clean the repo](../../../user/project/repository/reducing_the_repo_size_using_git.md), + but it's not necessary for the purposes of this guide. + +1. Install Git LFS in the mirror repository: + + ```shell + git lfs install + ``` + +1. [Unprotect the default branch](../../../user/project/protected_branches.md), + so that we can force-push the rewritten repository: + + 1. Navigate to your project's **Settings > Repository** and + expand **Protected Branches**. + 1. Scroll down to locate the protected branches and click + **Unprotect** the default branch. + +1. Force-push to GitLab: + + ```shell + git push --force + ``` + +1. Track the files you want with LFS: + + ```shell + # cd path/to/upstream/repo: + cd test-git-lfs-repo-migration + # You may need to reset your local copy with upstream's `master` after force-pushing from the mirror: + git reset --hard origin/master + # Track the files with LFS: + git lfs track "*.gif" "*.png" "*.jpg" "*.psd" "*.mp4" "img/" + ``` + + Now all existing the files you converted, as well as the new + ones you add, will be properly tracked with LFS. + +1. [Re-protect the default branch](../../../user/project/protected_branches.md): + + 1. Navigate to your project's **Settings > Repository** and + expand **Protected Branches**. + 1. Select the default branch from the **Branch** dropdown menu, + and set up the + **Allowed to push** and **Allowed to merge** rules. + 1. Click **Protect**. + + + +## References + +- [Getting Started with Git LFS](https://about.gitlab.com/blog/2017/01/30/getting-started-with-git-lfs-tutorial/) +- [Migrate from Git Annex to Git LFS](migrate_from_git_annex_to_git_lfs.md) +- [GitLab's Git LFS user documentation](index.md) +- [GitLab's Git LFS administrator documentation](../../../administration/lfs/lfs_administration.md) +- Alternative method to [migrate an existing repo to Git LFS](https://github.com/git-lfs/git-lfs/wiki/Tutorial#migrating-existing-repository-data-to-lfs) + + diff --git a/doc/topics/git/migrate_to_git_lfs/index.md b/doc/topics/git/migrate_to_git_lfs/index.md index 9c73b4dfde7..ff60603ae58 100644 --- a/doc/topics/git/migrate_to_git_lfs/index.md +++ b/doc/topics/git/migrate_to_git_lfs/index.md @@ -1,174 +1,5 @@ --- -type: tutorial, concepts -description: "How to migrate an existing Git repository to Git LFS with BFG." -last_updated: 2019-07-11 +redirect_to: '../lfs/migrate_to_git_lfs.md' --- -# Migrate a Git repo into Git LFS with BFG - -Using Git LFS can help you to reduce the size of your Git -repository and improve its performance. - -However, simply adding the -large files that are already in your repository to Git LFS, -will not actually reduce the size of your repository because -the files are still referenced by previous commits. - -Through the method described on this document, first migrate -to Git LFS with a tool such as the open source community-maintained [BFG](https://rtyley.github.io/bfg-repo-cleaner/) -through a mirror repo, then clean up the repository's history, -and lastly create LFS tracking rules to prevent new binary files -from being added. - -This tutorial was inspired by the guide -[Use BFG to migrate a repo to Git LFS](https://confluence.atlassian.com/bitbucket/use-bfg-to-migrate-a-repo-to-git-lfs-834233484.html). -For more information on Git LFS, see the [references](#references) -below. - -CAUTION: **Warning:** -The method described on this guide rewrites Git history. Make -sure to back up your repo before beginning and use it at your -own risk. - -## Requirements - -Before beginning, make sure: - -- You have enough LFS storage for the files you want to convert. - Storage is required for the entire history of all files. -- All the team members you share the repository with have pushed all changes. - Branches based on the repository before applying this method cannot be merged. - Branches based on the repo before applying this method cannot be merged. - -To follow this tutorial, you'll need: - -- Maintainer permissions to the existing Git repository - you'd like to migrate to LFS with access through the command line. -- [Git](https://git-scm.com/book/en/v2/Getting-Started-Installing-Git) - and [Java Runtime Environment](https://www.java.com/en/download/manual.jsp) - (Java 7 or above) installed locally. -- BFG installed locally: - - ```shell - brew install bfg - ``` - -- Git LFS installed locally: - - ```shell - brew install git-lfs - ``` - -NOTE: **Note:** -This guide was tested on macOS Mojave. - -## Steps - -Consider an example upstream project, `git@gitlab.com:gitlab-tests/test-git-lfs-repo-migration.git`. - -1. Back up your repository: - - Create a copy of your repository so that you can - recover it in case something goes wrong. - -1. Clone `--mirror` the repo: - - Cloning with the mirror flag will create a bare repository. - This ensures you get all the branches within the repo. - - It creates a directory called `.git` - (in our example, `test-git-lfs-repo-migration.git`), - mirroring the upstream project: - - ```shell - git clone --mirror git@gitlab.com:gitlab-tests/test-git-lfs-repo-migration.git - ``` - -1. Convert the Git history with BFG: - - ```shell - bfg --convert-to-git-lfs "*.{png,mp4,jpg,gif}" --no-blob-protection test-git-lfs-repo-migration.git - ``` - - It is scanning all the history, and looking for any files with - that extension, and then converting them to an LFS pointer. - -1. Clean up the repository: - - ```shell - # cd path/to/mirror/repo: - cd test-git-lfs-repo-migration.git - # clean up the repo: - git reflog expire --expire=now --all && git gc --prune=now --aggressive - ``` - - You can also take a look on how to further [clean the repo](../../../user/project/repository/reducing_the_repo_size_using_git.md), - but it's not necessary for the purposes of this guide. - -1. Install Git LFS in the mirror repository: - - ```shell - git lfs install - ``` - -1. [Unprotect the default branch](../../../user/project/protected_branches.md), - so that we can force-push the rewritten repository: - - 1. Navigate to your project's **Settings > Repository** and - expand **Protected Branches**. - 1. Scroll down to locate the protected branches and click - **Unprotect** the default branch. - -1. Force-push to GitLab: - - ```shell - git push --force - ``` - -1. Track the files you want with LFS: - - ```shell - # cd path/to/upstream/repo: - cd test-git-lfs-repo-migration - # You may need to reset your local copy with upstream's `master` after force-pushing from the mirror: - git reset --hard origin/master - # Track the files with LFS: - git lfs track "*.gif" "*.png" "*.jpg" "*.psd" "*.mp4" "img/" - ``` - - Now all existing the files you converted, as well as the new - ones you add, will be properly tracked with LFS. - -1. [Re-protect the default branch](../../../user/project/protected_branches.md): - - 1. Navigate to your project's **Settings > Repository** and - expand **Protected Branches**. - 1. Select the default branch from the **Branch** dropdown menu, - and set up the - **Allowed to push** and **Allowed to merge** rules. - 1. Click **Protect**. - - - -## References - -- [Getting Started with Git LFS](https://about.gitlab.com/blog/2017/01/30/getting-started-with-git-lfs-tutorial/) -- [Migrate from Git Annex to Git LFS](../../../administration/lfs/migrate_from_git_annex_to_git_lfs.md) -- [GitLab's Git LFS user documentation](../../../administration/lfs/manage_large_binaries_with_git_lfs.md) -- [GitLab's Git LFS administrator documentation](../../../administration/lfs/lfs_administration.md) -- Alternative method to [migrate an existing repo to Git LFS](https://github.com/git-lfs/git-lfs/wiki/Tutorial#migrating-existing-repository-data-to-lfs) - - +This document was moved to [another location](../lfs/migrate_to_git_lfs.md). -- cgit v1.2.1