From 899807f604f7923cc40a64cdd0f61c4248b8870d Mon Sep 17 00:00:00 2001 From: Marin Jankovski Date: Fri, 20 Nov 2015 16:10:08 +0100 Subject: Update required version of lfs client and separate the docs for users and admins. --- doc/README.md | 1 + doc/workflow/README.md | 1 + doc/workflow/git_lfs.md | 135 --------------------- doc/workflow/lfs/lfs_administration.md | 41 +++++++ .../lfs/manage_large_binaries_with_git_lfs.md | 125 +++++++++++++++++++ lib/gitlab/lfs/response.rb | 2 +- spec/lib/gitlab/lfs/lfs_router_spec.rb | 2 +- 7 files changed, 170 insertions(+), 137 deletions(-) delete mode 100644 doc/workflow/git_lfs.md create mode 100644 doc/workflow/lfs/lfs_administration.md create mode 100644 doc/workflow/lfs/manage_large_binaries_with_git_lfs.md diff --git a/doc/README.md b/doc/README.md index 0f6866475f7..58ab5dd08e0 100644 --- a/doc/README.md +++ b/doc/README.md @@ -50,6 +50,7 @@ - [Welcome message](customization/welcome_message.md) Add a custom welcome message to the sign-in page. - [Reply by email](incoming_email/README.md) Allow users to comment on issues and merge requests by replying to notification emails. - [Migrate GitLab CI to CE/EE](migrate_ci_to_ce/README.md) Follow this guide to migrate your existing GitLab CI data to GitLab CE/EE. +- [Git LFS configuration](workflow/lfs/lfs_administration.md) ## Contributor documentation diff --git a/doc/workflow/README.md b/doc/workflow/README.md index c1a4f64981f..a6b4d951188 100644 --- a/doc/workflow/README.md +++ b/doc/workflow/README.md @@ -17,3 +17,4 @@ - [Milestones](milestones.md) - [Merge Requests](merge_requests.md) - ["Work In Progress" Merge Requests](wip_merge_requests.md) +- [Manage large binaries with Git LFS](lfs/manage_large_binaries_with_git_lfs.md) diff --git a/doc/workflow/git_lfs.md b/doc/workflow/git_lfs.md deleted file mode 100644 index 616a71522ae..00000000000 --- a/doc/workflow/git_lfs.md +++ /dev/null @@ -1,135 +0,0 @@ -# Git LFS - -Managing large files such as audio, video and graphics files has always been one of the shortcomings of Git. -The general recommendation is to not have Git repositories larger than 1GB to preserve performance. - -GitLab already supports [managing large files with git annex](http://doc.gitlab.com/ee/workflow/git_annex.html) (EE only), however in certain -environments it is not always convenient to use different commands to differentiate between the large files and regular ones. - -Git LFS makes this simpler for the end user by removing the requirement to learn new commands. - - -## How it works - -Git LFS client talks with the GitLab server over HTTPS. It uses HTTP Basic Authentication to authorize client requests. -Once the request is authorized, Git LFS client receives instructions from where to fetch or where to push the large file. - -## Requirements - -* Git LFS is supported in GitLab starting with version 8.2 -* Git LFS [client](https://git-lfs.github.com) version 0.6.0 and up - -## GitLab and Git LFS - -### Configuration - -Git LFS objects can be large in size. By default, they are stored on the server GitLab is installed on. - -There are two configuration options to help GitLab server administrators: - -* Enabling/disabling Git LFS support -* Changing the location of LFS object storage - -#### Omnibus packages - -In `/etc/gitlab/gitlab.rb`: - -```ruby -gitlab_rails['lfs_enabled'] = false -gitlab_rails['lfs_storage_path'] = "/mnt/storage/lfs-objects" -``` - -#### Installations from source - -In `config/gitlab.yml`: - -```yaml - lfs: - enabled: false - storage_path: /mnt/storage/lfs-objects -``` - -## Known limitations - -* Git LFS v1 original API is not supported since it was deprecated early in LFS development, starting with Git LFS version 0.6.0 -* When SSH is set as a remote, Git LFS objects still go through HTTPS -* Any Git LFS request will ask for HTTPS credentials to be provided so good Git credentials store is recommended -* Currently, storing GitLab Git LFS objects on a non-local storage (like S3 buckets) is not supported -* Git LFS always assumes HTTPS so if you have GitLab server on HTTP you will have to add the URL to Git config manually (see #troubleshooting-tips) - -## Using Git LFS - -Lets take a look at the workflow when you need to check large files into your Git repository with Git LFS: -For example, if you want to upload a very large file and check it into your Git repository: - -```bash -git clone git@gitlab.example.com:group/project.git -git lfs init # initialize the Git LFS project project -git lfs track "*.iso" # select the file extensions that you want to treat as large files -``` - -Once a certain file extension is marked for tracking as a LFS object you can use Git as usual without having to redo the command to track a file with the same extension: - -```bash -cp ~/tmp/debian.iso ./ # copy a large file into the current directory -git add . # add the large file to the project -git commit -am "Added Debian iso" # commit the file meta data -git push origin master # sync the git repo and large file to the GitLab server -``` - -Cloning the repository works the same as before. Git automatically detects the LFS-tracked files and clones them via HTTP. If you performed the git clone command with a SSH URL, you have to enter your GitLab credentials for HTTP authentication. - -```bash -git clone git@gitlab.example.com:group/project.git -``` - - -## Troubleshooting - -### error: Repository or object not found - -There are a couple of reasons why this error can occur: - -* Wrong version of LFS client used: - -Check the version of Git LFS on the client machine with `git lfs version`. Only version 0.6.0 and newer are supported. - -* Project is using deprecated LFS API - -Check the Git config of the project for traces of deprecated API with `git lfs -l`. If `batch = false` is set in the config, remove the line and try using Git LFS client newer than 0.6.0. - -### Invalid status for : 501 - -When attempting to push a LFS object to a GitLab server that doesn't have Git LFS support enabled, server will return status `error 501`. Check with your GitLab administrator why Git LFS is not enabled on the server. See [Configuration section](#configuration) for instructions on how to enable LFS support. - -### getsockopt: connection refused - -If you push a LFS object to a project and you receive an error similar to: `Post /info/lfs/objects/batch: dial tcp IP: getsockopt: connection refused`, -the LFS client is trying to reach GitLab through HTTPS. However, your GitLab instance is being served on HTTP. - -This behaviour is caused by Git LFS using HTTPS connections by default when a `lfsurl` is not set in the Git config. - -To prevent this from happening, set the lfs url in project Git config: - -```bash - -git config --add lfs.url "http://gitlab.example.com/group/project.git/info/lfs/objects/batch" -``` - -### Credentials are always required when pushing an object - -Given that Git LFS uses HTTP Basic Authentication to authenticate the user pushing the LFS object on every push for every object, user HTTPS credentials are required. - -By default, Git has support for remembering the credentials for each repository you use. This is described in [Git credentials man pages](https://git-scm.com/docs/gitcredentials). - -For example, you can tell Git to remember the password for a period of time in which you expect to push the objects: - -```bash -git config --global credential.helper 'cache --timeout=3600' -``` - -This will remember the credentials for an hour after which Git operations will require re-authentication. - -If you are using OS X you can use `osxkeychain` to store and encrypt your credentials. For Windows, `wincred` is available. - -More details about various methods of storing the user credentials can be found on [Git Credential Storage documentation](https://git-scm.com/book/en/v2/Git-Tools-Credential-Storage) \ No newline at end of file diff --git a/doc/workflow/lfs/lfs_administration.md b/doc/workflow/lfs/lfs_administration.md new file mode 100644 index 00000000000..9fa307a9d5b --- /dev/null +++ b/doc/workflow/lfs/lfs_administration.md @@ -0,0 +1,41 @@ +## GitLab Git LFS Administration + +Documentation on how to use Git LFS are under [Managing large binary files with Git LFS doc](manage_large_binaries_with_git_lfs.md). + +## Requirements + +* Git LFS is supported in GitLab starting with version 8.2. +* Users need to install [Git LFS client](https://git-lfs.github.com) version 1.0.1 and up + +## Configuration + +Git LFS objects can be large in size. By default, they are stored on the server GitLab is installed on. + +There are two configuration options to help GitLab server administrators: + +* Enabling/disabling Git LFS support +* Changing the location of LFS object storage + +### Omnibus packages + +In `/etc/gitlab/gitlab.rb`: + +```ruby +gitlab_rails['lfs_enabled'] = false +gitlab_rails['lfs_storage_path'] = "/mnt/storage/lfs-objects" +``` + +### Installations from source + +In `config/gitlab.yml`: + +```yaml + lfs: + enabled: false + storage_path: /mnt/storage/lfs-objects +``` + +## Known limitations + +* Currently, storing GitLab Git LFS objects on a non-local storage (like S3 buckets) is not supported +* Currently, removing LFS objects from GitLab Git LFS storage is not supported diff --git a/doc/workflow/lfs/manage_large_binaries_with_git_lfs.md b/doc/workflow/lfs/manage_large_binaries_with_git_lfs.md new file mode 100644 index 00000000000..a93fd3dfb13 --- /dev/null +++ b/doc/workflow/lfs/manage_large_binaries_with_git_lfs.md @@ -0,0 +1,125 @@ +# Git LFS + +Managing large files such as audio, video and graphics files has always been one of the shortcomings of Git. +The general recommendation is to not have Git repositories larger than 1GB to preserve performance. + +GitLab already supports [managing large files with git annex](http://doc.gitlab.com/ee/workflow/git_annex.html) (EE only), however in certain +environments it is not always convenient to use different commands to differentiate between the large files and regular ones. + +Git LFS makes this simpler for the end user by removing the requirement to learn new commands. + +## How it works + +Git LFS client talks with the GitLab server over HTTPS. It uses HTTP Basic Authentication to authorize client requests. +Once the request is authorized, Git LFS client receives instructions from where to fetch or where to push the large file. + +## GitLab server configuration + +Documentation for GitLab instance administrators is under [LFS administration doc](lfs_administration.md). + +## Requirements + +* Git LFS is supported in GitLab starting with version 8.2 +* [Git LFS client](https://git-lfs.github.com) version 1.0.1 and up + +## Known limitations + +* Git LFS v1 original API is not supported since it was deprecated early in LFS development +* When SSH is set as a remote, Git LFS objects still go through HTTPS +* Any Git LFS request will ask for HTTPS credentials to be provided so good Git credentials store is recommended +* Git LFS always assumes HTTPS so if you have GitLab server on HTTP you will have to add the URL to Git config manually (see #troubleshooting) + +## Using Git LFS + +Lets take a look at the workflow when you need to check large files into your Git repository with Git LFS: +For example, if you want to upload a very large file and check it into your Git repository: + +```bash +git clone git@gitlab.example.com:group/project.git +git lfs init # initialize the Git LFS project project +git lfs track "*.iso" # select the file extensions that you want to treat as large files +``` + +Once a certain file extension is marked for tracking as a LFS object you can use Git as usual without having to redo the command to track a file with the same extension: + +```bash +cp ~/tmp/debian.iso ./ # copy a large file into the current directory +git add . # add the large file to the project +git commit -am "Added Debian iso" # commit the file meta data +git push origin master # sync the git repo and large file to the GitLab server +``` + +Cloning the repository works the same as before. Git automatically detects the LFS-tracked files and clones them via HTTP. If you performed the git clone command with a SSH URL, you have to enter your GitLab credentials for HTTP authentication. + +```bash +git clone git@gitlab.example.com:group/project.git +``` + +If you already cloned the repository and you want to get the latest LFS object that are on the remote repository, eg. from branch `master`: + +```bash +git lfs fetch master +``` + +## Troubleshooting + +### error: Repository or object not found + +There are a couple of reasons why this error can occur: + +* You don't have permissions to access certain LFS object + +Check if you have permissions to push to the project or fetch from the project. + +* Project is not allowed to access the LFS object + +Check if the LFS object you are trying to push to the project or fetch from the project is available to the project. + +* Project is using deprecated LFS API + +### Invalid status for : 501 + +Git LFS will log the failures into a log file. +To view this log file, while in project directory: + +```bash +git lfs logs last +``` + +If the status `error 501` is shown, it is because: + +* Git LFS support is not enabled on the GitLab server. Check with your GitLab administrator why Git LFS is not enabled on the server. See [LFS administration documentation](lfs_administration.md) for instructions on how to enable LFS support. + +* Git LFS client version is not supported by GitLab server. Check your Git LFS version with `git lfs version`. Check the Git config of the project for traces of deprecated API with `git lfs -l`. If `batch = false` is set in the config, remove the line and try to update your Git LFS client. Only version 1.0.1 and newer are supported. + +### getsockopt: connection refused + +If you push a LFS object to a project and you receive an error similar to: `Post /info/lfs/objects/batch: dial tcp IP: getsockopt: connection refused`, +the LFS client is trying to reach GitLab through HTTPS. However, your GitLab instance is being served on HTTP. + +This behaviour is caused by Git LFS using HTTPS connections by default when a `lfsurl` is not set in the Git config. + +To prevent this from happening, set the lfs url in project Git config: + +```bash + +git config --add lfs.url "http://gitlab.example.com/group/project.git/info/lfs/objects/batch" +``` + +### Credentials are always required when pushing an object + +Given that Git LFS uses HTTP Basic Authentication to authenticate the user pushing the LFS object on every push for every object, user HTTPS credentials are required. + +By default, Git has support for remembering the credentials for each repository you use. This is described in [Git credentials man pages](https://git-scm.com/docs/gitcredentials). + +For example, you can tell Git to remember the password for a period of time in which you expect to push the objects: + +```bash +git config --global credential.helper 'cache --timeout=3600' +``` + +This will remember the credentials for an hour after which Git operations will require re-authentication. + +If you are using OS X you can use `osxkeychain` to store and encrypt your credentials. For Windows, `wincred` is available. + +More details about various methods of storing the user credentials can be found on [Git Credential Storage documentation](https://git-scm.com/book/en/v2/Git-Tools-Credential-Storage). diff --git a/lib/gitlab/lfs/response.rb b/lib/gitlab/lfs/response.rb index aceef53ba60..c18dfbd485d 100644 --- a/lib/gitlab/lfs/response.rb +++ b/lib/gitlab/lfs/response.rb @@ -57,7 +57,7 @@ module Gitlab 501, { "Content-Type" => "application/json; charset=utf-8" }, [JSON.dump({ - 'message' => 'Server supports batch API only, please update your Git LFS client to version 0.6.0 and up.', + 'message' => 'Server supports batch API only, please update your Git LFS client to version 1.0.1 and up.', 'documentation_url' => "#{Gitlab.config.gitlab.url}/help", })] ] diff --git a/spec/lib/gitlab/lfs/lfs_router_spec.rb b/spec/lib/gitlab/lfs/lfs_router_spec.rb index 92598559aea..5b13d65c7c0 100644 --- a/spec/lib/gitlab/lfs/lfs_router_spec.rb +++ b/spec/lib/gitlab/lfs/lfs_router_spec.rb @@ -26,7 +26,7 @@ describe Gitlab::Lfs::Router do let(:sample_oid) { "b68143e6463773b1b6c6fd009a76c32aeec041faff32ba2ed42fd7f708a17f80" } let(:sample_size) { 499013 } - let(:respond_with_deprecated) {[ 501, { "Content-Type"=>"application/json; charset=utf-8" }, ["{\"message\":\"Server supports batch API only, please update your Git LFS client to version 0.6.0 and up.\",\"documentation_url\":\"#{Gitlab.config.gitlab.url}/help\"}"]]} + let(:respond_with_deprecated) {[ 501, { "Content-Type"=>"application/json; charset=utf-8" }, ["{\"message\":\"Server supports batch API only, please update your Git LFS client to version 1.0.1 and up.\",\"documentation_url\":\"#{Gitlab.config.gitlab.url}/help\"}"]]} let(:respond_with_disabled) {[ 501, { "Content-Type"=>"application/json; charset=utf-8" }, ["{\"message\":\"Git LFS is not enabled on this GitLab server, contact your admin.\",\"documentation_url\":\"#{Gitlab.config.gitlab.url}/help\"}"]]} describe 'when lfs is disabled' do -- cgit v1.2.1