From d27e400cb9ecf4423a0c713f4c3b084c154640b9 Mon Sep 17 00:00:00 2001 From: Marin Jankovski Date: Tue, 17 Nov 2015 12:39:06 +0100 Subject: LFS doc, first draft --- doc/workflow/git_lfs.md | 127 ++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 127 insertions(+) create mode 100644 doc/workflow/git_lfs.md diff --git a/doc/workflow/git_lfs.md b/doc/workflow/git_lfs.md new file mode 100644 index 00000000000..6202822ea2d --- /dev/null +++ b/doc/workflow/git_lfs.md @@ -0,0 +1,127 @@ +# Git LFS + +Managing large files such as audio, video and graphics files has always been one of the shortcomings of Git. +The general recommendation is to not have Git repositories larger than 1GB to preserve performance. + +GitLab already supports [managing large files with git annex](http://doc.gitlab.com/ee/workflow/git_annex.html) (EE only), however in certain +environments it is not always convenient to use different commands to differentiate between the large files and regular ones. + +Git LFS makes this simpler for the end user by removing the requirement to learn new commands + + +## How it works + +Git LFS client talks with the GitLab server over HTTPS. It uses HTTP Basic Authentication to authorize client requests. +Once the request is authorized, Git LFS client receives instructions from where to fetch/where to push the large file. + +## Requirements + +* Git LFS is supported in GitLab starting with version 8.2 +* Git LFS client version 0.6.0 and up + +## GitLab and Git LFS + +### Configuration + +Git LFS objects can be large in size and they are stored on GitLab server storage. + +There are two configuration options to help GitLab server administrators: + +* Enabling/disabling Git LFS support +* Changing the location of LFS object storage + +#### Omnibus packages + +In `/etc/gitlab/gitlab.rb`: + +```ruby +gitlab_rails['lfs_enabled'] = false +gitlab_rails['lfs_storage_path'] = "/mnt/storage/lfs-objects" +``` + +#### Installations from source + +In `config/gitlab.yml`: + +```yaml + lfs: + enabled: false + storage_path: /mnt/storage/lfs-objects +``` + +### Known limitations + +* Git LFS v1 original API is not supported since it was deprecated early in LFS development, starting with Git LFS version 0.6.0 +* When SSH is set as a remote, Git LFS objects still go through HTTPS +* Any Git LFS request will ask for HTTPS credentials to be provided so good Git credentials store is recommended +* Currently, storing GitLab Git LFS objects on a non-local storage (like S3 buckets) is not supported +* Git LFS always assumes HTTPS so if you have GitLab server on HTTP you will have to add the url to Git config manually (see #troubleshooting-tips) + +## Using Git LFS + +Lets take a look at the workflow when you need to check large files into your Git repository with Git LFS: +For example, if you want to upload a very large file and check it into your Git repository: + +```bash +git clone git@gitlab.example.com:group/project.git +git lfs init # initialize the Git LFS project project +git lfs track "*.iso" # select the file extensions that you want to treat as large files +cp ~/tmp/debian.iso ./ # copy a large file into the current directory +git add . # add the large file to git annex +git commit -am "Added Debian iso" # commit the file meta data +git push origin master # sync the git repo and large file to the GitLab server +``` + +Downloading a single large file is also very simple: + +```bash +git clone git@gitlab.example.com:group/project.git +git lfs fetch debian.iso # download the large file +``` + + +## Troubleshooting tips + +### error: Repository or object not found + +Few reasons why this error can occur: + +1. Check the version of Git LFS on the client machine, `git lfs version`. Only version 0.6.0 and up are supported. +1. Check the Git config for traces of deprecated API, `git lfs -l`. If `batch = false` remove the line and try using Git LFS client > 0.6.0 + +### Invalid status for : 501 + +When attempting to push a LFS object to a GitLab server that doesn't have Git LFS support enabled, server will return status `error 501`. Check with your GitLab administrator why Git LFS is not enabled on the server + +### getsockopt: connection refused + +When pushing a LFS object and you receive an error similar to: `Post /info/lfs/objects/batch: dial tcp IP: getsockopt: connection refused`, +LFS client is trying to reach GitLab through HTTPS but your GitLab is being served on HTTP. +This behaviour is caused by Git LFS using HTTPS connections by default when it doesn't have a `lfsurl` set in the Git config. + +To go around this issue set the lfs url in git config: + +```bash + +git config --add lfs.url "http://gitlab.example.com/group/project.git/info/lfs/objects/batch" +``` + +### Credentials are always required when pushing an object + +Given that Git LFS uses HTTP Basic Authentication to authenticate the user pushing the LFS object on every push for every object, user HTTPS credentials are required. + +By default, Git has support for remembering the credentials for each repository you use. This is described in [Git credentials man pages](https://git-scm.com/docs/gitcredentials). + +For example, you can tell Git to remember the password for a period of time in which you expect to push the objects: + +```bash +git config --global credential.helper 'cache --timeout=3600' +``` + +This will remember the credentials for an hour after which Git operations will require re-authentication. + +If you are using OS X you can use `osxkeychain` to store and encrypt your credentials. For Windows, `wincred` is available. + +More details about various methods of storing the user credentials can be found on [Git Credential Storage documentation](https://git-scm.com/book/en/v2/Git-Tools-Credential-Storage) + + -- cgit v1.2.1 From 3d762c2e92f42a2c35cfbc90c87caedc8d60518a Mon Sep 17 00:00:00 2001 From: Sytse Sijbrandij Date: Tue, 17 Nov 2015 12:19:04 -0800 Subject: More labels since we get more specializations. --- PROCESS.md | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-) diff --git a/PROCESS.md b/PROCESS.md index a4b0c83644b..1408c8302ee 100644 --- a/PROCESS.md +++ b/PROCESS.md @@ -34,13 +34,16 @@ The most important thing is making sure valid issues receive feedback from the d ## Workflow labels -Workflow labels are purposely not very detailed since that would be hard to keep updated as you would need to re-evaluate them after every comment. We optionally use functional labels on demand when want to group related issues to get an overview (for example all issues related to RVM, to tackle them in one go) and to add details to the issue. +Workflow labels are purposely not very detailed since that would be hard to keep updated as you would need to re-evaluate them after every comment. We optionally use functional labels on demand when want to group related issues to get an overview (for example all issues related to RVM, to tackle them in one go) and to add details to the issue. - *Awaiting feedback*: Feedback pending from the reporter - *Awaiting confirmation of fix*: The issue should already be solved in **master** (generally you can avoid this workflow item and just close the issue right away) - *Attached MR*: There is a MR attached and the discussion should happen there - We need to let issues stay in sync with the MR's. We can do this with a "Closing #XXXX" or "Fixes #XXXX" comment in the MR. We can't close the issue when there is a merge request because sometimes a MR is not good and we just close the MR, then the issue must stay. -- *Awaiting developer action/feedback*: Issue needs to be fixed or clarified by a developer +- *Developer*: needs help from a developer +- *UX* needs needs help from a UX designer +- *Frontend* needs help from a Front-end engineer +- *Graphics* needs help from a Graphics designer ## Functional labels -- cgit v1.2.1 From 1e74a12c2f1ed024bb7bd0d1963011833b7404ca Mon Sep 17 00:00:00 2001 From: Sytse Sijbrandij Date: Tue, 17 Nov 2015 14:43:51 -0800 Subject: Example workflow. --- PROCESS.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/PROCESS.md b/PROCESS.md index 1408c8302ee..482ad5fe9e1 100644 --- a/PROCESS.md +++ b/PROCESS.md @@ -45,6 +45,8 @@ Workflow labels are purposely not very detailed since that would be hard to keep - *Frontend* needs help from a Front-end engineer - *Graphics* needs help from a Graphics designer +Example workflow: when a UX designer provided a design but it needs frontend work they remove the UX label and add the frontend label. + ## Functional labels These labels describe what development specialities are involved such as: PostgreSQL, UX, LDAP. -- cgit v1.2.1 From 6c62f182c61e9f5f2c1cd3e98c647f246bf6880b Mon Sep 17 00:00:00 2001 From: Marin Jankovski Date: Wed, 18 Nov 2015 10:32:29 +0100 Subject: Updates to the lfs doc. --- doc/workflow/git_lfs.md | 37 ++++++++++++++++++++++++------------- 1 file changed, 24 insertions(+), 13 deletions(-) diff --git a/doc/workflow/git_lfs.md b/doc/workflow/git_lfs.md index 6202822ea2d..7304735f574 100644 --- a/doc/workflow/git_lfs.md +++ b/doc/workflow/git_lfs.md @@ -12,7 +12,7 @@ Git LFS makes this simpler for the end user by removing the requirement to learn ## How it works Git LFS client talks with the GitLab server over HTTPS. It uses HTTP Basic Authentication to authorize client requests. -Once the request is authorized, Git LFS client receives instructions from where to fetch/where to push the large file. +Once the request is authorized, Git LFS client receives instructions from where to fetch or where to push the large file. ## Requirements @@ -23,7 +23,7 @@ Once the request is authorized, Git LFS client receives instructions from where ### Configuration -Git LFS objects can be large in size and they are stored on GitLab server storage. +Git LFS objects can be large in size. By default, they are stored on the server GitLab is installed on. There are two configuration options to help GitLab server administrators: @@ -49,7 +49,7 @@ In `config/gitlab.yml`: storage_path: /mnt/storage/lfs-objects ``` -### Known limitations +## Known limitations * Git LFS v1 original API is not supported since it was deprecated early in LFS development, starting with Git LFS version 0.6.0 * When SSH is set as a remote, Git LFS objects still go through HTTPS @@ -66,8 +66,13 @@ For example, if you want to upload a very large file and check it into your Git git clone git@gitlab.example.com:group/project.git git lfs init # initialize the Git LFS project project git lfs track "*.iso" # select the file extensions that you want to treat as large files +``` + +Once a certain file extension is marked for tracking as a LFS object you can use Git as usual without having to redo the command to track a file with the same extension: + +```bash cp ~/tmp/debian.iso ./ # copy a large file into the current directory -git add . # add the large file to git annex +git add . # add the large file to the project git commit -am "Added Debian iso" # commit the file meta data git push origin master # sync the git repo and large file to the GitLab server ``` @@ -80,26 +85,32 @@ git lfs fetch debian.iso # download the large file ``` -## Troubleshooting tips +## Troubleshooting ### error: Repository or object not found -Few reasons why this error can occur: +There are a couple of reasons why this error can occur: -1. Check the version of Git LFS on the client machine, `git lfs version`. Only version 0.6.0 and up are supported. -1. Check the Git config for traces of deprecated API, `git lfs -l`. If `batch = false` remove the line and try using Git LFS client > 0.6.0 +* Wrong version of LFS client used: + +Check the version of Git LFS on the client machine with `git lfs version`. Only version 0.6.0 and newer are supported. + +* Project is using deprecated LFS API + +Check the Git config of the project for traces of deprecated API with `git lfs -l`. If `batch = false` is set in the config, remove the line and try using Git LFS client newer than 0.6.0. ### Invalid status for : 501 -When attempting to push a LFS object to a GitLab server that doesn't have Git LFS support enabled, server will return status `error 501`. Check with your GitLab administrator why Git LFS is not enabled on the server +When attempting to push a LFS object to a GitLab server that doesn't have Git LFS support enabled, server will return status `error 501`. Check with your GitLab administrator why Git LFS is not enabled on the server. See [Configuration section](#configuration) for instructions on how to enable LFS support. ### getsockopt: connection refused -When pushing a LFS object and you receive an error similar to: `Post /info/lfs/objects/batch: dial tcp IP: getsockopt: connection refused`, -LFS client is trying to reach GitLab through HTTPS but your GitLab is being served on HTTP. -This behaviour is caused by Git LFS using HTTPS connections by default when it doesn't have a `lfsurl` set in the Git config. +If you push a LFS object to a project and you receive an error similar to: `Post /info/lfs/objects/batch: dial tcp IP: getsockopt: connection refused`, +the LFS client is trying to reach GitLab through HTTPS. However, your GitLab instance is being served on HTTP. + +This behaviour is caused by Git LFS using HTTPS connections by default when a `lfsurl` is not set in the Git config. -To go around this issue set the lfs url in git config: +To prevent this from happening, set the lfs url in project Git config: ```bash -- cgit v1.2.1 From 60e45651425369d7e55b85984ac8f5045d8bdef6 Mon Sep 17 00:00:00 2001 From: Job van der Voort Date: Wed, 18 Nov 2015 10:03:08 +0000 Subject: missing . --- doc/workflow/git_lfs.md | 6 ++---- 1 file changed, 2 insertions(+), 4 deletions(-) diff --git a/doc/workflow/git_lfs.md b/doc/workflow/git_lfs.md index 7304735f574..4990a9b5aac 100644 --- a/doc/workflow/git_lfs.md +++ b/doc/workflow/git_lfs.md @@ -6,7 +6,7 @@ The general recommendation is to not have Git repositories larger than 1GB to pr GitLab already supports [managing large files with git annex](http://doc.gitlab.com/ee/workflow/git_annex.html) (EE only), however in certain environments it is not always convenient to use different commands to differentiate between the large files and regular ones. -Git LFS makes this simpler for the end user by removing the requirement to learn new commands +Git LFS makes this simpler for the end user by removing the requirement to learn new commands. ## How it works @@ -133,6 +133,4 @@ This will remember the credentials for an hour after which Git operations will r If you are using OS X you can use `osxkeychain` to store and encrypt your credentials. For Windows, `wincred` is available. -More details about various methods of storing the user credentials can be found on [Git Credential Storage documentation](https://git-scm.com/book/en/v2/Git-Tools-Credential-Storage) - - +More details about various methods of storing the user credentials can be found on [Git Credential Storage documentation](https://git-scm.com/book/en/v2/Git-Tools-Credential-Storage) \ No newline at end of file -- cgit v1.2.1