Merge branch 'revert-satellites' into 'master'

Revert satellites

Return satellites to master for 7.14

We remove satellites in next release (8.0)

See merge request !1136

8 files changed, 36 insertions, 48 deletions

diff --git a/doc/development/architecture.md b/doc/development/architecture.md
index c00d290371e..541af487bb1 100644
--- a/doc/development/architecture.md
+++ b/doc/development/architecture.md
@@ -56,9 +56,9 @@ To serve repositories over SSH there's an add-on application called gitlab-shell
 
 A typical install of GitLab will be on GNU/Linux. It uses Nginx or Apache as a web front end to proxypass the Unicorn web server. By default, communication between Unicorn and the front end is via a Unix domain socket but forwarding requests via TCP is also supported. The web front end accesses `/home/git/gitlab/public` bypassing the Unicorn server to serve static pages, uploads (e.g. avatar images or attachments), and precompiled assets. GitLab serves web pages and a [GitLab API](https://gitlab.com/gitlab-org/gitlab-ce/tree/master/doc/api) using the Unicorn web server. It uses Sidekiq as a job queue which, in turn, uses redis as a non-persistent database backend for job information, meta data, and incoming jobs.
 
-The GitLab web app uses MySQL or PostgreSQL for persistent database information (e.g. users, permissions, issues, other meta data). GitLab stores the bare git repositories it serves in `/home/git/repositories` by default. It also keeps default branch and hook information with the bare repository.
+The GitLab web app uses MySQL or PostgreSQL for persistent database information (e.g. users, permissions, issues, other meta data). GitLab stores the bare git repositories it serves in `/home/git/repositories` by default. It also keeps default branch and hook information with the bare repository. `/home/git/gitlab-satellites` keeps checked out repositories when performing actions such as a merge request, editing files in the web interface, etc.
 
-When serving repositories over HTTP/HTTPS GitLab utilizes the GitLab API to resolve authorization and access as well as serving git objects.
+The satellite repository is used by the web interface for editing repositories and the wiki which is also a git repository. When serving repositories over HTTP/HTTPS GitLab utilizes the GitLab API to resolve authorization and access as well as serving git objects.
 
 The add-on component gitlab-shell serves repositories over SSH. It manages the SSH keys within `/home/git/.ssh/authorized_keys` which should not be manually edited. gitlab-shell accesses the bare repositories directly to serve git objects and communicates with redis to submit jobs to Sidekiq for GitLab to process. gitlab-shell queries the GitLab API to determine authorization and access.
 
@@ -129,7 +129,7 @@ Note: `/home/git/` is shorthand for `/home/git`.
 
 gitlabhq (includes Unicorn and Sidekiq logs)
 
-- `/home/git/gitlab/log/` contains `application.log`, `production.log`, `sidekiq.log`, `unicorn.stdout.log`, `githost.log` and `unicorn.stderr.log` normally.
+- `/home/git/gitlab/log/` contains `application.log`, `production.log`, `sidekiq.log`, `unicorn.stdout.log`, `githost.log`, `satellites.log`, and `unicorn.stderr.log` normally.
 
 gitlab-shell
 
diff --git a/doc/install/installation.md b/doc/install/installation.md
index 55b6f216dde..6e2bb4191ca 100644
--- a/doc/install/installation.md
+++ b/doc/install/installation.md
@@ -216,6 +216,10 @@ We recommend using a PostgreSQL database. For MySQL check [MySQL setup guide](da
     sudo chmod -R u+rwX,go-w log/
     sudo chmod -R u+rwX tmp/
 
+    # Create directory for satellites
+    sudo -u git -H mkdir /home/git/gitlab-satellites
+    sudo chmod u+rwx,g=rx,o-rwx /home/git/gitlab-satellites
+
     # Make sure GitLab can write to the tmp/pids/ and tmp/sockets/ directories
     sudo chmod -R u+rwX tmp/pids/
     sudo chmod -R u+rwX tmp/sockets/
diff --git a/doc/install/requirements.md b/doc/install/requirements.md
index aa0d03b75bc..a78590d512a 100644
--- a/doc/install/requirements.md
+++ b/doc/install/requirements.md
@@ -40,7 +40,7 @@ We love [JRuby](http://jruby.org/) and [Rubinius](http://rubini.us/) but GitLab
 
 ### Storage
 
-The necessary hard drive space largely depends on the size of the repos you want to store in GitLab but as a *rule of thumb* you should have at least as much free space as all your repos combined take up. 
+The necessary hard drive space largely depends on the size of the repos you want to store in GitLab but as a *rule of thumb* you should have at least twice as much free space as all your repos combined take up. You need twice the storage because [GitLab satellites](structure.md) contain an extra copy of each repo.
 
 If you want to be flexible about growing your hard drive space in the future consider mounting it using LVM so you can add more hard drives when you need them.
 
@@ -113,4 +113,4 @@ On a very active server (10,000 active users) the Sidekiq process can use 1GB+ o
 
 ### Common UI problems with IE
 
-If you experience UI issues with Internet Explorer, please make sure that you have the `Compatibility View` mode disabled.
\ No newline at end of file
+If you experience UI issues with Internet Explorer, please make sure that you have the `Compatibility View` mode disabled.
diff --git a/doc/install/structure.md b/doc/install/structure.md
index d58b0040eef..5c03f073c18 100644
--- a/doc/install/structure.md
+++ b/doc/install/structure.md
@@ -6,14 +6,16 @@ This is the directory structure you will end up with following the instructions
     |   |-- git
     |       |-- .ssh
     |       |-- gitlab
+    |       |-- gitlab-satellites
     |       |-- gitlab-shell
     |       |-- repositories
 
 * `/home/git/.ssh` - contains openssh settings.  Specifically the `authorized_keys` file managed by gitlab-shell.
 * `/home/git/gitlab` - GitLab core software.
+* `/home/git/gitlab-satellites` - checked out repositories for merge requests and file editing from web UI.  This can be treated as a temporary files directory.
 * `/home/git/gitlab-shell` - Core add-on component of GitLab.  Maintains SSH cloning and other functionality.
 * `/home/git/repositories` - bare repositories for all projects organized by namespace.  This is where the git repositories which are pushed/pulled are maintained for all projects.  **This area is critical data for projects.  [Keep a backup](../raketasks/backup_restore.md)**
 
-*Note: the default locations for repositories can be configured in `config/gitlab.yml` of GitLab and `config.yml` of gitlab-shell.*
+*Note: the default locations for gitlab-satellites and repositories can be configured in `config/gitlab.yml` of GitLab and `config.yml` of gitlab-shell.*
 
 To see a more in-depth overview see the [GitLab architecture doc](../development/architecture.md).
diff --git a/doc/logs/logs.md b/doc/logs/logs.md
index 27937e51764..83c32b09253 100644
--- a/doc/logs/logs.md
+++ b/doc/logs/logs.md
@@ -51,6 +51,16 @@ December 03, 2014 13:20 -> ERROR -> Command failed [1]: /usr/bin/git --git-dir=/
 error: failed to push some refs to '/Users/vsizov/gitlab-development-kit/repositories/gitlabhq/gitlab_git.git'
 ```
 
+#### satellites.log
+This file lives in `/var/log/gitlab/gitlab-rails/satellites.log` for omnibus package or in `/home/git/gitlab/log/satellites.log` for installations from the source.
+
+In some cases GitLab should perform write actions to git repository, for example when it is needed to merge the merge request or edit a file with online editor. If something went wrong you can look into this file to find out what exactly happened.
+```
+October 07, 2014 11:36: Failed to create satellite for Chesley Weimann III / project1817
+October 07, 2014 11:36: PID: 1872: git clone /Users/vsizov/gitlab-development-kit/gitlab/tmp/tests/repositories/conrad6841/gitlabhq.git /Users/vsizov/gitlab-development-kit/gitlab/tmp/tests/gitlab-satellites/conrad6841/gitlabhq
+October 07, 2014 11:36: PID: 1872: -> fatal: repository '/Users/vsizov/gitlab-development-kit/gitlab/tmp/tests/repositories/conrad6841/gitlabhq.git' does not exist
+```
+
 #### sidekiq.log
 This file lives in `/var/log/gitlab/gitlab-rails/sidekiq.log` for omnibus package or in `/home/git/gitlab/log/sidekiq.log` for installations from the source.
 
@@ -89,4 +99,4 @@ W, [2015-02-13T07:16:01.313000 #9094]  WARN -- : Unicorn::WorkerKiller send SIGQ
 I, [2015-02-13T07:16:01.530733 #9047]  INFO -- : reaped #<Process::Status: pid 9094 exit 0> worker=1
 I, [2015-02-13T07:16:01.534501 #13379]  INFO -- : worker=1 spawned pid=13379
 I, [2015-02-13T07:16:01.534848 #13379]  INFO -- : worker=1 ready
-```
+```
\ No newline at end of file
diff --git a/doc/raketasks/maintenance.md b/doc/raketasks/maintenance.md
index d9dce2af480..69171cd1765 100644
--- a/doc/raketasks/maintenance.md
+++ b/doc/raketasks/maintenance.md
@@ -105,11 +105,24 @@ Log directory writable? ... yes
 Tmp directory writable? ... yes
 Init script exists? ... yes
 Init script up-to-date? ... yes
+Projects have satellites? ... yes
 Redis version >= 2.0.0? ... yes
 
 Checking GitLab ... Finished
 ```
 
+## (Re-)Create satellite repositories
+
+This will create satellite repositories for all your projects.
+
+If necessary, remove the `repo_satellites` directory and rerun the commands below.
+
+```
+sudo -u git -H mkdir -p /home/git/gitlab-satellites
+sudo -u git -H bundle exec rake gitlab:satellites:create RAILS_ENV=production
+sudo chmod u+rwx,g=rx,o-rwx /home/git/gitlab-satellites
+```
+
 ## Rebuild authorized_keys file
 
 In some case it is necessary to rebuild the `authorized_keys` file.
diff --git a/doc/workflow/README.md b/doc/workflow/README.md
index 5b8d72dfd34..3915198ad2a 100644
--- a/doc/workflow/README.md
+++ b/doc/workflow/README.md
@@ -13,5 +13,4 @@
 - [Project users](add-user/add-user.md)
 - [Protected branches](protected_branches.md)
 - [Web Editor](web_editor.md)
-- [Merge Requests](merge_requests.md)
 - ["Work In Progress" Merge Requests](wip_merge_requests.md)
diff --git a/doc/workflow/merge_requests.md b/doc/workflow/merge_requests.md
deleted file mode 100644
index 751e19da7f1..00000000000
--- a/doc/workflow/merge_requests.md
+++ /dev/null
@@ -1,40 +0,0 @@
-# Merge Requests
-
-Merge requests allow you to exchange changes you made to source code
-
-## Checkout merge requests locally
-
-Locate the section for your GitLab remote in the `.git/config` file. It looks like this:
-
-```
-[remote "origin"]
-  url = https://gitlab.com/gitlab-org/gitlab-ce.git
-  fetch = +refs/heads/*:refs/remotes/origin/*
-```
-
-Now add the line `fetch = +refs/merge-requests/*/head:refs/remotes/origin/merge-requests/*` to this section. 
-
-It should looks like this: 
-
-```
-[remote "origin"]
-  url = https://gitlab.com/gitlab-org/gitlab-ce.git
-  fetch = +refs/heads/*:refs/remotes/origin/*
-  fetch = +refs/merge-requests/*/head:refs/remotes/origin/merge-requests/*
-```
-
-Now you can fetch all the merge requests requests:
-
-```
-$ git fetch origin
-From https://gitlab.com/gitlab-org/gitlab-ce.git
- * [new ref]         refs/merge-requests/1/head -> origin/merge-requests/1
- * [new ref]         refs/merge-requests/2/head -> origin/merge-requests/2
-...
-```
-
-To check out a particular merge request:
-
-```
-$ git checkout origin/merge-requests/1
-```