Port repository mirroring from EEgithub-import-doc

11 files changed, 245 insertions, 5 deletions

diff --git a/doc/workflow/repository_mirroring.md b/doc/workflow/repository_mirroring.md
index dbe63144e38..aaddbe4fbf5 100644
--- a/doc/workflow/repository_mirroring.md
+++ b/doc/workflow/repository_mirroring.md
@@ -1,6 +1,6 @@
 # Repository mirroring
 
-Repository Mirroring is a way to mirror repositories from external sources.
+Repository mirroring is a way to mirror repositories from external sources.
 It can be used to mirror all branches, tags, and commits that you have
 in your repository.
 
@@ -34,13 +34,200 @@ A few things/limitations to consider:
 - The Git LFS objects will not be synced. You'll need to push/pull them
   manually.
 
-## Use-case
+## Use cases
 
+- You migrated to GitLab but still need to keep your project in another source.
+  In that case, you can simply set it up to mirror to GitLab (pull) and all the
+  essential history of commits, tags and branches will be available in your
+  GitLab instance.
 - You have old projects in another source that you don't use actively anymore,
   but don't want to remove for archiving purposes. In that case, you can create
   a push mirror so that your active GitLab repository can push its changes to the
   old location.
 
+## Pulling from a remote repository **[STARTER]**
+
+>[Introduced][ee-51] in GitLab Enterprise Edition 8.2.
+
+You can set up a repository to automatically have its branches, tags, and commits
+updated from an upstream repository. This is useful when a repository you're
+interested in is located on a different server, and you want to be able to
+browse its content and its activity using the familiar GitLab interface.
+
+When creating a new project, you can enable repository mirroring when you choose
+to import the repository from "Any repo by URL". Enter the full URL of the Git
+repository to pull from and click on the **Mirror repository** checkbox.
+
+![New project](repository_mirroring/repository_mirroring_new_project.png)
+
+For an existing project, you can set up mirror pulling by visiting your project's
+**Settings ➔ Repository** and searching for the "Pull from a remote repository"
+section. Check the "Mirror repository" box and hit **Save changes** at the bottom.
+You have a few options to choose from one being the user who will be the author
+of all events in the activity feed that are the result of an update. This user
+needs to have at least [master access][perms] to the project. Another option is
+whether you want to trigger builds for mirror updates.
+
+![Pull settings](repository_mirroring/repository_mirroring_pull_settings.png)
+
+Since the repository on GitLab functions as a mirror of the upstream repository,
+you are advised not to push commits directly to the repository on GitLab.
+Instead, any commits should be pushed to the upstream repository, and will end
+up in the GitLab repository automatically within a certain period of time
+or when a [forced update](#forcing-an-update) is initiated.
+
+If you do manually update a branch in the GitLab repository, the branch will
+become diverged from upstream, and GitLab will no longer automatically update
+this branch to prevent any changes from being lost.
+
+![Diverged branch](repository_mirroring/repository_mirroring_diverged_branch.png)
+
+### Trigger update using API **[STARTER]**
+
+>[Introduced][ee-3453] in GitLab Enterprise Edition 10.3.
+
+Pull mirroring uses polling to detect new branches and commits added upstream,
+often many minutes afterwards. If you notify GitLab by [API][pull-api], updates
+will be pulled immediately.
+
+Read the [Pull Mirror Trigger API docs][pull-api].
+
+### Pull only protected branches **[STARTER]**
+
+>[Introduced][ee-3326] in GitLab Enterprise Edition 10.3.
+
+You can choose to only pull the protected branches from your remote repository to GitLab.
+
+To use this option go to your project's repository settings page under pull mirror.
+
+### Overwrite diverged branches **[STARTER]**
+
+>[Introduced][ee-4559] in GitLab Enterprise Edition 10.6.
+
+You can choose to always update your local branch with the remote version even
+if your local version has diverged from the remote.
+
+To use this option go to your project's repository settings page under pull mirror.
+
+### Hard failure **[STARTER]**
+
+>[Introduced][ee-3117] in GitLab Enterprise Edition 10.2.
+
+Once a mirror gets retried 14 times in a row, it will get marked as hard failed,
+this will become visible in either the project main dashboard or in the
+pull mirror settings page.
+
+![Hard failed mirror main notice](repository_mirroring/repository_mirroring_hard_failed_main.png)
+
+![Hard failed mirror settings notice](repository_mirroring/repository_mirroring_hard_failed_settings.png)
+
+When a project is hard failed, it will no longer get picked up for mirroring.
+A user can resume the project mirroring again by either [forcing an update](#forcing-an-update)
+or by changing the import URL in repository settings.
+
+### SSH authentication **[STARTER]**
+
+> [Introduced][ee-2551] in GitLab Starter 9.5
+
+If you're mirroring over SSH (i.e., an `ssh://` URL), you can authenticate using
+password-based authentication, just as over HTTPS, but you can also use public
+key authentication. This is often more secure than password authentication,
+especially when the source repository supports [Deploy Keys][deploy-key].
+
+To get started, navigate to **Settings ➔ Repository ➔ Pull from a remote repository**,
+enable mirroring (if not already enabled) and enter an `ssh://` URL.
+
+> **NOTE**: SCP-style URLs, e.g., `git@example.com:group/project.git`, are not
+supported at this time.
+
+Entering the URL adds two features to the page - `Fingerprints` and
+`SSH public key authentication`:
+
+![Pull settings for SSH](repository_mirroring/repository_mirroring_pull_settings_for_ssh.png)
+
+SSH authentication is mutual. You have to prove to the server that you're
+allowed to access the repository, but the server also has to prove to *you* that
+it's who it claims to be. You provide your credentials as a password or public
+key. The server that the source repository resides on provides its credentials
+as a "host key", the fingerprint of which needs to be verified manually.
+
+Press the `Detect host keys` button. GitLab will fetch the host keys from the
+server, and display the fingerprints to you:
+
+![Detect SSH host keys](repository_mirroring/repository_mirroring_detect_host_keys.png)
+
+You now need to verify that the fingerprints are those you expect. GitLab.com
+and other code hosting sites publish their fingerprints in the open for you
+to check:
+
+* [AWS CodeCommit](http://docs.aws.amazon.com/codecommit/latest/userguide/regions.html#regions-fingerprints)
+* [Bitbucket](https://confluence.atlassian.com/bitbucket/use-the-ssh-protocol-with-bitbucket-cloud-221449711.html#UsetheSSHprotocolwithBitbucketCloud-KnownhostorBitbucket%27spublickeyfingerprints)
+* [GitHub](https://help.github.com/articles/github-s-ssh-key-fingerprints/)
+* [GitLab.com](https://about.gitlab.com/gitlab-com/settings/#ssh-host-keys-fingerprints)
+* [Launchpad](https://help.launchpad.net/SSHFingerprints)
+* [Savannah](http://savannah.gnu.org/maintenance/SshAccess/)
+* [SourceForge](https://sourceforge.net/p/forge/documentation/SSH%20Key%20Fingerprints/)
+
+Other providers will vary. If you're running on-premises GitLab, or otherwise
+have access to the source server, you can securely gather the key fingerprints:
+
+```
+$ cat /etc/ssh/ssh_host*pub | ssh-keygen -E md5 -l -f -
+256 MD5:f4:28:9f:23:99:15:21:1b:bf:ed:1f:8e:a0:76:b2:9d root@example.com (ECDSA)
+256 MD5:e6:eb:45:8a:3c:59:35:5f:e9:5b:80:12:be:7e:22:73 root@example.com (ED25519)
+2048 MD5:3f:72:be:3d:62:03:5c:62:83:e8:6e:14:34:3a:85:1d root@example.com (RSA)
+```
+
+(You may need to exclude `-E md5` for some older versions of SSH).
+
+If you're an SSH expert and already have a `known_hosts` file you'd like to use
+unaltered, then you can skip these steps. Just press the "Show advanced" button
+and paste in the file contents:
+
+![Advanced SSH host key management](repository_mirroring/repository_mirroring_pull_advanced_host_keys.png)
+
+Once you've **carefully verified** that all the fingerprints match your trusted
+source, you can press `Save changes`. This will record the host keys, along with
+the person who verified them (you!) and the date:
+
+![SSH host keys submitted](repository_mirroring/repository_mirroring_ssh_host_keys_verified.png)
+
+When pulling changes from the source repository, GitLab will now check that at
+least one of the stored host keys matches before connecting. This can prevent
+malicious code from being injected into your mirror, or your password being
+stolen!
+
+To use SSH public key authentication, you'll also need to choose that option
+from the authentication methods dropdown. GitLab will generate a 4096-bit RSA
+key and display the public component of that key to you:
+
+![SSH public key authentication](repository_mirroring/repository_mirroring_ssh_public_key_authentication.png)
+
+You then need to add the public SSH key to the source repository configuration.
+If the source is hosted on GitLab, you should add it as a [Deploy Key][deploy-key].
+Other sources may require you to add the key to your user's `authorized_keys`
+file - just paste the entire `ssh-rsa AAA.... user@host` block into the file on
+its own line and save it.
+
+Once the public key is set up on the source repository, press `Save changes` and your
+mirror will begin working.
+
+If you need to change the key at any time, you can press the `Regenerate key`
+button to do so. You'll have to update the source repository with the new key
+to keep the mirror running.
+
+### How it works
+
+Once you activate the pull mirroring feature, the mirror will be inserted into
+a queue. A scheduler will start every minute and schedule a fixed amount of
+mirrors for update, based on the configured maximum capacity.
+
+If the mirror successfully updates it will be enqueued once again with a small
+backoff period.
+
+If the mirror fails (eg: branch diverged from upstream), the project's backoff
+period will be penalized each time it fails up to a maximum amount of time.
+
 ## Pushing to a remote repository **[STARTER]**
 
 >[Introduced](https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/249) in
@@ -83,7 +270,7 @@ To use this option go to your project's repository settings page under push mirr
 To set up a mirror from GitLab to GitHub, you need to follow these steps:
 
 1. Create a [GitHub personal access token](https://help.github.com/articles/creating-a-personal-access-token-for-the-command-line/) with the "public_repo" box checked:
-    
+
     ![edit personal access token GitHub](repository_mirroring/repository_mirroring_github_edit_personal_access_token.png)
 
 1. Fill in the "Git repository URL" with the personal access token replacing the password `https://GitHubUsername:GitHubPersonalAccessToken@github.com/group/project.git`:
@@ -94,7 +281,7 @@ To set up a mirror from GitLab to GitHub, you need to follow these steps:
 1. And either wait or trigger the "Update Now" button:
 
     ![update now](repository_mirroring/repository_mirroring_gitlab_push_to_a_remote_repository_update_now.png)
-    
+
 ## Forcing an update
 
 While mirrors are scheduled to update automatically, you can always force an update
@@ -105,7 +292,60 @@ by using the **Update now** button which is exposed in various places:
 - in the tags page
 - in the **Mirror repository** settings page
 
+## Bidirectional mirroring
+
+CAUTION: **Warning:**
+There is no bidirectional support without conflicts. If you
+configure a repository to pull and push to a second remote, there is no
+guarantee that it will update correctly on both remotes. If you configure
+a repository for bidirectional mirroring, you should consider when conflicts
+occur who and how they will be resolved.
+
+Rewriting any mirrored commit on either remote will cause conflicts and
+mirroring to fail. This can be prevented by [only pulling protected branches](
+#pull-only-protected-branches) and [only pushing protected branches](
+#push-only-protected-branches). You should protect the branches you wish to
+mirror on both remotes to prevent conflicts caused by rewriting history.
+
+Bidirectional mirroring also creates a race condition where commits to the same
+branch in close proximity will cause conflicts. The race condition can be
+mitigated by reducing the mirroring delay by using a Push event webhook to
+trigger an immediate pull to GitLab. Push mirroring from GitLab is rate limited
+to once per minute when only push mirroring protected branches.
+
+It may be possible to implement a locking mechanism using the server-side
+`pre-receive` hook to prevent the race condition. Read about [configuring
+custom Git hooks][hooks] on the GitLab server.
+
+### Mirroring with Perforce via GitFusion
+
+CAUTION: **Warning:**
+Bidirectional mirroring should not be used as a permanent
+configuration. There is no bidirectional mirroring without conflicts.
+Refer to [Migrating from Perforce Helix][perforce] for alternative migration
+approaches.
+
+GitFusion provides a Git interface to Perforce which can be used by GitLab to
+bidirectionally mirror projects with GitLab. This may be useful in some
+situations when migrating from Perforce to GitLab where overlapping Perforce
+workspaces cannot be migrated simultaneously to GitLab.
+
+If using mirroring with Perforce you should only mirror protected branches.
+Perforce will reject any pushes that rewrite history. It is recommended that
+only the fewest number of branches are mirrored due to the performance
+limitations of GitFusion.
+
+[ee-51]: https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/51
+[ee-2551]: https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/2551
+[ee-3117]: https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/3117
+[ee-3326]: https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/3326
 [ee-3350]: https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/3350
+[ee-3453]: https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/3453
+[ee-4559]: https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/4559
 [ce-18715]: https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/18715
 [perms]: ../user/permissions.md
-
+[hooks]: ../administration/custom_hooks.md
+[deploy-key]: ../ssh/README.md#deploy-keys
+[webhook]: ../user/project/integrations/webhooks.md#push-events
+[pull-api]: ../api/projects.md#start-the-pull-mirroring-process-for-a-project
+[perforce]: ../user/project/import/perforce.md
diff --git a/doc/workflow/repository_mirroring/repository_mirroring_detect_host_keys.png b/doc/workflow/repository_mirroring/repository_mirroring_detect_host_keys.png
new file mode 100644
index 00000000000..333648942f8
--- /dev/null
+++ b/doc/workflow/repository_mirroring/repository_mirroring_detect_host_keys.png
diff --git a/doc/workflow/repository_mirroring/repository_mirroring_diverged_branch.png b/doc/workflow/repository_mirroring/repository_mirroring_diverged_branch.png
new file mode 100644
index 00000000000..45c9bce0889
--- /dev/null
+++ b/doc/workflow/repository_mirroring/repository_mirroring_diverged_branch.png
diff --git a/doc/workflow/repository_mirroring/repository_mirroring_hard_failed_main.png b/doc/workflow/repository_mirroring/repository_mirroring_hard_failed_main.png
new file mode 100644
index 00000000000..99d429a1802
--- /dev/null
+++ b/doc/workflow/repository_mirroring/repository_mirroring_hard_failed_main.png
diff --git a/doc/workflow/repository_mirroring/repository_mirroring_hard_failed_settings.png b/doc/workflow/repository_mirroring/repository_mirroring_hard_failed_settings.png
new file mode 100644
index 00000000000..0ab07afa3cc
--- /dev/null
+++ b/doc/workflow/repository_mirroring/repository_mirroring_hard_failed_settings.png
diff --git a/doc/workflow/repository_mirroring/repository_mirroring_new_project.png b/doc/workflow/repository_mirroring/repository_mirroring_new_project.png
new file mode 100644
index 00000000000..43bf304838f
--- /dev/null
+++ b/doc/workflow/repository_mirroring/repository_mirroring_new_project.png
diff --git a/doc/workflow/repository_mirroring/repository_mirroring_pull_advanced_host_keys.png b/doc/workflow/repository_mirroring/repository_mirroring_pull_advanced_host_keys.png
new file mode 100644
index 00000000000..5da5a7436bb
--- /dev/null
+++ b/doc/workflow/repository_mirroring/repository_mirroring_pull_advanced_host_keys.png
diff --git a/doc/workflow/repository_mirroring/repository_mirroring_pull_settings.png b/doc/workflow/repository_mirroring/repository_mirroring_pull_settings.png
new file mode 100644
index 00000000000..4b9085302a1
--- /dev/null
+++ b/doc/workflow/repository_mirroring/repository_mirroring_pull_settings.png
diff --git a/doc/workflow/repository_mirroring/repository_mirroring_pull_settings_for_ssh.png b/doc/workflow/repository_mirroring/repository_mirroring_pull_settings_for_ssh.png
new file mode 100644
index 00000000000..8c2efdafa43
--- /dev/null
+++ b/doc/workflow/repository_mirroring/repository_mirroring_pull_settings_for_ssh.png
diff --git a/doc/workflow/repository_mirroring/repository_mirroring_ssh_host_keys_verified.png b/doc/workflow/repository_mirroring/repository_mirroring_ssh_host_keys_verified.png
new file mode 100644
index 00000000000..93f3a532a0e
--- /dev/null
+++ b/doc/workflow/repository_mirroring/repository_mirroring_ssh_host_keys_verified.png
diff --git a/doc/workflow/repository_mirroring/repository_mirroring_ssh_public_key_authentication.png b/doc/workflow/repository_mirroring/repository_mirroring_ssh_public_key_authentication.png
new file mode 100644
index 00000000000..6997ad511d9
--- /dev/null
+++ b/doc/workflow/repository_mirroring/repository_mirroring_ssh_public_key_authentication.png