Merge branch 'master' into eReGeBe/gitlab-ce-feature/milestone-md

14 files changed, 869 insertions, 17 deletions

diff --git a/doc/README.md b/doc/README.md
index e6ac4794827..e358da1c424 100644
--- a/doc/README.md
+++ b/doc/README.md
@@ -41,6 +41,8 @@
 - [Git LFS configuration](workflow/lfs/lfs_administration.md)
 - [Housekeeping](administration/housekeeping.md) Keep your Git repository tidy and fast.
 - [GitLab Performance Monitoring](monitoring/performance/introduction.md) Configure GitLab and InfluxDB for measuring performance metrics
+- [Sidekiq Troubleshooting](administration/troubleshooting/sidekiq.md) Debug when Sidekiq appears hung and is not processing jobs
+- [High Availability](administration/high_availability/README.md) Configure multiple servers for scaling or high availability
 
 ## Contributor documentation
 
diff --git a/doc/administration/high_availability/README.md b/doc/administration/high_availability/README.md
new file mode 100644
index 00000000000..43d85ffb775
--- /dev/null
+++ b/doc/administration/high_availability/README.md
@@ -0,0 +1,35 @@
+# High Availability
+
+GitLab supports several different types of clustering and high-availability.
+The solution you choose will be based on the level of scalability and
+availability you require. The easiest solutions are scalable, but not necessarily
+highly available.
+
+## Architecture
+
+### Active/Passive
+
+For pure high-availability/failover with no scaling you can use an
+active/passive configuration. This utilizes DRBD (Distributed Replicated
+Block Device) to keep all data in sync. DRBD requires a low latency link to
+remain in sync. It is not advisable to attempt to run DRBD between data centers
+or in different cloud availability zones.
+
+Components/Servers Required:
+
+- 2 servers/virtual machines (one active/one passive)
+
+### Active/Active
+
+This architecture scales easily because all application servers handle
+user requests simultaneously. The database, Redis, and GitLab application are
+all deployed on separate servers. The configuration is **only** highly-available
+if the database, Redis and storage are also configured as such.
+
+**Steps to configure active/active:**
+
+1. [Configure the database](database.md)
+1. [Configure Redis](redis.md)
+1. [Configure NFS](nfs.md)
+1. [Configure the GitLab application servers](gitlab.md)
+1. [Configure the load balancers](load_balancer.md)
diff --git a/doc/administration/high_availability/database.md b/doc/administration/high_availability/database.md
new file mode 100644
index 00000000000..538dada1bae
--- /dev/null
+++ b/doc/administration/high_availability/database.md
@@ -0,0 +1,116 @@
+# Configuring a Database for GitLab HA
+
+You can choose to install and manage a database server (PostgreSQL/MySQL)
+yourself, or you can use GitLab Omnibus packages to help. GitLab recommends
+PostgreSQL. This is the database that will be installed if you use the
+Omnibus package to manage your database.
+
+## Configure your own database server
+
+If you're hosting GitLab on a cloud provider, you can optionally use a
+managed service for PostgreSQL. For example, AWS offers a managed Relational
+Database Service (RDS) that runs PostgreSQL.
+
+If you use a cloud-managed service, or provide your own PostgreSQL:
+
+1. Set up a `gitlab` username with a password of your choice. The `gitlab` user
+   needs privileges to create the `gitlabhq_production` database.
+1. Configure the GitLab application servers with the appropriate details.
+   This step is covered in [Configuring GitLab for HA](gitlab.md)
+
+## Configure using Omnibus
+
+1. Download/install GitLab Omnibus using **steps 1 and 2** from
+   [GitLab downloads](https://about.gitlab.com/downloads). Do not complete other
+   steps on the download page.
+1. Create/edit `/etc/gitlab/gitlab.rb` and use the following configuration.
+   Be sure to change the `external_url` to match your eventual GitLab front-end
+   URL.
+
+    ```ruby
+    external_url 'https://gitlab.example.com'
+
+    # Disable all components except PostgreSQL
+    postgresql['enable'] = true
+    bootstrap['enable'] = false
+    nginx['enable'] = false
+    unicorn['enable'] = false
+    sidekiq['enable'] = false
+    redis['enable'] = false
+    gitlab_workhorse['enable'] = false
+    mailroom['enable'] = false
+
+    # PostgreSQL configuration
+    postgresql['sql_password'] = 'DB password'
+    postgresql['md5_auth_cidr_addresses'] = ['0.0.0.0/0']
+    postgresql['listen_address'] = '0.0.0.0'
+    ```
+
+1. Run `sudo gitlab-ctl reconfigure` to install and configure PostgreSQL.
+
+    > **Note**: This `reconfigure` step will result in some errors.
+      That's OK - don't be alarmed.
+
+1. Open a database prompt:
+
+    ```
+    su - gitlab-psql
+    /bin/bash
+    psql -h /var/opt/gitlab/postgresql -d template1
+
+    # Output:
+
+    psql (9.2.15)
+    Type "help" for help.
+
+    template1=#
+    ```
+
+1. Run the following command at the database prompt and you will be asked to
+   enter the new password for the PostgreSQL superuser.
+
+    ```
+    \password
+
+    # Output:
+
+    Enter new password:
+    Enter it again:
+    ```
+
+1. Similarly, set the password for the `gitlab` database user. Use the same
+   password that you specified in the `/etc/gitlab/gitlab.rb` file for
+   `postgresql['sql_password']`.
+
+    ```
+    \password gitlab
+
+    # Output:
+
+    Enter new password:
+    Enter it again:
+    ```
+
+1. Enable the `pg_trgm` extension:
+    ```
+    CREATE EXTENSION pg_trgm;
+
+    # Output:
+
+    CREATE EXTENSION
+    ```
+1. Exit the database prompt by typing `\q` and Enter.
+1. Exit the `gitlab-psql` user by running `exit` twice.
+1. Run `sudo gitlab-ctl reconfigure` a final time.
+1. Run `touch /etc/gitlab/skip-auto-migrations` to prevent database migrations
+   from running on upgrade. Only the primary GitLab application server should
+   handle migrations.
+
+---
+
+Read more on high-availability configuration:
+
+1. [Configure Redis](redis.md)
+1. [Configure NFS](nfs.md)
+1. [Configure the GitLab application servers](gitlab.md)
+1. [Configure the load balancers](load_balancer.md)
diff --git a/doc/administration/high_availability/gitlab.md b/doc/administration/high_availability/gitlab.md
new file mode 100644
index 00000000000..8a881ce8863
--- /dev/null
+++ b/doc/administration/high_availability/gitlab.md
@@ -0,0 +1,131 @@
+# Configuring GitLab for HA
+
+Assuming you have already configured a database, Redis, and NFS, you can
+configure the GitLab application server(s) now. Complete the steps below
+for each GitLab application server in your environment.
+
+> **Note:** There is some additional configuration near the bottom for
+  secondary GitLab application servers. It's important to read and understand
+  these additional steps before proceeding with GitLab installation.
+
+1. If necessary, install the NFS client utility packages using the following
+   commands:
+
+    ```
+    # Ubuntu/Debian
+    apt-get install nfs-common
+
+    # CentOS/Red Hat
+    yum install nfs-utils nfs-utils-lib
+    ```
+
+1. Specify the necessary NFS shares. Mounts are specified in
+   `/etc/fstab`. The exact contents of `/etc/fstab` will depend on how you chose
+   to configure your NFS server. See [NFS documentation](nfs.md) for the various
+   options. Here is an example snippet to add to `/etc/fstab`:
+
+    ```
+    10.1.0.1:/var/opt/gitlab/.ssh /var/opt/gitlab/.ssh nfs defaults,soft,rsize=1048576,wsize=1048576,noatime,nobootwait,lookupcache=positive 0 2
+    10.1.0.1:/var/opt/gitlab/gitlab-rails/uploads /var/opt/gitlab/gitlab-rails/uploads nfs defaults,soft,rsize=1048576,wsize=1048576,noatime,nobootwait,lookupcache=positive 0 2
+    10.1.0.1:/var/opt/gitlab/gitlab-rails/shared /var/opt/gitlab/gitlab-rails/shared nfs defaults,soft,rsize=1048576,wsize=1048576,noatime,nobootwait,lookupcache=positive 0 2
+    10.1.0.1:/var/opt/gitlab/gitlab-ci/builds /var/opt/gitlab/gitlab-ci/builds nfs defaults,soft,rsize=1048576,wsize=1048576,noatime,nobootwait,lookupcache=positive 0 2
+    10.1.1.1:/var/opt/gitlab/git-data /var/opt/gitlab/git-data nfs defaults,soft,rsize=1048576,wsize=1048576,noatime,nobootwait,lookupcache=positive 0 2
+    ```
+
+1. Create the shared directories. These may be different depending on your NFS
+   mount locations.
+
+    ```
+    mkdir -p /var/opt/gitlab/.ssh /var/opt/gitlab/gitlab-rails/uploads /var/opt/gitlab/gitlab-rails/shared /var/opt/gitlab/gitlab-ci/builds /var/opt/gitlab/git-data
+    ```
+
+1. Download/install GitLab Omnibus using **steps 1 and 2** from
+   [GitLab downloads](https://about.gitlab.com/downloads). Do not complete other
+   steps on the download page.
+1. Create/edit `/etc/gitlab/gitlab.rb` and use the following configuration.
+   Be sure to change the `external_url` to match your eventual GitLab front-end
+   URL. Depending your the NFS configuration, you may need to change some GitLab
+   data locations. See [NFS documentation](nfs.md) for `/etc/gitlab/gitlab.rb`
+   configuration values for various scenarios. The example below assumes you've
+   added NFS mounts in the default data locations.
+    
+    ```ruby
+    external_url 'https://gitlab.example.com'
+
+    # Prevent GitLab from starting if NFS data mounts are not available
+    high_availability['mountpoint'] = '/var/opt/gitlab/git-data'
+    
+    # Disable components that will not be on the GitLab application server
+    postgresql['enable'] = false
+    redis['enable'] = false
+    
+    # PostgreSQL connection details
+    gitlab_rails['db_adapter'] = 'postgresql'
+    gitlab_rails['db_encoding'] = 'unicode'
+    gitlab_rails['db_host'] = '10.1.0.5' # IP/hostname of database server
+    gitlab_rails['db_password'] = 'DB password'
+    
+    # Redis connection details
+    gitlab_rails['redis_port'] = '6379'
+    gitlab_rails['redis_host'] = '10.1.0.6' # IP/hostname of Redis server
+    gitlab_rails['redis_password'] = 'Redis Password'
+    ```
+
+1. Run `sudo gitlab-ctl reconfigure` to compile the configuration.
+
+## Primary GitLab application server
+
+As a final step, run the setup rake task on the first GitLab application server.
+It is not necessary to run this on additional application servers.
+
+1. Initialize the database by running `sudo gitlab-rake gitlab:setup`.
+
+> **WARNING:** Only run this setup task on **NEW** GitLab instances because it
+  will wipe any existing data.
+
+> **Note:** When you specify `https` in the `external_url`, as in the example
+  above, GitLab assumes you have SSL certificates in `/etc/gitlab/ssl/`. If
+  certificates are not present, Nginx will fail to start. See
+  [Nginx documentation](http://docs.gitlab.com/omnibus/settings/nginx.html#enable-https)
+  for more information.
+
+## Additional configuration for secondary GitLab application servers
+
+Secondary GitLab servers (servers configured **after** the first GitLab server)
+need some additional configuration.
+
+1. Configure shared secrets. These values can be obtained from the primary
+   GitLab server in `/etc/gitlab/gitlab-secrets.json`. Add these to
+   `/etc/gitlab/gitlab.rb` **prior to** running the first `reconfigure` in
+   the steps above.
+
+    ```ruby
+    gitlab_shell['secret_token'] = 'fbfb19c355066a9afb030992231c4a363357f77345edd0f2e772359e5be59b02538e1fa6cae8f93f7d23355341cea2b93600dab6d6c3edcdced558fc6d739860'
+    gitlab_rails['secret_token'] = 'b719fe119132c7810908bba18315259ed12888d4f5ee5430c42a776d840a396799b0a5ef0a801348c8a357f07aa72bbd58e25a84b8f247a25c72f539c7a6c5fa'
+    gitlab_ci['secret_key_base'] = '6e657410d57c71b4fc3ed0d694e7842b1895a8b401d812c17fe61caf95b48a6d703cb53c112bc01ebd197a85da81b18e29682040e99b4f26594772a4a2c98c6d'
+    gitlab_ci['db_key_base'] = 'bf2e47b68d6cafaef1d767e628b619365becf27571e10f196f98dc85e7771042b9203199d39aff91fcb6837c8ed83f2a912b278da50999bb11a2fbc0fba52964'
+    ```
+
+1. Run `touch /etc/gitlab/skip-auto-migrations` to prevent database migrations
+   from running on upgrade. Only the primary GitLab application server should
+   handle migrations.
+
+## Troubleshooting
+
+- `mount: wrong fs type, bad option, bad superblock on`
+
+You have not installed the necessary NFS client utilities. See step 1 above.
+
+- `mount: mount point /var/opt/gitlab/... does not exist`
+
+This particular directory does not exist on the NFS server. Ensure
+the share is exported and exists on the NFS server and try to remount.
+
+---
+
+Read more on high-availability configuration:
+
+1. [Configure the database](database.md)
+1. [Configure Redis](redis.md)
+1. [Configure NFS](nfs.md)
+1. [Configure the load balancers](load_balancer.md)
diff --git a/doc/administration/high_availability/load_balancer.md b/doc/administration/high_availability/load_balancer.md
new file mode 100644
index 00000000000..b1fe34ed9a1
--- /dev/null
+++ b/doc/administration/high_availability/load_balancer.md
@@ -0,0 +1,63 @@
+# Load Balancer for GitLab HA
+
+In an active/active GitLab configuration, you will need a load balancer to route
+traffic to the application servers. The specifics on which load balancer to use
+or the exact configuration is beyond the scope of GitLab documentation. We hope
+that if you're managing HA systems like GitLab you have a load balancer of
+choice already. Some examples including HAProxy (open-source), F5 Big-IP LTM,
+and Citrix Net Scaler. This documentation will outline what ports and protocols
+you need to use with GitLab.
+
+## Basic ports
+
+| LB Port | Backend Port | Protocol   |
+| ------- | ------------ | --------   |
+| 80      | 80           | HTTP       |
+| 443     | 443          | HTTPS [^1] |
+| 22      | 22           | TCP        |
+
+## GitLab Pages Ports
+
+If you're using GitLab Pages you will need some additional port configurations.
+GitLab Pages requires a separate VIP. Configure DNS to point the
+`pages_external_url` from `/etc/gitlab/gitlab.rb` at the new VIP. See the
+[GitLab Pages documentation][gitlab-pages] for more information.
+
+| LB Port | Backend Port | Protocol |
+| ------- | ------------ | -------- |
+| 80      | Varies [^2]  | HTTP     |
+| 443     | Varies [^2]  | TCP [^3] |
+
+## Alternate SSH Port
+
+Some organizations have policies against opening SSH port 22. In this case,
+it may be helpful to configure an alternate SSH hostname that allows users
+to use SSH on port 443. An alternate SSH hostname will require a new VIP
+compared to the other GitLab HTTP configuration above.
+
+Configure DNS for an alternate SSH hostname such as altssh.gitlab.example.com.
+
+| LB Port | Backend Port | Protocol |
+| ------- | ------------ | -------- |
+| 443     | 22           | TCP      |
+
+---
+
+Read more on high-availability configuration:
+
+1. [Configure the database](database.md)
+1. [Configure Redis](redis.md)
+1. [Configure NFS](nfs.md)
+1. [Configure the GitLab application servers](gitlab.md)
+
+[^1]: When using HTTPS protocol for port 443, you will need to add an SSL
+      certificate to the load balancers. If you wish to terminate SSL at the
+      GitLab application server instead, use TCP protocol.
+[^2]: The backend port for GitLab Pages depends on the
+      `gitlab_pages['external_http']` and `gitlab_pages['external_https']`
+      setting. See [GitLab Pages documentation][gitlab-pages] for more details.
+[^3]: Port 443 for GitLab Pages should always use the TCP protocol. Users can
+      configure custom domains with custom SSL, which would not be possible
+      if SSL was terminated at the load balancer.
+
+[gitlab-pages]: http://doc.gitlab.com/ee/pages/administration.html
diff --git a/doc/administration/high_availability/nfs.md b/doc/administration/high_availability/nfs.md
new file mode 100644
index 00000000000..e4e124e200a
--- /dev/null
+++ b/doc/administration/high_availability/nfs.md
@@ -0,0 +1,116 @@
+# NFS
+
+## Required NFS Server features
+
+**File locking**: GitLab **requires** file locking which is only supported
+natively in NFS version 4. NFSv3 also supports locking as long as
+Linux Kernel 2.6.5+ is used. We recommend using version 4 and do not
+specifically test NFSv3.
+
+**no_root_squash**: NFS normally changes the `root` user to `nobody`. This is
+a good security measure when NFS shares will be accessed by many different
+users. However, in this case only GitLab will use the NFS share so it
+is safe. GitLab requires the `no_root_squash` setting because we need to
+manage file permissions automatically. Without the setting you will receive
+errors when the Omnibus package tries to alter permissions. Note that GitLab
+and other bundled components do **not** run as `root` but as non-privileged
+users. The requirement for `no_root_squash` is to allow the Omnibus package to
+set ownership and permissions on files, as needed.
+
+### Recommended options
+
+When you define your NFS exports, we recommend you also add the following
+options:
+
+- `sync` - Force synchronous behavior. Default is asynchronous and under certain
+  circumstances it could lead to data loss if a failure occurs before data has
+  synced.
+
+## Client mount options
+
+Below is an example of an NFS mount point we use on GitLab.com:
+
+```
+10.1.1.1:/var/opt/gitlab/git-data /var/opt/gitlab/git-data nfs4 defaults,soft,rsize=1048576,wsize=1048576,noatime,nobootwait,lookupcache=positive 0 2
+```
+
+Notice several options that you should consider using:
+
+| Setting | Description |
+| ------- | ----------- |
+| `nobootwait` | Don't halt boot process waiting for this mount to become available
+| `lookupcache=positive` | Tells the NFS client to honor `positive` cache results but invalidates any `negative` cache results. Negative cache results cause problems with Git. Specifically, a `git push` can fail to register uniformly across all NFS clients. The negative cache causes the clients to 'remember' that the files did not exist previously.
+
+## Mount locations
+
+When using default Omnibus configuration you will need to share 5 data locations
+between all GitLab cluster nodes. No other locations should be shared. The
+following are the 5 locations you need to mount:
+
+| Location | Description |
+| -------- | ----------- |
+| `/var/opt/gitlab/git-data` | Git repository data. This will account for a large portion of your data
+| `/var/opt/gitlab/.ssh` | SSH `authorized_keys` file and keys used to import repositories from some other Git services
+| `/var/opt/gitlab/gitlab-rails/uploads` | User uploaded attachments
+| `/var/opt/gitlab/gitlab-rails/shared` | Build artifacts, GitLab Pages, LFS objects, temp files, etc. If you're using LFS this may also account for a large portion of your data
+| `/var/opt/gitlab/gitlab-ci/builds` | GitLab CI build traces
+
+Other GitLab directories should not be shared between nodes. They contain
+node-specific files and GitLab code that does not need to be shared. To ship
+logs to a central location consider using remote syslog. GitLab Omnibus packages
+provide configuration for [UDP log shipping][udp-log-shipping].
+
+### Consolidating mount points
+
+If you don't want to configure 5-6 different NFS mount points, you have a few
+alternative options.
+
+#### Change default file locations
+
+Omnibus allows you to configure the file locations. With custom configuration
+you can specify just one main mountpoint and have all of these locations
+as subdirectories. Mount `/gitlab-data` then use the following Omnibus
+configuration to move each data location to a subdirectory:
+
+```ruby
+user['home'] = '/gitlab-data/home'
+git_data_dir '/gitlab-data/git-data'
+gitlab_rails['shared_path'] = '/gitlab-data/shared'
+gitlab_rails['uploads_directory'] = "/gitlab-data/uploads"
+gitlab_ci['builds_directory'] = '/gitlab-data/builds'
+```
+
+To move the `git` home directory, all GitLab services must be stopped. Run
+`gitlab-ctl stop && initctl stop gitlab-runsvdir`. Then continue with the
+reconfigure.
+
+Run `sudo gitlab-ctl reconfigure` to start using the central location. Please
+be aware that if you had existing data you will need to manually copy/rsync it
+to these new locations and then restart GitLab.
+
+#### Bind mounts
+
+Bind mounts provide a way to specify just one NFS mount and then
+bind the default GitLab data locations to the NFS mount. Start by defining your
+single NFS mount point as you normally would in `/etc/fstab`. Let's assume your
+NFS mount point is `/gitlab-data`. Then, add the following bind mounts in
+`/etc/fstab`:
+
+```bash
+/gitlab-data/git-data /var/opt/gitlab/git-data none bind 0 0
+/gitlab-data/.ssh /var/opt/gitlab/.ssh none bind 0 0
+/gitlab-data/uploads /var/opt/gitlab/gitlab-rails/uploads none bind 0 0
+/gitlab-data/shared /var/opt/gitlab/gitlab-rails/shared none bind 0 0
+/gitlab-data/builds /var/opt/gitlab/gitlab-ci/builds none bind 0 0
+```
+
+---
+
+Read more on high-availability configuration:
+
+1. [Configure the database](database.md)
+1. [Configure Redis](redis.md)
+1. [Configure the GitLab application servers](gitlab.md)
+1. [Configure the load balancers](load_balancer.md)
+
+[udp-log-shipping]: http://doc.gitlab.com/omnibus/settings/logs.html#udp-log-shipping-gitlab-enterprise-edition-only "UDP log shipping"
diff --git a/doc/administration/high_availability/redis.md b/doc/administration/high_availability/redis.md
new file mode 100644
index 00000000000..d89a1e582ca
--- /dev/null
+++ b/doc/administration/high_availability/redis.md
@@ -0,0 +1,62 @@
+# Configuring Redis for GitLab HA
+
+You can choose to install and manage Redis yourself, or you can use GitLab
+Omnibus packages to help.
+
+## Configure your own Redis server
+
+If you're hosting GitLab on a cloud provider, you can optionally use a
+managed service for Redis. For example, AWS offers a managed ElastiCache service
+that runs Redis.
+
+> **Note:** Redis does not require authentication by default. See
+  [Redis Security](http://redis.io/topics/security) documentation for more
+  information. We recommend using a combination of a Redis password and tight
+  firewall rules to secure your Redis service.
+
+## Configure using Omnibus
+
+1. Download/install GitLab Omnibus using **steps 1 and 2** from
+   [GitLab downloads](https://about.gitlab.com/downloads). Do not complete other
+   steps on the download page.
+1. Create/edit `/etc/gitlab/gitlab.rb` and use the following configuration.
+   Be sure to change the `external_url` to match your eventual GitLab front-end
+   URL.
+
+    ```ruby
+      external_url 'https://gitlab.example.com'
+
+      # Disable all components except PostgreSQL
+      redis['enable'] = true
+      bootstrap['enable'] = false
+      nginx['enable'] = false
+      unicorn['enable'] = false
+      sidekiq['enable'] = false
+      postgresql['enable'] = false
+      gitlab_workhorse['enable'] = false
+      mailroom['enable'] = false
+
+      # Redis configuration
+      redis['port'] = 6379
+      redis['bind'] = '0.0.0.0'
+
+      # If you wish to use Redis authentication (recommended)
+      redis['password'] = 'Redis Password'
+    ```
+
+1. Run `sudo gitlab-ctl reconfigure` to install and configure PostgreSQL.
+
+    > **Note**: This `reconfigure` step will result in some errors.
+      That's OK - don't be alarmed.
+1. Run `touch /etc/gitlab/skip-auto-migrations` to prevent database migrations
+   from running on upgrade. Only the primary GitLab application server should
+   handle migrations.
+
+---
+
+Read more on high-availability configuration:
+
+1. [Configure the database](database.md)
+1. [Configure NFS](nfs.md)
+1. [Configure the GitLab application servers](gitlab.md)
+1. [Configure the load balancers](load_balancer.md)
diff --git a/doc/administration/repository_checks.md b/doc/administration/repository_checks.md
index 61bf8ce6161..3411e4af6a7 100644
--- a/doc/administration/repository_checks.md
+++ b/doc/administration/repository_checks.md
@@ -1,7 +1,8 @@
 # Repository checks
 
 >**Note:**
-This feature was [introduced][ce-3232] in GitLab 8.7.
+This feature was [introduced][ce-3232] in GitLab 8.7. It is OFF by
+default because it still causes too many false alarms.
 
 Git has a built-in mechanism, [git fsck][git-fsck], to verify the
 integrity of all data commited to a repository. GitLab administrators
diff --git a/doc/administration/troubleshooting/sidekiq.md b/doc/administration/troubleshooting/sidekiq.md
new file mode 100644
index 00000000000..134a7583762
--- /dev/null
+++ b/doc/administration/troubleshooting/sidekiq.md
@@ -0,0 +1,162 @@
+# Troubleshooting Sidekiq
+
+Sidekiq is the background job processor GitLab uses to asynchronously run
+tasks. When things go wrong it can be difficult to troubleshoot. These
+situations also tend to be high-pressure because a production system job queue
+may be filling up. Users will notice when this happens because new branches
+may not show up and merge requests may not be updated. The following are some
+troubleshooting steps that will help you diagnose the bottleneck.
+
+> **Note:** GitLab administrators/users should consider working through these
+debug steps with GitLab Support so the backtraces can be analyzed by our team.
+It may reveal a bug or necessary improvement in GitLab.
+
+> **Note:** In any of the backtraces, be weary of suspecting cases where every
+  thread appears to be waiting in the database, Redis, or waiting to acquire
+  a mutex. This **may** mean there's contention in the database, for example,
+  but look for one thread that is different than the rest. This other thread
+  may be using all available CPU, or have a Ruby Global Interpreter Lock,
+  preventing other threads from continuing.
+
+## Thread dump
+
+Send the Sidekiq process ID the `TTIN` signal and it will output thread
+backtraces in the log file.
+
+```
+kill -TTIN <sidekiq_pid>
+```
+
+Check in `/var/log/gitlab/sidekiq/current` or `$GITLAB_HOME/log/sidekiq.log` for
+the backtrace output. The backtraces will be lengthy and generally start with
+several `WARN` level messages. Here's an example of a single thread's backtrace:
+
+```
+2016-04-13T06:21:20.022Z 31517 TID-orn4urby0 WARN: ActiveRecord::RecordNotFound: Couldn't find Note with 'id'=3375386
+2016-04-13T06:21:20.022Z 31517 TID-orn4urby0 WARN: /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activerecord-4.2.5.2/lib/active_record/core.rb:155:in `find'
+/opt/gitlab/embedded/service/gitlab-rails/app/workers/new_note_worker.rb:7:in `perform'
+/opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/sidekiq-4.0.1/lib/sidekiq/processor.rb:150:in `execute_job'
+/opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/sidekiq-4.0.1/lib/sidekiq/processor.rb:132:in `block (2 levels) in process'
+/opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/sidekiq-4.0.1/lib/sidekiq/middleware/chain.rb:127:in `block in invoke'
+/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/sidekiq_middleware/memory_killer.rb:17:in `call'
+/opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/sidekiq-4.0.1/lib/sidekiq/middleware/chain.rb:129:in `block in invoke'
+/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/sidekiq_middleware/arguments_logger.rb:6:in `call'
+...
+```
+
+In some cases Sidekiq may be hung and unable to respond to the `TTIN` signal.
+Move on to other troubleshooting methods if this happens.
+
+## Process profiling with `perf`
+
+Linux has a process profiling tool called `perf` that is helpful when a certain
+process is eating up a lot of CPU. If you see high CPU usage and Sidekiq won't
+respond to the `TTIN` signal, this is a good next step.
+
+If `perf` is not installed on your system, install it with `apt-get` or `yum`:
+
+```
+# Debian
+sudo apt-get install linux-tools
+
+# Ubuntu (may require these additional Kernel packages)
+sudo apt-get install linux-tools-common linux-tools-generic linux-tools-`uname -r`
+
+# Red Hat/CentOS
+sudo yum install perf
+```
+
+Run perf against the Sidekiq PID:
+
+```
+sudo perf record -p <sidekiq_pid>
+```
+
+Let this run for 30-60 seconds and then press Ctrl-C. Then view the perf report:
+
+```
+sudo perf report
+
+# Sample output
+Samples: 348K of event 'cycles', Event count (approx.): 280908431073
+ 97.69%            ruby  nokogiri.so         [.] xmlXPathNodeSetMergeAndClear
+  0.18%            ruby  libruby.so.2.1.0    [.] objspace_malloc_increase
+  0.12%            ruby  libc-2.12.so        [.] _int_malloc
+  0.10%            ruby  libc-2.12.so        [.] _int_free
+```
+
+Above you see sample output from a perf report. It shows that 97% of the CPU is
+being spent inside Nokogiri and `xmlXPathNodeSetMergeAndClear`. For something
+this obvious you should then go investigate what job in GitLab would use
+Nokogiri and XPath. Combine with `TTIN` or `gdb` output to show the
+corresponding Ruby code where this is happening.
+
+## The GNU Project Debugger (gdb)
+
+`gdb` can be another effective tool for debugging Sidekiq. It gives you a little
+more interactive way to look at each thread and see what's causing problems.
+
+> **Note:** Attaching to a process with `gdb` will suspends the normal operation
+  of the process (Sidekiq will not process jobs while `gdb` is attached).
+
+Start by attaching to the Sidekiq PID:
+
+```
+gdb -p <sidekiq_pid>
+```
+
+Then gather information on all the threads:
+
+```
+info threads
+
+# Example output
+30 Thread 0x7fe5fbd63700 (LWP 26060) 0x0000003f7cadf113 in poll () from /lib64/libc.so.6
+29 Thread 0x7fe5f2b3b700 (LWP 26533) 0x0000003f7ce0b68c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
+28 Thread 0x7fe5f2a3a700 (LWP 26534) 0x0000003f7ce0ba5e in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
+27 Thread 0x7fe5f2939700 (LWP 26535) 0x0000003f7ce0b68c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
+26 Thread 0x7fe5f2838700 (LWP 26537) 0x0000003f7ce0b68c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
+25 Thread 0x7fe5f2737700 (LWP 26538) 0x0000003f7ce0b68c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
+24 Thread 0x7fe5f2535700 (LWP 26540) 0x0000003f7ce0b68c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
+23 Thread 0x7fe5f2434700 (LWP 26541) 0x0000003f7ce0b68c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
+22 Thread 0x7fe5f2232700 (LWP 26543) 0x0000003f7ce0b68c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
+21 Thread 0x7fe5f2131700 (LWP 26544) 0x00007fe5f7b570f0 in xmlXPathNodeSetMergeAndClear ()
+from /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/nokogiri-1.6.7.2/lib/nokogiri/nokogiri.so
+...
+```
+
+If you see a suspicious thread, like the Nokogiri one above, you may want
+to get more information:
+
+```
+thread 21
+bt
+
+# Example output
+#0  0x00007ff0d6afe111 in xmlXPathNodeSetMergeAndClear () from /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/nokogiri-1.6.7.2/lib/nokogiri/nokogiri.so
+#1  0x00007ff0d6b0b836 in xmlXPathNodeCollectAndTest () from /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/nokogiri-1.6.7.2/lib/nokogiri/nokogiri.so
+#2  0x00007ff0d6b09037 in xmlXPathCompOpEval () from /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/nokogiri-1.6.7.2/lib/nokogiri/nokogiri.so
+#3  0x00007ff0d6b09017 in xmlXPathCompOpEval () from /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/nokogiri-1.6.7.2/lib/nokogiri/nokogiri.so
+#4  0x00007ff0d6b092e0 in xmlXPathCompOpEval () from /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/nokogiri-1.6.7.2/lib/nokogiri/nokogiri.so
+#5  0x00007ff0d6b0bc37 in xmlXPathRunEval () from /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/nokogiri-1.6.7.2/lib/nokogiri/nokogiri.so
+#6  0x00007ff0d6b0be5f in xmlXPathEvalExpression () from /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/nokogiri-1.6.7.2/lib/nokogiri/nokogiri.so
+#7  0x00007ff0d6a97dc3 in evaluate (argc=2, argv=0x1022d058, self=<value optimized out>) at xml_xpath_context.c:221
+#8  0x00007ff0daeab0ea in vm_call_cfunc_with_frame (th=0x1022a4f0, reg_cfp=0x1032b810, ci=<value optimized out>) at vm_insnhelper.c:1510
+```
+
+To output a backtrace from all threads at once:
+
+```
+apply all thread bt
+```
+
+## Check for blocking queries
+
+Sometimes the speed at which Sidekiq processes jobs can be so fast that it can
+cause database contention. Check for blocking queries when backtraces above
+show that many threads are stuck in the database adapter.
+
+The PostgreSQL wiki has details on the query you can run to see blocking
+queries. The query is different based on PostgreSQL version. See
+[Lock Monitoring](https://wiki.postgresql.org/wiki/Lock_Monitoring) for
+the query details.
diff --git a/doc/downgrade_ee_to_ce/README.md b/doc/downgrade_ee_to_ce/README.md
new file mode 100644
index 00000000000..3625c4191b8
--- /dev/null
+++ b/doc/downgrade_ee_to_ce/README.md
@@ -0,0 +1,82 @@
+# Downgrading from EE to CE
+
+If you ever decide to downgrade your Enterprise Edition back to the Community
+Edition, there are a few steps you need take before installing the CE package
+on top of the current EE package, or, if you are in an installation from source,
+before you change remotes and fetch the latest CE code.
+
+## Disable Enterprise-only features
+
+First thing to do is to disable the following features.
+
+### Authentication mechanisms
+
+Kerberos and Atlassian Crowd are only available on the Enterprise Edition, so
+you should disable these mechanisms before downgrading and you should provide
+alternative authentication methods to your users.
+
+### Git Annex
+
+Git Annex is also only available on the Enterprise Edition. This means that if
+you have repositories that use Git Annex to store large files, these files will
+no longer be easily available via Git. You should consider migrating these
+repositories to use Git LFS before downgrading to the Community Edition.
+
+### Remove Jenkins CI Service entries from the database
+
+The `JenkinsService` class is only available on the Enterprise Edition codebase,
+so if you downgrade to the Community Edition, you'll come across the following
+error:
+
+```
+Completed 500 Internal Server Error in 497ms (ActiveRecord: 32.2ms)
+
+ActionView::Template::Error (The single-table inheritance mechanism failed to locate the subclass: 'JenkinsService'. This
+error is raised because the column 'type' is reserved for storing the class in case of inheritance. Please rename this
+column if you didn't intend it to be used for storing the inheritance class or overwrite Service.inheritance_column to
+use another column for that information.)
+```
+
+All services are created automatically for every project you have, so in order
+to avoid getting this error, you need to remove all instances of the
+`JenkinsService` from your database:
+
+**Omnibus Installation**
+
+```
+$ sudo gitlab-rails runner "Service.where(type: 'JenkinsService').delete_all"
+```
+
+**Source Installation**
+
+```
+$ bundle exec rails runner "Service.where(type: 'JenkinsService').delete_all" production
+```
+
+## Downgrade to CE
+
+After performing the above mentioned steps, you are now ready to downgrade your
+GitLab installation to the Community Edition.
+
+**Omnibus Installation**
+
+To downgrade an Omnibus installation, it is sufficient to install the Community
+Edition package on top of the currently installed one. You can do this manually,
+by directly [downloading the package](https://packages.gitlab.com/gitlab/gitlab-ce)
+you need, or by adding our CE package repository and following the
+[CE installation instructions](https://about.gitlab.com/downloads/).
+
+**Source Installation**
+
+To downgrade a source installation, you need to replace the current remote of
+your GitLab installation with the Community Edition's remote, fetch the latest
+changes, and checkout the latest stable branch:
+
+```
+$ git remote set-url origin git@gitlab.com:gitlab-org/gitlab-ce.git
+$ git fetch --all
+$ git checkout 8-x-stable
+```
+
+Remember to follow the correct [update guides](../update/README.md) to make
+sure all dependencies are up to date.
diff --git a/doc/update/8.6-to-8.7.md b/doc/update/8.6-to-8.7.md
index 8599133a726..4a2c6ea91d2 100644
--- a/doc/update/8.6-to-8.7.md
+++ b/doc/update/8.6-to-8.7.md
@@ -45,8 +45,8 @@ sudo -u git -H git checkout 8-7-stable-ee
 
 ```bash
 cd /home/git/gitlab-shell
-sudo -u git -H git fetch --all
-sudo -u git -H git checkout v2.7.0
+sudo -u git -H git fetch --all --tags
+sudo -u git -H git checkout v2.7.2
 ```
 
 ### 5. Update gitlab-workhorse
diff --git a/doc/update/README.md b/doc/update/README.md
index 0241f036830..a770633c9b8 100644
--- a/doc/update/README.md
+++ b/doc/update/README.md
@@ -1,18 +1,95 @@
-Depending on the installation method and your GitLab version, there are multiple update guides. Choose one that fits your needs.
+# Updating GitLab
+
+Depending on the installation method and your GitLab version, there are multiple
+update guides.
+
+There are currently 3 official ways to install GitLab:
+
+- Omnibus packages
+- Source installation
+- Docker installation
+
+Based on your installation, choose a section below that fits your needs.
+
+---
+
+<!-- START doctoc generated TOC please keep comment here to allow auto update -->
+<!-- DON'T EDIT THIS SECTION, INSTEAD RE-RUN doctoc TO UPDATE -->
+**Table of Contents**  *generated with [DocToc](https://github.com/thlorenz/doctoc)*
+
+- [Omnibus Packages](#omnibus-packages)
+- [Installation from source](#installation-from-source)
+- [Installation using Docker](#installation-using-docker)
+- [Upgrading between editions](#upgrading-between-editions)
+    - [Community to Enterprise Edition](#community-to-enterprise-edition)
+    - [Enterprise to Community Edition](#enterprise-to-community-edition)
+- [Miscellaneous](#miscellaneous)
+
+<!-- END doctoc generated TOC please keep comment here to allow auto update -->
 
 ## Omnibus Packages
 
-- [Omnibus update guide](https://gitlab.com/gitlab-org/omnibus-gitlab/blob/master/doc/update.md) contains the steps needed to update a GitLab [package](https://about.gitlab.com/downloads/).
+- The [Omnibus update guide](http://doc.gitlab.com/omnibus/update/README.html)
+  contains the steps needed to update an Omnibus GitLab package.
 
 ## Installation from source
 
-- [The individual upgrade guides](https://gitlab.com/gitlab-org/gitlab-ce/tree/master/doc/update) are for those who have installed GitLab from source.
-- [The CE to EE update guides](https://gitlab.com/subscribers/gitlab-ee/tree/master/doc/update) are for subscribers of the Enterprise Edition only. The steps are very similar to a version upgrade: stop the server, get the code, update config files for the new functionality, install libs and do migrations, update the init script, start the application and check the application status.
-- [Upgrader](upgrader.md) is an automatic ruby script that performs the update for installations from source.
-- [Patch versions](patch_versions.md) guide includes the steps needed for a patch version, eg. 6.2.0 to 6.2.1.
+- [Upgrading Community Edition from source][source-ce] - The individual
+  upgrade guides are for those who have installed GitLab CE from source.
+- [Upgrading Enterprise Edition from source][source-ee] - The individual
+  upgrade guides are for those who have installed GitLab EE from source.
+- [Patch versions](patch_versions.md) guide includes the steps needed for a
+  patch version, eg. 6.2.0 to 6.2.1, and apply to both Community and Enterprise
+  Editions.
+
+## Installation using Docker
+
+GitLab provides official Docker images for both Community and Enterprise
+editions. They are based on the Omnibus package and instructions on how to
+update them are in [a separate document][omnidocker].
+
+## Upgrading between editions
+
+GitLab comes in two flavors: [Community Edition][ce] which is MIT licensed,
+and [Enterprise Edition][ee] which builds on top of the Community Edition and
+includes extra features mainly aimed at organizations with more than 100 users.
+
+Below you can find some guides to help you change editions easily.
+
+### Community to Enterprise Edition
+
+>**Note:**
+The following guides are for subscribers of the Enterprise Edition only.
+
+If you wish to upgrade your GitLab installation from Community to Enterprise
+Edition, follow the guides below based on the installation method:
+
+- [Source CE to EE update guides][source-ee] - Find your version, and follow the
+  `-ce-to-ee.md` guide. The steps are very similar to a version upgrade: stop
+  the server, get the code, update config files for the new functionality,
+  install libraries and do migrations, update the init script, start the
+  application and check its status.
+- [Omnibus CE to EE][omni-ce-ee] - Follow this guide to update your Omnibus
+  GitLab Community Edition to the Enterprise Edition.
+
+### Enterprise to Community Edition
+
+If you need to downgrade your Enterprise Edition installation back to Community
+Edition, you can follow [this guide][ee-ce] to make the process as smooth as
+possible.
 
 ## Miscellaneous
 
-- [MySQL to PostgreSQL](mysql_to_postgresql.md) guides you through migrating your database from MySQL to PostgreSQL.
-- [MySQL installation guide](https://gitlab.com/gitlab-org/gitlab-ce/blob/master/doc/install/database_mysql.md) contains additional information about configuring GitLab to work with a MySQL database.
+- [MySQL to PostgreSQL](mysql_to_postgresql.md) guides you through migrating
+  your database from MySQL to PostgreSQL.
+- [MySQL installation guide](../install/database_mysql.md) contains additional
+  information about configuring GitLab to work with a MySQL database.
 - [Restoring from backup after a failed upgrade](restore_after_failure.md)
+
+[omnidocker]: http://doc.gitlab.com/omnibus/docker/README.html
+[source-ee]: https://gitlab.com/gitlab-org/gitlab-ee/tree/master/doc/update
+[source-ce]: https://gitlab.com/gitlab-org/gitlab-ce/tree/master/doc/update
+[ee-ce]: ../downgrade_ee_to_ce/README.md
+[ce]: https://about.gitlab.com/features/#community
+[ee]: https://about.gitlab.com/features/#enterprise
+[omni-ce-ee]: http://doc.gitlab.com/omnibus/update/README.html#from-community-edition-to-enterprise-edition
diff --git a/doc/workflow/cherry_pick_changes.md b/doc/workflow/cherry_pick_changes.md
index b0ca0879643..4a499009842 100644
--- a/doc/workflow/cherry_pick_changes.md
+++ b/doc/workflow/cherry_pick_changes.md
@@ -1,6 +1,7 @@
 # Cherry-pick changes
 
-_**Note:** This feature was [introduced][ce-3514] in GitLab 8.7._
+>**Note:**
+This feature was [introduced][ce-3514] in GitLab 8.7.
 
 ---
 
diff --git a/doc/workflow/importing/import_projects_from_github.md b/doc/workflow/importing/import_projects_from_github.md
index f693f430a42..e670e415c71 100644
--- a/doc/workflow/importing/import_projects_from_github.md
+++ b/doc/workflow/importing/import_projects_from_github.md
@@ -1,7 +1,8 @@
 # Import your project from GitHub to GitLab

 

-_**Note:** In order to enable the GitHub import setting, you should first

-enable the [GitHub integration][gh-import] in your GitLab instance._

+>**Note:**

+In order to enable the GitHub import setting, you should first

+enable the [GitHub integration][gh-import] in your GitLab instance.

 

 At its current state, GitHub importer can import:

 

@@ -10,10 +11,13 @@ At its current state, GitHub importer can import:
 - the issues (introduced in GitLab 7.7)

 - the pull requests (introduced in GitLab 8.4)

 - the wiki pages (introduced in GitLab 8.4)

+- the milestones (introduced in GitLab 8.7)

+- the labels (introduced in GitLab 8.7)

 

-It is not yet possible to import your labels, milestones and cross-repository

-pull requests (those from forks). We are working on improving this in the near

-future.

+With GitLab 8.7+, references to pull requests and issues are preserved.

+

+It is not yet possible to import your cross-repository pull requests (those from

+forks). We are working on improving this in the near future.

 

 The importer page is visible when you [create a new project][new-project].

 Click on the **GitHub** link and you will be redirected to GitHub for