Merge remote-tracking branch 'upstream/master' into 32815--Add-Custom-CI-Config-Path

* upstream/master: (123 commits)
  Backport changes to Projects::IssuesController and the search bar
  bugfix: use `require_dependency` to bring in DeclarativePolicy
  Resolve "Select branch dropdown is too close to branch name"
  Clean up issuable lists
  Defer project destroys within a namespace in Groups::DestroyService#async_execute
  Fixed new navgiation bar logo height in Safari
  Resolve "Issue dropdown persists when adding issue number to issue description"
  Move verification to block level instead of paragraph
  Revert "Merge branch 'dm-drop-default-scope-on-sortable-finders' into 'master'"
  Added code for defining SHA attributes
  Minor edits
  Job details won't scroll horizontally to show long lines
  Run mysql tests on stable preperation branches like 9-3-stable-patch-2
  Bring back branches badge to main project page
  optimize translation content based on comments
  supplement traditional chinese in taiwan translation
  Inserts exact matches of username, email and name to the top of the user search list
  Remove Namespace model default scope override and write additional test to Project search
  optimize translation content based on comments
  Limit OpenGraph image size to 64x64
  ...

10 files changed, 200 insertions, 18 deletions

diff --git a/doc/README.md b/doc/README.md
index ab8ea192a26..fa755852304 100644
--- a/doc/README.md
+++ b/doc/README.md
@@ -1,12 +1,24 @@
-# GitLab Community Edition
+# GitLab Documentation
 
-[GitLab](https://about.gitlab.com/) is a Git-based fully featured platform
-for software development.
+Welcome to [GitLab](https://about.gitlab.com/), a Git-based fully featured
+platform for software development!
 
-**GitLab Community Edition (CE)** is an opensource product, self-hosted, free to use.
-All [GitLab products](https://about.gitlab.com/products/) contain the features
-available in GitLab CE. Premium features are available in
-[GitLab Enterprise Edition (EE)](https://about.gitlab.com/gitlab-ee/).
+We offer four different products for you and your company:
+
+- **GitLab Community Edition (CE)** is an [opensource product](https://gitlab.com/gitlab-org/gitlab-ce/),
+self-hosted, free to use. Every feature available in GitLab CE is also available on GitLab Enterprise Edition (Starter and Premium) and GitLab.com.
+- **GitLab Enterprise Edition (EE)** is an [opencore product](https://gitlab.com/gitlab-org/gitlab-ee/),
+self-hosted, fully featured solution of GitLab, available under distinct [subscriptions](https://about.gitlab.com/products/): **GitLab Enterprise Edition Starter (EES)** and **GitLab Enterprise Edition Premium (EEP)**.
+- **GitLab.com**: SaaS GitLab solution, with [free and paid subscriptions](https://about.gitlab.com/gitlab-com/). GitLab.com is hosted by GitLab, Inc., and administrated by GitLab (users don't have access to admin settings).
+
+**GitLab EE** contains all features available in **GitLab CE**,
+plus premium features available in each version: **Enterprise Edition Starter**
+(**EES**) and **Enterprise Edition Premium** (**EEP**). Everything available in
+**EES** is also available in **EEP**.
+
+**Note:** _We are unifying the documentation for CE and EE. To check if certain feature is
+available in CE or EE, look for a note right below the page title containing the GitLab
+version which introduced that feature._
 
 ----
 
@@ -125,7 +137,7 @@ have access to GitLab administration tools and settings.
 - [Access restrictions](user/admin_area/settings/visibility_and_access_controls.md#enabled-git-access-protocols): Define which Git access protocols can be used to talk to GitLab
 - [Authentication/Authorization](topics/authentication/index.md#gitlab-administrators): Enforce 2FA, configure external authentication with LDAP, SAML, CAS and additional Omniauth providers.
 
-### GitLab admins' superpowers
+### Features
 
 - [Container Registry](administration/container_registry.md): Configure Docker Registry with GitLab.
 - [Custom Git hooks](administration/custom_hooks.md): Custom Git hooks (on the filesystem) for when webhooks aren't enough.
diff --git a/doc/api/features.md b/doc/api/features.md
index 89b8d3ac948..558869255cc 100644
--- a/doc/api/features.md
+++ b/doc/api/features.md
@@ -58,6 +58,10 @@ POST /features/:name
 | --------- | ---- | -------- | ----------- |
 | `name` | string | yes | Name of the feature to create or update |
 | `value` | integer/string | yes | `true` or `false` to enable/disable, or an integer for percentage of time |
+| `feature_group` | string | no | A Feature group name |
+| `user` | string | no | A GitLab username |
+
+Note that `feature_group` and `user` are mutually exclusive.
 
 ```bash
 curl --data "value=30" --header "PRIVATE-TOKEN: 9koXpg98eAheJpvBs5tK" https://gitlab.example.com/api/v4/features/new_library
diff --git a/doc/api/namespaces.md b/doc/api/namespaces.md
index 4ad6071a0ed..8133251dffe 100644
--- a/doc/api/namespaces.md
+++ b/doc/api/namespaces.md
@@ -29,22 +29,30 @@ Example response:
   {
     "id": 1,
     "path": "user1",
-    "kind": "user"
+    "kind": "user",
+    "full_path": "user1"
   },
   {
     "id": 2,
     "path": "group1",
-    "kind": "group"
+    "kind": "group",
+    "full_path": "group1",
+    "parent_id": "null",
+    "members_count_with_descendants": 2
   },
   {
     "id": 3,
     "path": "bar",
     "kind": "group",
     "full_path": "foo/bar",
+    "parent_id": "9",
+    "members_count_with_descendants": 5
   }
 ]
 ```
 
+**Note**: `members_count_with_descendants` are presented only for group masters/owners.
+
 ## Search for namespace
 
 Get all namespaces that match a string in their name or path.
@@ -72,6 +80,8 @@ Example response:
     "path": "twitter",
     "kind": "group",
     "full_path": "twitter",
+    "parent_id": "null",
+    "members_count_with_descendants": 2
   }
 ]
 ```
diff --git a/doc/ci/ssh_keys/README.md b/doc/ci/ssh_keys/README.md
index befaa06e918..cf25a8b618f 100644
--- a/doc/ci/ssh_keys/README.md
+++ b/doc/ci/ssh_keys/README.md
@@ -34,9 +34,9 @@ instructions to [generate an SSH key](../../ssh/README.md). Do not add a
 passphrase to the SSH key, or the `before_script` will prompt for it.
 
 Then, create a new **Secret Variable** in your project settings on GitLab
-following **Settings > Variables**. As **Key** add the name `SSH_PRIVATE_KEY`
-and in the **Value** field paste the content of your _private_ key that you
-created earlier.
+following **Settings > Pipelines** and look for the "Secret Variables" section.
+As **Key** add the name `SSH_PRIVATE_KEY` and in the **Value** field paste the 
+content of your _private_ key that you created earlier.
 
 It is also good practice to check the server's own public key to make sure you
 are not being targeted by a man-in-the-middle attack. To do this, add another
diff --git a/doc/development/README.md b/doc/development/README.md
index 9496a87d84d..a2a07c37ced 100644
--- a/doc/development/README.md
+++ b/doc/development/README.md
@@ -54,6 +54,7 @@
 - [Polymorphic Associations](polymorphic_associations.md)
 - [Single Table Inheritance](single_table_inheritance.md)
 - [Background Migrations](background_migrations.md)
+- [Storing SHA1 Hashes As Binary](sha1_as_binary.md)
 
 ## i18n
 
diff --git a/doc/development/policies.md b/doc/development/policies.md
new file mode 100644
index 00000000000..62141356f59
--- /dev/null
+++ b/doc/development/policies.md
@@ -0,0 +1,116 @@
+# `DeclarativePolicy` framework
+
+The DeclarativePolicy framework is designed to assist in performance of policy checks, and to enable ease of extension for EE. The DSL code in `app/policies` is what `Ability.allowed?` uses to check whether a particular action is allowed on a subject.
+
+The policy used is based on the subject's class name - so `Ability.allowed?(user, :some_ability, project)` will create a `ProjectPolicy` and check permissions on that.
+
+## Managing Permission Rules
+
+Permissions are broken into two parts: `conditions` and `rules`. Conditions are boolean expressions that can access the database and the environment, while rules are statically configured combinations of expressions and other rules that enable or prevent certain abilities. For an ability to be allowed, it must be enabled by at least one rule, and not prevented by any.
+
+
+### Conditions
+
+Conditions are defined by the `condition` method, and are given a name and a block. The block will be executed in the context of the policy object - so it can access `@user` and `@subject`, as well as call any methods defined on the policy. Note that `@user` may be nil (in the anonymous case), but `@subject` is guaranteed to be a real instance of the subject class.
+
+``` ruby
+class FooPolicy < BasePolicy
+  condition(:is_public) do
+    # @subject guaranteed to be an instance of Foo
+    @subject.public?
+  end
+
+  # instance methods can be called from the condition as well
+  condition(:thing) { check_thing }
+
+  def check_thing
+    # ...
+  end
+end
+```
+
+When you define a condition, a predicate method is defined on the policy to check whether that condition passes - so in the above example, an instance of `FooPolicy` will also respond to `#is_public?` and `#thing?`.
+
+Conditions are cached according to their scope. Scope and ordering will be covered later.
+
+### Rules
+
+A `rule` is a logical combination of conditions and other rules, that are configured to enable or prevent certain abilities. It is important to note that the rule configuration is static - a rule's logic cannot touch the database or know about `@user` or `@subject`. This allows us to cache only at the condition level. Rules are specified through the `rule` method, which takes a block of DSL configuration, and returns an object that responds to `#enable` or `#prevent`:
+
+``` ruby
+class FooPolicy < BasePolicy
+  # ...
+
+  rule { is_public }.enable :read
+  rule { thing }.prevent :read
+
+  # equivalently,
+  rule { is_public }.policy do
+    enable :read
+  end
+
+  rule { ~thing }.policy do
+    prevent :read
+  end
+end
+```
+
+Within the rule DSL, you can use:
+
+* A regular word mentions a condition by name - a rule that is in effect when that condition is truthy.
+* `~` indicates negation
+* `&` and `|` are logical combinations, also available as `all?(...)` and `any?(...)`
+* `can?(:other_ability)` delegates to the rules that apply to `:other_ability`. Note that this is distinct from the instance method `can?`, which can check dynamically - this only configures a delegation to another ability.
+
+## Scores, Order, Performance
+
+To see how the rules get evaluated into a judgment, it is useful in a console to use `policy.debug(:some_ability)`. This will print the rules in the order they are evaluated.
+
+When a policy is asked whether a particular ability is allowed (`policy.allowed?(:some_ability)`), it does not necessarily have to compute all the conditions on the policy. First, only the rules relevant to that particular ability are selected. Then, the execution model takes advantage of short-circuiting, and attempts to sort rules based on a heuristic of how expensive they will be to calculate. The sorting is dynamic and cache-aware, so that previously calculated conditions will be considered first, before computing other conditions.
+
+## Scope
+
+Sometimes, a condition will only use data from `@user` or only from `@subject`. In this case, we want to change the scope of the caching, so that we don't recalculate conditions unnecessarily. For example, given:
+
+``` ruby
+class FooPolicy < BasePolicy
+  condition(:expensive_condition) { @subject.expensive_query? }
+
+  rule { expensive_condition }.enable :some_ability
+end
+```
+
+Naively, if we call `Ability.can?(user1, :some_ability, foo)` and `Ability.can?(user2, :some_ability, foo)`, we would have to calculate the condition twice - since they are for different users. But if we use the `scope: :subject` option:
+
+``` ruby
+  condition(:expensive_condition, scope: :subject) { @subject.expensive_query? }
+```
+
+then the result of the condition will be cached globally only based on the subject - so it will not be calculated repeatedly for different users. Similarly, `scope: :user` will cache only based on the user.
+
+**DANGER**: If you use a `:scope` option when the condition actually uses data from
+both user and subject (including a simple anonymous check!) your result will be cached at too global of a scope and will result in cache bugs.
+
+Sometimes we are checking permissions for a lot of users for one subject, or a lot of subjects for one user. In this case, we want to set a *preferred scope* - i.e. tell the system that we prefer rules that can be cached on the repeated parameter. For example, in `Ability.users_that_can_read_project`:
+
+``` ruby
+def users_that_can_read_project(users, project)
+  DeclarativePolicy.subject_scope do
+    users.select { |u| allowed?(u, :read_project, project) }
+  end
+end
+```
+
+This will, for example, prefer checking `project.public?` to checking `user.admin?`.
+
+## Delegation
+
+Delegation is the inclusion of rules from another policy, on a different subject. For example,
+
+``` ruby
+class FooPolicy < BasePolicy
+  delegate { @subject.project }
+end
+```
+
+will include all rules from `ProjectPolicy`. The delegated conditions will be evaluated with the correct delegated subject, and will be sorted along with the regular rules in the policy. Note that only the relevant rules for a particular ability will actually be considered.
diff --git a/doc/development/sha1_as_binary.md b/doc/development/sha1_as_binary.md
new file mode 100644
index 00000000000..3151cc29bbc
--- /dev/null
+++ b/doc/development/sha1_as_binary.md
@@ -0,0 +1,36 @@
+# Storing SHA1 Hashes As Binary
+
+Storing SHA1 hashes as strings is not very space efficient. A SHA1 as a string
+requires at least 40 bytes, an additional byte to store the encoding, and
+perhaps more space depending on the internals of PostgreSQL and MySQL.
+
+On the other hand, if one were to store a SHA1 as binary one would only need 20
+bytes for the actual SHA1, and 1 or 4 bytes of additional space (again depending
+on database internals). This means that in the best case scenario we can reduce
+the space usage by 50%.
+
+To make this easier to work with you can include the concern `ShaAttribute` into
+a model and define a SHA attribute using the `sha_attribute` class method. For
+example:
+
+```ruby
+class Commit < ActiveRecord::Base
+  include ShaAttribute
+
+  sha_attribute :sha
+end
+```
+
+This allows you to use the value of the `sha` attribute as if it were a string,
+while storing it as binary. This means that you can do something like this,
+without having to worry about converting data to the right binary format:
+
+```ruby
+commit = Commit.find_by(sha: '88c60307bd1f215095834f09a1a5cb18701ac8ad')
+commit.sha = '971604de4cfa324d91c41650fabc129420c8d1cc'
+commit.save
+```
+
+There is however one requirement: the column used to store the SHA has _must_ be
+a binary type. For Rails this means you need to use the `:binary` type instead
+of `:text` or `:string`.
diff --git a/doc/install/installation.md b/doc/install/installation.md
index 84af6432889..992ff162efb 100644
--- a/doc/install/installation.md
+++ b/doc/install/installation.md
@@ -294,9 +294,9 @@ sudo usermod -aG redis git
 ### Clone the Source
 
     # Clone GitLab repository
-    sudo -u git -H git clone https://gitlab.com/gitlab-org/gitlab-ce.git -b 9-2-stable gitlab
+    sudo -u git -H git clone https://gitlab.com/gitlab-org/gitlab-ce.git -b 9-3-stable gitlab
 
-**Note:** You can change `9-2-stable` to `master` if you want the *bleeding edge* version, but never install master on a production server!
+**Note:** You can change `9-3-stable` to `master` if you want the *bleeding edge* version, but never install master on a production server!
 
 ### Configure It
 
diff --git a/doc/update/9.2-to-9.3.md b/doc/update/9.2-to-9.3.md
index 0c32e4db53f..8fbcc892fd5 100644
--- a/doc/update/9.2-to-9.3.md
+++ b/doc/update/9.2-to-9.3.md
@@ -117,7 +117,7 @@ cd /home/git/gitlab
 sudo -u git -H git checkout 9-3-stable-ee
 ```
 
-### 5. Update gitlab-shell
+### 7. Update gitlab-shell
 
 ```bash
 cd /home/git/gitlab-shell
@@ -127,7 +127,7 @@ sudo -u git -H git checkout v$(</home/git/gitlab/GITLAB_SHELL_VERSION)
 sudo -u git -H bin/compile
 ```
 
-### 6. Update gitlab-workhorse
+### 8. Update gitlab-workhorse
 
 Install and compile gitlab-workhorse. This requires
 [Go 1.5](https://golang.org/dl) which should already be on your system from
@@ -143,7 +143,7 @@ sudo -u git -H git checkout v$(</home/git/gitlab/GITLAB_WORKHORSE_VERSION)
 sudo -u git -H make
 ```
 
-### 7. Update Gitaly
+### 9. Update Gitaly
 
 If you have not yet set up Gitaly then follow [Gitaly section of the installation
 guide](../install/installation.md#install-gitaly).
diff --git a/doc/user/profile/personal_access_tokens.md b/doc/user/profile/personal_access_tokens.md
index 9488ce1ef30..f28c034e74c 100644
--- a/doc/user/profile/personal_access_tokens.md
+++ b/doc/user/profile/personal_access_tokens.md
@@ -14,6 +14,9 @@ accepted method of authentication when you have
 Once you have your token, [pass it to the API][usage] using either the
 `private_token` parameter or the `PRIVATE-TOKEN` header.
 
+The expiration of personal access tokens happens on the date you define,
+at midnight UTC.
+
 ## Creating a personal access token
 
 You can create as many personal access tokens as you like from your GitLab