Add more storage statistics

This adds counters for build artifacts and LFS objects, and moves
the preexisting repository_size and commit_count from the projects
table into a new project_statistics table.

The counters are displayed in the administration area for projects
and groups, and also available through the API for admins (on */all)
and normal users (on */owned)

The statistics are updated through ProjectCacheWorker, which can now
do more granular updates with the new :statistics argument.

4 files changed, 25 insertions, 1 deletions

diff --git a/doc/administration/build_artifacts.md b/doc/administration/build_artifacts.md
index 3ba8387c7f0..cca422892ec 100644
--- a/doc/administration/build_artifacts.md
+++ b/doc/administration/build_artifacts.md
@@ -88,3 +88,9 @@ artifacts through the [Admin area settings](../user/admin_area/settings/continuo
 
 [reconfigure gitlab]: restart_gitlab.md "How to restart GitLab"
 [restart gitlab]: restart_gitlab.md "How to restart GitLab"
+
+## Storage statistics
+
+You can see the total storage used for build artifacts on groups and projects
+in the administration area, as well as through the [groups](../api/groups.md)
+and [projects APIs](../api/projects.md).
diff --git a/doc/api/groups.md b/doc/api/groups.md
index 134d7bda22f..bc737bff8ee 100644
--- a/doc/api/groups.md
+++ b/doc/api/groups.md
@@ -13,6 +13,7 @@ Parameters:
 | `search` | string | no | Return list of authorized groups matching the search criteria |
 | `order_by` | string | no | Order groups by `name` or `path`. Default is `name` |
 | `sort` | string | no | Order groups in `asc` or `desc` order. Default is `asc` |
+| `statistics` | boolean | no | Include group statistics (admins only) |
 
 ```
 GET /groups
@@ -31,7 +32,6 @@ GET /groups
 
 You can search for groups by name or path, see below.
 
-=======
 ## List owned groups
 
 Get a list of groups which are owned by the authenticated user.
@@ -40,6 +40,12 @@ Get a list of groups which are owned by the authenticated user.
 GET /groups/owned
 ```
 
+Parameters:
+
+| Attribute | Type | Required | Description |
+| --------- | ---- | -------- | ----------- |
+| `statistics` | boolean | no | Include group statistics |
+
 ## List a group's projects
 
 Get a list of projects in this group.
diff --git a/doc/api/projects.md b/doc/api/projects.md
index edffad555a5..122075bbd11 100644
--- a/doc/api/projects.md
+++ b/doc/api/projects.md
@@ -307,6 +307,8 @@ Parameters:
 | `order_by` | string | no | Return projects ordered by `id`, `name`, `path`, `created_at`, `updated_at`, or `last_activity_at` fields. Default is `created_at` |
 | `sort` | string | no | Return projects sorted in `asc` or `desc` order. Default is `desc` |
 | `search` | string | no | Return list of authorized projects matching the search criteria |
+| `simple` | boolean | no | Return only the ID, URL, name, and path of each project |
+| `statistics` | boolean | no | Include project statistics |
 
 ### List starred projects
 
@@ -325,6 +327,7 @@ Parameters:
 | `order_by` | string | no | Return projects ordered by `id`, `name`, `path`, `created_at`, `updated_at`, or `last_activity_at` fields. Default is `created_at` |
 | `sort` | string | no | Return projects sorted in `asc` or `desc` order. Default is `desc` |
 | `search` | string | no | Return list of authorized projects matching the search criteria |
+| `simple` | boolean | no | Return only the ID, URL, name, and path of each project |
 
 ### List ALL projects
 
@@ -343,6 +346,7 @@ Parameters:
 | `order_by` | string | no | Return projects ordered by `id`, `name`, `path`, `created_at`, `updated_at`, or `last_activity_at` fields. Default is `created_at` |
 | `sort` | string | no | Return projects sorted in `asc` or `desc` order. Default is `desc` |
 | `search` | string | no | Return list of authorized projects matching the search criteria |
+| `statistics` | boolean | no | Include project statistics |
 
 ### Get single project
 
diff --git a/doc/workflow/lfs/lfs_administration.md b/doc/workflow/lfs/lfs_administration.md
index b3c73e947f0..5f6a718135d 100644
--- a/doc/workflow/lfs/lfs_administration.md
+++ b/doc/workflow/lfs/lfs_administration.md
@@ -40,6 +40,12 @@ In `config/gitlab.yml`:
     storage_path: /mnt/storage/lfs-objects
 ```
 
+## Storage statistics
+
+You can see the total storage used for LFS objects on groups and projects
+in the administration area, as well as through the [groups](../api/groups.md)
+and [projects APIs](../api/projects.md).
+
 ## Known limitations
 
 * Currently, storing GitLab Git LFS objects on a non-local storage (like S3 buckets)
@@ -47,3 +53,5 @@ In `config/gitlab.yml`:
 * Currently, removing LFS objects from GitLab Git LFS storage is not supported
 * LFS authentications via SSH was added with GitLab 8.12
 * Only compatible with the GitLFS client versions 1.1.0 and up, or 1.0.2.
+* The storage statistics currently count each LFS object multiple times for
+  every project linking to it