summaryrefslogtreecommitdiff
path: root/doc
diff options
context:
space:
mode:
authorGrzegorz Bizon <grzegorz@gitlab.com>2018-03-29 11:07:54 +0000
committerGrzegorz Bizon <grzegorz@gitlab.com>2018-03-29 11:07:54 +0000
commit5501dfd13ffda6abe8ff191a65964abbd4f3bb30 (patch)
tree3b25faf2d41893bc67e972fa56a89b6efd3ac0dc /doc
parentf4d81536ac26f75e0aad248ad95c31c9e1f2680b (diff)
parent8230b774b5a6375d9b84c71e72c8e41e3bfd7fad (diff)
downloadgitlab-ce-5501dfd13ffda6abe8ff191a65964abbd4f3bb30.tar.gz
Merge branch 'master' into 'feature/gb/variables-expressions-in-only-except'
Conflicts: app/models/ci/build.rb
Diffstat (limited to 'doc')
-rw-r--r--doc/README.md6
-rw-r--r--doc/administration/auth/jwt.md72
-rw-r--r--doc/administration/index.md5
-rw-r--r--doc/administration/job_artifacts.md122
-rw-r--r--doc/administration/job_traces.md42
-rw-r--r--doc/administration/raketasks/uploads/migrate.md74
-rw-r--r--doc/administration/uploads.md209
-rw-r--r--doc/api/commits.md68
-rw-r--r--doc/api/runners.md7
-rw-r--r--doc/ci/docker/using_docker_build.md91
-rw-r--r--doc/ci/examples/README.md4
-rw-r--r--doc/ci/examples/code_climate.md7
-rw-r--r--doc/ci/quick_start/README.md4
-rw-r--r--doc/ci/runners/README.md45
-rw-r--r--doc/ci/yaml/README.md5
-rw-r--r--doc/development/ee_features.md272
-rw-r--r--doc/development/new_fe_guide/style/javascript.md194
-rw-r--r--doc/integration/omniauth.md1
-rw-r--r--doc/raketasks/README.md1
-rw-r--r--doc/topics/autodevops/index.md18
-rw-r--r--doc/user/profile/preferences.md4
-rw-r--r--doc/user/project/clusters/index.md4
-rw-r--r--doc/user/project/index.md14
-rw-r--r--doc/user/project/integrations/prometheus_library/kubernetes.md15
-rw-r--r--doc/user/project/issue_board.md2
-rw-r--r--doc/user/project/merge_requests/img/remove_source_branch_status.pngbin0 -> 32649 bytes
-rw-r--r--doc/user/project/merge_requests/index.md16
-rw-r--r--doc/user/project/pipelines/settings.md9
-rw-r--r--doc/workflow/lfs/lfs_administration.md116
29 files changed, 1360 insertions, 67 deletions
diff --git a/doc/README.md b/doc/README.md
index be805a2ccc4..604f7244a34 100644
--- a/doc/README.md
+++ b/doc/README.md
@@ -211,9 +211,9 @@ straight away.
### GitLab self-hosted
-With GitLab self-hosted, you deploy your own GitLab instance on-premises or on a private cloud of your choice. GitLab self-hosted is available for [free and with paid subscriptions](https://about.gitlab.com/products/): Libre, Starter, Premium, and Ultimate.
+With GitLab self-hosted, you deploy your own GitLab instance on-premises or on a private cloud of your choice. GitLab self-hosted is available for [free and with paid subscriptions](https://about.gitlab.com/products/): Core, Starter, Premium, and Ultimate.
-Every feature available in Libre is also available in Starter, Premium, and Ultimate.
+Every feature available in Core is also available in Starter, Premium, and Ultimate.
Starter features are also available in Premium and Ultimate, and Premium features are also
available in Ultimate.
@@ -227,7 +227,7 @@ GitLab.com subscriptions grants access
to the same features available in GitLab self-hosted, **expect
[administration](administration/index.md) tools and settings**:
-- GitLab.com Free includes the same features available in GitLab Libre
+- GitLab.com Free includes the same features available in Core
- GitLab.com Bronze includes the same features available in GitLab Starter
- GitLab.com Silver includes the same features available in GitLab Premium
- GitLab.com Gold includes the same features available in GitLab Ultimate
diff --git a/doc/administration/auth/jwt.md b/doc/administration/auth/jwt.md
new file mode 100644
index 00000000000..b51e705ab52
--- /dev/null
+++ b/doc/administration/auth/jwt.md
@@ -0,0 +1,72 @@
+# JWT OmniAuth provider
+
+To enable the JWT OmniAuth provider, you must register your application with JWT.
+JWT will provide you with a secret key for you to use.
+
+1. On your GitLab server, open the configuration file.
+
+ For Omnibus GitLab:
+
+ ```sh
+ sudo editor /etc/gitlab/gitlab.rb
+ ```
+
+ For installations from source:
+
+ ```sh
+ cd /home/git/gitlab
+ sudo -u git -H editor config/gitlab.yml
+ ```
+
+1. See [Initial OmniAuth Configuration](../../integration/omniauth.md#initial-omniauth-configuration) for initial settings.
+1. Add the provider configuration.
+
+ For Omnibus GitLab:
+
+ ```ruby
+ gitlab_rails['omniauth_providers'] = [
+ { name: 'jwt',
+ app_secret: 'YOUR_APP_SECRET',
+ args: {
+ algorithm: 'HS256',
+ uid_claim: 'email',
+ required_claims: ["name", "email"],
+ info_maps: { name: "name", email: "email" },
+ auth_url: 'https://example.com/',
+ valid_within: nil,
+ }
+ }
+ ]
+ ```
+
+ For installation from source:
+
+ ```
+ - { name: 'jwt',
+ app_secret: 'YOUR_APP_SECRET',
+ args: {
+ algorithm: 'HS256',
+ uid_claim: 'email',
+ required_claims: ["name", "email"],
+ info_map: { name: "name", email: "email" },
+ auth_url: 'https://example.com/',
+ valid_within: nil,
+ }
+ }
+ ```
+
+ NOTE: **Note:** For more information on each configuration option refer to
+ the [OmniAuth JWT usage documentation](https://github.com/mbleigh/omniauth-jwt#usage).
+
+1. Change `YOUR_APP_SECRET` to the client secret and set `auth_url` to your redirect URL.
+1. Save the configuration file.
+1. [Reconfigure GitLab][] or [restart GitLab][] for the changes to take effect if you
+ installed GitLab via Omnibus or from source respectively.
+
+On the sign in page there should now be a JWT icon below the regular sign in form.
+Click the icon to begin the authentication process. JWT will ask the user to
+sign in and authorize the GitLab application. If everything goes well, the user
+will be redirected to GitLab and will be signed in.
+
+[reconfigure GitLab]: ../restart_gitlab.md#omnibus-gitlab-reconfigure
+[restart GitLab]: ../restart_gitlab.md#installations-from-source
diff --git a/doc/administration/index.md b/doc/administration/index.md
index 69efaf75140..60a45426636 100644
--- a/doc/administration/index.md
+++ b/doc/administration/index.md
@@ -11,8 +11,8 @@ available through [different subscriptions](https://about.gitlab.com/products/).
You can [install GitLab CE or GitLab EE](https://about.gitlab.com/installation/ce-or-ee/),
but the features you'll have access to depend on the subscription you choose
-(Libre, Starter, Premium, or Ultimate). GitLab Community Edition installations
-only have access to Libre features.
+(Core, Starter, Premium, or Ultimate). GitLab Community Edition installations
+only have access to Core features.
GitLab.com is administered by GitLab, Inc., therefore, only GitLab team members have
access to its admin configurations. If you're a GitLab.com user, please check the
@@ -111,6 +111,7 @@ server with IMAP authentication on Ubuntu, to be used with Reply by email.
- [Enable/disable GitLab CI/CD](../ci/enable_or_disable_ci.md#site-wide-admin-setting): Enable or disable GitLab CI/CD for your instance.
- [GitLab CI/CD admin settings](../user/admin_area/settings/continuous_integration.md): Define max artifacts size and expiration time.
- [Job artifacts](job_artifacts.md): Enable, disable, and configure job artifacts (a set of files and directories which are outputted by a job when it completes successfully).
+- [Job traces](job_traces.md): Information about the job traces (logs).
- [Artifacts size and expiration](../user/admin_area/settings/continuous_integration.md#maximum-artifacts-size): Define maximum artifacts limits and expiration date.
- [Register Shared and specific Runners](../ci/runners/README.md#registering-a-shared-runner): Learn how to register and configure Shared and specific Runners to your own instance.
- [Shared Runners pipelines quota](../user/admin_area/settings/continuous_integration.md#shared-runners-pipeline-minutes-quota): Limit the usage of pipeline minutes for Shared Runners.
diff --git a/doc/administration/job_artifacts.md b/doc/administration/job_artifacts.md
index d86a54daadd..ac3a12930c3 100644
--- a/doc/administration/job_artifacts.md
+++ b/doc/administration/job_artifacts.md
@@ -87,10 +87,124 @@ _The artifacts are stored by default in
### Using object storage
+>**Notes:**
+- [Introduced][ee-1762] in [GitLab Premium][eep] 9.4.
+- Since version 9.5, artifacts are [browsable], when object storage is enabled.
+ 9.4 lacks this feature.
> Available in [GitLab Premium](https://about.gitlab.com/products/) and
[GitLab.com Silver](https://about.gitlab.com/gitlab-com/).
+> Since version 10.6, available in [GitLab CE](https://about.gitlab.com/products/)
+
+If you don't want to use the local disk where GitLab is installed to store the
+artifacts, you can use an object storage like AWS S3 instead.
+This configuration relies on valid AWS credentials to be configured already.
+Use an [Object storage option][os] like AWS S3 to store job artifacts.
+
+### Object Storage Settings
+
+For source installations the following settings are nested under `artifacts:` and then `object_store:`. On omnibus installs they are prefixed by `artifacts_object_store_`.
+
+| Setting | Description | Default |
+|---------|-------------|---------|
+| `enabled` | Enable/disable object storage | `false` |
+| `remote_directory` | The bucket name where Artfacts will be stored| |
+| `background_upload` | Set to false to disable automatic upload. Option may be removed once upload is direct to S3 | `true` |
+| `proxy_download` | Set to true to enable proxying all files served. Option allows to reduce egress traffic as this allows clients to download directly from remote storage instead of proxying all data | `false` |
+| `connection` | Various connection options described below | |
+
+#### S3 compatible connection settings
+
+The connection settings match those provided by [Fog](https://github.com/fog), and are as follows:
+
+| Setting | Description | Default |
+|---------|-------------|---------|
+| `provider` | Always `AWS` for compatible hosts | AWS |
+| `aws_access_key_id` | AWS credentials, or compatible | |
+| `aws_secret_access_key` | AWS credentials, or compatible | |
+| `region` | AWS region | us-east-1 |
+| `host` | S3 compatible host for when not using AWS, e.g. `localhost` or `storage.example.com` | s3.amazonaws.com |
+| `endpoint` | Can be used when configuring an S3 compatible service such as [Minio](https://www.minio.io), by entering a URL such as `http://127.0.0.1:9000` | (optional) |
+| `path_style` | Set to true to use `host/bucket_name/object` style paths instead of `bucket_name.host/object`. Leave as false for AWS S3 | false |
+
+**In Omnibus installations:**
+
+_The artifacts are stored by default in
+`/var/opt/gitlab/gitlab-rails/shared/artifacts`._
+
+1. Edit `/etc/gitlab/gitlab.rb` and add the following lines by replacing with
+ the values you want:
+
+ ```ruby
+ gitlab_rails['artifacts_enabled'] = true
+ gitlab_rails['artifacts_object_store_enabled'] = true
+ gitlab_rails['artifacts_object_store_remote_directory'] = "artifacts"
+ gitlab_rails['artifacts_object_store_connection'] = {
+ 'provider' => 'AWS',
+ 'region' => 'eu-central-1',
+ 'aws_access_key_id' => 'AWS_ACCESS_KEY_ID',
+ 'aws_secret_access_key' => 'AWS_SECRET_ACCESS_KEY'
+ }
+ ```
+
+ NOTE: For GitLab 9.4+, if you are using AWS IAM profiles, be sure to omit the
+ AWS access key and secret acces key/value pairs. For example:
+
+ ```ruby
+ gitlab_rails['artifacts_object_store_connection'] = {
+ 'provider' => 'AWS',
+ 'region' => 'eu-central-1',
+ 'use_iam_profile' => true
+ }
+ ```
+
+1. Save the file and [reconfigure GitLab][] for the changes to take effect.
+1. Migrate any existing local artifacts to the object storage:
+
+ ```bash
+ gitlab-rake gitlab:artifacts:migrate
+ ```
+
+ Currently this has to be executed manually and it will allow you to
+ migrate the existing artifacts to the object storage, but all new
+ artifacts will still be stored on the local disk. In the future
+ you will be given an option to define a default storage artifacts for all
+ new files.
+
+---
+
+**In installations from source:**
+
+_The artifacts are stored by default in
+`/home/git/gitlab/shared/artifacts`._
+
+1. Edit `/home/git/gitlab/config/gitlab.yml` and add or amend the following
+ lines:
+
+ ```yaml
+ artifacts:
+ enabled: true
+ object_store:
+ enabled: true
+ remote_directory: "artifacts" # The bucket name
+ connection:
+ provider: AWS # Only AWS supported at the moment
+ aws_access_key_id: AWS_ACESS_KEY_ID
+ aws_secret_access_key: AWS_SECRET_ACCESS_KEY
+ region: eu-central-1
+ ```
+
+1. Save the file and [restart GitLab][] for the changes to take effect.
+1. Migrate any existing local artifacts to the object storage:
+
+ ```bash
+ sudo -u git -H bundle exec rake gitlab:artifacts:migrate RAILS_ENV=production
+ ```
-Use an [Object storage option][ee-os] like AWS S3 to store job artifacts.
+ Currently this has to be executed manually and it will allow you to
+ migrate the existing artifacts to the object storage, but all new
+ artifacts will still be stored on the local disk. In the future
+ you will be given an option to define a default storage artifacts for all
+ new files.
## Expiring artifacts
@@ -194,7 +308,7 @@ When clicking on a specific file, [GitLab Workhorse] extracts it
from the archive and the download begins. This implementation saves space,
memory and disk I/O.
-[reconfigure gitlab]: restart_gitlab.md "How to restart GitLab"
-[restart gitlab]: restart_gitlab.md "How to restart GitLab"
+[reconfigure gitlab]: restart_gitlab.md#omnibus-gitlab-reconfigure "How to reconfigure Omnibus GitLab"
+[restart gitlab]: restart_gitlab.md#installations-from-source "How to restart GitLab"
[gitlab workhorse]: https://gitlab.com/gitlab-org/gitlab-workhorse "GitLab Workhorse repository"
-[ee-os]: https://docs.gitlab.com/ee/administration/job_artifacts.html#using-object-storage
+[os]: https://docs.gitlab.com/administration/job_artifacts.html#using-object-storage
diff --git a/doc/administration/job_traces.md b/doc/administration/job_traces.md
new file mode 100644
index 00000000000..84a1ffeec98
--- /dev/null
+++ b/doc/administration/job_traces.md
@@ -0,0 +1,42 @@
+# Job traces (logs)
+
+By default, all job traces (logs) are saved to `/var/opt/gitlab/gitlab-ci/builds`
+and `/home/git/gitlab/builds` for Omnibus packages and installations from source
+respectively. The job logs are organized by year and month (for example, `2017_03`),
+and then by project ID.
+
+There isn't a way to automatically expire old job logs, but it's safe to remove
+them if they're taking up too much space. If you remove the logs manually, the
+job output in the UI will be empty.
+
+## Changing the job traces location
+
+To change the location where the job logs will be stored, follow the steps below.
+
+**In Omnibus installations:**
+
+1. Edit `/etc/gitlab/gitlab.rb` and add or amend the following line:
+
+ ```
+ gitlab_ci['builds_directory'] = '/mnt/to/gitlab-ci/builds'
+ ```
+
+1. Save the file and [reconfigure GitLab][] for the changes to take effect.
+
+---
+
+**In installations from source:**
+
+1. Edit `/home/git/gitlab/config/gitlab.yml` and add or amend the following lines:
+
+ ```yaml
+ gitlab_ci:
+ # The location where build traces are stored (default: builds/).
+ # Relative paths are relative to Rails.root.
+ builds_path: path/to/builds/
+ ```
+
+1. Save the file and [restart GitLab][] for the changes to take effect.
+
+[reconfigure gitlab]: restart_gitlab.md#omnibus-gitlab-reconfigure "How to reconfigure Omnibus GitLab"
+[restart gitlab]: restart_gitlab.md#installations-from-source "How to restart GitLab"
diff --git a/doc/administration/raketasks/uploads/migrate.md b/doc/administration/raketasks/uploads/migrate.md
new file mode 100644
index 00000000000..0cd33ffc122
--- /dev/null
+++ b/doc/administration/raketasks/uploads/migrate.md
@@ -0,0 +1,74 @@
+# Uploads Migrate Rake Task
+
+## Migrate to Object Storage
+
+After [configuring the object storage](../../uploads.md#using-object-storage) for GitLab's uploads, you may use this task to migrate existing uploads from the local storage to the remote storage.
+
+>**Note:**
+All of the processing will be done in a background worker and requires **no downtime**.
+
+This tasks uses 3 parameters to find uploads to migrate.
+
+>**Note:**
+These parameters are mainly internal to GitLab's structure, you may want to refer to the task list instead below.
+
+Parameter | Type | Description
+--------- | ---- | -----------
+`uploader_class` | string | Type of the uploader to migrate from
+`model_class` | string | Type of the model to migrate from
+`mount_point` | string/symbol | Name of the model's column on which the uploader is mounted on.
+
+This task also accepts some environment variables which you can use to override
+certain values:
+
+Variable | Type | Description
+-------- | ---- | -----------
+`BATCH` | integer | Specifies the size of the batch. Defaults to 200.
+
+** Omnibus Installation**
+
+```bash
+# gitlab-rake gitlab:uploads:migrate[uploader_class, model_class, mount_point]
+
+# Avatars
+gitlab-rake "gitlab:uploads:migrate[AvatarUploader, Project, :avatar]"
+gitlab-rake "gitlab:uploads:migrate[AvatarUploader, Group, :avatar]"
+gitlab-rake "gitlab:uploads:migrate[AvatarUploader, User, :avatar]"
+
+# Attachments
+gitlab-rake "gitlab:uploads:migrate[AttachmentUploader, Note, :attachment]"
+gitlab-rake "gitlab:uploads:migrate[AttachmentUploader, Appearance, :logo]"
+gitlab-rake "gitlab:uploads:migrate[AttachmentUploader, Appearance, :header_logo]"
+
+# Markdown
+gitlab-rake "gitlab:uploads:migrate[FileUploader, Project]"
+gitlab-rake "gitlab:uploads:migrate[PersonalFileUploader, Snippet]"
+gitlab-rake "gitlab:uploads:migrate[NamespaceFileUploader, Snippet]"
+gitlab-rake "gitlab:uploads:migrate[FileUploader, MergeRequest]"
+```
+
+**Source Installation**
+
+>**Note:**
+Use `RAILS_ENV=production` for every task.
+
+```bash
+# sudo -u git -H bundle exec rake gitlab:uploads:migrate
+
+# Avatars
+sudo -u git -H bundle exec rake "gitlab:uploads:migrate[AvatarUploader, Project, :avatar]"
+sudo -u git -H bundle exec rake "gitlab:uploads:migrate[AvatarUploader, Group, :avatar]"
+sudo -u git -H bundle exec rake "gitlab:uploads:migrate[AvatarUploader, User, :avatar]"
+
+# Attachments
+sudo -u git -H bundle exec rake "gitlab:uploads:migrate[AttachmentUploader, Note, :attachment]"
+sudo -u git -H bundle exec rake "gitlab:uploads:migrate[AttachmentUploader, Appearance, :logo]"
+sudo -u git -H bundle exec rake "gitlab:uploads:migrate[AttachmentUploader, Appearance, :header_logo]"
+
+# Markdown
+sudo -u git -H bundle exec rake "gitlab:uploads:migrate[FileUploader, Project]"
+sudo -u git -H bundle exec rake "gitlab:uploads:migrate[PersonalFileUploader, Snippet]"
+sudo -u git -H bundle exec rake "gitlab:uploads:migrate[NamespaceFileUploader, Snippet]"
+sudo -u git -H bundle exec rake "gitlab:uploads:migrate[FileUploader, MergeRequest]"
+
+```
diff --git a/doc/administration/uploads.md b/doc/administration/uploads.md
new file mode 100644
index 00000000000..a82735cc72c
--- /dev/null
+++ b/doc/administration/uploads.md
@@ -0,0 +1,209 @@
+# Uploads administration
+
+>**Notes:**
+Uploads represent all user data that may be sent to GitLab as a single file. As an example, avatars and notes' attachments are uploads. Uploads are integral to GitLab functionality, and therefore cannot be disabled.
+
+### Using local storage
+
+>**Notes:**
+This is the default configuration
+
+To change the location where the uploads are stored locally, follow the steps
+below.
+
+---
+
+**In Omnibus installations:**
+
+>**Notes:**
+For historical reasons, uploads are stored into a base directory, which by default is `uploads/-/system`. It is strongly discouraged to change this configuration option on an existing GitLab installation.
+
+_The uploads are stored by default in `/var/opt/gitlab/gitlab-rails/public/uploads/-/system`._
+
+1. To change the storage path for example to `/mnt/storage/uploads`, edit
+ `/etc/gitlab/gitlab.rb` and add the following line:
+
+ ```ruby
+ gitlab_rails['uploads_storage_path'] = "/mnt/storage/"
+ gitlab_rails['uploads_base_dir'] = "uploads"
+ ```
+
+1. Save the file and [reconfigure GitLab][] for the changes to take effect.
+
+---
+
+**In installations from source:**
+
+_The uploads are stored by default in
+`/home/git/gitlab/public/uploads/-/system`._
+
+1. To change the storage path for example to `/mnt/storage/uploads`, edit
+ `/home/git/gitlab/config/gitlab.yml` and add or amend the following lines:
+
+ ```yaml
+ uploads:
+ storage_path: /mnt/storage
+ base_dir: uploads
+ ```
+
+1. Save the file and [restart GitLab][] for the changes to take effect.
+
+### Using object storage
+
+>**Notes:**
+- [Introduced][ee-3867] in [GitLab Enterprise Edition Premium][eep] 10.5.
+
+If you don't want to use the local disk where GitLab is installed to store the
+uploads, you can use an object storage provider like AWS S3 instead.
+This configuration relies on valid AWS credentials to be configured already.
+
+### Object Storage Settings
+
+For source installations the following settings are nested under `uploads:` and then `object_store:`. On omnibus installs they are prefixed by `uploads_object_store_`.
+
+| Setting | Description | Default |
+|---------|-------------|---------|
+| `enabled` | Enable/disable object storage | `false` |
+| `remote_directory` | The bucket name where Uploads will be stored| |
+| `background_upload` | Set to false to disable automatic upload. Option may be removed once upload is direct to S3 | `true` |
+| `proxy_download` | Set to true to enable proxying all files served. Option allows to reduce egress traffic as this allows clients to download directly from remote storage instead of proxying all data | `false` |
+| `connection` | Various connection options described below | |
+
+#### S3 compatible connection settings
+
+The connection settings match those provided by [Fog](https://github.com/fog), and are as follows:
+
+| Setting | Description | Default |
+|---------|-------------|---------|
+| `provider` | Always `AWS` for compatible hosts | AWS |
+| `aws_access_key_id` | AWS credentials, or compatible | |
+| `aws_secret_access_key` | AWS credentials, or compatible | |
+| `region` | AWS region | us-east-1 |
+| `host` | S3 compatible host for when not using AWS, e.g. `localhost` or `storage.example.com` | s3.amazonaws.com |
+| `endpoint` | Can be used when configuring an S3 compatible service such as [Minio](https://www.minio.io), by entering a URL such as `http://127.0.0.1:9000` | (optional) |
+| `path_style` | Set to true to use `host/bucket_name/object` style paths instead of `bucket_name.host/object`. Leave as false for AWS S3 | false |
+
+**In Omnibus installations:**
+
+_The uploads are stored by default in
+`/var/opt/gitlab/gitlab-rails/public/uploads/-/system`._
+
+1. Edit `/etc/gitlab/gitlab.rb` and add the following lines by replacing with
+ the values you want:
+
+ ```ruby
+ gitlab_rails['uploads_object_store_enabled'] = true
+ gitlab_rails['uploads_object_store_remote_directory'] = "uploads"
+ gitlab_rails['uploads_object_store_connection'] = {
+ 'provider' => 'AWS',
+ 'region' => 'eu-central-1',
+ 'aws_access_key_id' => 'AWS_ACCESS_KEY_ID',
+ 'aws_secret_access_key' => 'AWS_SECRET_ACCESS_KEY'
+ }
+ ```
+
+>**Note:**
+If you are using AWS IAM profiles, be sure to omit the AWS access key and secret acces key/value pairs.
+
+ ```ruby
+ gitlab_rails['uploads_object_store_connection'] = {
+ 'provider' => 'AWS',
+ 'region' => 'eu-central-1',
+ 'use_iam_profile' => true
+ }
+ ```
+
+1. Save the file and [reconfigure GitLab][] for the changes to take effect.
+1. Migrate any existing local uploads to the object storage:
+
+>**Notes:**
+These task complies with the `BATCH` environment variable to process uploads in batch (200 by default). All of the processing will be done in a background worker and requires **no downtime**.
+
+ ```bash
+ # gitlab-rake gitlab:uploads:migrate[uploader_class, model_class, mount_point]
+
+ # Avatars
+ gitlab-rake "gitlab:uploads:migrate[AvatarUploader, Project, :avatar]"
+ gitlab-rake "gitlab:uploads:migrate[AvatarUploader, Group, :avatar]"
+ gitlab-rake "gitlab:uploads:migrate[AvatarUploader, User, :avatar]"
+
+ # Attachments
+ gitlab-rake "gitlab:uploads:migrate[AttachmentUploader, Note, :attachment]"
+ gitlab-rake "gitlab:uploads:migrate[AttachmentUploader, Appearance, :logo]"
+ gitlab-rake "gitlab:uploads:migrate[AttachmentUploader, Appearance, :header_logo]"
+
+ # Markdown
+ gitlab-rake "gitlab:uploads:migrate[FileUploader, Project]"
+ gitlab-rake "gitlab:uploads:migrate[PersonalFileUploader, Snippet]"
+ gitlab-rake "gitlab:uploads:migrate[NamespaceFileUploader, Snippet]"
+ gitlab-rake "gitlab:uploads:migrate[FileUploader, MergeRequest]"
+ ```
+
+ Currently this has to be executed manually and it will allow you to
+ migrate the existing uploads to the object storage, but all new
+ uploads will still be stored on the local disk. In the future
+ you will be given an option to define a default storage for all
+ new files.
+
+---
+
+**In installations from source:**
+
+_The uploads are stored by default in
+`/home/git/gitlab/public/uploads/-/system`._
+
+1. Edit `/home/git/gitlab/config/gitlab.yml` and add or amend the following
+ lines:
+
+ ```yaml
+ uploads:
+ object_store:
+ enabled: true
+ remote_directory: "uploads" # The bucket name
+ connection:
+ provider: AWS # Only AWS supported at the moment
+ aws_access_key_id: AWS_ACESS_KEY_ID
+ aws_secret_access_key: AWS_SECRET_ACCESS_KEY
+ region: eu-central-1
+ ```
+
+1. Save the file and [restart GitLab][] for the changes to take effect.
+1. Migrate any existing local uploads to the object storage:
+
+>**Notes:**
+
+- These task comply with the `BATCH` environment variable to process uploads in batch (200 by default). All of the processing will be done in a background worker and requires **no downtime**.
+
+- To migrate in production use `RAILS_ENV=production` environment variable.
+
+ ```bash
+ # sudo -u git -H bundle exec rake gitlab:uploads:migrate
+
+ # Avatars
+ sudo -u git -H bundle exec rake "gitlab:uploads:migrate[AvatarUploader, Project, :avatar]"
+ sudo -u git -H bundle exec rake "gitlab:uploads:migrate[AvatarUploader, Group, :avatar]"
+ sudo -u git -H bundle exec rake "gitlab:uploads:migrate[AvatarUploader, User, :avatar]"
+
+ # Attachments
+ sudo -u git -H bundle exec rake "gitlab:uploads:migrate[AttachmentUploader, Note, :attachment]"
+ sudo -u git -H bundle exec rake "gitlab:uploads:migrate[AttachmentUploader, Appearance, :logo]"
+ sudo -u git -H bundle exec rake "gitlab:uploads:migrate[AttachmentUploader, Appearance, :header_logo]"
+
+ # Markdown
+ sudo -u git -H bundle exec rake "gitlab:uploads:migrate[FileUploader, Project]"
+ sudo -u git -H bundle exec rake "gitlab:uploads:migrate[PersonalFileUploader, Snippet]"
+ sudo -u git -H bundle exec rake "gitlab:uploads:migrate[NamespaceFileUploader, Snippet]"
+ sudo -u git -H bundle exec rake "gitlab:uploads:migrate[FileUploader, MergeRequest]"
+
+ ```
+
+ Currently this has to be executed manually and it will allow you to
+ migrate the existing uploads to the object storage, but all new
+ uploads will still be stored on the local disk. In the future
+ you will be given an option to define a default storage for all
+ new files.
+
+[reconfigure gitlab]: restart_gitlab.md#omnibus-gitlab-reconfigure "How to reconfigure Omnibus GitLab"
+[restart gitlab]: restart_gitlab.md#installations-from-source "How to restart GitLab"
+[eep]: https://about.gitlab.com/gitlab-ee/ "GitLab Enterprise Edition Premium"
+[ee-3867]: https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/3867
diff --git a/doc/api/commits.md b/doc/api/commits.md
index 55c673fd06a..a6b96ba539f 100644
--- a/doc/api/commits.md
+++ b/doc/api/commits.md
@@ -536,6 +536,74 @@ Example response:
}
```
+## List Merge Requests associated with a commit
+
+Get a list of Merge Requests related to the specified commit.
+
+```
+GET /projects/:id/repository/commits/:sha/merge_requests
+```
+
+| Attribute | Type | Required | Description |
+| --------- | ---- | -------- | ----------- |
+| `id` | integer/string | yes | The ID or [URL-encoded path of the project](README.md#namespaced-path-encoding) owned by the authenticated user
+| `sha` | string | yes | The commit SHA
+
+
+```bash
+curl --header "PRIVATE-TOKEN: 9koXpg98eAheJpvBs5tK" "https://gitlab.example.com/api/v4/projects/5/repository/commits/af5b13261899fb2c0db30abdd0af8b07cb44fdc5/merge_requests"
+```
+
+Example response:
+
+```json
+[
+ {
+ "id":45,
+ "iid":1,
+ "project_id":35,
+ "title":"Add new file",
+ "description":"",
+ "state":"opened",
+ "created_at":"2018-03-26T17:26:30.916Z",
+ "updated_at":"2018-03-26T17:26:30.916Z",
+ "target_branch":"master",
+ "source_branch":"test-branch",
+ "upvotes":0,
+ "downvotes":0,
+ "author" : {
+ "web_url" : "https://gitlab.example.com/thedude",
+ "name" : "Jeff Lebowski",
+ "avatar_url" : "https://gitlab.example.com/uploads/user/avatar/28/The-Big-Lebowski-400-400.png",
+ "username" : "thedude",
+ "state" : "active",
+ "id" : 28
+ },
+ "assignee":null,
+ "source_project_id":35,
+ "target_project_id":35,
+ "labels":[ ],
+ "work_in_progress":false,
+ "milestone":null,
+ "merge_when_pipeline_succeeds":false,
+ "merge_status":"can_be_merged",
+ "sha":"af5b13261899fb2c0db30abdd0af8b07cb44fdc5",
+ "merge_commit_sha":null,
+ "user_notes_count":0,
+ "discussion_locked":null,
+ "should_remove_source_branch":null,
+ "force_remove_source_branch":false,
+ "web_url":"http://https://gitlab.example.com/root/test-project/merge_requests/1",
+ "time_stats":{
+ "time_estimate":0,
+ "total_time_spent":0,
+ "human_time_estimate":null,
+ "human_total_time_spent":null
+ }
+ }
+]
+```
+
[ce-6096]: https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/6096 "Multi-file commit"
[ce-8047]: https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/8047
[ce-15026]: https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/15026
diff --git a/doc/api/runners.md b/doc/api/runners.md
index 7495c6cdedb..f384ac57bfe 100644
--- a/doc/api/runners.md
+++ b/doc/api/runners.md
@@ -153,7 +153,8 @@ Example response:
"mysql"
],
"version": null,
- "access_level": "ref_protected"
+ "access_level": "ref_protected",
+ "maximum_timeout": 3600
}
```
@@ -174,6 +175,7 @@ PUT /runners/:id
| `run_untagged` | boolean | no | Flag indicating the runner can execute untagged jobs |
| `locked` | boolean | no | Flag indicating the runner is locked |
| `access_level` | string | no | The access_level of the runner; `not_protected` or `ref_protected` |
+| `maximum_timeout` | integer | no | Maximum timeout set when this Runner will handle the job |
```
curl --request PUT --header "PRIVATE-TOKEN: 9koXpg98eAheJpvBs5tK" "https://gitlab.example.com/api/v4/runners/6" --form "description=test-1-20150125-test" --form "tag_list=ruby,mysql,tag1,tag2"
@@ -211,7 +213,8 @@ Example response:
"tag2"
],
"version": null,
- "access_level": "ref_protected"
+ "access_level": "ref_protected",
+ "maximum_timeout": null
}
```
diff --git a/doc/ci/docker/using_docker_build.md b/doc/ci/docker/using_docker_build.md
index 22afcb9199d..183808641c0 100644
--- a/doc/ci/docker/using_docker_build.md
+++ b/doc/ci/docker/using_docker_build.md
@@ -1,26 +1,29 @@
-# Using Docker Build
+# Building Docker images with GitLab CI/CD
-GitLab CI allows you to use Docker Engine to build and test docker-based projects.
+GitLab CI/CD allows you to use Docker Engine to build and test docker-based projects.
-**This also allows to you to use `docker-compose` and other docker-enabled tools.**
+TIP: **Tip:**
+This also allows to you to use `docker-compose` and other docker-enabled tools.
One of the new trends in Continuous Integration/Deployment is to:
-1. create an application image,
-1. run tests against the created image,
-1. push image to a remote registry, and
-1. deploy to a server from the pushed image.
+1. Create an application image
+1. Run tests against the created image
+1. Push image to a remote registry
+1. Deploy to a server from the pushed image
-It's also useful when your application already has the `Dockerfile` that can be used to create and test an image:
+It's also useful when your application already has the `Dockerfile` that can be
+used to create and test an image:
```bash
-$ docker build -t my-image dockerfiles/
-$ docker run my-docker-image /script/to/run/tests
-$ docker tag my-image my-registry:5000/my-image
-$ docker push my-registry:5000/my-image
+docker build -t my-image dockerfiles/
+docker run my-docker-image /script/to/run/tests
+docker tag my-image my-registry:5000/my-image
+docker push my-registry:5000/my-image
```
-This requires special configuration of GitLab Runner to enable `docker` support during jobs.
+This requires special configuration of GitLab Runner to enable `docker` support
+during jobs.
## Runner Configuration
@@ -74,8 +77,8 @@ GitLab Runner then executes job scripts as the `gitlab-runner` user.
5. You can now use `docker` command and install `docker-compose` if needed.
-> **Note:**
-* By adding `gitlab-runner` to the `docker` group you are effectively granting `gitlab-runner` full root permissions.
+NOTE: **Note:**
+By adding `gitlab-runner` to the `docker` group you are effectively granting `gitlab-runner` full root permissions.
For more information please read [On Docker security: `docker` group considered harmful](https://www.andreas-jung.com/contents/on-docker-security-docker-group-considered-harmful).
### Use docker-in-docker executor
@@ -259,8 +262,66 @@ aware of the following implications:
docker run --rm -t -i -v $(pwd)/src:/home/app/src test-image:latest run_app_tests
```
+## Making docker-in-docker builds faster with Docker layer caching
+
+When using docker-in-docker, Docker will download all layers of your image every
+time you create a build. Recent versions of Docker (Docker 1.13 and above) can
+use a pre-existing image as a cache during the `docker build` step, considerably
+speeding up the build process.
+
+### How Docker caching works
+
+When running `docker build`, each command in `Dockerfile` results in a layer.
+These layers are kept around as a cache and can be reused if there haven't been
+any changes. Change in one layer causes all subsequent layers to be recreated.
+
+You can specify a tagged image to be used as a cache source for the `docker build`
+command by using the `--cache-from` argument. Multiple images can be specified
+as a cache source by using multiple `--cache-from` arguments. Keep in mind that
+any image that's used with the `--cache-from` argument must first be pulled
+(using `docker pull`) before it can be used as a cache source.
+
+### Using Docker caching
+
+Here's a simple `.gitlab-ci.yml` file showing how Docker caching can be utilized:
+
+```yaml
+image: docker:latest
+
+services:
+ - docker:dind
+
+variables:
+ CONTAINER_IMAGE: registry.gitlab.com/$CI_PROJECT_PATH
+ DOCKER_DRIVER: overlay2
+
+before_script:
+ - docker login -u gitlab-ci-token -p $CI_JOB_TOKEN registry.gitlab.com
+
+build:
+ stage: build
+ script:
+ - docker pull $CONTAINER_IMAGE:latest || true
+ - docker build --cache-from $CONTAINER_IMAGE:latest --tag $CONTAINER_IMAGE:$CI_BUILD_REF --tag $CONTAINER_IMAGE:latest .
+ - docker push $CONTAINER_IMAGE:$CI_BUILD_REF
+ - docker push $CONTAINER_IMAGE:latest
+```
+
+The steps in the `script` section for the `build` stage can be summed up to:
+
+1. The first command tries to pull the image from the registry so that it can be
+ used as a cache for the `docker build` command.
+1. The second command builds a Docker image using the pulled image as a
+ cache (notice the `--cache-from $CONTAINER_IMAGE:latest` argument) if
+ available, and tags it.
+1. The last two commands push the tagged Docker images to the container registry
+ so that they may also be used as cache for subsequent builds.
+
## Using the OverlayFS driver
+NOTE: **Note:**
+The shared Runners on GitLab.com use the `overlay2` driver by default.
+
By default, when using `docker:dind`, Docker uses the `vfs` storage driver which
copies the filesystem on every run. This is a very disk-intensive operation
which can be avoided if a different driver is used, for example `overlay2`.
diff --git a/doc/ci/examples/README.md b/doc/ci/examples/README.md
index c1e258aedca..de60cd27cd1 100644
--- a/doc/ci/examples/README.md
+++ b/doc/ci/examples/README.md
@@ -49,6 +49,10 @@ There's also a collection of repositories with [example projects](https://gitlab
**(Ultimate)** [Scan your code for vulnerabilities](https://docs.gitlab.com/ee/ci/examples/sast.html)
+## Dependency Scanning
+
+**(Ultimate)** [Scan your dependencies for vulnerabilities](https://docs.gitlab.com/ee/ci/examples/dependency_scanning.html)
+
## Container Scanning
[Scan your Docker images for vulnerabilities](container_scanning.md)
diff --git a/doc/ci/examples/code_climate.md b/doc/ci/examples/code_climate.md
index ec5e5afb8c6..64a759a9a99 100644
--- a/doc/ci/examples/code_climate.md
+++ b/doc/ci/examples/code_climate.md
@@ -15,13 +15,8 @@ codequality:
services:
- docker:dind
script:
- - docker pull codeclimate/codeclimate
- export SP_VERSION=$(echo "$CI_SERVER_VERSION" | sed 's/^\([0-9]*\)\.\([0-9]*\).*/\1-\2-stable/')
- - docker run
- --env SOURCE_CODE="$PWD" \
- --volume "$PWD":/code \
- --volume /var/run/docker.sock:/var/run/docker.sock \
- "registry.gitlab.com/gitlab-org/security-products/codequality:$SP_VERSION" /code
+ - docker run --env SOURCE_CODE="$PWD" --volume "$PWD":/code --volume /var/run/docker.sock:/var/run/docker.sock "registry.gitlab.com/gitlab-org/security-products/codequality:$SP_VERSION" /code
artifacts:
paths: [codeclimate.json]
```
diff --git a/doc/ci/quick_start/README.md b/doc/ci/quick_start/README.md
index e504b81eae8..f64e868d390 100644
--- a/doc/ci/quick_start/README.md
+++ b/doc/ci/quick_start/README.md
@@ -104,8 +104,8 @@ Jobs are used to create jobs, which are then picked by
What is important is that each job is run independently from each other.
-If you want to check whether your `.gitlab-ci.yml` file is valid, there is a
-Lint tool under the page `/ci/lint` of your GitLab instance. You can also find
+If you want to check whether the `.gitlab-ci.yml` of your project is valid, there is a
+Lint tool under the page `/ci/lint` of your project namespace. You can also find
a "CI Lint" button to go to this page under **CI/CD ➔ Pipelines** and
**Pipelines ➔ Jobs** in your project.
diff --git a/doc/ci/runners/README.md b/doc/ci/runners/README.md
index 7a7b50b294d..b91aa334ff3 100644
--- a/doc/ci/runners/README.md
+++ b/doc/ci/runners/README.md
@@ -231,6 +231,38 @@ To make a Runner pick tagged/untagged jobs:
1. Check the **Run untagged jobs** option
1. Click **Save changes** for the changes to take effect
+### Setting maximum job timeout for a Runner
+
+For each Runner you can specify a _maximum job timeout_. Such timeout,
+if smaller than [project defined timeout], will take the precedence. This
+feature can be used to prevent Shared Runner from being appropriated
+by a project by setting a ridiculous big timeout (e.g. one week).
+
+When not configured, Runner will not override project timeout.
+
+How this feature will work:
+
+**Example 1 - Runner timeout bigger than project timeout**
+
+1. You set the _maximum job timeout_ for a Runner to 24 hours
+1. You set the _CI/CD Timeout_ for a project to **2 hours**
+1. You start a job
+1. The job, if running longer, will be timeouted after **2 hours**
+
+**Example 2 - Runner timeout not configured**
+
+1. You remove the _maximum job timeout_ configuration from a Runner
+1. You set the _CI/CD Timeout_ for a project to **2 hours**
+1. You start a job
+1. The job, if running longer, will be timeouted after **2 hours**
+
+**Example 3 - Runner timeout smaller than project timeout**
+
+1. You set the _maximum job timeout_ for a Runner to **30 minutes**
+1. You set the _CI/CD Timeout_ for a project to 2 hours
+1. You start a job
+1. The job, if running longer, will be timeouted after **30 minutes**
+
### Be careful with sensitive information
With some [Runner Executors](https://docs.gitlab.com/runner/executors/README.html),
@@ -259,12 +291,6 @@ Mentioned briefly earlier, but the following things of Runners can be exploited.
We're always looking for contributions that can mitigate these
[Security Considerations](https://docs.gitlab.com/runner/security/).
-[install]: http://docs.gitlab.com/runner/install/
-[fifo]: https://en.wikipedia.org/wiki/FIFO_(computing_and_electronics)
-[register]: http://docs.gitlab.com/runner/register/
-[protected branches]: ../../user/project/protected_branches.md
-[protected tags]: ../../user/project/protected_tags.md
-
## Determining the IP address of a Runner
> [Introduced](https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/17286) in GitLab 10.6.
@@ -297,3 +323,10 @@ You can find the IP address of a Runner for a specific project by:
1. On the details page you should see a row for "IP Address"
![specific Runner IP address](img/specific_runner_ip_address.png)
+
+[install]: http://docs.gitlab.com/runner/install/
+[fifo]: https://en.wikipedia.org/wiki/FIFO_(computing_and_electronics)
+[register]: http://docs.gitlab.com/runner/register/
+[protected branches]: ../../user/project/protected_branches.md
+[protected tags]: ../../user/project/protected_tags.md
+[project defined timeout]: ../../user/project/pipelines/settings.html#timeout
diff --git a/doc/ci/yaml/README.md b/doc/ci/yaml/README.md
index 3382fbc2d12..9aa443fa69d 100644
--- a/doc/ci/yaml/README.md
+++ b/doc/ci/yaml/README.md
@@ -1545,8 +1545,9 @@ capitalization, the commit will be created but the pipeline will be skipped.
## Validate the .gitlab-ci.yml
-Each instance of GitLab CI has an embedded debug tool called Lint.
-You can find the link under `/ci/lint` of your gitlab instance.
+Each instance of GitLab CI has an embedded debug tool called Lint, which validates the
+content of your `.gitlab-ci.yml` files. You can find the Lint under the page `ci/lint` of your
+project namespace (e.g, `http://gitlab-example.com/gitlab-org/project-123/ci/lint`)
## Using reserved keywords
diff --git a/doc/development/ee_features.md b/doc/development/ee_features.md
index fea92e740cb..3ba03d2d591 100644
--- a/doc/development/ee_features.md
+++ b/doc/development/ee_features.md
@@ -33,6 +33,26 @@ rest of the code should be as close to the CE files as possible.
[single code base]: https://gitlab.com/gitlab-org/gitlab-ee/issues/2952#note_41016454
+### EE-specific comments
+
+When complete separation can't be achieved with the `ee/` directory, you can wrap
+code in EE specific comments to designate the difference from CE/EE and add
+some context for someone resolving a conflict.
+
+```rb
+# EE-specific start
+stub_licensed_features(variable_environment_scope: true)
+# EE specific end
+```
+
+```haml
+-# EE-specific start
+= render 'ci/variables/environment_scope', form_field: form_field, variable: variable
+-# EE-specific end
+```
+
+EE-specific comments should not be backported to CE.
+
### Detection of EE-only files
For each commit (except on `master`), the `ee-files-location-check` CI job tries
@@ -350,6 +370,255 @@ class beneath the `EE` module just as you would normally.
For example, if CE has LDAP classes in `lib/gitlab/ldap/` then you would place
EE-specific LDAP classes in `ee/lib/ee/gitlab/ldap`.
+### Code in `lib/api/`
+
+It can be very tricky to extend EE features by a single line of `prepend`,
+and for each different [Grape](https://github.com/ruby-grape/grape) feature,
+we might need different strategies to extend it. To apply different strategies
+easily, we would use `extend ActiveSupport::Concern` in the EE module.
+
+Put the EE module files following
+[EE features based on CE features](#ee-features-based-on-ce-features).
+
+#### EE API routes
+
+For EE API routes, we put them in a `prepended` block:
+
+``` ruby
+module EE
+ module API
+ module MergeRequests
+ extend ActiveSupport::Concern
+
+ prepended do
+ params do
+ requires :id, type: String, desc: 'The ID of a project'
+ end
+ resource :projects, requirements: ::API::API::PROJECT_ENDPOINT_REQUIREMENTS do
+ # ...
+ end
+ end
+ end
+ end
+end
+```
+
+Note that due to namespace differences, we need to use the full qualifier for some
+constants.
+
+#### EE params
+
+We can define `params` and utilize `use` in another `params` definition to
+include params defined in EE. However, we need to define the "interface" first
+in CE in order for EE to override it. We don't have to do this in other places
+due to `prepend`, but Grape is complex internally and we couldn't easily do
+that, so we'll follow regular object-oriented practices that we define the
+interface first here.
+
+For example, suppose we have a few more optional params for EE, given this CE
+API code:
+
+``` ruby
+module API
+ class MergeRequests < Grape::API
+ # EE::API::MergeRequests would override the following helpers
+ helpers do
+ params :optional_params_ee do
+ end
+ end
+
+ prepend EE::API::MergeRequests
+
+ params :optional_params do
+ # CE specific params go here...
+
+ use :optional_params_ee
+ end
+ end
+end
+```
+
+And then we could override it in EE module:
+
+``` ruby
+module EE
+ module API
+ module MergeRequests
+ extend ActiveSupport::Concern
+
+ prepended do
+ helpers do
+ params :optional_params_ee do
+ # EE specific params go here...
+ end
+ end
+ end
+ end
+ end
+end
+```
+
+This way, the only difference between CE and EE for that API file would be
+`prepend EE::API::MergeRequests`.
+
+#### EE helpers
+
+To make it easy for an EE module to override the CE helpers, we need to define
+those helpers we want to extend first. Try to do that immediately after the
+class definition to make it easy and clear:
+
+``` ruby
+module API
+ class JobArtifacts < Grape::API
+ # EE::API::JobArtifacts would override the following helpers
+ helpers do
+ def authorize_download_artifacts!
+ authorize_read_builds!
+ end
+ end
+
+ prepend EE::API::JobArtifacts
+ end
+end
+```
+
+And then we can follow regular object-oriented practices to override it:
+
+``` ruby
+module EE
+ module API
+ module JobArtifacts
+ extend ActiveSupport::Concern
+
+ prepended do
+ helpers do
+ def authorize_download_artifacts!
+ super
+ check_cross_project_pipelines_feature!
+ end
+ end
+ end
+ end
+ end
+end
+```
+
+#### EE-specific behaviour
+
+Sometimes we need EE-specific behaviour in some of the APIs. Normally we could
+use EE methods to override CE methods, however API routes are not methods and
+therefore can't be simply overridden. We need to extract them into a standalone
+method, or introduce some "hooks" where we could inject behavior in the CE
+route. Something like this:
+
+``` ruby
+module API
+ class MergeRequests < Grape::API
+ helpers do
+ # EE::API::MergeRequests would override the following helpers
+ def update_merge_request_ee(merge_request)
+ end
+ end
+
+ prepend EE::API::MergeRequests
+
+ put ':id/merge_requests/:merge_request_iid/merge' do
+ merge_request = find_project_merge_request(params[:merge_request_iid])
+
+ # ...
+
+ update_merge_request_ee(merge_request)
+
+ # ...
+ end
+ end
+end
+```
+
+Note that `update_merge_request_ee` doesn't do anything in CE, but
+then we could override it in EE:
+
+``` ruby
+module EE
+ module API
+ module MergeRequests
+ extend ActiveSupport::Concern
+
+ prepended do
+ helpers do
+ def update_merge_request_ee(merge_request)
+ # ...
+ end
+ end
+ end
+ end
+ end
+end
+```
+
+#### EE `route_setting`
+
+It's very hard to extend this in an EE module, and this is simply storing
+some meta-data for a particular route. Given that, we could simply leave the
+EE `route_setting` in CE as it won't hurt and we are just not going to use
+those meta-data in CE.
+
+We could revisit this policy when we're using `route_setting` more and whether
+or not we really need to extend it from EE. For now we're not using it much.
+
+#### Utilizing class methods for setting up EE-specific data
+
+Sometimes we need to use different arguments for a particular API route, and we
+can't easily extend it with an EE module because Grape has different context in
+different blocks. In order to overcome this, we could use class methods from the
+API class.
+
+For example, in one place we need to pass an extra argument to
+`at_least_one_of` so that the API could consider an EE-only argument as the
+least argument. This is not quite beautiful but it's working:
+
+``` ruby
+module API
+ class MergeRequests < Grape::API
+ def self.update_params_at_least_one_of
+ %i[
+ assignee_id
+ description
+ ]
+ end
+
+ prepend EE::API::MergeRequests
+
+ params do
+ at_least_one_of(*::API::MergeRequests.update_params_at_least_one_of)
+ end
+ end
+end
+```
+
+And then we could easily extend that argument in the EE class method:
+
+``` ruby
+module EE
+ module API
+ module MergeRequests
+ extend ActiveSupport::Concern
+
+ class_methods do
+ def update_params_at_least_one_of
+ super.push(*%i[
+ squash
+ ])
+ end
+ end
+ end
+ end
+end
+```
+
+It could be annoying if we need this for a lot of routes, but it might be the
+simplest solution right now.
+
### Code in `spec/`
When you're testing EE-only features, avoid adding examples to the
@@ -405,12 +674,13 @@ to avoid conflicts during CE to EE merge.
}
}
-/* EE-specific styles */
+// EE-specific start
.section-body.ee-section-body {
.section-title {
background: $gl-header-color-cyan;
}
}
+// EE-specific end
```
## gitlab-svgs
diff --git a/doc/development/new_fe_guide/style/javascript.md b/doc/development/new_fe_guide/style/javascript.md
index 480d50a211f..57efd9353bc 100644
--- a/doc/development/new_fe_guide/style/javascript.md
+++ b/doc/development/new_fe_guide/style/javascript.md
@@ -1,3 +1,195 @@
# JavaScript style guide
-> TODO: Add content
+We use [Airbnb's JavaScript Style Guide][airbnb-style-guide] and it's accompanying linter to manage most of our JavaScript style guidelines.
+
+In addition to the style guidelines set by Airbnb, we also have a few specific rules listed below.
+
+> **Tip:**
+You can run eslint locally by running `yarn eslint`
+
+## Arrays
+
+<a name="avoid-foreach"></a><a name="1.1"></a>
+- [1.1](#avoid-foreach) **Avoid ForEach when mutating data** Use `map`, `reduce` or `filter` instead of `forEach` when mutating data. This will minimize mutations in functions ([which is aligned with Airbnb's style guide][airbnb-minimize-mutations])
+
+```
+// bad
+users.forEach((user, index) => {
+ user.id = index;
+});
+
+// good
+const usersWithId = users.map((user, index) => {
+ return Object.assign({}, user, { id: index });
+});
+```
+
+## Functions
+
+<a name="limit-params"></a><a name="2.1"></a>
+- [2.1](#limit-params) **Limit number of parameters** If your function or method has more than 3 parameters, use an object as a parameter instead.
+
+```
+// bad
+function a(p1, p2, p3) {
+ // ...
+};
+
+// good
+function a(p) {
+ // ...
+};
+```
+
+## Classes & constructors
+
+<a name="avoid-constructor-side-effects"></a><a name="3.1"></a>
+- [3.1](#avoid-constructor-side-effects) **Avoid side effects in constructors** Avoid making some operations in the `constructor`, such as asynchronous calls, API requests and DOM manipulations. Prefer moving them into separate functions. This will make tests easier to write and code easier to maintain.
+
+ ```javascript
+ // bad
+ class myClass {
+ constructor(config) {
+ this.config = config;
+ axios.get(this.config.endpoint)
+ }
+ }
+
+ // good
+ class myClass {
+ constructor(config) {
+ this.config = config;
+ }
+
+ makeRequest() {
+ axios.get(this.config.endpoint)
+ }
+ }
+ const instance = new myClass();
+ instance.makeRequest();
+
+ ```
+
+<a name="avoid-classes-to-handle-dom-events"></a><a name="3.2"></a>
+- [3.2](#avoid-classes-to-handle-dom-events) **Avoid classes to handle DOM events** If the only purpose of the class is to bind a DOM event and handle the callback, prefer using a function.
+
+```
+// bad
+class myClass {
+ constructor(config) {
+ this.config = config;
+ }
+
+ init() {
+ document.addEventListener('click', () => {});
+ }
+}
+
+// good
+
+const myFunction = () => {
+ document.addEventListener('click', () => {
+ // handle callback here
+ });
+}
+```
+
+<a name="element-container"></a><a name="3.3"></a>
+- [3.3](#element-container) **Pass element container to constructor** When your class manipulates the DOM, receive the element container as a parameter.
+This is more maintainable and performant.
+
+```
+// bad
+class a {
+ constructor() {
+ document.querySelector('.b');
+ }
+}
+
+// good
+class a {
+ constructor(options) {
+ options.container.querySelector('.b');
+ }
+}
+```
+
+## Type Casting & Coercion
+
+<a name="use-parseint"></a><a name="4.1"></a>
+- [4.1](#use-parseint) **Use ParseInt** Use `ParseInt` when converting a numeric string into a number.
+
+```
+// bad
+Number('10')
+
+
+// good
+parseInt('10', 10);
+```
+
+## CSS Selectors
+
+<a name="use-js-prefix"></a><a name="5.1"></a>
+- [5.1](#use-js-prefix) **Use js prefix** If a CSS class is only being used in JavaScript as a reference to the element, prefix the class name with `js-`
+
+```
+// bad
+<button class="add-user"></button>
+
+// good
+<button class="js-add-user"></button>
+```
+
+## Modules
+
+<a name="use-absolute-paths"></a><a name="6.1"></a>
+- [6.1](#use-absolute-paths) **Use absolute paths for nearby modules** Use absolute paths if the module you are importing is less than two levels up.
+
+```
+// bad
+import GitLabStyleGuide from '~/guides/GitLabStyleGuide';
+
+// good
+import GitLabStyleGuide from '../GitLabStyleGuide';
+```
+
+<a name="use-relative-paths"></a><a name="6.2"></a>
+- [6.2](#use-relative-paths) **Use relative paths for distant modules** If the module you are importing is two or more levels up, use a relative path instead of an absolute path.
+
+```
+// bad
+import GitLabStyleGuide from '../../../guides/GitLabStyleGuide';
+
+// good
+import GitLabStyleGuide from '~/GitLabStyleGuide';
+```
+
+<a name="global-namespace"></a><a name="6.3"></a>
+- [6.3](#global-namespace) **Do not add to global namespace**
+
+<a name="domcontentloaded"></a><a name="6.4"></a>
+- [6.4](#domcontentloaded) **Do not use DOMContentLoaded in non-page modules** Imported modules should act the same each time they are loaded. `DOMContentLoaded` events are only allowed on modules loaded in the `/pages/*` directory because those are loaded dynamically with webpack.
+
+## Security
+
+<a name="avoid-xss"></a><a name="7.1"></a>
+- [7.1](#avoid-xss) **Avoid XSS** Do not use `innerHTML`, `append()` or `html()` to set content. It opens up too many vulnerabilities.
+
+## ESLint
+
+<a name="disable-eslint-file"></a><a name="8.1"></a>
+- [8.1](#disable-eslint-file) **Disabling ESLint in new files** Do not disable ESLint when creating new files. Existing files may have existing rules disabled due to legacy compatibility reasons but they are in the process of being refactored.
+
+<a name="disable-eslint-rule"></a><a name="8.2"></a>
+- [8.2](#disable-eslint-rule) **Disabling ESLint rule** Do not disable specific ESLint rules. Due to technical debt, you may disable the following rules only if you are invoking/instantiating existing code modules
+
+ - [no-new][no-new]
+ - [class-method-use-this][class-method-use-this]
+
+> Note: Disable these rules on a per line basis. This makes it easier to refactor in the future. E.g. use `eslint-disable-next-line` or `eslint-disable-line`
+
+[airbnb-style-guide]: https://github.com/airbnb/javascript
+[airbnb-minimize-mutations]: https://github.com/airbnb/javascript#testing--for-real
+[no-new]: http://eslint.org/docs/rules/no-new
+[class-method-use-this]: http://eslint.org/docs/rules/class-methods-use-this
diff --git a/doc/integration/omniauth.md b/doc/integration/omniauth.md
index 20087a981f9..3edde3de83d 100644
--- a/doc/integration/omniauth.md
+++ b/doc/integration/omniauth.md
@@ -32,6 +32,7 @@ contains some settings that are common for all providers.
- [Auth0](auth0.md)
- [Authentiq](../administration/auth/authentiq.md)
- [OAuth2Generic](oauth2_generic.md)
+- [JWT](../administration/auth/jwt.md)
## Initial OmniAuth Configuration
diff --git a/doc/raketasks/README.md b/doc/raketasks/README.md
index 2f916f5dea7..90187617c41 100644
--- a/doc/raketasks/README.md
+++ b/doc/raketasks/README.md
@@ -14,3 +14,4 @@ comments: false
- [Webhooks](web_hooks.md)
- [Import](import.md) of git repositories in bulk
- [Rebuild authorized_keys file](http://docs.gitlab.com/ce/raketasks/maintenance.html#rebuild-authorized_keys-file) task for administrators
+- [Migrate Uploads](../administration/raketasks/uploads/migrate.md)
diff --git a/doc/topics/autodevops/index.md b/doc/topics/autodevops/index.md
index 4dc3adc1441..e88b787187c 100644
--- a/doc/topics/autodevops/index.md
+++ b/doc/topics/autodevops/index.md
@@ -20,6 +20,7 @@ project in an easy and automatic way:
1. [Auto Test](#auto-test)
1. [Auto Code Quality](#auto-code-quality)
1. [Auto SAST (Static Application Security Testing)](#auto-sast)
+1. [Auto Dependency Scanning](#auto-dependency-scanning)
1. [Auto Container Scanning](#auto-container-scanning)
1. [Auto Review Apps](#auto-review-apps)
1. [Auto DAST (Dynamic Application Security Testing)](#auto-dast)
@@ -95,7 +96,7 @@ Auto Deploy, and Auto Monitoring will be silently skipped.
The Auto DevOps base domain is required if you want to make use of [Auto
Review Apps](#auto-review-apps) and [Auto Deploy](#auto-deploy). It is defined
-either under the project's CI/CD settings while
+either under the project's CI/CD settings while
[enabling Auto DevOps](#enabling-auto-devops) or in instance-wide settings in
the CI/CD section.
It can also be set at the project or group level as a variable, `AUTO_DEVOPS_DOMAIN`.
@@ -209,7 +210,7 @@ target branches are also
> Introduced in [GitLab Ultimate][ee] 10.3.
Static Application Security Testing (SAST) uses the
-[gl-sast Docker image](https://gitlab.com/gitlab-org/gl-sast) to run static
+[SAST Docker image](https://gitlab.com/gitlab-org/security-products/sast) to run static
analysis on the current code and checks for potential security issues. Once the
report is created, it's uploaded as an artifact which you can later download and
check out.
@@ -217,6 +218,19 @@ check out.
In GitLab Ultimate, any security warnings are also
[shown in the merge request widget](https://docs.gitlab.com/ee/user/project/merge_requests/sast.html).
+### Auto Dependency Scanning
+
+> Introduced in [GitLab Ultimate][ee] 10.7.
+
+Dependency Scanning uses the
+[Dependency Scanning Docker image](https://gitlab.com/gitlab-org/security-products/dependency-scanning)
+to run analysis on the project dependencies and checks for potential security issues. Once the
+report is created, it's uploaded as an artifact which you can later download and
+check out.
+
+In GitLab Ultimate, any security warnings are also
+[shown in the merge request widget](https://docs.gitlab.com/ee/user/project/merge_requests/dependency_scanning.html).
+
### Auto Container Scanning
> Introduced in GitLab 10.4.
diff --git a/doc/user/profile/preferences.md b/doc/user/profile/preferences.md
index 022d6317555..930e506802a 100644
--- a/doc/user/profile/preferences.md
+++ b/doc/user/profile/preferences.md
@@ -41,7 +41,7 @@ select few, the amount of activity on the default Dashboard page can be
overwhelming. Changing this setting allows you to redefine what your default
dashboard will be.
-You have 6 options here that you can use for your default dashboard view:
+You have 8 options here that you can use for your default dashboard view:
- Your projects (default)
- Starred projects
@@ -49,6 +49,8 @@ You have 6 options here that you can use for your default dashboard view:
- Starred projects' activity
- Your groups
- Your [Todos]
+- Assigned Issues
+- Assigned Merge Requests
### Project home page content
diff --git a/doc/user/project/clusters/index.md b/doc/user/project/clusters/index.md
index bd9bcfadb99..716787532fc 100644
--- a/doc/user/project/clusters/index.md
+++ b/doc/user/project/clusters/index.md
@@ -71,7 +71,7 @@ You need Master [permissions] and above to access the Kubernetes page.
To add an existing Kubernetes cluster to your project:
1. Navigate to your project's **CI/CD > Kubernetes** page.
-1. Click on **Add Kuberntes cluster**.
+1. Click on **Add Kubernetes cluster**.
1. Click on **Add an existing Kubernetes cluster** and fill in the details:
- **Kubernetes cluster name** (required) - The name you wish to give the cluster.
- **Environment scope** (required)- The
@@ -101,7 +101,7 @@ To add an existing Kubernetes cluster to your project:
- If you or someone created a secret specifically for the project, usually
with limited permissions, the secret's namespace and project namespace may
be the same.
-1. Finally, click the **Create Kuberntes cluster** button.
+1. Finally, click the **Create Kubernetes cluster** button.
After a few moments, your cluster should be created. If something goes wrong,
you will be notified.
diff --git a/doc/user/project/index.md b/doc/user/project/index.md
index 175a8975ae1..f94e93dd7d8 100644
--- a/doc/user/project/index.md
+++ b/doc/user/project/index.md
@@ -128,11 +128,9 @@ and Git push/pull redirects.
Depending on the situation, different things apply.
-When [renaming a user](../profile/index.md#changing-your-username) or
-[changing a group path](../group/index.md#changing-a-group-s-path):
+When [renaming a user](../profile/index.md#changing-your-username),
+[changing a group path](../group/index.md#changing-a-group-s-path) or [renaming a repository](settings/index.md#renaming-a-repository):
-- **The redirect to the new URL is permanent**, which means that the original
- namespace can't be claimed again by any group or user.
- Existing web URLs for the namespace and anything under it (e.g., projects) will
redirect to the new URLs.
- Starting with GitLab 10.3, existing Git remote URLs for projects under the
@@ -141,9 +139,5 @@ When [renaming a user](../profile/index.md#changing-your-username) or
your remote will be displayed instead of rejecting your action.
This means that any automation scripts, or Git clients will continue to
work after a rename, making any transition a lot smoother.
- To avoid pulling from or pushing to an entirely incorrect repository, the old
- path will be reserved.
-
-When [renaming-a-repository](settings/index.md#renaming-a-repository), the same
-things apply, except for the Git push/pull actions which will be rejected with a
-warning message to change to the new remote URL.
+- The redirects will be available as long as the original path is not claimed by
+ another group, user or project.
diff --git a/doc/user/project/integrations/prometheus_library/kubernetes.md b/doc/user/project/integrations/prometheus_library/kubernetes.md
index 8ac753c07bf..6b190deaa6c 100644
--- a/doc/user/project/integrations/prometheus_library/kubernetes.md
+++ b/doc/user/project/integrations/prometheus_library/kubernetes.md
@@ -11,10 +11,17 @@ integration services must be enabled.
## Metrics supported
-| Name | Query |
-| ---- | ----- |
-| Average Memory Usage (MB) | avg(sum(container_memory_usage_bytes{container_name!="POD",pod_name=~"^%{ci_environment_slug}-([^c].*|c([^a]|a([^n]|n([^a]|a([^r]|r[^y])))).*|)-(.*)",namespace="%{kube_namespace}"}) by (job)) without (job) / count(avg(container_memory_usage_bytes{container_name!="POD",pod_name=~"^%{ci_environment_slug}-([^c].*|c([^a]|a([^n]|n([^a]|a([^r]|r[^y])))).*|)-(.*)",namespace="%{kube_namespace}"}) without (job)) /1024/1024 |
-| Average CPU Utilization (%) | avg(sum(rate(container_cpu_usage_seconds_total{container_name!="POD",pod_name=~"^%{ci_environment_slug}-([^c].*|c([^a]|a([^n]|n([^a]|a([^r]|r[^y])))).*|)-(.*)",namespace="%{kube_namespace}"}[15m])) by (job)) without (job) / count(sum(rate(container_cpu_usage_seconds_total{container_name!="POD",pod_name=~"^%{ci_environment_slug}-([^c].*|c([^a]|a([^n]|n([^a]|a([^r]|r[^y])))).*|)-(.*)",namespace="%{kube_namespace}"}[15m])) by (pod_name)) |
+- Average Memory Usage (MB):
+
+ ```
+ avg(sum(container_memory_usage_bytes{container_name!="POD",pod_name=~"^%{ci_environment_slug}-([^c].*|c([^a]|a([^n]|n([^a]|a([^r]|r[^y])))).*|)-(.*)",namespace="%{kube_namespace}"}) by (job)) without (job) / count(avg(container_memory_usage_bytes{container_name!="POD",pod_name=~"^%{ci_environment_slug}-([^c].*|c([^a]|a([^n]|n([^a]|a([^r]|r[^y])))).*|)-(.*)",namespace="%{kube_namespace}"}) without (job)) /1024/1024
+ ```
+
+- Average CPU Utilization (%):
+
+ ```
+ avg(sum(rate(container_cpu_usage_seconds_total{container_name!="POD",pod_name=~"^%{ci_environment_slug}-([^c].*|c([^a]|a([^n]|n([^a]|a([^r]|r[^y])))).*|)-(.*)",namespace="%{kube_namespace}"}[15m])) by (job)) without (job) / count(sum(rate(container_cpu_usage_seconds_total{container_name!="POD",pod_name=~"^%{ci_environment_slug}-([^c].*|c([^a]|a([^n]|n([^a]|a([^r]|r[^y])))).*|)-(.*)",namespace="%{kube_namespace}"}[15m])) by (pod_name))
+ ```
## Configuring Prometheus to monitor for Kubernetes metrics
diff --git a/doc/user/project/issue_board.md b/doc/user/project/issue_board.md
index d403d5698a9..b4a842f33d6 100644
--- a/doc/user/project/issue_board.md
+++ b/doc/user/project/issue_board.md
@@ -251,7 +251,7 @@ Different issue board features are available in different [GitLab tiers](https:/
| Tier | Number of project issue boards | Board with configuration in project issue boards | Number of group issue boards | Board with configuration in group issue boards |
| --- | --- | --- | --- | --- |
-| Libre | 1 | No | 1 | No |
+| Core | 1 | No | 1 | No |
| Starter | Multiple | Yes | 1 | No |
| Premium | Multiple | Yes | Multiple | Yes |
| Ultimate | Multiple | Yes | Multiple | Yes |
diff --git a/doc/user/project/merge_requests/img/remove_source_branch_status.png b/doc/user/project/merge_requests/img/remove_source_branch_status.png
new file mode 100644
index 00000000000..1377fab54ec
--- /dev/null
+++ b/doc/user/project/merge_requests/img/remove_source_branch_status.png
Binary files differ
diff --git a/doc/user/project/merge_requests/index.md b/doc/user/project/merge_requests/index.md
index 10d67729734..3640d236db4 100644
--- a/doc/user/project/merge_requests/index.md
+++ b/doc/user/project/merge_requests/index.md
@@ -77,6 +77,22 @@ You can [search and filter the results](../../search/index.md#issues-and-merge-r
![Group Issues list view](img/group_merge_requests_list_view.png)
+## Removing the source branch
+
+When creating a merge request, select the "Remove source branch when merge
+request accepted" option and the source branch will be removed when the merge
+request is merged.
+
+This option is also visible in an existing merge request next to the merge
+request button and can be selected/deselected before merging. It's only visible
+to users with [Master permissions](../../permissions.md) in the source project.
+
+If the user viewing the merge request does not have the correct permissions to
+remove the source branch and the source branch is set for removal, the merge
+request widget will show the "Removes source branch" text.
+
+![Remove source branch status](img/remove_source_branch_status.png)
+
## Authorization for merge requests
There are two main ways to have a merge request flow with GitLab:
diff --git a/doc/user/project/pipelines/settings.md b/doc/user/project/pipelines/settings.md
index 43451844f2d..1052c9efa25 100644
--- a/doc/user/project/pipelines/settings.md
+++ b/doc/user/project/pipelines/settings.md
@@ -27,6 +27,13 @@ The default value is 60 minutes. Decrease the time limit if you want to impose
a hard limit on your jobs' running time or increase it otherwise. In any case,
if the job surpasses the threshold, it is marked as failed.
+### Timeout overriding on Runner level
+
+> - [Introduced][ce-17221] in GitLab 10.6.
+
+Project defined timeout (either specific timeout set by user or the default
+60 minutes timeout) may be [overridden on Runner level][timeout overriding].
+
## Custom CI config path
> - [Introduced][ce-12509] in GitLab 9.4.
@@ -152,5 +159,7 @@ into your `README.md`:
[var]: ../../../ci/yaml/README.md#git-strategy
[coverage report]: #test-coverage-parsing
+[timeout overriding]: ../../../ci/runners/README.html#setting-maximum-job-timeout-for-a-runner
[ce-9362]: https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/9362
[ce-12509]: https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/12509
+[ce-17221]: https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/17221
diff --git a/doc/workflow/lfs/lfs_administration.md b/doc/workflow/lfs/lfs_administration.md
index d768b73286d..cac3cb599dd 100644
--- a/doc/workflow/lfs/lfs_administration.md
+++ b/doc/workflow/lfs/lfs_administration.md
@@ -5,6 +5,7 @@ Documentation on how to use Git LFS are under [Managing large binary files with
## Requirements
* Git LFS is supported in GitLab starting with version 8.2.
+* Support for object storage, such as AWS S3, was introduced in 10.0.
* Users need to install [Git LFS client](https://git-lfs.github.com) version 1.0.1 and up.
## Configuration
@@ -12,16 +13,18 @@ Documentation on how to use Git LFS are under [Managing large binary files with
Git LFS objects can be large in size. By default, they are stored on the server
GitLab is installed on.
-There are two configuration options to help GitLab server administrators:
+There are various configuration options to help GitLab server administrators:
* Enabling/disabling Git LFS support
* Changing the location of LFS object storage
+* Setting up AWS S3 compatible object storage
### Omnibus packages
In `/etc/gitlab/gitlab.rb`:
```ruby
+# Change to true to enable lfs
gitlab_rails['lfs_enabled'] = false
# Optionally, change the storage path location. Defaults to
@@ -35,11 +38,115 @@ gitlab_rails['lfs_storage_path'] = "/mnt/storage/lfs-objects"
In `config/gitlab.yml`:
```yaml
+# Change to true to enable lfs
lfs:
enabled: false
storage_path: /mnt/storage/lfs-objects
```
+## Setting up S3 compatible object storage
+
+> **Note:** [Introduced][ee-2760] in [GitLab Premium][eep] 10.0.
+> Available in [GitLab CE][ce] 10.7
+
+It is possible to store LFS objects on remote object storage instead of on a local disk.
+
+This allows you to offload storage to an external AWS S3 compatible service, freeing up disk space locally. You can also host your own S3 compatible storage decoupled from GitLab, with with a service such as [Minio](https://www.minio.io/).
+
+Object storage currently transfers files first to GitLab, and then on the object storage in a second stage. This can be done either by using a rake task to transfer existing objects, or in a background job after each file is received.
+
+### Object Storage Settings
+
+For source installations the following settings are nested under `lfs:` and then `object_store:`. On omnibus installs they are prefixed by `lfs_object_store_`.
+
+| Setting | Description | Default |
+|---------|-------------|---------|
+| `enabled` | Enable/disable object storage | `false` |
+| `remote_directory` | The bucket name where LFS objects will be stored| |
+| `direct_upload` | Set to true to enable direct upload of LFS without the need of local shared storage. Option may be removed once we decide to support only single storage for all files. | `false` |
+| `background_upload` | Set to false to disable automatic upload. Option may be removed once upload is direct to S3 | `true` |
+| `proxy_download` | Set to true to enable proxying all files served. Option allows to reduce egress traffic as this allows clients to download directly from remote storage instead of proxying all data | `false` |
+| `connection` | Various connection options described below | |
+
+#### S3 compatible connection settings
+
+The connection settings match those provided by [Fog](https://github.com/fog), and are as follows:
+
+| Setting | Description | Default |
+|---------|-------------|---------|
+| `provider` | Always `AWS` for compatible hosts | AWS |
+| `aws_access_key_id` | AWS credentials, or compatible | |
+| `aws_secret_access_key` | AWS credentials, or compatible | |
+| `region` | AWS region | us-east-1 |
+| `host` | S3 compatible host for when not using AWS, e.g. `localhost` or `storage.example.com` | s3.amazonaws.com |
+| `endpoint` | Can be used when configuring an S3 compatible service such as [Minio](https://www.minio.io), by entering a URL such as `http://127.0.0.1:9000` | (optional) |
+| `path_style` | Set to true to use `host/bucket_name/object` style paths instead of `bucket_name.host/object`. Leave as false for AWS S3 | false |
+
+
+### From source
+
+1. Edit `/home/git/gitlab/config/gitlab.yml` and add or amend the following
+ lines:
+
+ ```yaml
+ lfs:
+ enabled: true
+ object_store:
+ enabled: false
+ remote_directory: lfs-objects # Bucket name
+ connection:
+ provider: AWS
+ aws_access_key_id: 1ABCD2EFGHI34JKLM567N
+ aws_secret_access_key: abcdefhijklmnopQRSTUVwxyz0123456789ABCDE
+ region: eu-central-1
+ # Use the following options to configure an AWS compatible host such as Minio
+ host: 'localhost'
+ endpoint: 'http://127.0.0.1:9000'
+ path_style: true
+ ```
+
+1. Save the file and [restart GitLab][] for the changes to take effect.
+1. Migrate any existing local LFS objects to the object storage:
+
+ ```bash
+ sudo -u git -H bundle exec rake gitlab:lfs:migrate RAILS_ENV=production
+ ```
+
+ This will migrate existing LFS objects to object storage. New LFS objects
+ will be forwarded to object storage unless
+ `gitlab_rails['lfs_object_store_background_upload']` is set to false.
+
+### In Omnibus
+
+1. Edit `/etc/gitlab/gitlab.rb` and add the following lines by replacing with
+ the values you want:
+
+ ```ruby
+ gitlab_rails['lfs_object_store_enabled'] = true
+ gitlab_rails['lfs_object_store_remote_directory'] = "lfs-objects"
+ gitlab_rails['lfs_object_store_connection'] = {
+ 'provider' => 'AWS',
+ 'region' => 'eu-central-1',
+ 'aws_access_key_id' => '1ABCD2EFGHI34JKLM567N',
+ 'aws_secret_access_key' => 'abcdefhijklmnopQRSTUVwxyz0123456789ABCDE',
+ # The below options configure an S3 compatible host instead of AWS
+ 'host' => 'localhost',
+ 'endpoint' => 'http://127.0.0.1:9000',
+ 'path_style' => true
+ }
+ ```
+
+1. Save the file and [reconfigure GitLab]s for the changes to take effect.
+1. Migrate any existing local LFS objects to the object storage:
+
+ ```bash
+ gitlab-rake gitlab:lfs:migrate
+ ```
+
+ This will migrate existing LFS objects to object storage. New LFS objects
+ will be forwarded to object storage unless
+ `gitlab_rails['lfs_object_store_background_upload']` is set to false.
+
## Storage statistics
You can see the total storage used for LFS objects on groups and projects
@@ -48,10 +155,13 @@ and [projects APIs](../../api/projects.md).
## Known limitations
-* Currently, storing GitLab Git LFS objects on a non-local storage (like S3 buckets)
- is not supported
* Support for removing unreferenced LFS objects was added in 8.14 onwards.
* LFS authentications via SSH was added with GitLab 8.12
* Only compatible with the GitLFS client versions 1.1.0 and up, or 1.0.2.
* The storage statistics currently count each LFS object multiple times for
every project linking to it
+
+[reconfigure gitlab]: ../../administration/restart_gitlab.md#omnibus-gitlab-reconfigure "How to reconfigure Omnibus GitLab"
+[restart gitlab]: ../../administration/restart_gitlab.md#installations-from-source "How to restart GitLab"
+[eep]: https://about.gitlab.com/products/ "GitLab Premium"
+[ee-2760]: https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/2760