Merge branch 'master' into update-todo-in-uiupdate-todo-in-ui

* master: (435 commits)
  Change occurrence of Sidekiq::Testing.inline!
  Fix order-dependent spec failure in appearance_spec.rb
  Put a failed example from appearance_spec in quarantine
  Cache PerformanceBar.allowed_user_ids list locally and in Redis
  Add Grafana to Admin > Monitoring menu when enabled
  Add changelog entry
  Add salesforce logo
  Move error_tracking_frontend specs to Jest
  Only save Peek session in Redis when Peek is enabled
  Migrate markdown header_spec.js to Jest
  Fix golint command in Go guide doc to be recursive
  Move images to their own dirs
  Gitlab -> GitLab
  Re-align CE and EE API docs
  Rename Release groups in issue_workflow.md
  Update api docs to finish aligning EE and CE docs
  Update locale.pot
  Update TODO: allow_collaboration column renaming
  Show upcoming status for releases
  Rebased and squashed commits
  ...

6 files changed, 233 insertions, 26 deletions

diff --git a/doc/user/application_security/container_scanning/index.md b/doc/user/application_security/container_scanning/index.md
index 4a2fb1d7190..9dfbe326f1d 100644
--- a/doc/user/application_security/container_scanning/index.md
+++ b/doc/user/application_security/container_scanning/index.md
@@ -206,6 +206,11 @@ vulnerabilities in your groups and projects. Read more about the
 Once a vulnerability is found, you can interact with it. Read more on how to
 [interact with the vulnerabilities](../index.md#interacting-with-the-vulnerabilities).
 
+## Vulnerabilities database update
+
+For more information about the vulnerabilities database update, check the
+[maintenance table](../index.md#maintenance-and-update-of-the-vulnerabilities-database).
+
 ## Troubleshooting
 
 ### docker: Error response from daemon: failed to copy xattrs
diff --git a/doc/user/application_security/dast/index.md b/doc/user/application_security/dast/index.md
index a722aa88f9d..2283efe3a44 100644
--- a/doc/user/application_security/dast/index.md
+++ b/doc/user/application_security/dast/index.md
@@ -259,3 +259,8 @@ vulnerabilities in your groups and projects. Read more about the
 
 Once a vulnerability is found, you can interact with it. Read more on how to
 [interact with the vulnerabilities](../index.md#interacting-with-the-vulnerabilities).
+
+## Vulnerabilities database update
+
+For more information about the vulnerabilities database update, check the
+[maintenance table](../index.md#maintenance-and-update-of-the-vulnerabilities-database).
diff --git a/doc/user/application_security/dependency_scanning/analyzers.md b/doc/user/application_security/dependency_scanning/analyzers.md
new file mode 100644
index 00000000000..937ded287e5
--- /dev/null
+++ b/doc/user/application_security/dependency_scanning/analyzers.md
@@ -0,0 +1,133 @@
+# Dependency Scanning Analyzers **[ULTIMATE]**
+
+Dependency Scanning relies on underlying third party tools that are wrapped into
+what we call "Analyzers". An analyzer is a
+[dedicated project](https://gitlab.com/gitlab-org/security-products/analyzers)
+that wraps a particular tool to:
+
+- Expose its detection logic.
+- Handle its execution.
+- Convert its output to the common format.
+
+This is achieved by implementing the [common API](https://gitlab.com/gitlab-org/security-products/analyzers/common).
+
+Dependency Scanning supports the following official analyzers:
+
+- [`bundler-audit`](https://gitlab.com/gitlab-org/security-products/analyzers/bundler-audit)
+- [`gemnasium`](https://gitlab.com/gitlab-org/security-products/analyzers/gemnasium)
+- [`gemnasium-maven`](https://gitlab.com/gitlab-org/security-products/analyzers/gemnasium-maven)
+- [`gemnasium-python`](https://gitlab.com/gitlab-org/security-products/analyzers/gemnasium-python)
+- [`retire.js`](https://gitlab.com/gitlab-org/security-products/analyzers/retire.js)
+
+The analyzers are published as Docker images that Dependency Scanning will use
+to launch dedicated containers for each analysis.
+
+Dependency Scanning is pre-configured with a set of **default images** that are
+maintained by GitLab, but users can also integrate their own **custom images**.
+
+## Official default analyzers
+
+Any custom change to the official analyzers can be achieved by using an
+[environment variable in your `.gitlab-ci.yml`](index.md#customizing-the-dependency-scanning-settings).
+
+### Using a custom Docker mirror
+
+You can switch to a custom Docker registry that provides the official analyzer
+images under a different prefix. For instance, the following instructs Dependency
+Scanning to pull `my-docker-registry/gl-images/gemnasium`
+instead of `registry.gitlab.com/gitlab-org/security-products/analyzers/gemnasium`.
+In `.gitlab-ci.yml` define:
+
+```yaml
+include:
+  template: Dependency-Scanning.gitlab-ci.yml
+
+variables:
+  DS_ANALYZER_IMAGE_PREFIX: my-docker-registry/gl-images
+```
+
+This configuration requires that your custom registry provides images for all
+the official analyzers.
+
+### Selecting specific analyzers
+
+You can select the official analyzers you want to run. Here's how to enable
+`bundler-audit` and `gemnasium` while disabling all the other default ones.
+In `.gitlab-ci.yml` define:
+
+```yaml
+include:
+  template: Dependency-Scanning.gitlab-ci.yml
+
+variables:
+  DS_DEFAULT_ANALYZERS: "bundler-audit,gemnasium"
+```
+
+`bundler-audit` runs first. When merging the reports, Dependency Scanning will
+remove the duplicates and will keep the `bundler-audit` entries.
+
+### Disabling default analyzers
+
+Setting `DS_DEFAULT_ANALYZERS` to an empty string will disable all the official
+default analyzers. In `.gitlab-ci.yml` define:
+
+```yaml
+include:
+  template: Dependency-Scanning.gitlab-ci.yml
+
+variables:
+  DS_DEFAULT_ANALYZERS: ""
+```
+
+That's needed when one totally relies on [custom analyzers](#custom-analyzers).
+
+## Custom analyzers
+
+You can provide your own analyzers as a comma separated list of Docker images.
+Here's how to add `analyzers/nugget` and `analyzers/perl` to the default images.
+In `.gitlab-ci.yml` define:
+
+```yaml
+include:
+  template: Dependency-Scanning.gitlab-ci.yml
+
+variables:
+  DS_ANALYZER_IMAGES: "my-docker-registry/analyzers/nugget,amy-docker-registry/nalyzers/perl"
+```
+
+The values must be the full path to the container registry images,
+like what you would feed to the `docker pull` command.
+
+NOTE: **Note:**
+This configuration doesn't benefit from the integrated detection step. Dependency
+Scanning has to fetch and spawn each Docker image to establish whether the
+custom analyzer can scan the source code.
+
+## Analyzers data
+
+The following table lists the data available for each official analyzer.
+
+| Property \ Tool                       |      Gemnasium     |    bundler-audit   |     Retire.js      |
+|---------------------------------------|:------------------:|:------------------:|:------------------:|
+| Severity                              | 𐄂                  | ✓                  | ✓                  |
+| Title                                 | ✓                  | ✓                  | ✓                  |
+| File                                  | ✓                  | ⚠                  | ✓                  |
+| Start line                            | 𐄂                  | 𐄂                  | 𐄂                  |
+| End line                              | 𐄂                  | 𐄂                  | 𐄂                  |
+| External ID (e.g., CVE)               | ✓                  | ✓                  | ⚠                  |
+| URLs                                  | ✓                  | ✓                  | ✓                  |
+| Internal doc/explanation              | ✓                  | 𐄂                  | 𐄂                  |
+| Solution                              | ✓                  | ✓                  | 𐄂                  |
+| Confidence                            | 𐄂                  | 𐄂                  | 𐄂                  |
+| Affected item (e.g. class or package) | ✓                  | ✓                  | ✓                  |
+| Source code extract                   | 𐄂                  | 𐄂                  | 𐄂                  |
+| Internal ID                           | ✓                  | 𐄂                  | 𐄂                  |
+| Date                                  | ✓                  | 𐄂                  | 𐄂                  |
+| Credits                               | ✓                  | 𐄂                  | 𐄂                  |
+
+- ✓ => we have that data
+- ⚠ => we have that data but it's partially reliable, or we need to extract that data from unstructured content
+- 𐄂 => we don't have that data or it would need to develop specific or inefficient/unreliable logic to obtain it.
+
+The values provided by these tools are heterogeneous so they are sometimes
+normalized into common values (e.g., `severity`, `confidence`, etc).
diff --git a/doc/user/application_security/dependency_scanning/index.md b/doc/user/application_security/dependency_scanning/index.md
index a4e5b19bdc7..9145e034dcb 100644
--- a/doc/user/application_security/dependency_scanning/index.md
+++ b/doc/user/application_security/dependency_scanning/index.md
@@ -46,17 +46,33 @@ this is enabled by default.
 
 The following languages and dependency managers are supported.
 
-| Language (package managers)                                                 | Scan tool                                                                                                                         |
-|-----------------------------------------------------------------------------|-----------------------------------------------------------------------------------------------------------------------------------|
-| JavaScript ([npm](https://www.npmjs.com/), [yarn](https://yarnpkg.com/en/)) | [gemnasium](https://gitlab.com/gitlab-org/security-products/gemnasium/general), [Retire.js](https://retirejs.github.io/retire.js)         |
-| Python ([pip](https://pip.pypa.io/en/stable/)) (only `requirements.txt` supported)  | [gemnasium](https://gitlab.com/gitlab-org/security-products/gemnasium/general)                                                            |
-| Ruby ([gem](https://rubygems.org/))                                         | [gemnasium](https://gitlab.com/gitlab-org/security-products/gemnasium/general), [bundler-audit](https://github.com/rubysec/bundler-audit) |
-| Java ([Maven](https://maven.apache.org/))                                   | [gemnasium](https://gitlab.com/gitlab-org/security-products/gemnasium/general)                                                            |
-| PHP ([Composer](https://getcomposer.org/))                                  | [gemnasium](https://gitlab.com/gitlab-org/security-products/gemnasium/general)                                                            |
-
-Some scanners require to send a list of project dependencies to GitLab's central
-servers to check for vulnerabilities. To learn more about this or to disable it,
-refer to the [GitLab Dependency Scanning tool documentation](https://gitlab.com/gitlab-org/security-products/dependency-scanning#remote-checks).
+| Language (package managers)  | Supported | Scan tool(s) |
+|----------------------------- | --------- | ------------ |
+| JavaScript ([npm](https://www.npmjs.com/), [yarn](https://yarnpkg.com/en/)) | yes | [gemnasium](https://gitlab.com/gitlab-org/security-products/gemnasium), [Retire.js](https://retirejs.github.io/retire.js)         |
+| Python ([pip](https://pip.pypa.io/en/stable/)) (only `requirements.txt` supported)  | yes | [gemnasium](https://gitlab.com/gitlab-org/security-products/gemnasium) |
+| Ruby ([gem](https://rubygems.org/)) | yes | [gemnasium](https://gitlab.com/gitlab-org/security-products/gemnasium), [bundler-audit](https://github.com/rubysec/bundler-audit) |
+| Java ([Maven](https://maven.apache.org/)) | yes | [gemnasium](https://gitlab.com/gitlab-org/security-products/gemnasium) |
+| PHP ([Composer](https://getcomposer.org/))  | yes | [gemnasium](https://gitlab.com/gitlab-org/security-products/gemnasium) |
+| Python ([poetry](https://poetry.eustace.io/)) | no ([issue](https://gitlab.com/gitlab-org/gitlab-ee/issues/7006 "Support Poetry in Dependency Scanning")) | not available |
+| Python ([Pipfile](https://docs.pipenv.org/en/latest/basics/)) | no ([issue](https://gitlab.com/gitlab-org/gitlab-ee/issues/11756 "Pipfile.lock support for Dependency Scanning"))| not available |
+| Go ([Golang](https://golang.org/)) | no ([issue](https://gitlab.com/gitlab-org/gitlab-ee/issues/7132 "Dependency Scanning for Go")) | not available |
+
+## Remote checks
+
+While some tools pull a local database to check vulnerabilities, some others
+like Gemnasium require sending data to GitLab central servers to analyze them:
+
+1. Gemnasium scans the dependencies of your project locally and sends a list of
+   packages to GitLab central servers.
+1. The servers return the list of known vulnerabilities for all versions of
+   these packages.
+1. The client picks up the relevant vulnerabilities by comparing with the versions
+   of the packages that are used by the project.
+
+The Gemnasium client does **NOT** send the exact package versions your project relies on.
+
+You can disable the remote checks by [using](#customizing-the-dependency-scanning-settings)
+the `DS_DISABLE_REMOTE_CHECKS` environment variable and setting it to `true`.
 
 ## Configuring Dependency Scanning
 
@@ -97,17 +113,10 @@ The report will be saved as a
 that you can later download and analyze. Due to implementation limitations, we
 always take the latest Dependency Scanning artifact available.
 
-Some security scanners require to send a list of project dependencies to GitLab
-central servers to check for vulnerabilities. To learn more about this or to
-disable it, check the
-[GitLab Dependency Scanning tool documentation](https://gitlab.com/gitlab-org/security-products/dependency-scanning#remote-checks).
-
 #### Customizing the Dependency Scanning settings
 
-The Dependency Scanning settings can be changed through environment variables by using the
+The Dependency Scanning settings can be changed through [environment variables](#available-variables) by using the
 [`variables`](../../../ci/yaml/README.md#variables) parameter in `.gitlab-ci.yml`.
-These variables are documented in the
-[Dependency Scanning tool documentation](https://gitlab.com/gitlab-org/security-products/dependency-scanning#settings).
 
 For example:
 
@@ -116,7 +125,7 @@ include:
   template: Dependency-Scanning.gitlab-ci.yml
 
 variables:
-  DEP_SCAN_DISABLE_REMOTE_CHECKS: true
+  DS_DISABLE_REMOTE_CHECKS: true
 ```
 
 Because template is [evaluated before](../../../ci/yaml/README.md#include) the pipeline
@@ -137,6 +146,24 @@ dependency_scanning:
     CI_DEBUG_TRACE: "true"
 ```
 
+#### Available variables
+
+Dependency Scanning can be [configured](#customizing-the-dependency-scanning-settings)
+using environment variables.
+
+| Environment variable                    | Function |
+|--------------------------------         |----------|
+| `DS_ANALYZER_IMAGES`                    | Comma separated list of custom images. The official default images are still enabled. Read more about [customizing analyzers](analyzers.md). |
+| `DS_ANALYZER_IMAGE_PREFIX`              | Override the name of the Docker registry providing the official default images (proxy). Read more about [customizing analyzers](analyzers.md). |
+| `DS_ANALYZER_IMAGE_TAG`                 | Override the Docker tag of the official default images. Read more about [customizing analyzers](analyzers.md). |
+| `DS_DEFAULT_ANALYZERS`                  | Override the names of the official default images. Read more about [customizing analyzers](analyzers.md). |
+| `DS_DISABLE_REMOTE_CHECKS`              | Do not send any data to GitLab. Used in the [Gemnasium analyzer](#remote-checks). |
+| `DS_PULL_ANALYZER_IMAGES`               | Pull the images from the Docker registry (set to `0` to disable). |
+| `DS_EXCLUDED_PATHS`                     | Exclude vulnerabilities from output based on the paths. A comma-separated list of patterns. Patterns can be globs, file or folder paths. Parent directories will also match patterns. |
+| `DS_DOCKER_CLIENT_NEGOTIATION_TIMEOUT`  | Time limit for Docker client negotiation. Timeouts are parsed using Go's [`ParseDuration`](https://golang.org/pkg/time/#ParseDuration). Valid time units are `ns`, `us` (or `µs`), `ms`, `s`, `m`, `h`. For example, `300ms`, `1.5h`, or `2h45m`. |
+| `DS_PULL_ANALYZER_IMAGE_TIMEOUT`        | Time limit when pulling the image of an analyzer. Timeouts are parsed using Go's [`ParseDuration`](https://golang.org/pkg/time/#ParseDuration). Valid time units are `ns`, `us` (or `µs`), `ms`, `s`, `m`, `h`. For example, `300ms`, `1.5h`, or `2h45m`. |
+| `DS_RUN_ANALYZER_TIMEOUT`               | Time limit when running an analyzer. Timeouts are parsed using Go's [`ParseDuration`](https://golang.org/pkg/time/#ParseDuration). Valid time units are `ns`, `us` (or `µs`), `ms`, `s`, `m`, `h`. For example, `300ms`, `1.5h`, or `2h45m`. |
+
 ### Manual job definition for GitLab 11.5 and later
 
 For GitLab 11.5 and GitLab Runner 11.5 and later, the following `dependency_scanning`
@@ -171,7 +198,7 @@ dependency_scanning:
       dependency_scanning: gl-dependency-scanning-report.json
 ```
 
-You can supply many other [settings variables](https://gitlab.com/gitlab-org/security-products/dependency-scanning#settings)
+You can supply many other [settings variables](#available-variables)
 via `docker run --env` to customize your job execution.
 
 ### Manual job definition for GitLab 11.4 and earlier (deprecated)
@@ -377,6 +404,11 @@ vulnerabilities in your groups and projects. Read more about the
 Once a vulnerability is found, you can interact with it. Read more on how to
 [interact with the vulnerabilities](../index.md#interacting-with-the-vulnerabilities).
 
+## Vulnerabilities database update
+
+For more information about the vulnerabilities database update, check the
+[maintenance table](../index.md#maintenance-and-update-of-the-vulnerabilities-database).
+
 ## Dependency List
 
 > [Introduced](https://gitlab.com/gitlab-org/gitlab-ee/issues/10075) in [GitLab Ultimate](https://about.gitlab.com/pricing/) 12.0.
@@ -388,6 +420,10 @@ supported by Gemnasium.
 
 To see the generated dependency list, navigate to your project's **Project > Dependency List**.
 
+## Versioning and release process
+
+Please check the [Release Process documentation](https://gitlab.com/gitlab-org/security-products/release/blob/master/docs/release_process.md).
+
 ## Contributing to the vulnerability database
 
 You can search the [gemnasium-db](https://gitlab.com/gitlab-org/security-products/gemnasium-db) project
diff --git a/doc/user/application_security/index.md b/doc/user/application_security/index.md
index 679847b76d7..69fa1ec5da6 100644
--- a/doc/user/application_security/index.md
+++ b/doc/user/application_security/index.md
@@ -10,7 +10,7 @@ high-level view on projects and groups, and start remediation processes when nee
 
 GitLab can scan and report any vulnerabilities found in your project.
 
-| Secure scanning tools                                                        | Description                                                            |
+| Secure scanning tool                                                         | Description                                                            |
 |:-----------------------------------------------------------------------------|:-----------------------------------------------------------------------|
 | [Container Scanning](container_scanning/index.md) **[ULTIMATE]**             | Scan Docker containers for known vulnerabilities.                      |
 | [Dependency Scanning](dependency_scanning/index.md) **[ULTIMATE]**           | Analyze your dependencies for known vulnerabilities.                   |
@@ -19,6 +19,29 @@ GitLab can scan and report any vulnerabilities found in your project.
 | [Security Dashboard](security_dashboard/index.md) **[ULTIMATE]**             | View vulnerabilities in all your projects and groups.                  |
 | [Static Application Security Testing (SAST)](sast/index.md) **[ULTIMATE]**   | Analyze source code for known vulnerabilities.                         |
 
+## Maintenance and update of the vulnerabilities database
+
+The various scanning tools and the vulnerabilities database are updated regularly.
+
+| Secure scanning tool                                         | Vulnerabilities database updates          |
+|:-------------------------------------------------------------|-------------------------------------------|
+| [Container Scanning](container_scanning/index.md)            | Uses `clair` underneath and the latest `clair-db` version is used for each job run by running the [`latest` docker image tag](https://gitlab.com/gitlab-org/gitlab-ee/blob/438a0a56dc0882f22bdd82e700554525f552d91b/lib/gitlab/ci/templates/Security/Container-Scanning.gitlab-ci.yml#L37). The `clair-db` database [is updated daily according to the author](https://github.com/arminc/clair-local-scan#clair-server-or-local). |
+| [Dependency Scanning](dependency_scanning/index.md)          | Relies on `bundler-audit` (for Rubygems), `retire.js` (for NPM packages) and `gemnasium` (GitLab's own tool for all libraries). `bundler-audit` and `retire.js` both fetch their vulnerabilities data from GitHub repositories, so vulnerabilities added to `ruby-advisory-db` and `retire.js` are immediately available. The tools themselves are updated once per month if there's a new version. The [Gemnasium DB](https://gitlab.com/gitlab-org/security-products/gemnasium-db) is updated at least once a week. |
+| [Dynamic Application Security Testing (DAST)](dast/index.md) | Updated weekly on Sundays. The underlying tool, `zaproxy`, downloads fresh rules at startup. |
+| [Static Application Security Testing (SAST)](sast/index.md)  | Relies exclusively on [the tools GitLab is wrapping](sast/index.md#supported-languages-and-frameworks). The underlying analyzers are updated at least once per month if a relevant update is available. The vulnerabilities database is updated by the upstream tools. |
+
+You don't have to update GitLab to benefit from the latest vulnerabilities definitions,
+but you may have to in the future.
+
+The security tools are released as Docker images, and the vendored job definitions
+to enable them are using the `x-y-stable` image tags that get overridden each time a new
+release of the tools is pushed. The Docker images are updated to match the
+previous GitLab releases, so they automatically get the latest versions of the
+scanning tools without the user having to do anything.
+
+This workflow comes with some drawbacks and there's a
+[plan to change this](https://gitlab.com/gitlab-org/gitlab-ee/issues/9725).
+
 ## Interacting with the vulnerabilities
 
 > Introduced in [GitLab Ultimate](https://about.gitlab.com/pricing) 10.8.
diff --git a/doc/user/application_security/sast/index.md b/doc/user/application_security/sast/index.md
index ec3f7fbde76..9074ac3f4a1 100644
--- a/doc/user/application_security/sast/index.md
+++ b/doc/user/application_security/sast/index.md
@@ -269,7 +269,7 @@ it highlighted:
           "url": "https://cwe.mitre.org/data/definitions/330.html"
         }
       ]
-    },   
+    },
     {
       "category": "sast",
       "message": "Probable insecure usage of temp file/directory.",
@@ -296,7 +296,7 @@ it highlighted:
           "url": "https://docs.openstack.org/bandit/latest/plugins/b108_hardcoded_tmp_directory.html"
         }
       ]
-    },    
+    },
   ],
   "remediations": []
 }
@@ -320,7 +320,7 @@ the report JSON unless stated otherwise. Presence of optional fields depends on
 | `vulnerabilities[].scanner`             | A node that describes the analyzer used to find this vulnerability. |
 | `vulnerabilities[].scanner.id`          | Id of the scanner as a snake_case string. |
 | `vulnerabilities[].scanner.name`        | Name of the scanner, for display purposes. |
-| `vulnerabilities[].location`            | A node that tells where the vulnerability is located. | 
+| `vulnerabilities[].location`            | A node that tells where the vulnerability is located. |
 | `vulnerabilities[].location.file`       | Path to the file where the vulnerability is located. Optional. |
 | `vulnerabilities[].location.start_line` | The first line of the code affected by the vulnerability. Optional. |
 | `vulnerabilities[].location.end_line`   | The last line of the code affected by the vulnerability. Optional. |
@@ -330,7 +330,7 @@ the report JSON unless stated otherwise. Presence of optional fields depends on
 | `vulnerabilities[].identifiers[].type`  | Type of the identifier. Possible values: common identifier types (among `cve`, `cwe`, `osvdb`, and `usn`) or analyzer-dependent ones (e.g., `bandit_test_id` for [Bandit analyzer](https://wiki.openstack.org/wiki/Security/Projects/Bandit)). |
 | `vulnerabilities[].identifiers[].name`  | Name of the identifier for display purposes. |
 | `vulnerabilities[].identifiers[].value` | Value of the identifier for matching purposes. |
-| `vulnerabilities[].identifiers[].url`   | URL to identifier's documentation. Optional. | 
+| `vulnerabilities[].identifiers[].url`   | URL to identifier's documentation. Optional. |
 
 ## Secret detection
 
@@ -363,3 +363,8 @@ vulnerabilities in your groups and projects. Read more about the
 
 Once a vulnerability is found, you can interact with it. Read more on how to
 [interact with the vulnerabilities](../index.md#interacting-with-the-vulnerabilities).
+
+## Vulnerabilities database update
+
+For more information about the vulnerabilities database update, check the
+[maintenance table](../index.md#maintenance-and-update-of-the-vulnerabilities-database).