diff options
Diffstat (limited to 'doc/development/integrations')
| -rw-r--r-- | doc/development/integrations/example_vuln.png | bin | 0 -> 102950 bytes | |||
| -rw-r--r-- | doc/development/integrations/secure.md | 164 | ||||
| -rw-r--r-- | doc/development/integrations/secure_partner_integration.md | 2 |
3 files changed, 104 insertions, 62 deletions
diff --git a/doc/development/integrations/example_vuln.png b/doc/development/integrations/example_vuln.png Binary files differnew file mode 100644 index 00000000000..f7a3c8b38f2 --- /dev/null +++ b/doc/development/integrations/example_vuln.png diff --git a/doc/development/integrations/secure.md b/doc/development/integrations/secure.md index b38e45778fb..b0e1e28ba8b 100644 --- a/doc/development/integrations/secure.md +++ b/doc/development/integrations/secure.md @@ -2,14 +2,23 @@ Integrating a security scanner into GitLab consists of providing end users with a [CI job definition](../../ci/yaml/README.md#introduction) -they can add to their CI configuration files, to scan their GitLab projects. +they can add to their CI configuration files to scan their GitLab projects. +This CI job should then output its results in a GitLab-specified format. These results are then +automatically presented in various places in GitLab, such as the Pipeline view, Merge Request +widget, and Security Dashboard. + The scanning job is usually based on a [Docker image](https://docs.docker.com/) that contains the scanner and all its dependencies in a self-contained environment. -This page documents requirements and guidelines for writing CI jobs implementing a security scanner, -as well as requirements and guidelines for the Docker image itself. + +This page documents requirements and guidelines for writing CI jobs that implement a security +scanner, as well as requirements and guidelines for the Docker image. ## Job definition +This section desribes several important fields to add to the security scanner's job +definition file. Full documentation on these and other available fields can be viewed +in the [CI documentation](../../ci/yaml/README.md#image). + ### Name For consistency, scanning jobs should be named after the scanner, in lower case. @@ -26,8 +35,8 @@ containing the security scanner. ### Script The [`script`](../../ci/yaml/README.md#script) keyword -is used to specify the command that the job runs. -Because the `script` cannot be left empty, it must be set to the command that performs the scan. +is used to specify the commands to run the scanner. +Because the `script` entry can't be left empty, it must be set to the command that performs the scan. It is not possible to rely on the predefined `ENTRYPOINT` and `CMD` of the Docker image to perform the scan automatically, without passing any command. @@ -53,44 +62,41 @@ so the [`allow_failure`](../../ci/yaml/README.md#allow_failure) parameter should ### Artifacts Scanning jobs must declare a report that corresponds to the type of scanning they perform, -using the [`artifacts:reports`](../../ci/yaml/README.md#artifactsreports) keyword. +using the [`artifacts:reports`](../../ci/pipelines/job_artifacts.md#artifactsreports) keyword. Valid reports are: `dependency_scanning`, `container_scanning`, `dast`, and `sast`. For example, here is the definition of a SAST job that generates a file named `gl-sast-report.json`, and uploads it as a SAST report: ```yaml -mysec_dependency_scanning: +mysec_sast_scanning: image: registry.gitlab.com/secure/mysec artifacts: reports: sast: gl-sast-report.json ``` -`gl-sast-report.json` is an example file path. See [the Output file section](#output-file) for more details. -It is processed as a SAST report because it is declared as such in the job definition. +Note that `gl-sast-report.json` is an example file path but any other file name can be used. See +[the Output file section](#output-file) for more details. It's processed as a SAST report because +it's declared under the `reports:sast` key in the job definition, not because of the file name. ### Policies -Scanning jobs should be skipped unless the corresponding feature is listed -in the `GITLAB_FEATURES` variable (comma-separated list of values). -So Dependency Scanning, Container Scanning, SAST, and DAST should be skipped -unless `GITLAB_FEATURES` contains `dependency_scanning`, `container_scanning`, `sast`, and `dast`, respectively. -See [GitLab CI/CD predefined variables](../../ci/variables/predefined_variables.md). - -Also, scanning jobs should be skipped when the corresponding variable prefixed with `_DISABLED` is present. -See `DEPENDENCY_SCANNING_DISABLED`, `CONTAINER_SCANNING_DISABLED`, `SAST_DISABLED`, and `DAST_DISABLED` -in [Auto DevOps documentation](../../topics/autodevops/customize.md#disable-jobs). +Certain GitLab workflows, such as [AutoDevOps](../../topics/autodevops/customize.md#disable-jobs), +define variables to indicate that given scans should be disabled. You can check for this by looking +for variables such as `DEPENDENCY_SCANNING_DISABLED`, `CONTAINER_SCANNING_DISABLED`, +`SAST_DISABLED`, and `DAST_DISABLED`. If appropriate based on the scanner type, you should then +disable running the custom scanner. -Finally, SAST and Dependency Scanning job definitions should use -`CI_PROJECT_REPOSITORY_LANGUAGES` (comma-separated list of values) -in order to skip the job when the language or technology is not supported. +GitLab also defines a `CI_PROJECT_REPOSITORY_LANGUAGES` variable, which provides the list of +languages in the repo. Depending on this value, your scanner may or may not do something different. Language detection currently relies on the [`linguist`](https://github.com/github/linguist) Ruby gem. See [GitLab CI/CD prefined variables](../../ci/variables/predefined_variables.md#variables-reference). -For instance, here is how to skip the Dependency Scanning job `mysec_dependency_scanning` -unless the project repository contains Java source code, -and the `dependency_scanning` feature is enabled: +#### Policy checking example + +This example shows how to skip a custom Dependency Scanning job, `mysec_dependency_scanning`, unless +the project repository contains Java source code and the `dependency_scanning` feature is enabled: ```yaml mysec_dependency_scanning: @@ -111,6 +117,8 @@ for a particular branch or when a particular set of files changes. The Docker image is a self-contained environment that combines the scanner with all the libraries and tools it depends on. +Packaging your scanner into a Docker image makes its dependencies and configuration always present, +regardless of the individual machine the scanner runs on. ### Image size @@ -144,7 +152,7 @@ It also generates text output on the standard output and standard error streams, All CI variables are passed to the scanner as environment variables. The scanned project is described by the [predefined CI variables](../../ci/variables/README.md). -#### SAST, Dependency Scanning +#### SAST and Dependency Scanning SAST and Dependency Scanning scanners must scan the files in the project directory, given by the `CI_PROJECT_DIR` variable. @@ -178,7 +186,7 @@ It is recommended to name the output file after the type of scanning, and to use Since all Secure reports are JSON files, it is recommended to use `.json` as a file extension. For instance, a suggested file name for a Dependency Scanning report is `gl-dependency-scanning.json`. -The [`artifacts:reports`](../../ci/yaml/README.md#artifactsreports) keyword +The [`artifacts:reports`](../../ci/pipelines/job_artifacts.md#artifactsreports) keyword of the job definition must be consistent with the file path where the Security report is written. For instance, if a Dependency Scanning analyzer writes its report to the CI project directory, and if this report file name is `depscan.json`, @@ -223,11 +231,8 @@ The DAST variant of the report JSON format is not documented at the moment. ### Version -The documentation of -[SAST](../../user/application_security/sast/index.md#reports-json-format), -[Dependency Scanning](../../user/application_security/dependency_scanning/index.md#reports-json-format), -and [Container Scanning](../../user/application_security/container_scanning/index.md#reports-json-format) -describes the Secure report format version. +This field specifies the version of the report schema you are using. Please reference individual scanner +pages for the specific versions to use. ### Vulnerabilities @@ -251,12 +256,17 @@ The `id` should not collide with any other scanner another integrator would prov #### Name, message, and description -The `name` and `message` fields contain a short description of the vulnerability, -whereas the `description` field provides more details. +The `name` and `message` fields contain a short description of the vulnerability. +The `description` field provides more details. -The `name` is context-free and contains no information on where the vulnerability has been found, +The `name` field is context-free and contains no information on where the vulnerability has been found, whereas the `message` may repeat the location. +As a visual example, this screenshot highlights where these fields are used when viewing a +vulnerability as part of a pipeline view. + + + For instance, a `message` for a vulnerability reported by Dependency Scanning gives information on the vulnerable dependency, which is redundant with the `location` field of the vulnerability. @@ -288,21 +298,17 @@ It should not repeat the other fields of the vulnerability object. In particular, the `description` should not repeat the `location` (what is affected) or the `solution` (how to mitigate the risk). -There is a proposal to remove either the `name` or the `message`, to remove ambiguities. -See [issue #36779](https://gitlab.com/gitlab-org/gitlab/issues/36779). - #### Solution -The `solution` field may contain instructions users should follow to fix the vulnerability or to mitigate the risk. -It is intended for users whereas the `remediations` objects are processed automatically by GitLab. +You can use the `solution` field to instruct users how to fix the identified vulnerability or to mitigate +the risk. End-users interact with this field, whereas GitLab automatically processes the +`remediations` objects. #### Identifiers -The `identifiers` array describes the vulnerability flaw that has been detected. -An identifier object has a `type` and a `value`; -these technical fields are used to tell if two identifiers are the same. -It also has a `name` and a `url`; -these fields are used to display the identifier in the user interface. +The `identifiers` array describes the detected vulnerability. An identifier object's `type` and +`value` fields are used to tell if two identifiers are the same. The user interface uses the +object's `name` and `url` fields to display the identifier. It is recommended to reuse the identifiers the GitLab scanners already define: @@ -316,18 +322,15 @@ It is recommended to reuse the identifiers the GitLab scanners already define: | [RHSA](https://access.redhat.com/errata/#/) | `rhsa` | RHSA-2020:0111 | | [ELSA](https://linux.oracle.com/security/) | `elsa` | ELSA-2020-0085 | -The generic identifiers listed above are defined in the [common library](https://gitlab.com/gitlab-org/security-products/analyzers/common); -this library is shared by the analyzers maintained by GitLab, -and this is where you can [contribute](https://gitlab.com/gitlab-org/security-products/analyzers/common/blob/master/issue/identifier.go) new generic identifiers. -Analyzers may also produce vendor-specific or product-specific identifiers; -these do not belong to the [common library](https://gitlab.com/gitlab-org/security-products/analyzers/common). +The generic identifiers listed above are defined in the [common library](https://gitlab.com/gitlab-org/security-products/analyzers/common), +which is shared by the analyzers that GitLab maintains. You can [contribute](https://gitlab.com/gitlab-org/security-products/analyzers/common/blob/master/issue/identifier.go) +new generic identifiers to if needed. Analyzers may also produce vendor-specific or product-specific +identifiers, which don't belong in the [common library](https://gitlab.com/gitlab-org/security-products/analyzers/common). The first item of the `identifiers` array is called the primary identifier. The primary identifier is particularly important, because it is used to -[track vulnerabilities](#tracking-merging-vulnerabilities) -as new commits are pushed to the repository. - -Identifiers are used to [merge duplicate vulnerabilities](#tracking-merging-vulnerabilities) +[track vulnerabilities](#tracking-and-merging-vulnerabilities) as new commits are pushed to the repository. +Identifiers are also used to [merge duplicate vulnerabilities](#tracking-and-merging-vulnerabilities) reported for the same commit, except for `CWE` and `WASC`. ### Location @@ -336,7 +339,7 @@ The `location` indicates where the vulnerability has been detected. The format of the location depends on the type of scanning. Internally GitLab extracts some attributes of the `location` to generate the **location fingerprint**, -which is used to [track vulnerabilities](#tracking-merging-vulnerabilities) +which is used to track vulnerabilities as new commits are pushed to the repository. The attributes used to generate the location fingerprint also depend on the type of scanning. @@ -426,12 +429,12 @@ combines `file`, `start_line`, and `end_line`, so these attributes are mandatory. All other attributes are optional. -### Tracking, merging vulnerabilities +### Tracking and merging vulnerabilities Users may give feedback on a vulnerability: -- they may dismiss a vulnerability if it does not apply to their projects -- or they may create an issue for a vulnerability, if there is a possible threat +- They may dismiss a vulnerability if it doesn't apply to their projects +- They may create an issue for a vulnerability if there's a possible threat GitLab tracks vulnerabilities so that user feedback is not lost when new Git commits are pushed to the repository. @@ -470,18 +473,57 @@ Valid values are: `Ignore`, `Unknown`, `Experimental`, `Low`, `Medium`, `High`, ### Remediations The `remediations` field of the report is an array of remediation objects. -Each remediation describes a patch that can be applied to automatically fix +Each remediation describes a patch that can be applied to +[automatically fix](../../user/application_security/#solutions-for-vulnerabilities-auto-remediation) a set of vulnerabilities. +Here is an example of a report that contains remediations. + +```json +{ + "vulnerabilities": [ + { + "category": "dependency_scanning", + "name": "Regular Expression Denial of Service", + "id": "123e4567-e89b-12d3-a456-426655440000", + "solution": "Upgrade to new versions.", + "scanner": { + "id": "gemnasium", + "name": "Gemnasium" + }, + "identifiers": [ + { + "type": "gemnasium", + "name": "Gemnasium-642735a5-1425-428d-8d4e-3c854885a3c9", + "value": "642735a5-1425-428d-8d4e-3c854885a3c9" + } + ] + } + ], + "remediations": [ + { + "fixes": [ + { + "id": "123e4567-e89b-12d3-a456-426655440000" + } + ], + "summary": "Upgrade to new version", + "diff": "ZGlmZiAtLWdpdCBhL3lhcm4ubG9jayBiL3lhcm4ubG9jawppbmRleCAwZWNjOTJmLi43ZmE0NTU0IDEwMDY0NAotLS0gYS95Y==" + } + ] +} +``` + #### Summary -The `summary` field is an overview of how the vulnerabilities can be fixed. +The `summary` field is an overview of how the vulnerabilities can be fixed. This field is required. #### Fixed vulnerabilities The `fixes` field is an array of objects that reference the vulnerabilities fixed by the -remediation. `fixes[].id` contains a fixed vulnerability's unique identifier. +remediation. `fixes[].id` contains a fixed vulnerability's [unique identifier](#id). This field is required. #### Diff -The `diff` field is a base64-encoded remediation code diff, compatible with [`git apply`](https://git-scm.com/docs/git-format-patch#_discussion). +The `diff` field is a base64-encoded remediation code diff, compatible with +[`git apply`](https://git-scm.com/docs/git-format-patch#_discussion). This field is required. diff --git a/doc/development/integrations/secure_partner_integration.md b/doc/development/integrations/secure_partner_integration.md index d8badda4125..59336b0e6a1 100644 --- a/doc/development/integrations/secure_partner_integration.md +++ b/doc/development/integrations/secure_partner_integration.md @@ -68,7 +68,7 @@ and complete an intgration with the Secure stage. 1. Ensure your pipeline jobs create a report artifact that GitLab can process to successfully display your own product's results with the rest of GitLab. - See detailed [technical directions](secure.md) for this step. - - Read more about [job report artifacts](../../ci/yaml/README.md#artifactsreports). + - Read more about [job report artifacts](../../ci/pipelines/job_artifacts.md#artifactsreports). - Read about [job artifacts](../../user/project/pipelines/job_artifacts.md). - Your report artifact must be in one of our currently supported formats. For more information, see the [documentation on reports](secure.md#report). |
