diff options
Diffstat (limited to 'doc/user/application_security')
24 files changed, 2198 insertions, 1602 deletions
diff --git a/doc/user/application_security/api_fuzzing/create_har_files.md b/doc/user/application_security/api_fuzzing/create_har_files.md index 445bcfb0444..e7eaf47121b 100644 --- a/doc/user/application_security/api_fuzzing/create_har_files.md +++ b/doc/user/application_security/api_fuzzing/create_har_files.md @@ -93,7 +93,7 @@ used in [Web API Fuzz Testing](index.md#http-archive-har). 1. Select **Export Data > Current Workspace**. 1. Select requests to include in the HAR file. 1. Select **Export**. -1. In the **Select Export Type** dropdown select **HAR -- HTTP Archive Format**. +1. In the **Select Export Type** dropdown list select **HAR -- HTTP Archive Format**. 1. Select **Done**. 1. Enter a location and filename for the HAR file. @@ -109,7 +109,7 @@ responses in HAR format. have an account, first create an account. 1. Browse pages that call an API. Fiddler automatically captures the requests. 1. Select one or more requests, then from the context menu, select **Export > Selected Sessions**. -1. In the **Choose Format** dropdown select **HTTPArchive v1.2**. +1. In the **Choose Format** dropdown list select **HTTPArchive v1.2**. 1. Enter a filename and select **Save**. Fiddler shows a popup message confirming the export has succeeded. diff --git a/doc/user/application_security/api_fuzzing/index.md b/doc/user/application_security/api_fuzzing/index.md index d542c2d4be5..03eed6fdbf8 100644 --- a/doc/user/application_security/api_fuzzing/index.md +++ b/doc/user/application_security/api_fuzzing/index.md @@ -1211,7 +1211,7 @@ Example usage for setting a `body-json` override: } ``` -Note that each JSON property name in the object `body-json` is set to a [JSON Path](https://goessner.net/articles/JsonPath/) +Each JSON property name in the object `body-json` is set to a [JSON Path](https://goessner.net/articles/JsonPath/) expression. The JSON Path expression `$.credentials.access-token` identifies the node to be overridden with the value `iddqd!42.$`. The override engine uses `body-json` when the request body has only [JSON](https://www.json.org/json-en.html) content. @@ -1250,7 +1250,7 @@ the second entry overrides an XML element: } ``` -Note that each JSON property name in the object `body-xml` is set to an +Each JSON property name in the object `body-xml` is set to an [XPath v2](https://www.w3.org/TR/xpath20/) expression. The XPath expression `/credentials/@isEnabled` identifies the attribute node to override with the value `true`. The XPath expression `/credentials/access-token/text()` identifies the @@ -1392,7 +1392,7 @@ It is also possible to write messages from your script to a log file that is col Adding some basic logging to your overrides script is useful in case the script fails unexpectedly during normal running of the job. The log file is automatically included as an artifact of the job, allowing you to download it after the job has finished. -Following our example, we provided `renew_token.py` in the environmental variable `FUZZAPI_OVERRIDES_CMD`. Please notice two things in the script: +Following our example, we provided `renew_token.py` in the environmental variable `FUZZAPI_OVERRIDES_CMD`. Notice two things in the script: - Log file is saved in the location indicated by the environment variable `CI_PROJECT_DIR`. - Log filename should match `gl-*.log`. @@ -1871,7 +1871,7 @@ variables: ##### Excluding two URLs and their child resources -In order to exclude the URLs: `http://target/api/buy` and `http://target/api/sell`, and their child resources. To provide multiple URLs we use the `,` character as follows: +To exclude the URLs: `http://target/api/buy` and `http://target/api/sell`, and their child resources. To provide multiple URLs we use the `,` character as follows: ```yaml stages: @@ -1888,7 +1888,11 @@ variables: ##### Excluding URL using regular expressions -In order to exclude exactly `https://target/api/v1/user/create` and `https://target/api/v2/user/create` or any other version (`v3`,`v4`, and more). We could use `https://target/api/v.*/user/create$`, in the previous regular expression `.` indicates any character and `*` indicates zero or more times, additionally `$` indicates that the URL should end there. +To exclude exactly `https://target/api/v1/user/create` and `https://target/api/v2/user/create` or any other version (`v3`,`v4`, and more), we could use `https://target/api/v.*/user/create$`. In the previous regular expression: + +- `.` indicates any character. +- `*` indicates zero or more times. +- `$` indicates that the URL should end there. ```yaml stages: @@ -2211,9 +2215,235 @@ Setting `SECURE_ANALYZERS_PREFIX` changes the Docker image registry location for For more information, see [Offline environments](../offline_deployments/index.md). +## Performance tuning and testing speed + +Security tools that perform API fuzz testing, such as API Fuzzing, perform testing by sending requests to an instance of your running application. The requests are mutated by our fuzzing engine to trigger unexpected behavior that might exist in your application. The speed of an API fuzzing test depends on the following: + +- How many requests per second can be sent to your application by our tooling +- How fast your application responds to requests +- How many requests must be sent to test the application + - How many operations your API is comprised of + - How many fields are in each operation (think JSON bodies, headers, query string, cookies, etc.) + +If API Fuzzing testing job still takes longer than expected after following the advice in this performance guide, reach out to support for further assistance. + +### Diagnosing performance issues + +The first step to resolving performance issues is to understand what is contributing to the slower-than-expected testing time. Some common issues we see are: + +- API Fuzzing is running on a slow or single-CPU GitLab Runner (GitLab Shared Runners are single-CPU) +- The application deployed to a slow/single-CPU instance and is not able to keep up with the testing load +- The application contains a slow operation that impacts the overall test speed (> 1/2 second) +- The application contains an operation that returns a large amount of data (> 500K+) +- The application contains a large number of operations (> 40) + +#### The application contains a slow operation that impacts the overall test speed (> 1/2 second) + +The API Fuzzing job output contains helpful information about how fast we are testing, how fast each operation being tested responds, and summary information. Let's take a look at some sample output to see how it can be used in tracking down performance issues: + +```shell +API Security: Loaded 10 operations from: assets/har-large-response/large_responses.har +API Security: +API Security: Testing operation [1/10]: 'GET http://target:7777/api/large_response_json'. +API Security: - Parameters: (Headers: 4, Query: 0, Body: 0) +API Security: - Request body size: 0 Bytes (0 bytes) +API Security: +API Security: Finished testing operation 'GET http://target:7777/api/large_response_json'. +API Security: - Excluded Parameters: (Headers: 0, Query: 0, Body: 0) +API Security: - Performed 767 requests +API Security: - Average response body size: 130 MB +API Security: - Average call time: 2 seconds and 82.69 milliseconds (2.082693 seconds) +API Security: - Time to complete: 14 minutes, 8 seconds and 788.36 milliseconds (848.788358 seconds) +``` + +This job console output snippet starts by telling us how many operations were found (10), followed by notifications that testing has started on a specific operation and a summary of the operation has been completed. The summary is the most interesting part of this log output. In the summary, we can see that it took API Fuzzing 767 requests to fully test this operation and its related fields. We can also see that the average response time was 2 seconds and the time to complete was 14 minutes for this one operation. + +An average response time of 2 seconds is a good initial indicator that this specific operation takes a long time to test. Further, we can see that the response body size is quite large. The large body size is the culprit here, transferring that much data on each request is what takes the majority of that 2 seconds. + +For this issue, the team might decide to: + +- Use a multi-CPU runner. Using a multi-CPU runner allows API Fuzzing to parallelize the work being performed. This helps lower the test time, but getting the test down under 10 minutes might still be problematic without moving to a high CPU machine due to how long the operation takes to test. + - Trade off between how many CPUs and cost. +- [Exclude this operation](#excluding-slow-operations) from the API Fuzzing test. While this is the simplest, it has the downside of a gap in security test coverage. +- [Exclude the operation from feature branch API Fuzzing tests, but include it in the default branch test](#excluding-operations-in-feature-branches-but-not-default-branch). +- [Split up the API Fuzzing testing into multiple jobs](#splitting-a-test-into-multiple-jobs). + +The likely solution is to use a combination of these solutions to reach an acceptable test time, assuming your team's requirements are in the 5-7 minute range. + +### Addressing performance issues + +The following sections document various options for addressing performance issues for API Fuzzing: + +- [Using a multi-CPU Runner](#using-a-multi-cpu-runner) +- [Excluding slow operations](#excluding-slow-operations) +- [Splitting a test into multiple jobs](#splitting-a-test-into-multiple-jobs) +- [Excluding operations in feature branches, but not default branch](#excluding-operations-in-feature-branches-but-not-default-branch) + +#### Using a multi-CPU Runner + +One of the easiest performance boosts can be achieved using a multi-CPU runner with API Fuzzing. This table shows statistics collected during benchmarking of a Java Spring Boot REST API. In this benchmark, the target and API Fuzzing share a single runner instance. + +| CPU Count | Request per Second | +|----------------------|--------------------| +| 1 CPU (Shared Runner)| 75 | +| 4 CPU | 255 | +| 8 CPU | 400 | + +As we can see from this table, increasing the CPU count of the runner can have a large impact on testing speed/performance. + +To use a multi-CPU typically requires deploying a self-managed GitLab Runner onto a multi-CPU machine or cloud compute instance. + +When multiple types of GitLab Runners are available for use, the various instances are commonly set up with tags that can be used in the job definition to select a type of runner. + +Here is an example job definition for API Fuzzing that adds a `tags` section with the tag `multi-cpu`. The job automatically extends the job definition included through the API Fuzzing template. + +```yaml +apifuzzer_fuzz: + tags: + - multi-cpu +``` + +To verify that API Fuzzing can detect multiple CPUs in the runner, download the `gl-api-security-scanner.log` file from a completed job's artifacts. Search the file for the string `Starting work item processor` and inspect the reported max DOP (degree of parallelism). The max DOP should be greater than or equal to the number of CPUs assigned to the runner. The value is never lower than 2, even on single CPU runners, unless forced through a configuration variable. If the value reported is less than the number of CPUs assigned to the runner, then something is wrong with the runner deployment. If unable to identify the problem, open a ticket with support to assist. + +Example log entry: + +`17:00:01.084 [INF] <Peach.Web.Core.Services.WebRunnerMachine> Starting work item processor with 2 max DOP` + +#### Excluding slow operations + +In the case of one or two slow operations, the team might decide to skip testing the operations. Excluding the operation is done using the `FUZZAPI_EXCLUDE_PATHS` configuration [variable as explained in this section.](#exclude-paths) + +In this example, we have an operation that returns a large amount of data. The operation is `GET http://target:7777/api/large_response_json`. To exclude it we provide the `FUZZAPI_EXCLUDE_PATHS` configuration variable with the path portion of our operation URL `/api/large_response_json`. + +To verify the operation is excluded, run the API Fuzzing job and review the job console output. It includes a list of included and excluded operations at the end of the test. + +```yaml +apifuzzer_fuzz: + variables: + FUZZAPI_EXCLUDE_PATHS: /api/large_response_json +``` + +Excluding operations from testing could allow some vulnerabilities to go undetected. +{: .alert .alert-warning} + +#### Splitting a test into multiple jobs + +Splitting a test into multiple jobs is supported by API Fuzzing through the use of [`FUZZAPI_EXCLUDE_PATHS`](#exclude-paths) and [`FUZZAPI_EXCLUDE_URLS`](#exclude-urls). When splitting a test up, a good pattern is to disable the `apifuzzer_fuzz` job and replace it with two jobs with identifying names. In this example we have two jobs, each job is testing a version of the API, so our names reflect that. However, this technique can be applied to any situation, not just with versions of an API. + +The rules we are using in the `apifuzzer_v1` and `apifuzzer_v2` jobs are copied from the [API Fuzzing template](https://gitlab.com/gitlab-org/gitlab/blob/master/lib/gitlab/ci/templates/Security/DAST-API.gitlab-ci.yml). + +```yaml +# Disable the main apifuzzer_fuzz job +apifuzzer_fuzz: + rules: + - if: $CI_COMMIT_BRANCH + when: never + +apifuzzer_v1: + extends: apifuzzer_fuzz + variables: + FUZZAPI_EXCLUDE_PATHS: /api/v1/** + rules: + rules: + - if: $API_FUZZING_DISABLED + when: never + - if: $API_FUZZING_DISABLED_FOR_DEFAULT_BRANCH && + $CI_DEFAULT_BRANCH == $CI_COMMIT_REF_NAME + when: never + - if: $CI_COMMIT_BRANCH && + $CI_GITLAB_FIPS_MODE == "true" + variables: + FUZZAPI_IMAGE_SUFFIX: "-fips" + - if: $CI_COMMIT_BRANCH + +apifuzzer_v2: + variables: + FUZZAPI_EXCLUDE_PATHS: /api/v2/** + rules: + rules: + - if: $API_FUZZING_DISABLED + when: never + - if: $API_FUZZING_DISABLED_FOR_DEFAULT_BRANCH && + $CI_DEFAULT_BRANCH == $CI_COMMIT_REF_NAME + when: never + - if: $CI_COMMIT_BRANCH && + $CI_GITLAB_FIPS_MODE == "true" + variables: + FUZZAPI_IMAGE_SUFFIX: "-fips" + - if: $CI_COMMIT_BRANCH +``` + +#### Excluding operations in feature branches, but not default branch + +In the case of one or two slow operations, the team might decide to skip testing the operations, or exclude them from feature branch tests, but include them for default branch tests. Excluding the operation is done using the `FUZZAPI_EXCLUDE_PATHS` configuration [variable as explained in this section.](#exclude-paths) + +In this example, we have an operation that returns a large amount of data. The operation is `GET http://target:7777/api/large_response_json`. To exclude it we provide the `FUZZAPI_EXCLUDE_PATHS` configuration variable with the path portion of our operation URL `/api/large_response_json`. Our configuration disables the main `apifuzzer_fuzz` job and creates two new jobs `apifuzzer_main` and `apifuzzer_branch`. The `apifuzzer_branch` is set up to exclude the long operation and only run on non-default branches (e.g. feature branches). The `apifuzzer_main` branch is set up to only execute on the default branch (`main` in this example). The `apifuzzer_branch` jobs run faster, allowing for quick development cycles, while the `apifuzzer_main` job which only runs on default branch builds, takes longer to run. + +To verify the operation is excluded, run the API Fuzzing job and review the job console output. It includes a list of included and excluded operations at the end of the test. + +```yaml +# Disable the main job so we can create two jobs with +# different names +apifuzzer_fuzz: + rules: + - if: $CI_COMMIT_BRANCH + when: never + +# API Fuzzing for feature branch work, excludes /api/large_response_json +apifuzzer_branch: + extends: apifuzzer_fuzz + variables: + FUZZAPI_EXCLUDE_PATHS: /api/large_response_json + rules: + rules: + - if: $API_FUZZING_DISABLED + when: never + - if: $API_FUZZING_DISABLED_FOR_DEFAULT_BRANCH && + $CI_DEFAULT_BRANCH == $CI_COMMIT_REF_NAME + when: never + - if: $CI_COMMIT_BRANCH && + $CI_GITLAB_FIPS_MODE == "true" + variables: + FUZZAPI_IMAGE_SUFFIX: "-fips" + - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH + when: never + - if: $CI_COMMIT_BRANCH + +# API Fuzzing for default branch (main in our case) +# Includes the long running operations +apifuzzer_main: + extends: apifuzzer_fuzz + rules: + - if: $API_FUZZING_DISABLED + when: never + - if: $API_FUZZING_DISABLED_FOR_DEFAULT_BRANCH && + $CI_DEFAULT_BRANCH == $CI_COMMIT_REF_NAME + when: never + - if: $CI_COMMIT_BRANCH && + $CI_GITLAB_FIPS_MODE == "true" + variables: + FUZZAPI_IMAGE_SUFFIX: "-fips" + - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH +``` + ## Troubleshooting -### `Error waiting for API Security 'http://127.0.0.1:5000' to become available` +### API Fuzzing job times out after N hours + +The top two reasons for the API Fuzzing job timing out are slow operations (> 1 second) and using a single-CPU runner for API Fuzzing (GitLab shared runners are single-CPU). Before you can diagnose the problem further, the job must complete so the output can be analyzed. We recommend to start with a multi-CPU runner first, then exclude portions of your API operations until the job completes and the output can be further reviewed. + +See the following documentation sections for assistance: + +- [Performance tuning and testing speed](#performance-tuning-and-testing-speed) +- [Using a multi-CPU Runner](#using-a-multi-cpu-runner) +- [Excluding operations by path](#exclude-paths) +- [Excluding slow operations](#excluding-slow-operations) + +### API Fuzzing job takes too long to complete + +See [Performance Tuning and Testing Speed](#performance-tuning-and-testing-speed) + +### Error waiting for API Security 'http://127.0.0.1:5000' to become available A bug exists in versions of the API Fuzzing analyzer prior to v1.6.196 that can cause a background process to fail under certain conditions. The solution is to update to a newer version of the API Fuzzing analyzer. @@ -2281,16 +2511,16 @@ For OpenAPI Specifications that are generated automatically validation errors ar 1. Identify the validation errors. 1. Use the [Swagger Editor](https://editor.swagger.io/) to identify validation problems in your specification. The visual nature of the Swagger Editor makes it easier to understand what needs to change. - 1. Alternatively, you can check the log output and look for schema validation warnings. They are prefixed with messages such as `OpenAPI 2.0 schema validation error` or `OpenAPI 3.0.x schema validation error`. Each failed validation provides extra information about `location` and `description`. Note that JSON Schema validation messages might not be easy to understand. This is why we recommend the use of editors to validate schema documents. + 1. Alternatively, you can check the log output and look for schema validation warnings. They are prefixed with messages such as `OpenAPI 2.0 schema validation error` or `OpenAPI 3.0.x schema validation error`. Each failed validation provides extra information about `location` and `description`. JSON Schema validation messages can be complex, and editors can help you validate schema documents. 1. Review the documentation for the OpenAPI generation your framework/tech stack is using. Identify the changes needed to produce a correct OpenAPI document. -1. Once the validation issues are resolved, re-run your pipeline. +1. After the validation issues are resolved, re-run your pipeline. **For manually created OpenAPI Specifications** 1. Identify the validation errors. 1. The simplest solution is to use a visual tool to edit and validate the OpenAPI document. For example the [Swagger Editor](https://editor.swagger.io/) highlights schema errors and possible solutions. - 1. Alternatively, you can check the log output and look for schema validation warnings. They are prefixed with messages such as `OpenAPI 2.0 schema validation error` or `OpenAPI 3.0.x schema validation error`. Each failed validation provides extra information about `location` and `description`. Correct each of the validation failures and then resubmit the OpenAPI doc. Note that JSON Schema validation message might not be easy to understand. This is why we recommend the use of editors to validate document. -1. Once the validation issues are resolved, re-run your pipeline. + 1. Alternatively, you can check the log output and look for schema validation warnings. They are prefixed with messages such as `OpenAPI 2.0 schema validation error` or `OpenAPI 3.0.x schema validation error`. Each failed validation provides extra information about `location` and `description`. Correct each of the validation failures and then resubmit the OpenAPI doc. JSON Schema validation messages can be complex, and editors can help you validate schema documents. +1. After the validation issues are resolved, re-run your pipeline. ### `Failed to start scanner session (version header not found)` @@ -2312,7 +2542,10 @@ The API Fuzzing analyzer outputs an error message when it cannot determine the t There is an order of precedence in which the API Fuzzing analyzer tries to get the target API when checking the different sources. First, it will try to use the `FUZZAPI_TARGET_URL`. If the environment variable has not been set, then the API Fuzzing analyzer will attempt to use the `environment_url.txt` file. If there is no file `environment_url.txt`, the API Fuzzing analyzer will then use the OpenAPI document contents and the URL provided in `FUZZAPI_OPENAPI` (if a URL is provided) to try to compute the target API. -The best-suited solution will depend on whether or not your target API changes for each deployment. In static environments, the target API is the same for each deployment, in this case please refer to the [static environment solution](#static-environment-solution). If the target API changes for each deployment a [dynamic environment solution](#dynamic-environment-solutions) should be applied. +The best-suited solution depends on whether or not your target API changes for each deployment: + +- If the target API is the same for each deployment (a static environment), use the [static environment solution](#static-environment-solution). +- If the target API changes for each deployment, use a [dynamic environment solution](#dynamic-environment-solutions). #### Static environment solution @@ -2417,10 +2650,10 @@ API Fuzzing uses the specified media types in the OpenAPI document to generate r ## Get support or request an improvement -To get support for your particular problem please use the [getting help channels](https://about.gitlab.com/get-help/). +To get support for your particular problem use the [getting help channels](https://about.gitlab.com/get-help/). The [GitLab issue tracker on GitLab.com](https://gitlab.com/gitlab-org/gitlab/-/issues) is the right place for bugs and feature proposals about API Security and API Fuzzing. -Please use `~"Category:API Security"` [label](../../../development/contributing/issue_workflow.md#labels) when opening a new issue regarding API fuzzing to ensure it is quickly reviewed by the right people. Please refer to our [review response SLO](https://about.gitlab.com/handbook/engineering/workflow/code-review/#review-response-slo) to understand when you should receive a response. +Use `~"Category:API Security"` [label](../../../development/contributing/issue_workflow.md#labels) when opening a new issue regarding API fuzzing to ensure it is quickly reviewed by the right people. Refer to our [review response SLO](https://about.gitlab.com/handbook/engineering/workflow/code-review/#review-response-slo) to understand when you should receive a response. [Search the issue tracker](https://gitlab.com/gitlab-org/gitlab/-/issues) for similar entries before submitting your own, there's a good chance somebody else had the same issue or feature proposal. Show your support with an award emoji and or join the discussion. @@ -2432,7 +2665,7 @@ When experiencing a behavior not working as expected, consider providing context - Scanner log file available as a job artifact named `gl-api-security-scanner.log`. WARNING: -**Sanitize data attached to a support issue**. Please remove sensitive information, including: credentials, passwords, tokens, keys, and secrets. +**Sanitize data attached to a support issue**. Remove sensitive information, including: credentials, passwords, tokens, keys, and secrets. ## Glossary diff --git a/doc/user/application_security/configuration/index.md b/doc/user/application_security/configuration/index.md index 5eb1b93eb76..d9ba4640855 100644 --- a/doc/user/application_security/configuration/index.md +++ b/doc/user/application_security/configuration/index.md @@ -55,7 +55,7 @@ You can configure the following security controls: - [Dynamic Application Security Testing](../dast/index.md) (DAST) - Select **Enable DAST** to configure DAST for the current project. - Select **Manage scans** to manage the saved DAST scans, site profiles, and scanner profiles. - For more details, read [DAST on-demand scans](../dast/index.md#on-demand-scans). + For more details, read [DAST on-demand scans](../dast/proxy-based.md#on-demand-scans). - [Dependency Scanning](../dependency_scanning/index.md) - Select **Configure with a merge request** to create a merge request with the changes required to enable Dependency Scanning. For more details, see [Enable Dependency Scanning via an automatic merge request](../dependency_scanning/index.md#enable-dependency-scanning-via-an-automatic-merge-request). diff --git a/doc/user/application_security/container_scanning/index.md b/doc/user/application_security/container_scanning/index.md index f6bd6157a28..6fc01a716b2 100644 --- a/doc/user/application_security/container_scanning/index.md +++ b/doc/user/application_security/container_scanning/index.md @@ -173,7 +173,7 @@ container_scanning: before_script: - ruby -r open-uri -e "IO.copy_stream(URI.open('https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip'), 'awscliv2.zip')" - unzip awscliv2.zip - - ./aws/install + - sudo ./aws/install - aws --version - export AWS_ECR_PASSWORD=$(aws ecr get-login-password --region region) @@ -259,7 +259,7 @@ including a large number of false positives. | `CS_IMAGE_SUFFIX` | `""` | Suffix added to `CS_ANALYZER_IMAGE`. If set to `-fips`, `FIPS-enabled` image is used for scan. See [FIPS-enabled images](#fips-enabled-images) for more details. [Introduced](https://gitlab.com/groups/gitlab-org/-/epics/7630) in GitLab 14.10. | All | | `CS_IGNORE_UNFIXED` | `"false"` | Ignore vulnerabilities that are not fixed. | All | | `CS_REGISTRY_INSECURE` | `"false"` | Allow access to insecure registries (HTTP only). Should only be set to `true` when testing the image locally. Works with all scanners, but the registry must listen on port `80/tcp` for Trivy to work. | All | -| `CS_SEVERITY_THRESHOLD` | `UNKNOWN` | Severity level threshold. The scanner outputs vulnerabilities with severity level higher than or equal to this threshold. Supported levels are Unknown, Low, Medium, High, and Critical. | Trivy | +| `CS_SEVERITY_THRESHOLD` | `UNKNOWN` | Severity level threshold. The scanner outputs vulnerabilities with severity level higher than or equal to this threshold. Supported levels are `UNKNOWN`, `LOW`, `MEDIUM`, `HIGH`, and `CRITICAL`. | Trivy | | <!-- start_remove The following content will be removed on remove_date: '2023-08-22' --> `DOCKER_IMAGE` | `$CI_APPLICATION_REPOSITORY:$CI_APPLICATION_TAG` | **Deprecated** will be removed in GitLab 16.0. Replaced by `CS_IMAGE`. The Docker image to be scanned. If set, this variable overrides the `$CI_APPLICATION_REPOSITORY` and `$CI_APPLICATION_TAG` variables. | All | | `DOCKER_PASSWORD` | `$CI_REGISTRY_PASSWORD` | **Deprecated** will be removed in GitLab 16.0. Replaced by `CS_REGISTRY_PASSWORD`. Password for accessing a Docker registry requiring authentication. The default is only set if `$DOCKER_IMAGE` resides at [`$CI_REGISTRY`](../../../ci/variables/predefined_variables.md). Not supported when [FIPS mode](../../../development/fips_compliance.md#enable-fips-mode) is enabled. | All | | `DOCKER_USER` | `$CI_REGISTRY_USER` | **Deprecated** will be removed in GitLab 16.0. Replaced by `CS_REGISTRY_USER`. Username for accessing a Docker registry requiring authentication. The default is only set if `$DOCKER_IMAGE` resides at [`$CI_REGISTRY`](../../../ci/variables/predefined_variables.md). Not supported when [FIPS mode](../../../development/fips_compliance.md#enable-fips-mode) is enabled. | All | @@ -359,46 +359,12 @@ The container-scanning analyzer can use different scanners, depending on the val The following options are available: -| Scanner name | `CS_ANALYZER_IMAGE` | -| ------------ | ------------------- | -| Default ([Trivy](https://github.com/aquasecurity/trivy)) | `registry.gitlab.com/security-products/container-scanning:5` | +| Scanner name | `CS_ANALYZER_IMAGE` | +|----------------------------------------------------------|--------------------------------------------------------------------| +| Default ([Trivy](https://github.com/aquasecurity/trivy)) | `registry.gitlab.com/security-products/container-scanning:5` | | [Grype](https://github.com/anchore/grype) | `registry.gitlab.com/security-products/container-scanning/grype:5` | | Trivy | `registry.gitlab.com/security-products/container-scanning/trivy:5` | -If you're migrating from a GitLab 13.x release to a GitLab 14.x release and have customized the -`container_scanning` job or its CI variables, you might need to perform these migration steps in -your CI file: - -1. Remove these variables: - - - `CS_MAJOR_VERSION` - - `CS_PROJECT` - - `SECURE_ANALYZERS_PREFIX` - -1. Review the `CS_ANALYZER_IMAGE` variable. It no longer depends on the variables above and its new - default value is `registry.gitlab.com/security-products/container-scanning:5`. If you have an - offline environment, see - [Running container scanning in an offline environment](#running-container-scanning-in-an-offline-environment). - -1. If present, remove the `.cs_common` and `container_scanning_new` configuration sections. - -1. If the `container_scanning` section is present, it's safer to create one from scratch based on - the new version of the [`Container-Scanning.gitlab-ci.yml` template](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/ci/templates/Security/Container-Scanning.gitlab-ci.yml). - Once finished, it should not have any variables that are only applicable to Klar or Clair. For a - complete list of supported variables, see [available variables](#available-cicd-variables). - -1. Make any [necessary customizations](#customizing-the-container-scanning-settings) - to the chosen scanner. We recommend that you minimize such customizations, as they might require - changes in future GitLab major releases. - -1. Trigger a new run of a pipeline that includes the `container_scanning` job. Inspect the job - output and ensure that the log messages do not mention Clair. - -NOTE: -Prior to the GitLab 14.0 release, any variable defined under the scope `container_scanning` is not -considered for scanners other than Clair. In GitLab 14.0 and later, all variables can be defined -either as a global variable or under `container_scanning`. - ### Setting the default branch image > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/338877) in GitLab 14.5. diff --git a/doc/user/application_security/dast/browser_based.md b/doc/user/application_security/dast/browser_based.md index 5a4acc78728..c0a97c0ff92 100644 --- a/doc/user/application_security/dast/browser_based.md +++ b/doc/user/application_security/dast/browser_based.md @@ -36,8 +36,8 @@ current DAST scanner is much more effective at finding and testing every page in To enable the browser-based analyzer: 1. Ensure the DAST [prerequisites](index.md#prerequisites) are met. -1. Include the [DAST CI/CD template](index.md#include-the-dast-template). -1. Set the target website using the [`DAST_WEBSITE` CI/CD variable](index.md#available-cicd-variables). +1. Include the [DAST CI/CD template](proxy-based.md#include-the-dast-template). +1. Set the target website using the [`DAST_WEBSITE` CI/CD variable](proxy-based.md#available-cicd-variables). 1. Set the CI/CD variable `DAST_BROWSER_SCAN` to `true`. Example extract of `.gitlab-ci.yml` file: @@ -79,7 +79,7 @@ The browser-based crawler can be configured using CI/CD variables. | `DAST_BROWSER_ELEMENT_TIMEOUT` | [Duration string](https://pkg.go.dev/time#ParseDuration) | `600ms` | The maximum amount of time to wait for an element before determining it is ready for analysis. | | `DAST_BROWSER_PAGE_READY_SELECTOR` | selector | `css:#page-is-ready` | Selector that when detected as visible on the page, indicates to the analyzer that the page has finished loading and the scan can continue. Note: When this selector is set, but the element is not found, the scanner waits for the period defined in `DAST_BROWSER_STABILITY_TIMEOUT` before continuing the scan. This can significantly increase scanning time if the element is not present on multiple pages within the site. | -The [DAST variables](index.md#available-cicd-variables) `SECURE_ANALYZERS_PREFIX`, `DAST_FULL_SCAN_ENABLED`, `DAST_AUTO_UPDATE_ADDONS`, `DAST_EXCLUDE_RULES`, `DAST_REQUEST_HEADERS`, `DAST_HTML_REPORT`, `DAST_MARKDOWN_REPORT`, `DAST_XML_REPORT`, +The [DAST variables](proxy-based.md#available-cicd-variables) `SECURE_ANALYZERS_PREFIX`, `DAST_FULL_SCAN_ENABLED`, `DAST_AUTO_UPDATE_ADDONS`, `DAST_EXCLUDE_RULES`, `DAST_REQUEST_HEADERS`, `DAST_HTML_REPORT`, `DAST_MARKDOWN_REPORT`, `DAST_XML_REPORT`, `DAST_AUTH_URL`, `DAST_USERNAME`, `DAST_PASSWORD`, `DAST_USERNAME_FIELD`, `DAST_PASSWORD_FIELD`, `DAST_FIRST_SUBMIT_FIELD`, `DAST_SUBMIT_FIELD`, `DAST_EXCLUDE_URLS`, `DAST_AUTH_VERIFICATION_URL`, `DAST_BROWSER_AUTH_VERIFICATION_SELECTOR`, `DAST_BROWSER_AUTH_VERIFICATION_LOGIN_FORM`, `DAST_BROWSER_AUTH_REPORT`, `DAST_INCLUDE_ALPHA_VULNERABILITIES`, `DAST_PATHS_FILE`, `DAST_PATHS`, `DAST_ZAP_CLI_OPTIONS`, and `DAST_ZAP_LOG_CONFIGURATION` are also compatible with browser-based crawler scans. diff --git a/doc/user/application_security/dast/dast_troubleshooting.md b/doc/user/application_security/dast/dast_troubleshooting.md index 194761797de..61a7520bf7c 100644 --- a/doc/user/application_security/dast/dast_troubleshooting.md +++ b/doc/user/application_security/dast/dast_troubleshooting.md @@ -20,7 +20,7 @@ A DAST job has two executing processes: Enable the `DAST_DEBUG` CI/CD variable to debug scripts. This can help when troubleshooting the job, and outputs statements indicating what percentage of the scan is complete. -For details on using variables, see [Overriding the DAST template](index.md#customize-dast-settings). +For details on using variables, see [Overriding the DAST template](proxy-based.md#customize-dast-settings). Debug mode of the ZAP server can be enabled using the `DAST_ZAP_LOG_CONFIGURATION` variable. The following table outlines examples of values that can be set and the effect that they have on the output that is logged. diff --git a/doc/user/application_security/dast/index.md b/doc/user/application_security/dast/index.md index 6d1b7beefc7..d78a8fca98f 100644 --- a/doc/user/application_security/dast/index.md +++ b/doc/user/application_security/dast/index.md @@ -13,16 +13,7 @@ application server or incorrect assumptions about security controls may not be visible from the source code. Dynamic Application Security Testing (DAST) examines applications for -vulnerabilities like these in deployed environments. DAST uses the open source -tool [OWASP Zed Attack Proxy](https://www.zaproxy.org/) for analysis. - -After DAST creates its report, GitLab evaluates it for discovered -vulnerabilities between the source and target branches. Relevant -findings are noted in the merge request. - -The comparison logic uses only the latest pipeline executed for the target -branch's base commit. Running the pipeline on other commits has no effect on -the merge request. +vulnerabilities like these in deployed environments. NOTE: To learn how four of the top six attacks were application-based and how @@ -30,16 +21,88 @@ to protect your organization, download our ["A Seismic Shift in Application Security"](https://about.gitlab.com/resources/whitepaper-seismic-shift-application-security/) whitepaper. -## DAST application analysis +## GitLab DAST + +GitLab provides the following DAST analyzers, one or more of which may be useful depending on the kind of application you're testing. + +For scanning websites, use one of: + +- The [DAST proxy-based analyzer](proxy-based.md) for scanning traditional applications serving simple HTML. The proxy-based analyzer can be run automatically or on-demand. +- The [DAST browser-based analyzer](browser_based.md) for scanning applications that make heavy use of JavaScript. This includes single page web applications. + +For scanning APIs, use: + +- The [DAST API analyzer](../dast_api/index.md) for scanning web APIs. Web API technologies such as GraphQL, REST, and SOAP are supported. + +Analyzers follow the architectural patterns described in [Secure your application](../index.md). +Each analyzer can be configured in the pipeline using a CI template and runs the scan in a Docker container. Scans output a [DAST report artifact](../../../ci/yaml/artifacts_reports.md#artifactsreportsdast) +which GitLab uses to determine discovered vulnerabilities based on differences between scan results on the source and target branches. + +### Getting started + +#### Prerequisites + +- [GitLab Runner](../../../ci/runners/index.md) available, with the + [`docker` executor](https://docs.gitlab.com/runner/executors/docker.html) on Linux/amd64. +- Target application deployed. For more details, read [Deployment options](#application-deployment-options). +- `dast` stage added to the CI/CD pipeline definition. This should be added after the deploy step, for example: + + ```yaml + stages: + - build + - test + - deploy + - dast + ``` + +#### Recommendations + +- Take care if your pipeline is configured to deploy to the same web server in each run. Running a DAST scan while a server is being updated will lead to inaccurate and non-deterministic results. +- Configure runners to use the [always pull policy](https://docs.gitlab.com/runner/executors/docker.html#using-the-always-pull-policy) to run the latest versions of the analyzers. +- By default, DAST downloads all artifacts defined by previous jobs in the pipeline. If + your DAST job does not rely on `environment_url.txt` to define the URL under test or any other files created + in previous jobs, we recommend you don't download artifacts. To avoid downloading + artifacts, extend the analyzer CI/CD job to specify no dependencies. For example, for the DAST proxy-based analyzer add the following to your `.gitlab-ci.yml` file: + + ```yaml + dast: + dependencies: [] + ``` + +#### Analyzer configuration + +Please see [DAST proxy-based analyzer](proxy-based.md), [DAST browser-based analyzer](browser_based.md) or [DAST API analyzer](../dast_api/index.md) for +analyzer-specific configuration instructions. + +### View scan results -DAST can analyze applications in two ways: +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/36332) in GitLab 13.1. + +Detected vulnerabilities are shown in [Merge requests](../index.md#view-security-scan-information-in-merge-requests), the [Pipeline security tab](../index.md#view-security-scan-information-in-the-pipeline-security-tab), +and the [Vulnerability report](../index.md#view-security-scan-information-in-the-vulnerability-report). -- Passive scan only (DAST default). DAST executes - [ZAP's Baseline Scan](https://www.zaproxy.org/docs/docker/baseline-scan/) and doesn't - actively attack your application. -- Passive and active scan. DAST can be [configured](#full-scan) to also perform an active scan - to attack your application and produce a more extensive security report. It can be very - useful when combined with [Review Apps](../../../ci/review_apps/index.md). +1. To see all vulnerabilities detected, either: + - From your project, select **Security & Compliance**, then **Vulnerability report**. + - From your pipeline, click on the **Security** tab. + - From the merge request, go to the **Security scanning** widget and click **Full report** tab. + +1. Select a DAST vulnerability's description. The following fields are examples of what a DAST analyzer may produce to aid investigation and rectification of the underlying cause. Each analyzer may output different fields. + + | Field | Description | + |:------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|:------------------------------------------------------------------ | + | Description | Description of the vulnerability. | + | Evidence | Evidence of the data found that verified the vulnerability. Often a snippet of the request or response, this can be used to help verify that the finding is a vulnerability. | + | Identifiers | Identifiers of the vulnerability. | + | Links | Links to further details of the detected vulnerability. | + | Method | HTTP method used to detect the vulnerability. | + | Project | Namespace and project in which the vulnerability was detected. | + | Request Headers | Headers of the request. | + | Response Headers | Headers of the response received from the application. | + | Response Status | Response status received from the application. | + | Scanner Type | Type of vulnerability report. | + | Severity | Severity of the vulnerability. | + | Solution | Details of a recommended solution to the vulnerability. | + | URL | URL at which the vulnerability was detected. | NOTE: A pipeline may consist of multiple jobs, including SAST and DAST scanning. If any job @@ -48,17 +111,19 @@ example, if the DAST job finishes but the SAST job fails, the security dashboard results. On failure, the analyzer outputs an [exit code](../../../development/integrations/secure.md#exit-code). -## Prerequisites +#### List URLs scanned -- [GitLab Runner](../../../ci/runners/index.md) available, with the -[`docker` executor](https://docs.gitlab.com/runner/executors/docker.html) on Linux/amd64. -- Target application deployed. For more details, read [Deployment options](#deployment-options). -- DAST runs in the `dast` stage, which must be added manually to your `.gitlab-ci.yml`. +When DAST completes scanning, the merge request page states the number of URLs scanned. +Select **View details** to view the web console output which includes the list of scanned URLs. + + -### Deployment options +### Application deployment options + +DAST requires a deployed application to be available to scan. Depending on the complexity of the target application, there are a few options as to how to deploy and configure -the DAST template. We provided a set of example applications with their configurations in our +the DAST template. A set of example applications have been provided with their configurations in the [DAST demonstrations](https://gitlab.com/gitlab-org/security-products/demos/dast/) project. #### Review Apps @@ -77,6 +142,8 @@ After your Docker build job completes and your image is added to your container By using service definitions in your `.gitlab-ci.yml`, you can scan services with the DAST analyzer. +When adding a `services` section to the job, the `alias` is used to define the hostname that can be used to access the service. In the following example, the `alias: yourapp` portion of the `dast` job definition means that the URL to the deployed application will use `yourapp` as the hostname (`https://yourapp/`). + ```yaml stages: - build @@ -105,6 +172,7 @@ dast: alias: yourapp variables: + DAST_WEBSITE: https://yourapp DAST_FULL_SCAN_ENABLED: "true" # do a full scan DAST_BROWSER_SCAN: "true" # use the browser-based GitLab DAST crawler ``` @@ -122,1323 +190,3 @@ services: # use services to link the container to the dast job - name: $CI_REGISTRY_IMAGE:$CI_COMMIT_SHA alias: yourapp ``` - -### DAST job order - -When using the `DAST.gitlab-ci.yml` template, the `dast` stage is run last as shown in -the example below. To ensure DAST scans the latest code, deploy your application -in a stage before the `dast` stage. - -```yaml - stages: - - build - - test - - deploy - - dast -``` - -Take care if your pipeline is configured to deploy to the same web server in each run. Running a -pipeline while another is still running could result in one pipeline overwriting the code from -another pipeline. The site to be scanned should be excluded from changes for the duration of a DAST -scan. The only changes to the site should be from the DAST scanner. - -Changes to the site during a scan from any of the following could lead to inaccurate results: - -- Users. -- Scheduled tasks. -- Database changes. -- Code changes. -- Other pipelines. -- Other scanners. - -## DAST run options - -You can use DAST to examine your web application: - -- Automatically, initiated by a merge request. -- Manually, initiated on demand. - -Some of the differences between these run options: - -| Automatic scan | On-demand scan | -|:-----------------------------------------------------------------|:------------------------------| -| DAST scan is initiated by a merge request. | DAST scan is initiated manually, outside the DevOps life cycle. | -| CI/CD variables are sourced from `.gitlab-ci.yml`. | CI/CD variables are provided in the UI. | -| All [DAST CI/CD variables](#available-cicd-variables) available. | Subset of [DAST CI/CD variables](#available-cicd-variables) available. | -| `DAST.gitlab-ci.yml` template. | `DAST-On-Demand-Scan.gitlab-ci.yml` template. | - -### Enable automatic DAST run - -To enable DAST to run automatically, either: - -- Enable [Auto DAST](../../../topics/autodevops/stages.md#auto-dast) (provided - by [Auto DevOps](../../../topics/autodevops/index.md)). -- [Include the DAST template](#include-the-dast-template) in your existing - `.gitlab-ci.yml` file. -- [Configure DAST using the UI](#configure-dast-using-the-ui). - -#### Include the DAST template - -> - This template was [updated](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/62597) to DAST_VERSION: 2 in GitLab 14.0. -> - This template was [updated](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/87183) to DAST_VERSION: 3 in GitLab 15.0. - -If you want to manually add DAST to your application, the DAST job is defined -in a CI/CD template file. Updates to the template are provided with GitLab -upgrades, allowing you to benefit from any improvements and additions. - -To include the DAST template: - -1. Select the CI/CD template you want to use: - - - [`DAST.gitlab-ci.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/ci/templates/Security/DAST.gitlab-ci.yml): - Stable version of the DAST CI/CD template. - - [`DAST.latest.gitlab-ci.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/ci/templates/Security/DAST.latest.gitlab-ci.yml): - Latest version of the DAST template. ([Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/254325) - in GitLab 13.8). - - WARNING: - The latest version of the template may include breaking changes. Use the - stable template unless you need a feature provided only in the latest template. - - For more information about template versioning, see the - [CI/CD documentation](../../../development/cicd/templates.md#latest-version). - -1. Add a `dast` stage to your GitLab CI stages configuration: - - ```yaml - stages: - - dast - ``` - -1. Add the template to GitLab, based on your version of GitLab: - - - In GitLab 11.9 and later, [include](../../../ci/yaml/index.md#includetemplate) - the template by adding the following to your `.gitlab-ci.yml` file: - - ```yaml - include: - - template: <template_file.yml> - - variables: - DAST_WEBSITE: https://example.com - ``` - - - In GitLab 11.8 and earlier, add the contents of the template to your - `.gitlab_ci.yml` file. - -1. Define the URL to be scanned by DAST by using one of these methods: - - - Set the `DAST_WEBSITE` [CI/CD variable](../../../ci/yaml/index.md#variables). - If set, this value takes precedence. - - - Add the URL in an `environment_url.txt` file at the root of your project. This is - useful for testing in dynamic environments. To run DAST against an application - dynamically created during a GitLab CI/CD pipeline, a job that runs prior to - the DAST scan must persist the application's domain in an `environment_url.txt` - file. DAST automatically parses the `environment_url.txt` file to find its - scan target. - - For example, in a job that runs prior to DAST, you could include code that - looks similar to: - - ```yaml - script: - - echo http://${CI_PROJECT_ID}-${CI_ENVIRONMENT_SLUG}.domain.com > environment_url.txt - artifacts: - paths: [environment_url.txt] - when: always - ``` - - You can see an example of this in our - [Auto DevOps CI YAML](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/ci/templates/Jobs/Deploy.gitlab-ci.yml) - file. - -The included template creates a `dast` job in your CI/CD pipeline and scans -your project's running application for possible vulnerabilities. - -The results are saved as a -[DAST report artifact](../../../ci/yaml/artifacts_reports.md#artifactsreportsdast) -that you can later download and analyze. Due to implementation limitations, we -always take the latest DAST artifact available. Behind the scenes, the -[GitLab DAST Docker image](https://gitlab.com/security-products/dast) -is used to run the tests on the specified URL and scan it for possible -vulnerabilities. - -By default, the DAST template uses the latest major version of the DAST Docker -image. Using the `DAST_VERSION` variable, you can choose how DAST updates: - -- Automatically update DAST with new features and fixes by pinning to a major - version (such as `1`). -- Only update fixes by pinning to a minor version (such as `1.6`). -- Prevent all updates by pinning to a specific version (such as `1.6.4`). - -Find the latest DAST versions on the [Releases](https://gitlab.com/gitlab-org/security-products/dast/-/releases) -page. - -#### Configure DAST using the UI - -You can enable or configure DAST settings using the UI. The generated settings are formatted so they -can be conveniently pasted into the `.gitlab-ci.yml` file. - -1. On the top bar, select **Main menu > Projects** and find your project. -1. On the left sidebar, select **Security & Compliance > Configuration**. -1. In the **Dynamic Application Security Testing (DAST)** section, select **Enable DAST** or - **Configure DAST**. -1. Select the desired **Scanner profile**, or select **Create scanner profile** and save a - scanner profile. For more details, see [scanner profiles](#scanner-profile). -1. Select the desired **Site profile**, or select **Create site profile** and save a site - profile. For more details, see [site profiles](#site-profile). -1. Select **Generate code snippet**. A modal opens with the YAML snippet corresponding to the - options you selected. -1. Do one of the following: - 1. To copy the snippet to your clipboard, select **Copy code only**. - 1. To add the snippet to your project's `.gitlab-ci.yml` file, select - **Copy code and open `.gitlab-ci.yml` file**. The Pipeline Editor opens. - 1. Paste the snippet into the `.gitlab-ci.yml` file. - 1. Select the **Lint** tab to confirm the edited `.gitlab-ci.yml` file is valid. - 1. Select the **Edit** tab, then select **Commit changes**. - -When the snippet is committed to the `.gitlab-ci.yml` file, pipelines include a DAST job. - -#### Crawling web applications dependent on JavaScript - -GitLab has released a new browser-based crawler, an add-on to DAST that uses a browser to crawl web applications for content. This crawler replaces the standard DAST Spider and Ajax Crawler, and uses the same authentication mechanisms as a normal DAST scan. - -The browser-based crawler crawls websites by browsing web pages as a user would. This approach works well with web applications that make heavy use of JavaScript, such as Single Page Applications. - -For more details, including setup instructions, see [DAST browser-based crawler](browser_based.md). - -### Full scan - -DAST can be configured to perform [ZAP Full Scan](https://www.zaproxy.org/docs/docker/full-scan/), which -includes both passive and active scanning against the same target website: - -```yaml -include: - - template: DAST.gitlab-ci.yml - -variables: - DAST_FULL_SCAN_ENABLED: "true" - DAST_BROWSER_SCAN: "true" # use the browser-based GitLab DAST crawler -``` - -If your DAST job exceeds the job timeout and you need to reduce the scan duration, we shared some -tips for optimizing DAST scans in a [blog post](https://about.gitlab.com/blog/2020/08/31/how-to-configure-dast-full-scans-for-complex-web-applications/). - -### API scan - -> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/10928) in GitLab 12.10. -> - A new DAST API scanning engine was introduced in GitLab 13.10. - -Using an API specification as a scan's target is a useful way to seed URLs for scanning an API. -Vulnerability rules in an API scan are different than those in a normal website scan. - -A new DAST API scanning engine is available in GitLab 13.12 and later. For more details, see [DAST API scanning engine](../dast_api). The new scanning engine supports REST, SOAP, GraphQL, and generic APIs using forms, XML, and JSON. Testing can be performed using OpenAPI, Postman Collections, and HTTP Archive (HAR) documents. - -The target API instance's base URL is provided by using the `DAST_API_TARGET_URL` variable or an `environment_url.txt` file. - -#### Specification format - -API scans support OpenAPI V2 and OpenAPI V3 specifications. You can define these specifications using `JSON` or `YAML`. - -#### Import API specification from a URL - -If your API specification is accessible at a URL, you can pass that URL in directly as the target. -The specification does not have to be hosted on the same host as the API being tested. - -```yaml -include: - - template: DAST-API.gitlab-ci.yml - -variables: - DAST_API_SPECIFICATION: http://my.api/api-specification.yml -``` - -#### Import API specification from a file - -If your API specification file is in your repository, you can provide its filename as the target. - -```yaml -dast: - variables: - GIT_STRATEGY: fetch - DAST_API_SPECIFICATION: api-specification.yml -``` - -#### Full API scan - -API scans support full scanning, which can be enabled by using the `DAST_FULL_SCAN_ENABLED` -CI/CD variable. Domain validation is not supported for full API scans. - -#### Host override - -Specifications often define a host, which contains a domain name and a port. The -host referenced may be different than the host of the API's review instance. -This can cause incorrect URLs to be imported, or a scan on an incorrect host. -Use the `DAST_API_HOST_OVERRIDE` CI/CD variable to override these values. - -WARNING: -When using the API host override feature, you cannot use the `$DAST_WEBSITE` variable to override the hostname. -A host override is _only_ supported when importing the API specification from a URL. Attempts to override the -host throw an error when the API specification is imported from a file. This is due to a limitation in the -ZAP OpenAPI extension. - -For example, with a OpenAPI V3 specification containing: - -```yaml -servers: - - url: https://api.host.com -``` - -If the test version of the API is running at `https://api-test.host.com`, then -the following DAST configuration can be used: - -```yaml -include: - - template: DAST-API.gitlab-ci.yml - -variables: - DAST_API_SPECIFICATION: http://api-test.host.com/api-specification.yml - DAST_API_HOST_OVERRIDE: api-test.host.com -``` - -#### Authentication using headers - -Tokens in request headers are often used as a way to authenticate API requests. -You can achieve this by using the `DAST_REQUEST_HEADERS` CI/CD variable. -Headers are applied to every request DAST makes. - -```yaml -include: - - template: DAST-API.gitlab-ci.yml - -variables: - DAST_API_SPECIFICATION: http://api-test.api.com/api-specification.yml - DAST_REQUEST_HEADERS: "Authorization: Bearer my.token" -``` - -### URL scan - -> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/214120) in GitLab 13.4. -> - [Improved](https://gitlab.com/gitlab-org/gitlab/-/issues/273141) in GitLab 13.11. - -A URL scan allows you to specify which parts of a website are scanned by DAST. - -#### Define the URLs to scan - -URLs to scan can be specified by either of the following methods: - -- Use `DAST_PATHS_FILE` CI/CD variable to specify the name of a file containing the paths. -- Use `DAST_PATHS` variable to list the paths. - -##### Use `DAST_PATHS_FILE` CI/CD variable - -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/258825) in GitLab 13.6. - -To define the URLs to scan in a file, create a plain text file with one path per line. - -```plaintext -page1.html -/page2.html -category/shoes/page1.html -``` - -To scan the URLs in that file, set the CI/CD variable `DAST_PATHS_FILE` to the path of that file. -The file can be checked into the project repository or generated as an artifact by a job that -runs before DAST. - -By default, DAST scans do not clone the project repository. Instruct the DAST job to clone -the project by setting `GIT_STRATEGY` to fetch. Give a file path relative to `CI_PROJECT_DIR` to `DAST_PATHS_FILE`. - -```yaml -include: - - template: DAST.gitlab-ci.yml - -variables: - GIT_STRATEGY: fetch - DAST_PATHS_FILE: url_file.txt # url_file.txt lives in the root directory of the project - DAST_BROWSER_SCAN: "true" # use the browser-based GitLab DAST crawler -``` - -##### Use `DAST_PATHS` CI/CD variable - -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/214120) in GitLab 13.4. - -To specify the paths to scan in a CI/CD variable, add a comma-separated list of the paths to the `DAST_PATHS` -variable. Note that you can only scan paths of a single host. - -```yaml -include: - - template: DAST.gitlab-ci.yml - -variables: - DAST_PATHS: "/page1.html,/category1/page1.html,/page3.html" - DAST_BROWSER_SCAN: "true" # use the browser-based GitLab DAST crawler -``` - -When using `DAST_PATHS` and `DAST_PATHS_FILE`, note the following: - -- `DAST_WEBSITE` must be defined when using either `DAST_PATHS_FILE` or `DAST_PATHS`. The paths listed in either use `DAST_WEBSITE` to build the URLs to scan -- Spidering is disabled when `DAST_PATHS` or `DAST_PATHS_FILE` are defined -- `DAST_PATHS_FILE` and `DAST_PATHS` cannot be used together -- The `DAST_PATHS` variable has a limit of about 130kb. If you have a list or paths - greater than this, use `DAST_PATHS_FILE`. - -#### Full Scan - -To perform a [full scan](#full-scan) on the listed paths, use the `DAST_FULL_SCAN_ENABLED` CI/CD variable. - -### List URLs scanned - -When DAST completes scanning, the merge request page states the number of URLs scanned. -Select **View details** to view the web console output which includes the list of scanned URLs. - - - -### View details of a vulnerability detected by DAST - -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/36332) in GitLab 13.1. - -Vulnerabilities detected by DAST occur in the live web application. Addressing these types of -vulnerabilities requires specific information. DAST provides the information required to -investigate and rectify the underlying cause. - -To view details of vulnerabilities detected by DAST: - -1. To see all vulnerabilities detected, either: - - Go to your project and select **Security & Compliance**. - - Go to the merge request and select the **Security** tab. - -1. Select a vulnerability's description. The following details are provided: - - | Field | Description | - |:-----------------|:------------------------------------------------------------------ | - | Description | Description of the vulnerability. | - | Project | Namespace and project in which the vulnerability was detected. | - | Method | HTTP method used to detect the vulnerability. | - | URL | URL at which the vulnerability was detected. | - | Request Headers | Headers of the request. | - | Response Status | Response status received from the application. | - | Response Headers | Headers of the response received from the application. | - | Evidence | Evidence of the data found that verified the vulnerability. Often a snippet of the request or response, this can be used to help verify that the finding is a vulnerability. | - | Identifiers | Identifiers of the vulnerability. | - | Severity | Severity of the vulnerability. | - | Scanner Type | Type of vulnerability report. | - | Links | Links to further details of the detected vulnerability. | - | Solution | Details of a recommended solution to the vulnerability (optional). | - -## Customize DAST settings - -You can customize the behavior of DAST using both CI/CD variables and command-line options. Use of CI/CD -variables overrides the values contained in the DAST template. - -### Customize DAST using CI/CD variables - -WARNING: -Beginning in GitLab 13.0, the use of [`only` and `except`](../../../ci/yaml/index.md#only--except) -is no longer supported. You must use [`rules`](../../../ci/yaml/index.md#rules) instead. - -The DAST settings can be changed through CI/CD variables by using the -[`variables`](../../../ci/yaml/index.md#variables) parameter in `.gitlab-ci.yml`. For details of -all DAST CI/CD variables, read [Available CI/CD variables](#available-cicd-variables). - -For example: - -```yaml -include: - - template: DAST.gitlab-ci.yml - -variables: - DAST_WEBSITE: https://example.com - DAST_SPIDER_MINS: 120 - DAST_BROWSER_SCAN: "true" # use the browser-based GitLab DAST crawler -``` - -Because the template is [evaluated before](../../../ci/yaml/index.md#include) the pipeline -configuration, the last mention of the variable takes precedence. - -#### Enable or disable rules - -A complete list of the rules that DAST uses to scan for vulnerabilities can be -found in the [ZAP documentation](https://www.zaproxy.org/docs/alerts/). - -`DAST_EXCLUDE_RULES` disables the rules with the given IDs. - -`DAST_ONLY_INCLUDE_RULES` restricts the set of rules used in the scan to -those with the given IDs. - -`DAST_EXCLUDE_RULES` and `DAST_ONLY_INCLUDE_RULES` are mutually exclusive and a -DAST scan with both configured exits with an error. - -By default, several rules are disabled because they either take a long time to -run or frequently generate false positives. The complete list of disabled rules -can be found in [`exclude_rules.yml`](https://gitlab.com/gitlab-org/security-products/dast/-/blob/main/src/config/exclude_rules.yml). - -The lists for `DAST_EXCLUDE_RULES` and `DAST_ONLY_INCLUDE_RULES` **must** be enclosed in double -quotes (`"`), otherwise they are interpreted as numeric values. - -#### Hide sensitive information - -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/36332) in GitLab 13.1. - -HTTP request and response headers may contain sensitive information, including cookies and -authorization credentials. By default, the following headers are masked: - -- `Authorization`. -- `Proxy-Authorization`. -- `Set-Cookie` (values only). -- `Cookie` (values only). - -Using the [`DAST_MASK_HTTP_HEADERS` CI/CD variable](#available-cicd-variables), you can list the -headers whose values you want masked. For details on how to mask headers, see -[Customizing the DAST settings](#customize-dast-settings). - -#### Use Mutual TLS - -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/299596) in GitLab 14.8. - -Mutual TLS allows a target application server to verify that requests are from a known source. Browser-based scans do not support Mutual TLS. - -**Requirements** - -- Base64-encoded PKCS12 certificate -- Password of the base64-encoded PKCS12 certificate - -To enable Mutual TLS: - -1. If the PKCS12 certificate is not already base64-encoded, convert it to base64 encoding. For security reasons, we recommend encoding the certificate locally, **not** using a web-hosted conversion service. For example, to encode the certificate on either macOS or Linux: - - ```shell - base64 <path-to-pkcs12-certificate-file> - ``` - -1. Create a [masked variable](../../../ci/variables/index.md) named `DAST_PKCS12_CERTIFICATE_BASE64` and store the base64-encoded PKCS12 certificate's value in that variable. -1. Create a masked variable `DAST_PKCS12_PASSWORD` and store the PKCS12 certificate's password in that variable. - -#### Available CI/CD variables - -These CI/CD variables are specific to DAST. They can be used to customize the behavior of DAST to your requirements. - -WARNING: -All customization of GitLab security scanning tools should be tested in a merge request before -merging these changes to the default branch. Failure to do so can give unexpected results, -including a large number of false positives. - -| CI/CD variable | Type | Description | -|:-------------------------------------------------|:--------------|:------------------------------| -| `DAST_ADVERTISE_SCAN` | boolean | Set to `true` to add a `Via` header to every request sent, advertising that the request was sent as part of a GitLab DAST scan. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/334947) in GitLab 14.1. | -| `DAST_AGGREGATE_VULNERABILITIES` | boolean | Vulnerability aggregation is set to `true` by default. To disable this feature and see each vulnerability individually set to `false`. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/254043) in GitLab 14.0. | -| `DAST_API_HOST_OVERRIDE` <sup>1</sup> | string | Used to override domains defined in API specification files. Only supported when importing the API specification from a URL. Example: `example.com:8080`. | -| `DAST_API_SPECIFICATION` <sup>1</sup> | URL or string | The API specification to import. The specification can be hosted at a URL, or the name of a file present in the `/zap/wrk` directory. The variable `DAST_WEBSITE` must be specified if this is omitted. | -| `DAST_AUTH_REPORT` <sup>2</sup> | boolean | Used in combination with exporting the `gl-dast-debug-auth-report.html` artifact to aid in debugging authentication issues. | -| `DAST_AUTH_EXCLUDE_URLS` <sup>2</sup> | URLs | **{warning}** **[Removed](https://gitlab.com/gitlab-org/gitlab/-/issues/289959)** in GitLab 14.0. Replaced by `DAST_EXCLUDE_URLS`. The URLs to skip during the authenticated scan; comma-separated. Regular expression syntax can be used to match multiple URLs. For example, `.*` matches an arbitrary character sequence. Not supported for API scans. | -| `DAST_AUTH_URL` <sup>1,2</sup> | URL | The URL of the page containing the sign-in HTML form on the target website. `DAST_USERNAME` and `DAST_PASSWORD` are submitted with the login form to create an authenticated scan. Not supported for API scans. Example: `https://login.example.com`. | -| `DAST_AUTH_VERIFICATION_LOGIN_FORM` <sup>2</sup> | boolean | Verifies successful authentication by checking for the lack of a login form once the login form has been submitted. | -| `DAST_AUTH_VERIFICATION_SELECTOR` <sup>2</sup> | selector | Verifies successful authentication by checking for presence of a selector once the login form has been submitted. Example: `css:.user-photo`. | -| `DAST_AUTH_VERIFICATION_URL` <sup>1,2</sup> | URL | A URL only accessible to logged in users that DAST can use to confirm successful authentication. If provided, DAST exits if it cannot access the URL. Example: `"http://example.com/loggedin_page"`. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/207335) in GitLab 13.8. | -| `DAST_AUTO_UPDATE_ADDONS` | boolean | ZAP add-ons are pinned to specific versions in the DAST Docker image. Set to `true` to download the latest versions when the scan starts. Default: `false`. | -| `DAST_BROWSER_PATH_TO_LOGIN_FORM` <sup>1,2</sup> | selector | Comma-separated list of selectors that are selected prior to attempting to enter `DAST_USERNAME` and `DAST_PASSWORD` into the login form. Example: `"css:.navigation-menu,css:.login-menu-item"`. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/326633) in GitLab 14.1. | -| `DAST_DEBUG` <sup>1</sup> | boolean | Enable debug message output. Default: `false`. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/12652) in GitLab 13.1. | -| `DAST_EXCLUDE_RULES` | string | Set to a comma-separated list of Vulnerability Rule IDs to exclude them from running during the scan. Rule IDs are numbers and can be found from the DAST log or on the [ZAP project](https://www.zaproxy.org/docs/alerts/). For example, `HTTP Parameter Override` has a rule ID of `10026`. Cannot be used when `DAST_ONLY_INCLUDE_RULES` is set. **Note:** In earlier versions of GitLab the excluded rules were executed but vulnerabilities they generated were suppressed. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/118641) in GitLab 12.10. | -| `DAST_EXCLUDE_URLS` <sup>1,2</sup> | URLs | The URLs to skip during the authenticated scan; comma-separated. Regular expression syntax can be used to match multiple URLs. For example, `.*` matches an arbitrary character sequence. Not supported for API scans. Example, `http://example.com/sign-out`. | -| `DAST_FIRST_SUBMIT_FIELD` <sup>2</sup> | string | The `id` or `name` of the element that when selected submits the username form of a multi-page login process. For example, `css:button[type='user-submit']`. [Introduced](https://gitlab.com/gitlab-org/gitlab-ee/issues/9894) in GitLab 12.4. | -| `DAST_FULL_SCAN_DOMAIN_VALIDATION_REQUIRED` | boolean | **{warning}** **[Removed](https://gitlab.com/gitlab-org/gitlab/-/issues/293595)** in GitLab 14.0. Set to `true` to require domain validation when running DAST full scans. Not supported for API scans. Default: `false` | -| `DAST_FULL_SCAN_ENABLED` <sup>1</sup> | boolean | Set to `true` to run a [ZAP Full Scan](https://github.com/zaproxy/zaproxy/wiki/ZAP-Full-Scan) instead of a [ZAP Baseline Scan](https://github.com/zaproxy/zaproxy/wiki/ZAP-Baseline-Scan). Default: `false` | -| `DAST_HTML_REPORT` | string | The filename of the HTML report written at the end of a scan. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/12652) in GitLab 13.1. | -| `DAST_INCLUDE_ALPHA_VULNERABILITIES` | boolean | Set to `true` to include alpha passive and active scan rules. Default: `false`. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/12652) in GitLab 13.1. | -| `DAST_MARKDOWN_REPORT` | string | The filename of the Markdown report written at the end of a scan. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/12652) in GitLab 13.1. | -| `DAST_MASK_HTTP_HEADERS` | string | Comma-separated list of request and response headers to be masked (GitLab 13.1). Must contain **all** headers to be masked. Refer to [list of headers that are masked by default](#hide-sensitive-information). | -| `DAST_MAX_URLS_PER_VULNERABILITY` | number | The maximum number of URLs reported for a single vulnerability. `DAST_MAX_URLS_PER_VULNERABILITY` is set to `50` by default. To list all the URLs set to `0`. [Introduced](https://gitlab.com/gitlab-org/security-products/dast/-/merge_requests/433) in GitLab 13.12. | -| `DAST_ONLY_INCLUDE_RULES` | string | Set to a comma-separated list of Vulnerability Rule IDs to configure the scan to run only them. Rule IDs are numbers and can be found from the DAST log or on the [ZAP project](https://www.zaproxy.org/docs/alerts/). Cannot be used when `DAST_EXCLUDE_RULES` is set. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/250651) in GitLab 13.12. | -| `DAST_PASSWORD` <sup>1,2</sup> | string | The password to authenticate to in the website. Example: `P@55w0rd!` | -| `DAST_PASSWORD_FIELD` <sup>1,2</sup> | string | The selector of password field at the sign-in HTML form. Example: `id:password` | -| `DAST_PATHS` | string | Set to a comma-separated list of URLs for DAST to scan. For example, `/page1.html,/category1/page3.html,/page2.html`. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/214120) in GitLab 13.4. | -| `DAST_PATHS_FILE` | string | The file path containing the paths within `DAST_WEBSITE` to scan. The file must be plain text with one path per line. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/258825) in GitLab 13.6. | -| `DAST_PKCS12_CERTIFICATE_BASE64` | string | The PKCS12 certificate used for sites that require Mutual TLS. Must be encoded as base64 text. | -| `DAST_PKCS12_PASSWORD` | string | The password of the certificate used in `DAST_PKCS12_CERTIFICATE_BASE64`. | -| `DAST_REQUEST_HEADERS` <sup>1</sup> | string | Set to a comma-separated list of request header names and values. Headers are added to every request made by DAST. For example, `Cache-control: no-cache,User-Agent: DAST/1.0` | -| `DAST_SKIP_TARGET_CHECK` | boolean | Set to `true` to prevent DAST from checking that the target is available before scanning. Default: `false`. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/229067) in GitLab 13.8. | -| `DAST_SPIDER_MINS` <sup>1</sup> | number | The maximum duration of the spider scan in minutes. Set to `0` for unlimited. Default: One minute, or unlimited when the scan is a full scan. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/12652) in GitLab 13.1. | -| `DAST_SPIDER_START_AT_HOST` | boolean | Set to `false` to prevent DAST from resetting the target to its host before scanning. When `true`, non-host targets `http://test.site/some_path` is reset to `http://test.site` before scan. Default: `true`. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/258805) in GitLab 13.6. | -| `DAST_SUBMIT_FIELD` <sup>2</sup> | string | The `id` or `name` of the element that when selected submits the login form or the password form of a multi-page login process. For example, `css:button[type='submit']`. [Introduced](https://gitlab.com/gitlab-org/gitlab-ee/issues/9894) in GitLab 12.4. | -| `DAST_TARGET_AVAILABILITY_TIMEOUT` <sup>1</sup> | number | Time limit in seconds to wait for target availability. | -| `DAST_USE_AJAX_SPIDER` <sup>1</sup> | boolean | Set to `true` to use the AJAX spider in addition to the traditional spider, useful for crawling sites that require JavaScript. Default: `false`. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/12652) in GitLab 13.1. | -| `DAST_USERNAME` <sup>1,2</sup> | string | The username to authenticate to in the website. Example: `admin` | -| `DAST_USERNAME_FIELD` <sup>1,2</sup> | string | The selector of username field at the sign-in HTML form. Example: `name:username` | -| `DAST_XML_REPORT` | string | The filename of the XML report written at the end of a scan. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/12652) in GitLab 13.1. | -| `DAST_WEBSITE` <sup>1</sup> | URL | The URL of the website to scan. The variable `DAST_API_SPECIFICATION` must be specified if this is omitted. | -| `DAST_ZAP_CLI_OPTIONS` | string | ZAP server command-line options. For example, `-Xmx3072m` would set the Java maximum memory allocation pool size. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/12652) in GitLab 13.1. | -| `DAST_ZAP_LOG_CONFIGURATION` | string | Set to a semicolon-separated list of additional log4j properties for the ZAP Server. Example: `logger.httpsender.name=org.parosproxy.paros.network.HttpSender;logger.httpsender.level=debug;logger.sitemap.name=org.parosproxy.paros.model.SiteMap;logger.sitemap.level=debug;` | -| `SECURE_ANALYZERS_PREFIX` | URL | Set the Docker registry base address from which to download the analyzer. | - -1. Available to an on-demand DAST scan. -1. Used for authentication. - -### Customize DAST using command-line options - -Not all DAST configuration is available via CI/CD variables. To find out all -possible options, run the following configuration. -Available command-line options are printed to the job log: - -```yaml -include: - template: DAST.gitlab-ci.yml - -dast: - script: - - /analyze --help -``` - -You must then overwrite the `script` command to pass in the appropriate -argument. For example, vulnerability definitions in alpha can be included with -`-a`. The following configuration includes those definitions: - -```yaml -include: - template: DAST.gitlab-ci.yml - -dast: - script: - - export DAST_WEBSITE=${DAST_WEBSITE:-$(cat environment_url.txt)} - - /analyze -a -t $DAST_WEBSITE -``` - -### Custom ZAProxy configuration - -The ZAProxy server contains many [useful configurable values](https://gitlab.com/gitlab-org/gitlab/-/issues/36437#note_245801885). -Many key/values for `-config` remain undocumented, but there is an untested list of -[possible keys](https://gitlab.com/gitlab-org/gitlab/-/issues/36437#note_244981023). -Note that these options are not supported by DAST, and may break the DAST scan -when used. An example of how to rewrite the Authorization header value with `TOKEN` follows: - -```yaml -include: - template: DAST.gitlab-ci.yml - -variables: - DAST_ZAP_CLI_OPTIONS: "-config replacer.full_list(0).description=auth -config replacer.full_list(0).enabled=true -config replacer.full_list(0).matchtype=REQ_HEADER -config replacer.full_list(0).matchstr=Authorization -config replacer.full_list(0).regex=false -config replacer.full_list(0).replacement=TOKEN" -``` - -## Authentication - -NOTE: -We highly recommend you configure the scanner to authenticate to the application. If you don't, it cannot check most of the application for security risks, as most -of your application is likely not accessible without authentication. We also recommend -you periodically confirm the scanner's authentication is still working, as this tends to break over -time due to authentication changes to the application. - -Create masked CI/CD variables to pass the credentials that DAST uses. -To create masked variables for the username and password, see [Create a custom variable in the UI](../../../ci/variables/index.md#custom-cicd-variables). -The key of the username variable must be `DAST_USERNAME`, -and the key of the password variable must be `DAST_PASSWORD`. - -After DAST has authenticated with the application, all cookies are collected from the web browser. -For each cookie a matching session token is created for use by ZAP. This ensures ZAP is recognized -by the application as correctly authenticated. - -Authentication supports single form logins, multi-step login forms, and authenticating to URLs outside of the configured target URL. - -WARNING: -**Never** run an authenticated scan against a production server. When an authenticated -scan is run, it may perform *any* function that the authenticated user can. This -includes actions like modifying and deleting data, submitting forms, and following links. -Only run an authenticated scan against a test server. - -### SSO - -DAST can authenticate to websites making use of SSO, with the following restrictions: - -- DAST cannot bypass a CAPTCHA if the authentication flow includes one. -- DAST cannot handle multi-factor authentication like one-time passwords (OTP) by using SMS or authenticator apps. -- DAST must get a cookie, or a local or session storage, with a sufficiently random value. - -The [authentication debug output](index.md#configure-the-authentication-debug-output) can be helpful for troubleshooting SSO authentication -with DAST. - -### Log in using automatic detection of the login form - -By providing a `DAST_USERNAME`, `DAST_PASSWORD`, and `DAST_AUTH_URL`, DAST attempts to authenticate to the -target application by locating the login form based on a determination about whether or not the form contains username or password fields. - -Automatic detection is "best-effort", and depending on the application being scanned may provide either a resilient login experience or one that fails to authenticate the user. - -Login process: - -1. The `DAST_AUTH_URL` is loaded into the browser, and any forms on the page are located. - 1. If a form contains a username and password field, `DAST_USERNAME` and `DAST_PASSWORD` is inputted into the respective fields, the form submit button is selected and the user is logged in. - 1. If a form contains only a username field, it is assumed that the login form is multi-step. - 1. The `DAST_USERNAME` is inputted into the username field and the form submit button is selected. - 1. The subsequent pages loads where it is expected that a form exists and contains a password field. If found, `DAST_PASSWORD` is inputted, form submit button is selected and the user is logged in. - -### Log in using explicit selection of the login form - -By providing a `DAST_USERNAME_FIELD`, `DAST_PASSWORD_FIELD`, and `DAST_SUBMIT_FIELD`, in addition to the fields required for automatic login, -DAST attempts to authenticate to the target application by locating the login form based on the selectors provided. -Most applications benefit from this approach to authentication. - -Login process: - -1. The `DAST_AUTH_URL` is loaded into the browser, and any forms on the page are located. - 1. If the `DAST_FIRST_SUBMIT_FIELD` is not defined, then `DAST_USERNAME` is inputted into `DAST_USERNAME_FIELD`, `DAST_PASSWORD` is inputted into `DAST_PASSWORD_FIELD`, `DAST_SUBMIT_FIELD` is selected and the user is logged in. - 1. If the `DAST_FIRST_SUBMIT_FIELD` is defined, then it is assumed that the login form is multi-step. - 1. The `DAST_USERNAME` is inputted into the `DAST_USERNAME_FIELD` field and the `DAST_FIRST_SUBMIT_FIELD` is selected. - 1. The subsequent pages loads where the `DAST_PASSWORD` is inputted into the `DAST_PASSWORD_FIELD` field, the `DAST_SUBMIT_FIELD` is selected and the user is logged in. - -### Verifying successful login - -Once the login form has been submitted, DAST determines if the login was successful. Unsuccessful attempts at authentication cause the scan to halt. - -Following the submission of the login form, authentication is determined to be unsuccessful when: - -- A `400` or `500` series HTTP response status code is returned. -- A new cookie/browser storage value determined to be sufficiently random has not been set. - -In addition to these checks, the user can configure their own verification checks. -Each of the following checks can be used in conjunction with one another, if none are configured by default the presence of a login form is checked. - -#### Verifying based on the URL - -When `DAST_AUTH_VERIFICATION_URL` is configured, the URL displayed in the browser tab post login form submission is directly compared to the URL in the CI/CD variable. -If these are not exactly the same, authentication is deemed to be unsuccessful. - -For example: - -```yaml -include: - - template: DAST.gitlab-ci.yml - -dast: - variables: - DAST_WEBSITE: "https://example.com" - DAST_BROWSER_SCAN: "true" # use the browser-based GitLab DAST crawler - ... - DAST_AUTH_VERIFICATION_URL: "https://example.com/user/welcome" -``` - -#### Verify based on presence of an element - -When `DAST_AUTH_VERIFICATION_SELECTOR` is configured, the page displayed in the browser tab is searched for an element described by the selector in the CI/CD variable. -If no element is found, authentication is deemed to be unsuccessful. - -For example: - -```yaml -include: - - template: DAST.gitlab-ci.yml - -dast: - variables: - DAST_WEBSITE: "https://example.com" - DAST_BROWSER_SCAN: "true" # use the browser-based GitLab DAST crawler - ... - DAST_AUTH_VERIFICATION_SELECTOR: "css:.welcome-user" -``` - -#### Verify based on presence of a login form - -When `DAST_AUTH_VERIFICATION_LOGIN_FORM` is configured, the page displayed in the browser tab is searched for a form that is detected to be a login form. -If any such form is found, authentication is deemed to be unsuccessful. - -For example: - -```yaml -include: - - template: DAST.gitlab-ci.yml - -dast: - variables: - DAST_WEBSITE: "https://example.com" - DAST_BROWSER_SCAN: "true" # use the browser-based GitLab DAST crawler - ... - DAST_AUTH_VERIFICATION_LOGIN_FORM: "true" -``` - -### View the login form - -Many web applications show the user the login form in a pop-up (modal) window. -For these applications, navigating to the form requires both: - -- A starting URL. -- A list of elements to select to display the modal window. - -When `DAST_BROWSER_PATH_TO_LOGIN_FORM` is present, like in this example: - -```yaml -include: - - template: DAST.gitlab-ci.yml - -dast: - variables: - DAST_WEBSITE: "https://my.site.com" - DAST_BROWSER_SCAN: "true" # use the browser-based GitLab DAST crawler - ... - DAST_AUTH_URL: "https://my.site.com/admin" - DAST_BROWSER_PATH_TO_LOGIN_FORM: "css:.navigation-menu,css:.login-menu-item" -``` - -DAST performs these actions: - -1. Load the `DAST_AUTH_URL` page, such as `https://my.site.com/admin`. -1. After the page loads, DAST selects elements found by the selectors described - in `DAST_BROWSER_PATH_TO_LOGIN_FORM`. This example opens the navigation menu - and selects the login menu, to display the login modal window. -1. To continue the authentication process, DAST fills in the username and password - on the login form. - -### Configure the authentication debug output - -It is often difficult to understand the cause of an authentication failure when running DAST in a CI/CD pipeline. -To assist users in debugging authentication issues, a debug report can be generated and saved as a job artifact. -This HTML report contains all steps made during the login process, along with HTTP requests and responses, the Document Object Model (DOM) and screenshots. - - - -An example configuration where the authentication debug report is exported may look like the following: - -```yaml -dast: - variables: - DAST_WEBSITE: "https://example.com" - DAST_BROWSER_SCAN: "true" # use the browser-based GitLab DAST crawler - ... - DAST_AUTH_REPORT: "true" - artifacts: - paths: [gl-dast-debug-auth-report.html] - when: always -``` - -### Selectors - -Selectors are used by CI/CD variables to specify the location of an element displayed on a page in a browser. -Selectors have the format `type`:`search string`. The crawler searches for the selector using the search string based on the type. - -| Selector type | Example | Description | -| ------------- | ---------------------------------- | ----------- | -| `css` | `css:.password-field` | Searches for a HTML element having the supplied CSS selector. Selectors should be as specific as possible for performance reasons. | -| `id` | `id:element` | Searches for an HTML element with the provided element ID. | -| `name` | `name:element` | Searches for an HTML element with the provided element name. | -| `xpath` | `xpath://input[@id="my-button"]/a` | Searches for a HTML element with the provided XPath. Note that XPath searches are expected to be less performant than other searches. | -| None provided | `a.click-me` | Defaults to searching using a CSS selector. | - -#### Find selectors with Google Chrome - -Chrome DevTools element selector tool is an effective way to find a selector. - -1. Open Chrome and navigate to the page where you would like to find a selector, for example, the login page for your site. -1. Open the `Elements` tab in Chrome DevTools with the keyboard shortcut `Command + Shift + c` in macOS or `Ctrl + Shift + c` in Windows. -1. Select the `Select an element in the page to select it` tool. -  -1. Select the field on your page that you would like to know the selector for. -1. Once the tool is active, highlight a field you wish to view the details of. -  -1. Once highlighted, you can see the element's details, including attributes that would make a good candidate for a selector. - -In this example, the `id="user_login"` appears to be a good candidate. You can use this as a selector as the DAST username field by setting -`DAST_USERNAME_FIELD: "id:user_login"`. - -#### Choose the right selector - -Judicious choice of selector leads to a scan that is resilient to the application changing. - -In order of preference, it is recommended to choose as selectors: - -- `id` fields. These are generally unique on a page, and rarely change. -- `name` fields. These are generally unique on a page, and rarely change. -- `class` values specific to the field, such as the selector `"css:.username"` for the `username` class on the username field. -- Presence of field specific data attributes, such as the selector, `"css:[data-username]"` when the `data-username` field has any value on the username field. -- Multiple `class` hierarchy values, such as the selector `"css:.login-form .username"` when there are multiple elements with class `username` but only one nested inside the element with the class `login-form`. - -When using selectors to locate specific fields we recommend you avoid searching on: - -- Any `id`, `name`, `attribute`, `class` or `value` that is dynamically generated. -- Generic class names, such as `column-10` and `dark-grey`. -- XPath searches as they are less performant than other selector searches. -- Unscoped searches, such as those beginning with `css:*` and `xpath://*`. - -### Bleeding-edge vulnerability definitions - -ZAP first creates rules in the `alpha` class. After a testing period with -the community, they are promoted to `beta`. DAST uses `beta` definitions by -default. To request `alpha` definitions, use the -`DAST_INCLUDE_ALPHA_VULNERABILITIES` CI/CD variable as shown in the -following configuration: - -```yaml -include: - template: DAST.gitlab-ci.yml - -variables: - DAST_INCLUDE_ALPHA_VULNERABILITIES: "true" -``` - -### Cloning the project's repository - -The DAST job does not require the project's repository to be present when running, so by default -[`GIT_STRATEGY`](../../../ci/runners/configure_runners.md#git-strategy) is set to `none`. - -## On-demand scans - -> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/218465) in GitLab 13.2. -> - [Improved](https://gitlab.com/gitlab-org/gitlab/-/issues/218465) in GitLab 13.3. -> - The saved scans feature was [introduced](https://gitlab.com/groups/gitlab-org/-/epics/5100) in GitLab 13.9. -> - The option to select a branch was [introduced](https://gitlab.com/groups/gitlab-org/-/epics/4847) in GitLab 13.10. -> - DAST branch selection [feature flag removed](https://gitlab.com/gitlab-org/gitlab/-/issues/322672) in GitLab 13.11. -> - Auditing for DAST profile management was [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/217872) in GitLab 14.1. - -An on-demand DAST scan runs outside the DevOps life cycle. Changes in your repository don't trigger -the scan. You must either start it manually, or schedule it to run. - -An on-demand DAST scan: - -- Can run a specific combination of a [site profile](#site-profile) and a - [scanner profile](#scanner-profile). -- Is associated with your project's default branch. -- Is saved on creation so it can be run later. - -### On-demand scan modes - -An on-demand scan can be run in active or passive mode: - -- _Passive mode_ is the default and runs a ZAP Baseline Scan. -- _Active mode_ runs a ZAP Full Scan which is potentially harmful to the site being scanned. To - minimize the risk of accidental damage, running an active scan requires a [validated site profile](#site-profile-validation). - -### View on-demand DAST scans - -To view running completed and scheduled on-demand DAST scans for a project, go to -**Security & Compliance > On-demand Scans** in the left sidebar. - -- To view both running and completed scans, select **All**. -- To view running scans only, select **Running**. -- To view finished scans, select **Finished**. A finished scan is a scan that either succeeded, - failed, or was canceled. -- To view scheduled scans, select **Scheduled**. It shows on-demand scans that have a schedule - set up. Those are _not_ included in the **All** tab. -- To view saved on-demand scan profiles, select **Scan library**. - Those are _not_ included in the **All** tab. - -#### Cancel an on-demand scan - -To cancel a pending or running on-demand scan, select **Cancel** (**{cancel}**) in the -on-demand scans list. - -#### Retry an on-demand scan - -To retry a scan that failed or succeeded with warnings, select **Retry** (**{retry}**) in the -on-demand scans list. - -#### View an on-demand scan's results - -To view a finished scan's results, select **View results** in the on-demand scans list. - -#### Edit an on-demand scan - -To edit an on-demand scan's settings, select **Edit** (**{pencil}**) in the **Scheduled** tab. - -### Run an on-demand DAST scan - -Prerequisites: - -- You must have permission to run an on-demand DAST scan against a protected branch. The default - branch is automatically protected. For more information, read - [Pipeline security on protected branches](../../../ci/pipelines/index.md#pipeline-security-on-protected-branches). -- A [scanner profile](#create-a-scanner-profile). -- A [site profile](#create-a-site-profile). -- If you are running an active scan the site profile must have been [validated](#validate-a-site-profile). - -You can run an on-demand scan immediately, once at a scheduled date and time or at a specified -frequency: - -- Every day -- Every week -- Every month -- Every 3 months -- Every 6 months -- Every year - -To run an on-demand scan immediately, either: - -- [Create and run an on-demand scan immediately](#create-and-run-an-on-demand-scan-immediately). -- [Run a previously saved on-demand scan](#run-a-saved-on-demand-scan). - -To run an on-demand scan either at a scheduled date or frequency, read -[Schedule an on-demand scan](#schedule-an-on-demand-scan). - -#### Create and run an on-demand scan immediately - -1. From your project's home page, go to **Security & Compliance > On-demand Scans** in the left - sidebar. -1. Select **New scan**. -1. Complete the **Scan name** and **Description** fields. -1. In GitLab 13.10 and later, select the desired branch from the **Branch** dropdown. -1. In **Scanner profile**, select a scanner profile from the dropdown. -1. In **Site profile**, select a site profile from the dropdown. -1. To run the on-demand scan immediately, select **Save and run scan**. Otherwise, select - **Save scan** to [run](#run-a-saved-on-demand-scan) it later. - -The on-demand DAST scan runs and the project's dashboard shows the results. - -#### Run a saved on-demand scan - -To run a saved on-demand scan: - -1. On the top bar, select **Main menu > Projects** and find your project. -1. On the left sidebar, select **Security & Compliance > On-demand Scans**. -1. Select the **Scan library** tab. -1. In the scan's row, select **Run scan**. - - If the branch saved in the scan no longer exists, you must first - [edit the scan](#edit-an-on-demand-scan), select a new branch, and save the edited scan. - -The on-demand DAST scan runs, and the project's dashboard shows the results. - -#### Schedule an on-demand scan - -> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/328749) in GitLab 14.3. [Deployed behind the `dast_on_demand_scans_scheduler` flag](../../../administration/feature_flags.md), disabled by default. -> - [Enabled on GitLab.com](https://gitlab.com/gitlab-org/gitlab/-/issues/328749) in GitLab 14.4. -> - [Enabled on self-managed](https://gitlab.com/gitlab-org/gitlab/-/issues/328749) in GitLab 14.4. -> - [Feature flag `dast_on_demand_scans_scheduler` removed](https://gitlab.com/gitlab-org/gitlab/-/issues/328749) in GitLab 14.5. - -To schedule a scan: - -1. On the top bar, select **Main menu > Projects** and find your project. -1. On the left sidebar, select **Security & Compliance > On-demand Scans**. -1. Select **New scan**. -1. Complete the **Scan name** and **Description** text boxes. -1. In GitLab 13.10 and later, from the **Branch** dropdown list, select the desired branch. -1. In the **Scanner profile** section, from the dropdown list, select a scanner profile. -1. In the **Site profile** section, from the dropdown list, select a site profile. -1. Select **Schedule scan**. -1. In the **Start time** section, select a time zone, date, and time. -1. From the **Repeats** dropdown list, select your desired frequency: - - To run the scan once, select **Never**. - - For a recurring scan, select any other option. -1. To run the on-demand scan immediately, select **Save and run scan**. To [run](#run-a-saved-on-demand-scan) it according to the schedule you set, select - **Save scan**. - -#### List saved on-demand scans - -To list saved on-demand scans: - -1. From your project's home page, go to **Security & Compliance > On-demand Scans**. -1. Select the **Scan library** tab. - -#### View details of an on-demand scan - -To view details of an on-demand scan: - -1. From your project's home page, go to **Security & Compliance > On-demand Scans**. -1. Select the **Scan library** tab. -1. In the saved scan's row select **More actions** (**{ellipsis_v}**), then select **Edit**. - -#### Edit an on-demand scan - -To edit an on-demand scan: - -1. From your project's home page, go to **Security & Compliance > On-demand Scans**. -1. Select the **Scan library** tab. -1. In the saved scan's row select **More actions** (**{ellipsis_v}**), then select **Edit**. -1. Edit the form. -1. Select **Save scan**. - -#### Delete an on-demand scan - -To delete an on-demand scan: - -1. From your project's home page, go to **Security & Compliance > On-demand Scans**. -1. Select the **Scan library** tab. -1. In the saved scan's row select **More actions** (**{ellipsis_v}**), then select **Delete**. -1. Select **Delete** to confirm the deletion. - -## Site profile - -A site profile defines the attributes and configuration details of the deployed application, -website, or API to be scanned by DAST. A site profile can be referenced in `.gitlab-ci.yml` and -on-demand scans. - -A site profile contains: - -- **Profile name**: A name you assign to the site to be scanned. While a site profile is referenced - in either `.gitlab-ci.yml` or an on-demand scan, it **cannot** be renamed. -- **Site type**: The type of target to be scanned, either website or API scan. -- **Target URL**: The URL that DAST runs against. -- **Excluded URLs**: A comma-separated list of URLs to exclude from the scan. -- **Request headers**: A comma-separated list of HTTP request headers, including names and values. These headers are added to every request made by DAST. -- **Authentication**: - - **Authenticated URL**: The URL of the page containing the sign-in HTML form on the target website. The username and password are submitted with the login form to create an authenticated scan. - - **Username**: The username used to authenticate to the website. - - **Password**: The password used to authenticate to the website. - - **Username form field**: The name of username field at the sign-in HTML form. - - **Password form field**: The name of password field at the sign-in HTML form. - - **Submit form field**: The `id` or `name` of the element that when selected submits the sign-in HTML form. - -When an API site type is selected, a [host override](#host-override) is used to ensure the API being scanned is on the same host as the target. This is done to reduce the risk of running an active scan against the wrong API. - -When configured, request headers and password fields are encrypted using [`aes-256-gcm`](https://en.wikipedia.org/wiki/Advanced_Encryption_Standard) before being stored in the database. -This data can only be read and decrypted with a valid secrets file. - -### Site profile validation - -> - Site profile validation [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/233020) in GitLab 13.8. -> - Meta tag validation [introduced](https://gitlab.com/groups/gitlab-org/-/epics/6460) in GitLab 14.2. - -Site profile validation reduces the risk of running an active scan against the wrong website. A site -must be validated before an active scan can run against it. The site validation methods are as -follows: - -- _Text file validation_ requires a text file be uploaded to the target site. The text file is - allocated a name and content that is unique to the project. The validation process checks the - file's content. -- _Header validation_ requires the header `Gitlab-On-Demand-DAST` be added to the target site, - with a value unique to the project. The validation process checks that the header is present, and - checks its value. -- _Meta tag validation_ requires the meta tag named `gitlab-dast-validation` be added to the target site, - with a value unique to the project. Make sure it's added to the `<head>` section of the page. The validation process checks that the meta tag is present, and - checks its value. - -All these methods are equivalent in functionality. Use whichever is feasible. - -In [GitLab 14.2 and later](https://gitlab.com/gitlab-org/gitlab/-/issues/324990), site profile -validation happens in a CI job using the [GitLab Runner](../../../ci/runners/index.md). - -### Create a site profile - -To create a site profile: - -1. From your project's home page, go to **Security & Compliance > Configuration**. -1. Select **Manage** in the **DAST Profiles** row. -1. Select **New > Site Profile**. -1. Complete the fields then select **Save profile**. - -The site profile is created. - -### Edit a site profile - -If a site profile is linked to a security policy, a user cannot edit the profile from this page. See -[Scan execution policies](../policies/scan-execution-policies.md) -for more information. - -When a validated site profile's file, header, or meta tag is edited, the site's -[validation status](#site-profile-validation) is revoked. - -To edit a site profile: - -1. From your project's home page, go to **Security & Compliance > Configuration**. -1. In the **DAST Profiles** row select **Manage**. -1. Select the **Site Profiles** tab. -1. In the profile's row select the **More actions** (**{ellipsis_v}**) menu, then select **Edit**. -1. Edit the fields then select **Save profile**. - -### Delete a site profile - -If a site profile is linked to a security policy, a user cannot delete the profile from this page. -See [Scan execution policies](../policies/scan-execution-policies.md) -for more information. - -To delete a site profile: - -1. From your project's home page, go to **Security & Compliance > Configuration**. -1. In the **DAST Profiles** row select **Manage**. -1. Select the **Site Profiles** tab. -1. In the profile's row, select the **More actions** (**{ellipsis_v}**) menu, then select **Delete**. -1. Select **Delete** to confirm the deletion. - -### Validate a site profile - -Validating a site is required to run an active scan. - -To validate a site profile: - -1. On the top bar, select **Main menu > Projects** and find your project. -1. On the left sidebar, select **Security & Compliance > Configuration**. -1. In the **Dynamic Application Security Testing (DAST)** section, select **Manage profiles**. -1. Select the **Site Profiles** tab. -1. In the profile's row, select **Validate**. -1. Select the validation method. - 1. For **Text file validation**: - 1. Download the validation file listed in **Step 2**. - 1. Upload the validation file to the host, to the location in **Step 3** or any location you - prefer. - 1. If required, edit the file location in **Step 3**. - 1. Select **Validate**. - 1. For **Header validation**: - 1. Select the clipboard icon in **Step 2**. - 1. Edit the header of the site to validate, and paste the clipboard content. - 1. Select the input field in **Step 3** and enter the location of the header. - 1. Select **Validate**. - 1. For **Meta tag validation**: - 1. Select the clipboard icon in **Step 2**. - 1. Edit the content of the site to validate, and paste the clipboard content. - 1. Select the input field in **Step 3** and enter the location of the meta tag. - 1. Select **Validate**. - -The site is validated and an active scan can run against it. A site profile's validation status is -revoked only when it's revoked manually, or its file, header, or meta tag is edited. - -### Retry a failed validation - -> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/322609) in GitLab 14.3. -> - [Deployed behind the `dast_failed_site_validations` flag](../../../administration/feature_flags.md), enabled by default. -> - [Feature flag `dast_failed_site_validations` removed](https://gitlab.com/gitlab-org/gitlab/-/issues/323961) in GitLab 14.4. - -Failed site validation attempts are listed on the **Site profiles** tab of the **Manage profiles** -page. - -To retry a site profile's failed validation: - -1. On the top bar, select **Main menu > Projects** and find your project. -1. On the left sidebar, select **Security & Compliance > Configuration**. -1. In the **Dynamic Application Security Testing (DAST)** section, select **Manage profiles**. -1. Select the **Site Profiles** tab. -1. In the profile's row, select **Retry validation**. - -### Revoke a site profile's validation status - -WARNING: -When a site profile's validation status is revoked, all site profiles that share the same URL also -have their validation status revoked. - -To revoke a site profile's validation status: - -1. From your project's home page, go to **Security & Compliance > Configuration**. -1. In the **DAST Profiles** row select **Manage**. -1. Beside the validated profile, select **Revoke validation**. - -The site profile's validation status is revoked. - -### Validated site profile headers - -The following are code samples of how you can provide the required site profile header in your -application. - -#### Ruby on Rails example for on-demand scan - -Here's how you can add a custom header in a Ruby on Rails application: - -```ruby -class DastWebsiteTargetController < ActionController::Base - def dast_website_target - response.headers['Gitlab-On-Demand-DAST'] = '0dd79c9a-7b29-4e26-a815-eaaf53fcab1c' - head :ok - end -end -``` - -#### Django example for on-demand scan - -Here's how you can add a -[custom header in Django](https://docs.djangoproject.com/en/2.2/ref/request-response/#setting-header-fields): - -```python -class DastWebsiteTargetView(View): - def head(self, *args, **kwargs): - response = HttpResponse() - response['Gitlab-On-Demand-DAST'] = '0dd79c9a-7b29-4e26-a815-eaaf53fcab1c' - - return response -``` - -#### Node (with Express) example for on-demand scan - -Here's how you can add a -[custom header in Node (with Express)](https://expressjs.com/en/5x/api.html#res.append): - -```javascript -app.get('/dast-website-target', function(req, res) { - res.append('Gitlab-On-Demand-DAST', '0dd79c9a-7b29-4e26-a815-eaaf53fcab1c') - res.send('Respond to DAST ping') -}) -``` - -## Scanner profile - -> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/222767) in GitLab 13.4. -> - [Added](https://gitlab.com/gitlab-org/gitlab/-/issues/225804) in GitLab 13.5: scan mode, AJAX spider, debug messages. - -A scanner profile defines the configuration details of a security scanner. A scanner profile can be -referenced in `.gitlab-ci.yml` and on-demand scans. - -A scanner profile contains: - -- **Profile name:** A name you give the scanner profile. For example, "Spider_15". While a scanner - profile is referenced in either `.gitlab-ci.yml` or an on-demand scan, it **cannot** be renamed. -- **Scan mode:** A passive scan monitors all HTTP messages (requests and responses) sent to the target. An active scan attacks the target to find potential vulnerabilities. -- **Spider timeout:** The maximum number of minutes allowed for the spider to traverse the site. -- **Target timeout:** The maximum number of seconds DAST waits for the site to be available before - starting the scan. -- **AJAX spider:** Run the AJAX spider, in addition to the traditional spider, to crawl the target site. -- **Debug messages:** Include debug messages in the DAST console output. - -### Create a scanner profile - -To create a scanner profile: - -1. From your project's home page, go to **Security & Compliance > Configuration**. -1. In the **DAST Profiles** row, select **Manage**. -1. Select **New > Scanner Profile**. -1. Complete the form. For details of each field, see [Scanner profile](#scanner-profile). -1. Select **Save profile**. - -### Edit a scanner profile - -If a scanner profile is linked to a security policy, a user cannot edit the profile from this page. -See [Scan execution policies](../policies/scan-execution-policies.md) -for more information. - -To edit a scanner profile: - -1. From your project's home page, go to **Security & Compliance > Configuration**. -1. In the **DAST Profiles** row, select **Manage**. -1. Select the **Scanner Profiles** tab. -1. In the scanner's row, select the **More actions** (**{ellipsis_v}**) menu, then select **Edit**. -1. Edit the form. -1. Select **Save profile**. - -### Delete a scanner profile - -If a scanner profile is linked to a security policy, a user cannot delete the profile from this -page. See [Scan execution policies](../policies/scan-execution-policies.md) -for more information. - -To delete a scanner profile: - -1. From your project's home page, go to **Security & Compliance > Configuration**. -1. In the **DAST Profiles** row, select **Manage**. -1. Select the **Scanner Profiles** tab. -1. In the scanner's row, select the **More actions** (**{ellipsis_v}**) menu, then select **Delete**. -1. Select **Delete**. - -## Auditing - -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/217872) in GitLab 14.1. - -The creation, updating, and deletion of DAST profiles, DAST scanner profiles, -and DAST site profiles are included in the [audit log](../../../administration/audit_events.md). - -## Reports - -The DAST tool outputs a `gl-dast-report.json` report file containing details of the scan and its results. -This file is included in the job's artifacts. JSON is the default format, but -you can output the report in Markdown, HTML, and XML formats. To specify an alternative -format, use a [CI/CD variable](#available-cicd-variables). You can also use a CI/CD variable -to configure the job to output the `gl-dast-debug-auth-report.html` file which helps when debugging -authentication issues. - -For details of the report's schema, see the [schema for DAST reports](https://gitlab.com/gitlab-org/security-products/security-report-schemas/-/blob/master/dist/dast-report-format.json). Example reports can be found in the -[DAST repository](https://gitlab.com/gitlab-org/security-products/dast/-/tree/main/test/end-to-end/expect). - -WARNING: -The JSON report artifacts are not a public API of DAST and their format is expected to change in the -future. - -## Optimizing DAST - -By default, DAST downloads all artifacts defined by previous jobs in the pipeline. If -your DAST job does not rely on `environment_url.txt` to define the URL under test or any other files created -in previous jobs, we recommend you don't download artifacts. To avoid downloading -artifacts, add the following to your `.gitlab-ci.yml` file: - -```yaml -dast: - dependencies: [] -``` diff --git a/doc/user/application_security/dast/proxy-based.md b/doc/user/application_security/dast/proxy-based.md new file mode 100644 index 00000000000..ec98b809fb7 --- /dev/null +++ b/doc/user/application_security/dast/proxy-based.md @@ -0,0 +1,1247 @@ +--- +stage: Secure +group: Dynamic Analysis +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments +type: reference, howto +--- + +# DAST proxy-based analyzer **(ULTIMATE)** + +The DAST proxy-based analyzer can be added to your [GitLab CI/CD](../../../ci/index.md) pipeline. +This helps you discover vulnerabilities in web applications that do not use JavaScript heavily. For applications that do, +please see the [DAST browser-based analyzer](browser_based.md). + +WARNING: +Do not run DAST scans against a production server. Not only can it perform *any* function that +a user can, such as clicking buttons or submitting forms, but it may also trigger bugs, leading to modification or loss of production data. Only run DAST scans against a test server. + +The analyzer uses the [OWASP Zed Attack Proxy](https://www.zaproxy.org/) (ZAP) to scan in two different ways: + +- Passive scan only (default). DAST executes + [ZAP's Baseline Scan](https://www.zaproxy.org/docs/docker/baseline-scan/) and doesn't + actively attack your application. +- Passive and active (or full) scan. DAST can be [configured](#full-scan) to also perform an active scan + to attack your application and produce a more extensive security report. It can be very + useful when combined with [Review Apps](../../../ci/review_apps/index.md). + +## DAST run options + +You can use DAST to examine your web application: + +- Automatically, initiated by a merge request. +- Manually, initiated on demand. + +Some of the differences between these run options: + +| Automatic scan | On-demand scan | +|:-----------------------------------------------------------------|:------------------------------| +| DAST scan is initiated by a merge request. | DAST scan is initiated manually, outside the DevOps life cycle. | +| CI/CD variables are sourced from `.gitlab-ci.yml`. | CI/CD variables are provided in the UI. | +| All [DAST CI/CD variables](#available-cicd-variables) available. | Subset of [DAST CI/CD variables](#available-cicd-variables) available. | +| `DAST.gitlab-ci.yml` template. | `DAST-On-Demand-Scan.gitlab-ci.yml` template. | + +### Enable automatic DAST run + +To enable DAST to run automatically, either: + +- Enable [Auto DAST](../../../topics/autodevops/stages.md#auto-dast) (provided + by [Auto DevOps](../../../topics/autodevops/index.md)). +- [Include the DAST template](#include-the-dast-template) in your existing + `.gitlab-ci.yml` file. +- [Configure DAST using the UI](#configure-dast-using-the-ui). + +#### Include the DAST template + +> - This template was [updated](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/62597) to DAST_VERSION: 2 in GitLab 14.0. +> - This template was [updated](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/87183) to DAST_VERSION: 3 in GitLab 15.0. + +If you want to manually add DAST to your application, the DAST job is defined +in a CI/CD template file. Updates to the template are provided with GitLab +upgrades, allowing you to benefit from any improvements and additions. + +To include the DAST template: + +1. Select the CI/CD template you want to use: + + - [`DAST.gitlab-ci.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/ci/templates/Security/DAST.gitlab-ci.yml): + Stable version of the DAST CI/CD template. + - [`DAST.latest.gitlab-ci.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/ci/templates/Security/DAST.latest.gitlab-ci.yml): + Latest version of the DAST template. ([Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/254325) + in GitLab 13.8). + + WARNING: + The latest version of the template may include breaking changes. Use the + stable template unless you need a feature provided only in the latest template. + + For more information about template versioning, see the + [CI/CD documentation](../../../development/cicd/templates.md#latest-version). + +1. Add a `dast` stage to your GitLab CI stages configuration: + + ```yaml + stages: + - dast + ``` + +1. Add the template to GitLab, based on your version of GitLab: + + - In GitLab 11.9 and later, [include](../../../ci/yaml/index.md#includetemplate) + the template by adding the following to your `.gitlab-ci.yml` file: + + ```yaml + include: + - template: <template_file.yml> + + variables: + DAST_WEBSITE: https://example.com + ``` + + - In GitLab 11.8 and earlier, add the contents of the template to your + `.gitlab_ci.yml` file. + +1. Define the URL to be scanned by DAST by using one of these methods: + + - Set the `DAST_WEBSITE` [CI/CD variable](../../../ci/yaml/index.md#variables). + If set, this value takes precedence. + + - Add the URL in an `environment_url.txt` file at the root of your project. This is + useful for testing in dynamic environments. To run DAST against an application + dynamically created during a GitLab CI/CD pipeline, a job that runs prior to + the DAST scan must persist the application's domain in an `environment_url.txt` + file. DAST automatically parses the `environment_url.txt` file to find its + scan target. + + For example, in a job that runs prior to DAST, you could include code that + looks similar to: + + ```yaml + script: + - echo http://${CI_PROJECT_ID}-${CI_ENVIRONMENT_SLUG}.domain.com > environment_url.txt + artifacts: + paths: [environment_url.txt] + when: always + ``` + + You can see an example of this in our + [Auto DevOps CI YAML](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/ci/templates/Jobs/Deploy.gitlab-ci.yml) + file. + +The included template creates a `dast` job in your CI/CD pipeline and scans +your project's running application for possible vulnerabilities. + +The results are saved as a +[DAST report artifact](../../../ci/yaml/artifacts_reports.md#artifactsreportsdast) +that you can later download and analyze. Due to implementation limitations, we +always take the latest DAST artifact available. Behind the scenes, the +[GitLab DAST Docker image](https://gitlab.com/security-products/dast) +is used to run the tests on the specified URL and scan it for possible +vulnerabilities. + +By default, the DAST template uses the latest major version of the DAST Docker +image. Using the `DAST_VERSION` variable, you can choose how DAST updates: + +- Automatically update DAST with new features and fixes by pinning to a major + version (such as `1`). +- Only update fixes by pinning to a minor version (such as `1.6`). +- Prevent all updates by pinning to a specific version (such as `1.6.4`). + +Find the latest DAST versions on the [Releases](https://gitlab.com/gitlab-org/security-products/dast/-/releases) +page. + +#### Configure DAST using the UI + +You can enable or configure DAST settings using the UI. The generated settings are formatted so they +can be conveniently pasted into the `.gitlab-ci.yml` file. + +1. On the top bar, select **Main menu > Projects** and find your project. +1. On the left sidebar, select **Security & Compliance > Configuration**. +1. In the **Dynamic Application Security Testing (DAST)** section, select **Enable DAST** or + **Configure DAST**. +1. Select the desired **Scanner profile**, or select **Create scanner profile** and save a + scanner profile. For more details, see [scanner profiles](#scanner-profile). +1. Select the desired **Site profile**, or select **Create site profile** and save a site + profile. For more details, see [site profiles](#site-profile). +1. Select **Generate code snippet**. A modal opens with the YAML snippet corresponding to the + options you selected. +1. Do one of the following: + 1. To copy the snippet to your clipboard, select **Copy code only**. + 1. To add the snippet to your project's `.gitlab-ci.yml` file, select + **Copy code and open `.gitlab-ci.yml` file**. The Pipeline Editor opens. + 1. Paste the snippet into the `.gitlab-ci.yml` file. + 1. Select the **Lint** tab to confirm the edited `.gitlab-ci.yml` file is valid. + 1. Select the **Edit** tab, then select **Commit changes**. + +When the snippet is committed to the `.gitlab-ci.yml` file, pipelines include a DAST job. + +### API scan + +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/10928) in GitLab 12.10. +> - A new DAST API scanning engine was introduced in GitLab 13.10. + +Using an API specification as a scan's target is a useful way to seed URLs for scanning an API. +Vulnerability rules in an API scan are different than those in a normal website scan. + +A new DAST API scanning engine is available in GitLab 13.12 and later. For more details, see [DAST API scanning engine](../dast_api). The new scanning engine supports REST, SOAP, GraphQL, and generic APIs using forms, XML, and JSON. Testing can be performed using OpenAPI, Postman Collections, and HTTP Archive (HAR) documents. + +The target API instance's base URL is provided by using the `DAST_API_TARGET_URL` variable or an `environment_url.txt` file. + +#### Specification format + +API scans support OpenAPI V2 and OpenAPI V3 specifications. You can define these specifications using `JSON` or `YAML`. + +#### Import API specification from a URL + +If your API specification is accessible at a URL, you can pass that URL in directly as the target. +The specification does not have to be hosted on the same host as the API being tested. + +```yaml +include: + - template: DAST-API.gitlab-ci.yml + +variables: + DAST_API_SPECIFICATION: http://my.api/api-specification.yml +``` + +#### Import API specification from a file + +If your API specification file is in your repository, you can provide its filename as the target. + +```yaml +dast: + variables: + GIT_STRATEGY: fetch + DAST_API_SPECIFICATION: api-specification.yml +``` + +#### Full API scan + +API scans support full scanning, which can be enabled by using the `DAST_FULL_SCAN_ENABLED` +CI/CD variable. Domain validation is not supported for full API scans. + +#### Host override + +Specifications often define a host, which contains a domain name and a port. The +host referenced may be different than the host of the API's review instance. +This can cause incorrect URLs to be imported, or a scan on an incorrect host. +Use the `DAST_API_HOST_OVERRIDE` CI/CD variable to override these values. + +WARNING: +When using the API host override feature, you cannot use the `$DAST_WEBSITE` variable to override the hostname. +A host override is _only_ supported when importing the API specification from a URL. Attempts to override the +host throw an error when the API specification is imported from a file. This is due to a limitation in the +ZAP OpenAPI extension. + +For example, with a OpenAPI V3 specification containing: + +```yaml +servers: + - url: https://api.host.com +``` + +If the test version of the API is running at `https://api-test.host.com`, then +the following DAST configuration can be used: + +```yaml +include: + - template: DAST-API.gitlab-ci.yml + +variables: + DAST_API_SPECIFICATION: http://api-test.host.com/api-specification.yml + DAST_API_HOST_OVERRIDE: api-test.host.com +``` + +#### Authentication using headers + +Tokens in request headers are often used as a way to authenticate API requests. +You can achieve this by using the `DAST_REQUEST_HEADERS` CI/CD variable. +Headers are applied to every request DAST makes. + +```yaml +include: + - template: DAST-API.gitlab-ci.yml + +variables: + DAST_API_SPECIFICATION: http://api-test.api.com/api-specification.yml + DAST_REQUEST_HEADERS: "Authorization: Bearer my.token" +``` + +### URL scan + +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/214120) in GitLab 13.4. +> - [Improved](https://gitlab.com/gitlab-org/gitlab/-/issues/273141) in GitLab 13.11. + +A URL scan allows you to specify which parts of a website are scanned by DAST. + +#### Define the URLs to scan + +URLs to scan can be specified by either of the following methods: + +- Use `DAST_PATHS_FILE` CI/CD variable to specify the name of a file containing the paths. +- Use `DAST_PATHS` variable to list the paths. + +##### Use `DAST_PATHS_FILE` CI/CD variable + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/258825) in GitLab 13.6. + +To define the URLs to scan in a file, create a plain text file with one path per line. + +```plaintext +page1.html +/page2.html +category/shoes/page1.html +``` + +To scan the URLs in that file, set the CI/CD variable `DAST_PATHS_FILE` to the path of that file. +The file can be checked into the project repository or generated as an artifact by a job that +runs before DAST. + +By default, DAST scans do not clone the project repository. Instruct the DAST job to clone +the project by setting `GIT_STRATEGY` to fetch. Give a file path relative to `CI_PROJECT_DIR` to `DAST_PATHS_FILE`. + +```yaml +include: + - template: DAST.gitlab-ci.yml + +variables: + GIT_STRATEGY: fetch + DAST_PATHS_FILE: url_file.txt # url_file.txt lives in the root directory of the project + DAST_BROWSER_SCAN: "true" # use the browser-based GitLab DAST crawler +``` + +##### Use `DAST_PATHS` CI/CD variable + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/214120) in GitLab 13.4. + +To specify the paths to scan in a CI/CD variable, add a comma-separated list of the paths to the `DAST_PATHS` +variable. Note that you can only scan paths of a single host. + +```yaml +include: + - template: DAST.gitlab-ci.yml + +variables: + DAST_PATHS: "/page1.html,/category1/page1.html,/page3.html" + DAST_BROWSER_SCAN: "true" # use the browser-based GitLab DAST crawler +``` + +When using `DAST_PATHS` and `DAST_PATHS_FILE`, note the following: + +- `DAST_WEBSITE` must be defined when using either `DAST_PATHS_FILE` or `DAST_PATHS`. The paths listed in either use `DAST_WEBSITE` to build the URLs to scan +- Spidering is disabled when `DAST_PATHS` or `DAST_PATHS_FILE` are defined +- `DAST_PATHS_FILE` and `DAST_PATHS` cannot be used together +- The `DAST_PATHS` variable has a limit of about 130kb. If you have a list or paths + greater than this, use `DAST_PATHS_FILE`. + +#### Full Scan + +To perform a [full scan](#full-scan) on the listed paths, use the `DAST_FULL_SCAN_ENABLED` CI/CD variable. + +## Customize DAST settings + +You can customize the behavior of DAST using both CI/CD variables and command-line options. Use of CI/CD +variables overrides the values contained in the DAST template. + +### Customize DAST using CI/CD variables + +WARNING: +Beginning in GitLab 13.0, the use of [`only` and `except`](../../../ci/yaml/index.md#only--except) +is no longer supported. You must use [`rules`](../../../ci/yaml/index.md#rules) instead. + +The DAST settings can be changed through CI/CD variables by using the +[`variables`](../../../ci/yaml/index.md#variables) parameter in `.gitlab-ci.yml`. For details of +all DAST CI/CD variables, read [Available CI/CD variables](#available-cicd-variables). + +For example: + +```yaml +include: + - template: DAST.gitlab-ci.yml + +variables: + DAST_WEBSITE: https://example.com + DAST_SPIDER_MINS: 120 + DAST_BROWSER_SCAN: "true" # use the browser-based GitLab DAST crawler +``` + +Because the template is [evaluated before](../../../ci/yaml/index.md#include) the pipeline +configuration, the last mention of the variable takes precedence. + +#### Enable or disable rules + +A complete list of the rules that DAST uses to scan for vulnerabilities can be +found in the [ZAP documentation](https://www.zaproxy.org/docs/alerts/). + +`DAST_EXCLUDE_RULES` disables the rules with the given IDs. + +`DAST_ONLY_INCLUDE_RULES` restricts the set of rules used in the scan to +those with the given IDs. + +`DAST_EXCLUDE_RULES` and `DAST_ONLY_INCLUDE_RULES` are mutually exclusive and a +DAST scan with both configured exits with an error. + +By default, several rules are disabled because they either take a long time to +run or frequently generate false positives. The complete list of disabled rules +can be found in [`exclude_rules.yml`](https://gitlab.com/gitlab-org/security-products/dast/-/blob/main/src/config/exclude_rules.yml). + +The lists for `DAST_EXCLUDE_RULES` and `DAST_ONLY_INCLUDE_RULES` **must** be enclosed in double +quotes (`"`), otherwise they are interpreted as numeric values. + +#### Hide sensitive information + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/36332) in GitLab 13.1. + +HTTP request and response headers may contain sensitive information, including cookies and +authorization credentials. By default, the following headers are masked: + +- `Authorization`. +- `Proxy-Authorization`. +- `Set-Cookie` (values only). +- `Cookie` (values only). + +Using the [`DAST_MASK_HTTP_HEADERS` CI/CD variable](#available-cicd-variables), you can list the +headers whose values you want masked. For details on how to mask headers, see +[Customizing the DAST settings](#customize-dast-settings). + +#### Use Mutual TLS + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/299596) in GitLab 14.8. + +Mutual TLS allows a target application server to verify that requests are from a known source. Browser-based scans do not support Mutual TLS. + +**Requirements** + +- Base64-encoded PKCS12 certificate +- Password of the base64-encoded PKCS12 certificate + +To enable Mutual TLS: + +1. If the PKCS12 certificate is not already base64-encoded, convert it to base64 encoding. For security reasons, we recommend encoding the certificate locally, **not** using a web-hosted conversion service. For example, to encode the certificate on either macOS or Linux: + + ```shell + base64 <path-to-pkcs12-certificate-file> + ``` + +1. Create a [masked variable](../../../ci/variables/index.md) named `DAST_PKCS12_CERTIFICATE_BASE64` and store the base64-encoded PKCS12 certificate's value in that variable. +1. Create a masked variable `DAST_PKCS12_PASSWORD` and store the PKCS12 certificate's password in that variable. + +#### Available CI/CD variables + +These CI/CD variables are specific to DAST. They can be used to customize the behavior of DAST to your requirements. + +WARNING: +All customization of GitLab security scanning tools should be tested in a merge request before +merging these changes to the default branch. Failure to do so can give unexpected results, +including a large number of false positives. + +| CI/CD variable | Type | Description | +|:-------------------------------------------------|:--------------|:------------------------------| +| `DAST_ADVERTISE_SCAN` | boolean | Set to `true` to add a `Via` header to every request sent, advertising that the request was sent as part of a GitLab DAST scan. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/334947) in GitLab 14.1. | +| `DAST_AGGREGATE_VULNERABILITIES` | boolean | Vulnerability aggregation is set to `true` by default. To disable this feature and see each vulnerability individually set to `false`. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/254043) in GitLab 14.0. | +| `DAST_API_HOST_OVERRIDE` <sup>1</sup> | string | Used to override domains defined in API specification files. Only supported when importing the API specification from a URL. Example: `example.com:8080`. | +| `DAST_API_SPECIFICATION` <sup>1</sup> | URL or string | The API specification to import. The specification can be hosted at a URL, or the name of a file present in the `/zap/wrk` directory. The variable `DAST_WEBSITE` must be specified if this is omitted. | +| `DAST_AUTH_REPORT` <sup>2</sup> | boolean | Used in combination with exporting the `gl-dast-debug-auth-report.html` artifact to aid in debugging authentication issues. | +| `DAST_AUTH_EXCLUDE_URLS` <sup>2</sup> | URLs | **{warning}** **[Removed](https://gitlab.com/gitlab-org/gitlab/-/issues/289959)** in GitLab 14.0. Replaced by `DAST_EXCLUDE_URLS`. The URLs to skip during the authenticated scan; comma-separated. Regular expression syntax can be used to match multiple URLs. For example, `.*` matches an arbitrary character sequence. Not supported for API scans. | +| `DAST_AUTH_URL` <sup>1,2</sup> | URL | The URL of the page containing the sign-in HTML form on the target website. `DAST_USERNAME` and `DAST_PASSWORD` are submitted with the login form to create an authenticated scan. Not supported for API scans. Example: `https://login.example.com`. | +| `DAST_AUTH_VERIFICATION_LOGIN_FORM` <sup>2</sup> | boolean | Verifies successful authentication by checking for the lack of a login form once the login form has been submitted. | +| `DAST_AUTH_VERIFICATION_SELECTOR` <sup>2</sup> | selector | Verifies successful authentication by checking for presence of a selector once the login form has been submitted. Example: `css:.user-photo`. | +| `DAST_AUTH_VERIFICATION_URL` <sup>1,2</sup> | URL | A URL only accessible to logged in users that DAST can use to confirm successful authentication. If provided, DAST exits if it cannot access the URL. Example: `"http://example.com/loggedin_page"`. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/207335) in GitLab 13.8. | +| `DAST_AUTO_UPDATE_ADDONS` | boolean | ZAP add-ons are pinned to specific versions in the DAST Docker image. Set to `true` to download the latest versions when the scan starts. Default: `false`. | +| `DAST_BROWSER_PATH_TO_LOGIN_FORM` <sup>1,2</sup> | selector | Comma-separated list of selectors that are selected prior to attempting to enter `DAST_USERNAME` and `DAST_PASSWORD` into the login form. Example: `"css:.navigation-menu,css:.login-menu-item"`. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/326633) in GitLab 14.1. | +| `DAST_DEBUG` <sup>1</sup> | boolean | Enable debug message output. Default: `false`. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/12652) in GitLab 13.1. | +| `DAST_EXCLUDE_RULES` | string | Set to a comma-separated list of Vulnerability Rule IDs to exclude them from running during the scan. Rule IDs are numbers and can be found from the DAST log or on the [ZAP project](https://www.zaproxy.org/docs/alerts/). For example, `HTTP Parameter Override` has a rule ID of `10026`. Cannot be used when `DAST_ONLY_INCLUDE_RULES` is set. **Note:** In earlier versions of GitLab the excluded rules were executed but vulnerabilities they generated were suppressed. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/118641) in GitLab 12.10. | +| `DAST_EXCLUDE_URLS` <sup>1,2</sup> | URLs | The URLs to skip during the authenticated scan; comma-separated. Regular expression syntax can be used to match multiple URLs. For example, `.*` matches an arbitrary character sequence. Not supported for API scans. Example, `http://example.com/sign-out`. | +| `DAST_FIRST_SUBMIT_FIELD` <sup>2</sup> | string | The `id` or `name` of the element that when selected submits the username form of a multi-page login process. For example, `css:button[type='user-submit']`. [Introduced](https://gitlab.com/gitlab-org/gitlab-ee/issues/9894) in GitLab 12.4. | +| `DAST_FULL_SCAN_DOMAIN_VALIDATION_REQUIRED` | boolean | **{warning}** **[Removed](https://gitlab.com/gitlab-org/gitlab/-/issues/293595)** in GitLab 14.0. Set to `true` to require domain validation when running DAST full scans. Not supported for API scans. Default: `false` | +| `DAST_FULL_SCAN_ENABLED` <sup>1</sup> | boolean | Set to `true` to run a [ZAP Full Scan](https://github.com/zaproxy/zaproxy/wiki/ZAP-Full-Scan) instead of a [ZAP Baseline Scan](https://github.com/zaproxy/zaproxy/wiki/ZAP-Baseline-Scan). Default: `false` | +| `DAST_HTML_REPORT` | string | The filename of the HTML report written at the end of a scan. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/12652) in GitLab 13.1. | +| `DAST_INCLUDE_ALPHA_VULNERABILITIES` | boolean | Set to `true` to include alpha passive and active scan rules. Default: `false`. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/12652) in GitLab 13.1. | +| `DAST_MARKDOWN_REPORT` | string | The filename of the Markdown report written at the end of a scan. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/12652) in GitLab 13.1. | +| `DAST_MASK_HTTP_HEADERS` | string | Comma-separated list of request and response headers to be masked (GitLab 13.1). Must contain **all** headers to be masked. Refer to [list of headers that are masked by default](#hide-sensitive-information). | +| `DAST_MAX_URLS_PER_VULNERABILITY` | number | The maximum number of URLs reported for a single vulnerability. `DAST_MAX_URLS_PER_VULNERABILITY` is set to `50` by default. To list all the URLs set to `0`. [Introduced](https://gitlab.com/gitlab-org/security-products/dast/-/merge_requests/433) in GitLab 13.12. | +| `DAST_ONLY_INCLUDE_RULES` | string | Set to a comma-separated list of Vulnerability Rule IDs to configure the scan to run only them. Rule IDs are numbers and can be found from the DAST log or on the [ZAP project](https://www.zaproxy.org/docs/alerts/). Cannot be used when `DAST_EXCLUDE_RULES` is set. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/250651) in GitLab 13.12. | +| `DAST_PASSWORD` <sup>1,2</sup> | string | The password to authenticate to in the website. Example: `P@55w0rd!` | +| `DAST_PASSWORD_FIELD` <sup>1,2</sup> | string | The selector of password field at the sign-in HTML form. Example: `id:password` | +| `DAST_PATHS` | string | Set to a comma-separated list of URLs for DAST to scan. For example, `/page1.html,/category1/page3.html,/page2.html`. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/214120) in GitLab 13.4. | +| `DAST_PATHS_FILE` | string | The file path containing the paths within `DAST_WEBSITE` to scan. The file must be plain text with one path per line. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/258825) in GitLab 13.6. | +| `DAST_PKCS12_CERTIFICATE_BASE64` | string | The PKCS12 certificate used for sites that require Mutual TLS. Must be encoded as base64 text. | +| `DAST_PKCS12_PASSWORD` | string | The password of the certificate used in `DAST_PKCS12_CERTIFICATE_BASE64`. | +| `DAST_REQUEST_HEADERS` <sup>1</sup> | string | Set to a comma-separated list of request header names and values. Headers are added to every request made by DAST. For example, `Cache-control: no-cache,User-Agent: DAST/1.0` | +| `DAST_SKIP_TARGET_CHECK` | boolean | Set to `true` to prevent DAST from checking that the target is available before scanning. Default: `false`. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/229067) in GitLab 13.8. | +| `DAST_SPIDER_MINS` <sup>1</sup> | number | The maximum duration of the spider scan in minutes. Set to `0` for unlimited. Default: One minute, or unlimited when the scan is a full scan. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/12652) in GitLab 13.1. | +| `DAST_SPIDER_START_AT_HOST` | boolean | Set to `false` to prevent DAST from resetting the target to its host before scanning. When `true`, non-host targets `http://test.site/some_path` is reset to `http://test.site` before scan. Default: `true`. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/258805) in GitLab 13.6. | +| `DAST_SUBMIT_FIELD` <sup>2</sup> | string | The `id` or `name` of the element that when selected submits the login form or the password form of a multi-page login process. For example, `css:button[type='submit']`. [Introduced](https://gitlab.com/gitlab-org/gitlab-ee/issues/9894) in GitLab 12.4. | +| `DAST_TARGET_AVAILABILITY_TIMEOUT` <sup>1</sup> | number | Time limit in seconds to wait for target availability. | +| `DAST_USE_AJAX_SPIDER` <sup>1</sup> | boolean | Set to `true` to use the AJAX spider in addition to the traditional spider, useful for crawling sites that require JavaScript. Default: `false`. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/12652) in GitLab 13.1. | +| `DAST_USERNAME` <sup>1,2</sup> | string | The username to authenticate to in the website. Example: `admin` | +| `DAST_USERNAME_FIELD` <sup>1,2</sup> | string | The selector of username field at the sign-in HTML form. Example: `name:username` | +| `DAST_XML_REPORT` | string | The filename of the XML report written at the end of a scan. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/12652) in GitLab 13.1. | +| `DAST_WEBSITE` <sup>1</sup> | URL | The URL of the website to scan. The variable `DAST_API_SPECIFICATION` must be specified if this is omitted. | +| `DAST_ZAP_CLI_OPTIONS` | string | ZAP server command-line options. For example, `-Xmx3072m` would set the Java maximum memory allocation pool size. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/12652) in GitLab 13.1. | +| `DAST_ZAP_LOG_CONFIGURATION` | string | Set to a semicolon-separated list of additional log4j properties for the ZAP Server. Example: `logger.httpsender.name=org.parosproxy.paros.network.HttpSender;logger.httpsender.level=debug;logger.sitemap.name=org.parosproxy.paros.model.SiteMap;logger.sitemap.level=debug;` | +| `SECURE_ANALYZERS_PREFIX` | URL | Set the Docker registry base address from which to download the analyzer. | + +1. Available to an on-demand DAST scan. +1. Used for authentication. + +### Customize DAST using command-line options + +Not all DAST configuration is available via CI/CD variables. To find out all +possible options, run the following configuration. +Available command-line options are printed to the job log: + +```yaml +include: + template: DAST.gitlab-ci.yml + +dast: + script: + - /analyze --help +``` + +You must then overwrite the `script` command to pass in the appropriate +argument. For example, vulnerability definitions in alpha can be included with +`-a`. The following configuration includes those definitions: + +```yaml +include: + template: DAST.gitlab-ci.yml + +dast: + script: + - export DAST_WEBSITE=${DAST_WEBSITE:-$(cat environment_url.txt)} + - /analyze -a -t $DAST_WEBSITE +``` + +### Custom ZAProxy configuration + +The ZAProxy server contains many [useful configurable values](https://gitlab.com/gitlab-org/gitlab/-/issues/36437#note_245801885). +Many key/values for `-config` remain undocumented, but there is an untested list of +[possible keys](https://gitlab.com/gitlab-org/gitlab/-/issues/36437#note_244981023). +Note that these options are not supported by DAST, and may break the DAST scan +when used. An example of how to rewrite the Authorization header value with `TOKEN` follows: + +```yaml +include: + template: DAST.gitlab-ci.yml + +variables: + DAST_ZAP_CLI_OPTIONS: "-config replacer.full_list(0).description=auth -config replacer.full_list(0).enabled=true -config replacer.full_list(0).matchtype=REQ_HEADER -config replacer.full_list(0).matchstr=Authorization -config replacer.full_list(0).regex=false -config replacer.full_list(0).replacement=TOKEN" +``` + +## Authentication + +NOTE: +We highly recommend you configure the scanner to authenticate to the application. If you don't, it cannot check most of the application for security risks, as most +of your application is likely not accessible without authentication. We also recommend +you periodically confirm the scanner's authentication is still working, as this tends to break over +time due to authentication changes to the application. + +Create masked CI/CD variables to pass the credentials that DAST uses. +To create masked variables for the username and password, see [Create a custom variable in the UI](../../../ci/variables/index.md#custom-cicd-variables). +The key of the username variable must be `DAST_USERNAME`, +and the key of the password variable must be `DAST_PASSWORD`. + +After DAST has authenticated with the application, all cookies are collected from the web browser. +For each cookie a matching session token is created for use by ZAP. This ensures ZAP is recognized +by the application as correctly authenticated. + +Authentication supports single form logins, multi-step login forms, and authenticating to URLs outside of the configured target URL. + +WARNING: +**Never** run an authenticated scan against a production server. When an authenticated +scan is run, it may perform *any* function that the authenticated user can. This +includes actions like modifying and deleting data, submitting forms, and following links. +Only run an authenticated scan against a test server. + +### SSO + +DAST can authenticate to websites making use of SSO, with the following restrictions: + +- DAST cannot bypass a CAPTCHA if the authentication flow includes one. +- DAST cannot handle multi-factor authentication like one-time passwords (OTP) by using SMS or authenticator apps. +- DAST must get a cookie, or a local or session storage, with a sufficiently random value. + +The [authentication debug output](#configure-the-authentication-debug-output) can be helpful for troubleshooting SSO authentication +with DAST. + +### Log in using automatic detection of the login form + +By providing a `DAST_USERNAME`, `DAST_PASSWORD`, and `DAST_AUTH_URL`, DAST attempts to authenticate to the +target application by locating the login form based on a determination about whether or not the form contains username or password fields. + +Automatic detection is "best-effort", and depending on the application being scanned may provide either a resilient login experience or one that fails to authenticate the user. + +Login process: + +1. The `DAST_AUTH_URL` is loaded into the browser, and any forms on the page are located. + 1. If a form contains a username and password field, `DAST_USERNAME` and `DAST_PASSWORD` is inputted into the respective fields, the form submit button is selected and the user is logged in. + 1. If a form contains only a username field, it is assumed that the login form is multi-step. + 1. The `DAST_USERNAME` is inputted into the username field and the form submit button is selected. + 1. The subsequent pages loads where it is expected that a form exists and contains a password field. If found, `DAST_PASSWORD` is inputted, form submit button is selected and the user is logged in. + +### Log in using explicit selection of the login form + +By providing a `DAST_USERNAME_FIELD`, `DAST_PASSWORD_FIELD`, and `DAST_SUBMIT_FIELD`, in addition to the fields required for automatic login, +DAST attempts to authenticate to the target application by locating the login form based on the selectors provided. +Most applications benefit from this approach to authentication. + +Login process: + +1. The `DAST_AUTH_URL` is loaded into the browser, and any forms on the page are located. + 1. If the `DAST_FIRST_SUBMIT_FIELD` is not defined, then `DAST_USERNAME` is inputted into `DAST_USERNAME_FIELD`, `DAST_PASSWORD` is inputted into `DAST_PASSWORD_FIELD`, `DAST_SUBMIT_FIELD` is selected and the user is logged in. + 1. If the `DAST_FIRST_SUBMIT_FIELD` is defined, then it is assumed that the login form is multi-step. + 1. The `DAST_USERNAME` is inputted into the `DAST_USERNAME_FIELD` field and the `DAST_FIRST_SUBMIT_FIELD` is selected. + 1. The subsequent pages loads where the `DAST_PASSWORD` is inputted into the `DAST_PASSWORD_FIELD` field, the `DAST_SUBMIT_FIELD` is selected and the user is logged in. + +### Verifying successful login + +Once the login form has been submitted, DAST determines if the login was successful. Unsuccessful attempts at authentication cause the scan to halt. + +Following the submission of the login form, authentication is determined to be unsuccessful when: + +- A `400` or `500` series HTTP response status code is returned. +- A new cookie/browser storage value determined to be sufficiently random has not been set. + +In addition to these checks, the user can configure their own verification checks. +Each of the following checks can be used in conjunction with one another, if none are configured by default the presence of a login form is checked. + +#### Verifying based on the URL + +When `DAST_AUTH_VERIFICATION_URL` is configured, the URL displayed in the browser tab post login form submission is directly compared to the URL in the CI/CD variable. +If these are not exactly the same, authentication is deemed to be unsuccessful. + +For example: + +```yaml +include: + - template: DAST.gitlab-ci.yml + +dast: + variables: + DAST_WEBSITE: "https://example.com" + DAST_BROWSER_SCAN: "true" # use the browser-based GitLab DAST crawler + ... + DAST_AUTH_VERIFICATION_URL: "https://example.com/user/welcome" +``` + +#### Verify based on presence of an element + +When `DAST_AUTH_VERIFICATION_SELECTOR` is configured, the page displayed in the browser tab is searched for an element described by the selector in the CI/CD variable. +If no element is found, authentication is deemed to be unsuccessful. + +For example: + +```yaml +include: + - template: DAST.gitlab-ci.yml + +dast: + variables: + DAST_WEBSITE: "https://example.com" + DAST_BROWSER_SCAN: "true" # use the browser-based GitLab DAST crawler + ... + DAST_AUTH_VERIFICATION_SELECTOR: "css:.welcome-user" +``` + +#### Verify based on presence of a login form + +When `DAST_AUTH_VERIFICATION_LOGIN_FORM` is configured, the page displayed in the browser tab is searched for a form that is detected to be a login form. +If any such form is found, authentication is deemed to be unsuccessful. + +For example: + +```yaml +include: + - template: DAST.gitlab-ci.yml + +dast: + variables: + DAST_WEBSITE: "https://example.com" + DAST_BROWSER_SCAN: "true" # use the browser-based GitLab DAST crawler + ... + DAST_AUTH_VERIFICATION_LOGIN_FORM: "true" +``` + +### View the login form + +Many web applications show the user the login form in a pop-up (modal) window. +For these applications, navigating to the form requires both: + +- A starting URL. +- A list of elements to select to display the modal window. + +When `DAST_BROWSER_PATH_TO_LOGIN_FORM` is present, like in this example: + +```yaml +include: + - template: DAST.gitlab-ci.yml + +dast: + variables: + DAST_WEBSITE: "https://my.site.com" + DAST_BROWSER_SCAN: "true" # use the browser-based GitLab DAST crawler + ... + DAST_AUTH_URL: "https://my.site.com/admin" + DAST_BROWSER_PATH_TO_LOGIN_FORM: "css:.navigation-menu,css:.login-menu-item" +``` + +DAST performs these actions: + +1. Load the `DAST_AUTH_URL` page, such as `https://my.site.com/admin`. +1. After the page loads, DAST selects elements found by the selectors described + in `DAST_BROWSER_PATH_TO_LOGIN_FORM`. This example opens the navigation menu + and selects the login menu, to display the login modal window. +1. To continue the authentication process, DAST fills in the username and password + on the login form. + +### Configure the authentication debug output + +It is often difficult to understand the cause of an authentication failure when running DAST in a CI/CD pipeline. +To assist users in debugging authentication issues, a debug report can be generated and saved as a job artifact. +This HTML report contains all steps made during the login process, along with HTTP requests and responses, the Document Object Model (DOM) and screenshots. + + + +An example configuration where the authentication debug report is exported may look like the following: + +```yaml +dast: + variables: + DAST_WEBSITE: "https://example.com" + DAST_BROWSER_SCAN: "true" # use the browser-based GitLab DAST crawler + ... + DAST_AUTH_REPORT: "true" + artifacts: + paths: [gl-dast-debug-auth-report.html] + when: always +``` + +### Selectors + +Selectors are used by CI/CD variables to specify the location of an element displayed on a page in a browser. +Selectors have the format `type`:`search string`. The crawler searches for the selector using the search string based on the type. + +| Selector type | Example | Description | +| ------------- | ---------------------------------- | ----------- | +| `css` | `css:.password-field` | Searches for a HTML element having the supplied CSS selector. Selectors should be as specific as possible for performance reasons. | +| `id` | `id:element` | Searches for an HTML element with the provided element ID. | +| `name` | `name:element` | Searches for an HTML element with the provided element name. | +| `xpath` | `xpath://input[@id="my-button"]/a` | Searches for a HTML element with the provided XPath. Note that XPath searches are expected to be less performant than other searches. | +| None provided | `a.click-me` | Defaults to searching using a CSS selector. | + +#### Find selectors with Google Chrome + +Chrome DevTools element selector tool is an effective way to find a selector. + +1. Open Chrome and navigate to the page where you would like to find a selector, for example, the login page for your site. +1. Open the `Elements` tab in Chrome DevTools with the keyboard shortcut `Command + Shift + c` in macOS or `Ctrl + Shift + c` in Windows. +1. Select the `Select an element in the page to select it` tool. +  +1. Select the field on your page that you would like to know the selector for. +1. Once the tool is active, highlight a field you wish to view the details of. +  +1. Once highlighted, you can see the element's details, including attributes that would make a good candidate for a selector. + +In this example, the `id="user_login"` appears to be a good candidate. You can use this as a selector as the DAST username field by setting +`DAST_USERNAME_FIELD: "id:user_login"`. + +#### Choose the right selector + +Judicious choice of selector leads to a scan that is resilient to the application changing. + +In order of preference, it is recommended to choose as selectors: + +- `id` fields. These are generally unique on a page, and rarely change. +- `name` fields. These are generally unique on a page, and rarely change. +- `class` values specific to the field, such as the selector `"css:.username"` for the `username` class on the username field. +- Presence of field specific data attributes, such as the selector, `"css:[data-username]"` when the `data-username` field has any value on the username field. +- Multiple `class` hierarchy values, such as the selector `"css:.login-form .username"` when there are multiple elements with class `username` but only one nested inside the element with the class `login-form`. + +When using selectors to locate specific fields we recommend you avoid searching on: + +- Any `id`, `name`, `attribute`, `class` or `value` that is dynamically generated. +- Generic class names, such as `column-10` and `dark-grey`. +- XPath searches as they are less performant than other selector searches. +- Unscoped searches, such as those beginning with `css:*` and `xpath://*`. + +### Bleeding-edge vulnerability definitions + +ZAP first creates rules in the `alpha` class. After a testing period with +the community, they are promoted to `beta`. DAST uses `beta` definitions by +default. To request `alpha` definitions, use the +`DAST_INCLUDE_ALPHA_VULNERABILITIES` CI/CD variable as shown in the +following configuration: + +```yaml +include: + template: DAST.gitlab-ci.yml + +variables: + DAST_INCLUDE_ALPHA_VULNERABILITIES: "true" +``` + +### Cloning the project's repository + +The DAST job does not require the project's repository to be present when running, so by default +[`GIT_STRATEGY`](../../../ci/runners/configure_runners.md#git-strategy) is set to `none`. + +## On-demand scans + +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/218465) in GitLab 13.2. +> - [Improved](https://gitlab.com/gitlab-org/gitlab/-/issues/218465) in GitLab 13.3. +> - The saved scans feature was [introduced](https://gitlab.com/groups/gitlab-org/-/epics/5100) in GitLab 13.9. +> - The option to select a branch was [introduced](https://gitlab.com/groups/gitlab-org/-/epics/4847) in GitLab 13.10. +> - DAST branch selection [feature flag removed](https://gitlab.com/gitlab-org/gitlab/-/issues/322672) in GitLab 13.11. +> - Auditing for DAST profile management was [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/217872) in GitLab 14.1. + +An on-demand DAST scan runs outside the DevOps life cycle. Changes in your repository don't trigger +the scan. You must either start it manually, or schedule it to run. + +An on-demand DAST scan: + +- Can run a specific combination of a [site profile](#site-profile) and a + [scanner profile](#scanner-profile). +- Is associated with your project's default branch. +- Is saved on creation so it can be run later. + +### On-demand scan modes + +An on-demand scan can be run in active or passive mode: + +- _Passive mode_ is the default and runs a ZAP Baseline Scan. +- _Active mode_ runs a ZAP Full Scan which is potentially harmful to the site being scanned. To + minimize the risk of accidental damage, running an active scan requires a [validated site profile](#site-profile-validation). + +### View on-demand DAST scans + +To view running completed and scheduled on-demand DAST scans for a project, go to +**Security & Compliance > On-demand Scans** in the left sidebar. + +- To view both running and completed scans, select **All**. +- To view running scans only, select **Running**. +- To view finished scans, select **Finished**. A finished scan is a scan that either succeeded, + failed, or was canceled. +- To view scheduled scans, select **Scheduled**. It shows on-demand scans that have a schedule + set up. Those are _not_ included in the **All** tab. +- To view saved on-demand scan profiles, select **Scan library**. + Those are _not_ included in the **All** tab. + +#### Cancel an on-demand scan + +To cancel a pending or running on-demand scan, select **Cancel** (**{cancel}**) in the +on-demand scans list. + +#### Retry an on-demand scan + +To retry a scan that failed or succeeded with warnings, select **Retry** (**{retry}**) in the +on-demand scans list. + +#### View an on-demand scan's results + +To view a finished scan's results, select **View results** in the on-demand scans list. + +#### Edit an on-demand scan + +To edit an on-demand scan's settings, select **Edit** (**{pencil}**) in the **Scheduled** tab. + +### Run an on-demand DAST scan + +Prerequisites: + +- You must have permission to run an on-demand DAST scan against a protected branch. The default + branch is automatically protected. For more information, read + [Pipeline security on protected branches](../../../ci/pipelines/index.md#pipeline-security-on-protected-branches). +- A [scanner profile](#create-a-scanner-profile). +- A [site profile](#create-a-site-profile). +- If you are running an active scan the site profile must have been [validated](#validate-a-site-profile). + +You can run an on-demand scan immediately, once at a scheduled date and time or at a specified +frequency: + +- Every day +- Every week +- Every month +- Every 3 months +- Every 6 months +- Every year + +To run an on-demand scan immediately, either: + +- [Create and run an on-demand scan immediately](#create-and-run-an-on-demand-scan-immediately). +- [Run a previously saved on-demand scan](#run-a-saved-on-demand-scan). + +To run an on-demand scan either at a scheduled date or frequency, read +[Schedule an on-demand scan](#schedule-an-on-demand-scan). + +#### Create and run an on-demand scan immediately + +1. From your project's home page, go to **Security & Compliance > On-demand Scans** in the left + sidebar. +1. Select **New scan**. +1. Complete the **Scan name** and **Description** fields. +1. In GitLab 13.10 and later, select the desired branch from the **Branch** dropdown list. +1. In **Scanner profile**, select a scanner profile from the dropdown list. +1. In **Site profile**, select a site profile from the dropdown list. +1. To run the on-demand scan immediately, select **Save and run scan**. Otherwise, select + **Save scan** to [run](#run-a-saved-on-demand-scan) it later. + +The on-demand DAST scan runs and the project's dashboard shows the results. + +#### Run a saved on-demand scan + +To run a saved on-demand scan: + +1. On the top bar, select **Main menu > Projects** and find your project. +1. On the left sidebar, select **Security & Compliance > On-demand Scans**. +1. Select the **Scan library** tab. +1. In the scan's row, select **Run scan**. + + If the branch saved in the scan no longer exists, you must first + [edit the scan](#edit-an-on-demand-scan), select a new branch, and save the edited scan. + +The on-demand DAST scan runs, and the project's dashboard shows the results. + +#### Schedule an on-demand scan + +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/328749) in GitLab 14.3. [Deployed behind the `dast_on_demand_scans_scheduler` flag](../../../administration/feature_flags.md), disabled by default. +> - [Enabled on GitLab.com](https://gitlab.com/gitlab-org/gitlab/-/issues/328749) in GitLab 14.4. +> - [Enabled on self-managed](https://gitlab.com/gitlab-org/gitlab/-/issues/328749) in GitLab 14.4. +> - [Feature flag `dast_on_demand_scans_scheduler` removed](https://gitlab.com/gitlab-org/gitlab/-/issues/328749) in GitLab 14.5. + +To schedule a scan: + +1. On the top bar, select **Main menu > Projects** and find your project. +1. On the left sidebar, select **Security & Compliance > On-demand Scans**. +1. Select **New scan**. +1. Complete the **Scan name** and **Description** text boxes. +1. In GitLab 13.10 and later, from the **Branch** dropdown list, select the desired branch. +1. In the **Scanner profile** section, from the dropdown list, select a scanner profile. +1. In the **Site profile** section, from the dropdown list, select a site profile. +1. Select **Schedule scan**. +1. In the **Start time** section, select a time zone, date, and time. +1. From the **Repeats** dropdown list, select your desired frequency: + - To run the scan once, select **Never**. + - For a recurring scan, select any other option. +1. To run the on-demand scan immediately, select **Save and run scan**. To [run](#run-a-saved-on-demand-scan) it according to the schedule you set, select + **Save scan**. + +#### List saved on-demand scans + +To list saved on-demand scans: + +1. From your project's home page, go to **Security & Compliance > On-demand Scans**. +1. Select the **Scan library** tab. + +#### View details of an on-demand scan + +To view details of an on-demand scan: + +1. From your project's home page, go to **Security & Compliance > On-demand Scans**. +1. Select the **Scan library** tab. +1. In the saved scan's row select **More actions** (**{ellipsis_v}**), then select **Edit**. + +#### Edit an on-demand scan + +To edit an on-demand scan: + +1. From your project's home page, go to **Security & Compliance > On-demand Scans**. +1. Select the **Scan library** tab. +1. In the saved scan's row select **More actions** (**{ellipsis_v}**), then select **Edit**. +1. Edit the form. +1. Select **Save scan**. + +#### Delete an on-demand scan + +To delete an on-demand scan: + +1. From your project's home page, go to **Security & Compliance > On-demand Scans**. +1. Select the **Scan library** tab. +1. In the saved scan's row select **More actions** (**{ellipsis_v}**), then select **Delete**. +1. Select **Delete** to confirm the deletion. + +## Site profile + +> - Scan method [Enabled on GitLab.com and self-managed](https://gitlab.com/gitlab-org/gitlab/-/issues/345837) in GitLab 15.6. +> - File URL [Enabled on GitLab.com and self-managed](https://gitlab.com/gitlab-org/gitlab/-/issues/345837) in GitLab 15.6. + +A site profile defines the attributes and configuration details of the deployed application, +website, or API to be scanned by DAST. A site profile can be referenced in `.gitlab-ci.yml` and +on-demand scans. + +A site profile contains: + +- **Profile name**: A name you assign to the site to be scanned. While a site profile is referenced + in either `.gitlab-ci.yml` or an on-demand scan, it **cannot** be renamed. +- **Site type**: The type of target to be scanned, either website or API scan. +- **Target URL**: The URL that DAST runs against. +- **Excluded URLs**: A comma-separated list of URLs to exclude from the scan. +- **Request headers**: A comma-separated list of HTTP request headers, including names and values. These headers are added to every request made by DAST. +- **Authentication**: + - **Authenticated URL**: The URL of the page containing the sign-in HTML form on the target website. The username and password are submitted with the login form to create an authenticated scan. + - **Username**: The username used to authenticate to the website. + - **Password**: The password used to authenticate to the website. + - **Username form field**: The name of username field at the sign-in HTML form. + - **Password form field**: The name of password field at the sign-in HTML form. + - **Submit form field**: The `id` or `name` of the element that when selected submits the sign-in HTML form. + +- **Scan method**: A type of method to perform API testing. The supported methods are OpenAPI, Postman Collections, and HTTP Archive (HAR) documents. +- **File URL**: The URL of the OpenAPI, Postman Collection, or HTTP Archive file. + +When an API site type is selected, a [host override](#host-override) is used to ensure the API being scanned is on the same host as the target. This is done to reduce the risk of running an active scan against the wrong API. + +When configured, request headers and password fields are encrypted using [`aes-256-gcm`](https://en.wikipedia.org/wiki/Advanced_Encryption_Standard) before being stored in the database. +This data can only be read and decrypted with a valid secrets file. + +### Site profile validation + +> - Site profile validation [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/233020) in GitLab 13.8. +> - Meta tag validation [introduced](https://gitlab.com/groups/gitlab-org/-/epics/6460) in GitLab 14.2. + +Site profile validation reduces the risk of running an active scan against the wrong website. A site +must be validated before an active scan can run against it. The site validation methods are as +follows: + +- _Text file validation_ requires a text file be uploaded to the target site. The text file is + allocated a name and content that is unique to the project. The validation process checks the + file's content. +- _Header validation_ requires the header `Gitlab-On-Demand-DAST` be added to the target site, + with a value unique to the project. The validation process checks that the header is present, and + checks its value. +- _Meta tag validation_ requires the meta tag named `gitlab-dast-validation` be added to the target site, + with a value unique to the project. Make sure it's added to the `<head>` section of the page. The validation process checks that the meta tag is present, and + checks its value. + +All these methods are equivalent in functionality. Use whichever is feasible. + +In [GitLab 14.2 and later](https://gitlab.com/gitlab-org/gitlab/-/issues/324990), site profile +validation happens in a CI job using the [GitLab Runner](../../../ci/runners/index.md). + +### Create a site profile + +To create a site profile: + +1. From your project's home page, go to **Security & Compliance > Configuration**. +1. Select **Manage** in the **DAST Profiles** row. +1. Select **New > Site Profile**. +1. Complete the fields then select **Save profile**. + +The site profile is created. + +### Edit a site profile + +If a site profile is linked to a security policy, a user cannot edit the profile from this page. See +[Scan execution policies](../policies/scan-execution-policies.md) +for more information. + +When a validated site profile's file, header, or meta tag is edited, the site's +[validation status](#site-profile-validation) is revoked. + +To edit a site profile: + +1. From your project's home page, go to **Security & Compliance > Configuration**. +1. In the **DAST Profiles** row select **Manage**. +1. Select the **Site Profiles** tab. +1. In the profile's row select the **More actions** (**{ellipsis_v}**) menu, then select **Edit**. +1. Edit the fields then select **Save profile**. + +### Delete a site profile + +If a site profile is linked to a security policy, a user cannot delete the profile from this page. +See [Scan execution policies](../policies/scan-execution-policies.md) +for more information. + +To delete a site profile: + +1. From your project's home page, go to **Security & Compliance > Configuration**. +1. In the **DAST Profiles** row select **Manage**. +1. Select the **Site Profiles** tab. +1. In the profile's row, select the **More actions** (**{ellipsis_v}**) menu, then select **Delete**. +1. Select **Delete** to confirm the deletion. + +### Validate a site profile + +Validating a site is required to run an active scan. + +To validate a site profile: + +1. On the top bar, select **Main menu > Projects** and find your project. +1. On the left sidebar, select **Security & Compliance > Configuration**. +1. In the **Dynamic Application Security Testing (DAST)** section, select **Manage profiles**. +1. Select the **Site Profiles** tab. +1. In the profile's row, select **Validate**. +1. Select the validation method. + 1. For **Text file validation**: + 1. Download the validation file listed in **Step 2**. + 1. Upload the validation file to the host, to the location in **Step 3** or any location you + prefer. + 1. If required, edit the file location in **Step 3**. + 1. Select **Validate**. + 1. For **Header validation**: + 1. Select the clipboard icon in **Step 2**. + 1. Edit the header of the site to validate, and paste the clipboard content. + 1. Select the input field in **Step 3** and enter the location of the header. + 1. Select **Validate**. + 1. For **Meta tag validation**: + 1. Select the clipboard icon in **Step 2**. + 1. Edit the content of the site to validate, and paste the clipboard content. + 1. Select the input field in **Step 3** and enter the location of the meta tag. + 1. Select **Validate**. + +The site is validated and an active scan can run against it. A site profile's validation status is +revoked only when it's revoked manually, or its file, header, or meta tag is edited. + +### Retry a failed validation + +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/322609) in GitLab 14.3. +> - [Deployed behind the `dast_failed_site_validations` flag](../../../administration/feature_flags.md), enabled by default. +> - [Feature flag `dast_failed_site_validations` removed](https://gitlab.com/gitlab-org/gitlab/-/issues/323961) in GitLab 14.4. + +Failed site validation attempts are listed on the **Site profiles** tab of the **Manage profiles** +page. + +To retry a site profile's failed validation: + +1. On the top bar, select **Main menu > Projects** and find your project. +1. On the left sidebar, select **Security & Compliance > Configuration**. +1. In the **Dynamic Application Security Testing (DAST)** section, select **Manage profiles**. +1. Select the **Site Profiles** tab. +1. In the profile's row, select **Retry validation**. + +### Revoke a site profile's validation status + +WARNING: +When a site profile's validation status is revoked, all site profiles that share the same URL also +have their validation status revoked. + +To revoke a site profile's validation status: + +1. From your project's home page, go to **Security & Compliance > Configuration**. +1. In the **DAST Profiles** row select **Manage**. +1. Beside the validated profile, select **Revoke validation**. + +The site profile's validation status is revoked. + +### Validated site profile headers + +The following are code samples of how you can provide the required site profile header in your +application. + +#### Ruby on Rails example for on-demand scan + +Here's how you can add a custom header in a Ruby on Rails application: + +```ruby +class DastWebsiteTargetController < ActionController::Base + def dast_website_target + response.headers['Gitlab-On-Demand-DAST'] = '0dd79c9a-7b29-4e26-a815-eaaf53fcab1c' + head :ok + end +end +``` + +#### Django example for on-demand scan + +Here's how you can add a +[custom header in Django](https://docs.djangoproject.com/en/2.2/ref/request-response/#setting-header-fields): + +```python +class DastWebsiteTargetView(View): + def head(self, *args, **kwargs): + response = HttpResponse() + response['Gitlab-On-Demand-DAST'] = '0dd79c9a-7b29-4e26-a815-eaaf53fcab1c' + + return response +``` + +#### Node (with Express) example for on-demand scan + +Here's how you can add a +[custom header in Node (with Express)](https://expressjs.com/en/5x/api.html#res.append): + +```javascript +app.get('/dast-website-target', function(req, res) { + res.append('Gitlab-On-Demand-DAST', '0dd79c9a-7b29-4e26-a815-eaaf53fcab1c') + res.send('Respond to DAST ping') +}) +``` + +## Scanner profile + +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/222767) in GitLab 13.4. +> - [Added](https://gitlab.com/gitlab-org/gitlab/-/issues/225804) in GitLab 13.5: scan mode, AJAX spider, debug messages. + +A scanner profile defines the configuration details of a security scanner. A scanner profile can be +referenced in `.gitlab-ci.yml` and on-demand scans. + +A scanner profile contains: + +- **Profile name:** A name you give the scanner profile. For example, "Spider_15". While a scanner + profile is referenced in either `.gitlab-ci.yml` or an on-demand scan, it **cannot** be renamed. +- **Scan mode:** A passive scan monitors all HTTP messages (requests and responses) sent to the target. An active scan attacks the target to find potential vulnerabilities. +- **Spider timeout:** The maximum number of minutes allowed for the spider to traverse the site. +- **Target timeout:** The maximum number of seconds DAST waits for the site to be available before + starting the scan. +- **AJAX spider:** Run the AJAX spider, in addition to the traditional spider, to crawl the target site. +- **Debug messages:** Include debug messages in the DAST console output. + +### Create a scanner profile + +To create a scanner profile: + +1. From your project's home page, go to **Security & Compliance > Configuration**. +1. In the **DAST Profiles** row, select **Manage**. +1. Select **New > Scanner Profile**. +1. Complete the form. For details of each field, see [Scanner profile](#scanner-profile). +1. Select **Save profile**. + +### Edit a scanner profile + +If a scanner profile is linked to a security policy, a user cannot edit the profile from this page. +See [Scan execution policies](../policies/scan-execution-policies.md) +for more information. + +To edit a scanner profile: + +1. From your project's home page, go to **Security & Compliance > Configuration**. +1. In the **DAST Profiles** row, select **Manage**. +1. Select the **Scanner Profiles** tab. +1. In the scanner's row, select the **More actions** (**{ellipsis_v}**) menu, then select **Edit**. +1. Edit the form. +1. Select **Save profile**. + +### Delete a scanner profile + +If a scanner profile is linked to a security policy, a user cannot delete the profile from this +page. See [Scan execution policies](../policies/scan-execution-policies.md) +for more information. + +To delete a scanner profile: + +1. From your project's home page, go to **Security & Compliance > Configuration**. +1. In the **DAST Profiles** row, select **Manage**. +1. Select the **Scanner Profiles** tab. +1. In the scanner's row, select the **More actions** (**{ellipsis_v}**) menu, then select **Delete**. +1. Select **Delete**. + +## Auditing + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/217872) in GitLab 14.1. + +The creation, updating, and deletion of DAST profiles, DAST scanner profiles, +and DAST site profiles are included in the [audit log](../../../administration/audit_events.md). + +## Reports + +The DAST tool outputs a `gl-dast-report.json` report file containing details of the scan and its results. +This file is included in the job's artifacts. JSON is the default format, but +you can output the report in Markdown, HTML, and XML formats. To specify an alternative +format, use a [CI/CD variable](#available-cicd-variables). You can also use a CI/CD variable +to configure the job to output the `gl-dast-debug-auth-report.html` file which helps when debugging +authentication issues. + +For details of the report's schema, see the [schema for DAST reports](https://gitlab.com/gitlab-org/security-products/security-report-schemas/-/blob/master/dist/dast-report-format.json). Example reports can be found in the +[DAST repository](https://gitlab.com/gitlab-org/security-products/dast/-/tree/main/test/end-to-end/expect). + +WARNING: +The JSON report artifacts are not a public API of DAST and their format is expected to change in the +future. diff --git a/doc/user/application_security/dast_api/index.md b/doc/user/application_security/dast_api/index.md index eae32f789b8..d77be0f0ca9 100644 --- a/doc/user/application_security/dast_api/index.md +++ b/doc/user/application_security/dast_api/index.md @@ -7,20 +7,19 @@ type: reference, howto # DAST API **(ULTIMATE)** -You can add dynamic application security testing (DAST) of web APIs to your -[GitLab CI/CD](../../../ci/index.md) pipelines. This helps you discover bugs and potential security -issues that other QA processes may miss. +> DAST API analyzer [became the default analyzer for on-demand DAST API scans](https://gitlab.com/groups/gitlab-org/-/epics/4254) in GitLab 15.6. -We recommend that you use DAST API testing in addition to [GitLab Secure](../index.md)'s -other security scanners and your own test processes. If you're using [GitLab CI/CD](../../../ci/index.md), -you can run DAST API tests as part your CI/CD workflow. +Perform Dynamic Application Security Testing (DAST) of web APIs to help discover bugs and potential +security issues that other QA processes may miss. Use DAST API tests in addition to +[GitLab Secure](../index.md)'s other security scanners and your own test processes. You can run DAST +API tests either as part your CI/CD workflow, [on-demand](../dast/proxy-based.md#on-demand-scans), or both. WARNING: -Do not run DAST API testing against a production server. Not only can it perform *any* function that +Do not run DAST API testing against a production server. Not only can it perform _any_ function that the API can, it may also trigger bugs in the API. This includes actions like modifying and deleting data. Only run DAST API against a test server. -You can run DAST API scanning against the following web API types: +DAST API can test the following web API types: - REST API - SOAP @@ -29,9 +28,9 @@ You can run DAST API scanning against the following web API types: ## When DAST API scans run -DAST API scanning runs in the `dast` stage by default. To ensure DAST API scanning examines the latest -code, ensure your CI/CD pipeline deploys changes to a test environment in a stage before the `dast` -stage. +When run in your CI/CD pipeline, DAST API scanning runs in the `dast` stage by default. To ensure +DAST API scanning examines the latest code, ensure your CI/CD pipeline deploys changes to a test +environment in a stage before the `dast` stage. If your pipeline is configured to deploy to the same web server on each run, running a pipeline while another is still running could cause a race condition in which one pipeline overwrites the @@ -2057,9 +2056,232 @@ Setting `SECURE_ANALYZERS_PREFIX` changes the Docker image registry location for For more information, see [Offline environments](../offline_deployments/index.md). +## Performance tuning and testing speed + +Security tools that perform dynamic analysis testing, such as DAST API, perform testing by sending requests to an instance of your running application. The requests are engineered to test for specific vulnerabilities that might exist in your application. The speed of a dynamic analysis test depends on the following: + +- How many requests per second can be sent to your application by our tooling +- How fast your application responds to requests +- How many requests must be sent to test the application + - How many operations your API is comprised of + - How many fields are in each operation (think JSON bodies, headers, query string, cookies, etc.) + +If the DAST API testing job still takes longer than expected reach after following the advice in this performance guide, reach out to support for further assistance. + +### Diagnosing performance issues + +The first step to resolving performance issues is to understand what is contributing to the slower-than-expected testing time. Some common issues we see are: + +- DAST API is running on a slow or single-CPU GitLab Runner (GitLab Shared Runners are single-CPU) +- The application deployed to a slow/single-CPU instance and is not able to keep up with the testing load +- The application contains a slow operation that impacts the overall test speed (> 1/2 second) +- The application contains an operation that returns a large amount of data (> 500K+) +- The application contains a large number of operations (> 40) + +#### The application contains a slow operation that impacts the overall test speed (> 1/2 second) + +The DAST API job output contains helpful information about how fast we are testing, how fast each operation being tested responds, and summary information. Let's take a look at some sample output to see how it can be used in tracking down performance issues: + +```shell +API Security: Loaded 10 operations from: assets/har-large-response/large_responses.har +API Security: +API Security: Testing operation [1/10]: 'GET http://target:7777/api/large_response_json'. +API Security: - Parameters: (Headers: 4, Query: 0, Body: 0) +API Security: - Request body size: 0 Bytes (0 bytes) +API Security: +API Security: Finished testing operation 'GET http://target:7777/api/large_response_json'. +API Security: - Excluded Parameters: (Headers: 0, Query: 0, Body: 0) +API Security: - Performed 767 requests +API Security: - Average response body size: 130 MB +API Security: - Average call time: 2 seconds and 82.69 milliseconds (2.082693 seconds) +API Security: - Time to complete: 14 minutes, 8 seconds and 788.36 milliseconds (848.788358 seconds) +``` + +This job console output snippet starts by telling us how many operations were found (10), followed by notifications that testing has started on a specific operation and a summary of the operation has been completed. The summary is the most interesting part of this log output. In the summary, we can see that it took DAST API 767 requests to fully test this operation and its related fields. We can also see that the average response time was 2 seconds and the time to complete was 14 minutes for this one operation. + +An average response time of 2 seconds is a good initial indicator that this specific operation takes a long time to test. Further, we can see that the response body size is quite large. The large body size is the culprit here, transferring that much data on each request is what takes the majority of that 2 seconds. + +For this issue, the team might decide to: + +- Use a multi-CPU runner. Using a multi-CPU runner allows DAST API to parallelize the work being performed. This helps lower the test time, but getting the test down under 10 minutes might still be problematic without moving to a high CPU machine due to how long the operation takes to test. + - Trade off between how many CPUs and cost. +- [Exclude this operation](#excluding-slow-operations) from the DAST API test. While this is the simplest, it has the downside of a gap in security test coverage. +- [Exclude the operation from feature branch DAST API tests, but include it in the default branch test](#excluding-operations-in-feature-branches-but-not-default-branch). +- [Split up the DAST API testing into multiple jobs](#splitting-a-test-into-multiple-jobs). + +The likely solution is to use a combination of these solutions to reach an acceptable test time, assuming your team's requirements are in the 5-7 minute range. + +### Addressing performance issues + +The following sections document various options for addressing performance issues for DAST API: + +- [Using a multi-CPU Runner](#using-a-multi-cpu-runner) +- [Excluding slow operations](#excluding-slow-operations) +- [Splitting a test into multiple jobs](#splitting-a-test-into-multiple-jobs) +- [Excluding operations in feature branches, but not default branch](#excluding-operations-in-feature-branches-but-not-default-branch) + +#### Using a multi-CPU Runner + +One of the easiest performance boosts can be achieved using a multi-CPU runner with DAST API. This table shows statistics collected during benchmarking of a Java Spring Boot REST API. In this benchmark, the target and DAST API share a single runner instance. + +| CPU Count | Request per Second | +|----------------------|--------------------| +| 1 CPU (Shared Runner)| 75 | +| 4 CPU | 255 | +| 8 CPU | 400 | + +As we can see from this table, increasing the CPU count of the runner can have a large impact on testing speed/performance. + +To use a multi-CPU typically requires deploying a self-managed GitLab Runner onto a multi-CPU machine or cloud compute instance. + +When multiple types of GitLab Runners are available for use, the various instances are commonly set up with tags that can be used in the job definition to select a type of runner. + +Here is an example job definition for DAST API that adds a `tags` section with the tag `multi-cpu`. The job automatically extends the job definition included through the DAST API template. + +```yaml +dast_api: + tags: + - multi-cpu +``` + +To verify that DAST API can detect multiple CPUs in the runner, download the `gl-api-security-scanner.log` file from a completed job's artifacts. Search the file for the string `Starting work item processor` and inspect the reported max DOP (degree of parallelism). The max DOP should be greater than or equal to the number of CPUs assigned to the runner. The value is never lower than 2, even on single CPU runners, unless forced through a configuration variable. If the value reported is less than the number of CPUs assigned to the runner, then something is wrong with the runner deployment. If unable to identify the problem, open a ticket with support to assist. + +Example log entry: + +`17:00:01.084 [INF] <Peach.Web.Core.Services.WebRunnerMachine> Starting work item processor with 2 max DOP` + +#### Excluding slow operations + +In the case of one or two slow operations, the team might decide to skip testing the operations. Excluding the operation is done using the `DAST_API_EXCLUDE_PATHS` configuration [variable as explained in this section.](#exclude-paths) + +In this example, we have an operation that returns a large amount of data. The operation is `GET http://target:7777/api/large_response_json`. To exclude it we provide the `DAST_API_EXCLUDE_PATHS` configuration variable with the path portion of our operation URL `/api/large_response_json`. + +To verify the operation is excluded, run the DAST API job and review the job console output. It includes a list of included and excluded operations at the end of the test. + +```yaml +dast_api: + variables: + DAST_API_EXCLUDE_PATHS: /api/large_response_json +``` + +Excluding operations from testing could allow some vulnerabilities to go undetected. +{: .alert .alert-warning} + +#### Splitting a test into multiple jobs + +Splitting a test into multiple jobs is supported by DAST API through the use of [`DAST_API_EXCLUDE_PATHS`](#exclude-paths) and [`DAST_API_EXCLUDE_URLS`](#exclude-urls). When splitting a test up, a good pattern is to disable the `dast_api` job and replace it with two jobs with identifying names. In this example we have two jobs, each job is testing a version of the API, so our names reflect that. However, this technique can be applied to any situation, not just with versions of an API. + +The rules we are using in the `dast_api_v1` and `dast_api_v2` jobs are copied from the [DAST API template](https://gitlab.com/gitlab-org/gitlab/blob/master/lib/gitlab/ci/templates/Security/DAST-API.gitlab-ci.yml). + +```yaml +# Disable the main dast_api job +dast_api: + rules: + - if: $CI_COMMIT_BRANCH + when: never + +dast_api_v1: + extends: dast_api + variables: + DAST_API_EXCLUDE_PATHS: /api/v1/** + rules: + - if: $DAST_API_DISABLED + when: never + - if: $DAST_API_DISABLED_FOR_DEFAULT_BRANCH && + $CI_DEFAULT_BRANCH == $CI_COMMIT_REF_NAME + when: never + - if: $CI_COMMIT_BRANCH && + $CI_GITLAB_FIPS_MODE == "true" + variables: + DAST_API_IMAGE_SUFFIX: "-fips" + - if: $CI_COMMIT_BRANCH + +dast_api_v2: + variables: + DAST_API_EXCLUDE_PATHS: /api/v2/** + rules: + - if: $DAST_API_DISABLED + when: never + - if: $DAST_API_DISABLED_FOR_DEFAULT_BRANCH && + $CI_DEFAULT_BRANCH == $CI_COMMIT_REF_NAME + when: never + - if: $CI_COMMIT_BRANCH && + $CI_GITLAB_FIPS_MODE == "true" + variables: + DAST_API_IMAGE_SUFFIX: "-fips" + - if: $CI_COMMIT_BRANCH +``` + +#### Excluding operations in feature branches, but not default branch + +In the case of one or two slow operations, the team might decide to skip testing the operations, or exclude them from feature branch tests, but include them for default branch tests. Excluding the operation is done using the `DAST_API_EXCLUDE_PATHS` configuration [variable as explained in this section.](#exclude-paths) + +In this example, we have an operation that returns a large amount of data. The operation is `GET http://target:7777/api/large_response_json`. To exclude it we provide the `DAST_API_EXCLUDE_PATHS` configuration variable with the path portion of our operation URL `/api/large_response_json`. Our configuration disables the main `dast_api` job and creates two new jobs `dast_api_main` and `dast_api_branch`. The `dast_api_branch` is set up to exclude the long operation and only run on non-default branches (e.g. feature branches). The `dast_api_main` branch is set up to only execute on the default branch (`main` in this example). The `dast_api_branch` jobs run faster, allowing for quick development cycles, while the `dast_api_main` job which only runs on default branch builds, takes longer to run. + +To verify the operation is excluded, run the DAST API job and review the job console output. It includes a list of included and excluded operations at the end of the test. + +```yaml +# Disable the main job so we can create two jobs with +# different names +dast_api: + rules: + - if: $CI_COMMIT_BRANCH + when: never + +# DAST API for feature branch work, excludes /api/large_response_json +dast_api_branch: + extends: dast_api + variables: + DAST_API_EXCLUDE_PATHS: /api/large_response_json + rules: + - if: $DAST_API_DISABLED + when: never + - if: $DAST_API_DISABLED_FOR_DEFAULT_BRANCH && + $CI_DEFAULT_BRANCH == $CI_COMMIT_REF_NAME + when: never + - if: $CI_COMMIT_BRANCH && + $CI_GITLAB_FIPS_MODE == "true" + variables: + DAST_API_IMAGE_SUFFIX: "-fips" + - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH + when: never + - if: $CI_COMMIT_BRANCH + +# DAST API for default branch (main in our case) +# Includes the long running operations +dast_api_main: + extends: dast_api + rules: + - if: $DAST_API_DISABLED + when: never + - if: $DAST_API_DISABLED_FOR_DEFAULT_BRANCH && + $CI_DEFAULT_BRANCH == $CI_COMMIT_REF_NAME + when: never + - if: $CI_COMMIT_BRANCH && + $CI_GITLAB_FIPS_MODE == "true" + variables: + DAST_API_IMAGE_SUFFIX: "-fips" + - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH +``` + ## Troubleshooting -### `Error waiting for API Security 'http://127.0.0.1:5000' to become available` +### DAST API job times out after N hours + +The top two reasons for the DAST API job timing out are slow operations (> 1 second) and using a single-CPU runner for DAST API (GitLab shared runners are single-CPU). Before you can diagnose the problem further, the job must complete so the output can be analyzed. We recommend to start with a multi-CPU runner first, then exclude portions of your API operations until the job completes and the output can be further reviewed. + +See the following documentation sections for assistance: + +- [Performance tuning and testing speed](#performance-tuning-and-testing-speed) +- [Using a multi-CPU Runner](#using-a-multi-cpu-runner) +- [Excluding operations by path](#exclude-paths) +- [Excluding slow operations](#excluding-slow-operations) + +### DAST API job takes too long to complete + +See [Performance Tuning and Testing Speed](#performance-tuning-and-testing-speed) + +### Error waiting for API Security 'http://127.0.0.1:5000' to become available A bug exists in versions of the DAST API analyzer prior to v1.6.196 that can cause a background process to fail under certain conditions. The solution is to update to a newer version of the DAST API analyzer. diff --git a/doc/user/application_security/dependency_scanning/index.md b/doc/user/application_security/dependency_scanning/index.md index a3c6c46b081..4ed9ceedb4d 100644 --- a/doc/user/application_security/dependency_scanning/index.md +++ b/doc/user/application_security/dependency_scanning/index.md @@ -221,7 +221,7 @@ table.supported-languages ul { <td>N</td> </tr> <tr> - <td rowspan="2">JavaScript</td> + <td rowspan="2">JavaScript and TypeScript</td> <td>All versions</td> <td><a href="https://www.npmjs.com/">npm</a></td> <td> @@ -501,11 +501,11 @@ From GitLab 14.8 the `gemnasium` analyzer scans supported JavaScript projects fo #### Go -When scanning a Go project, gemnasium invokes a builder and attempts to generate a [build list](https://go.dev/ref/mod#glos-build-list) using -[Minimal Version Selection](https://go.dev/ref/mod#glos-minimal-version-selection). If a non-fatal error is encountered, the build process signals -that the execution should proceed and falls back to parsing the available `go.sum` file. +Multiple files are supported. When a `go.mod` file is detected, the analyzer attempts to generate a [build list](https://go.dev/ref/mod#glos-build-list) using +[Minimal Version Selection](https://go.dev/ref/mod#glos-minimal-version-selection). If a non-fatal error is encountered, the analyzer falls back to parsing the +available `go.sum` file. The process is repeated for every detected `go.mod` and `go.sum` file. -#### PHP, Go, C, C++, .NET, C#, Ruby, JavaScript +#### PHP, C, C++, .NET, C#, Ruby, JavaScript The analyzer for these languages supports multiple lockfiles. @@ -525,7 +525,7 @@ The [Security Scanner Integration](../../../development/integrations/secure.md) To enable dependency scanning for GitLab 11.9 and later, you must [include](../../../ci/yaml/index.md#includetemplate) the -[`Dependency-Scanning.gitlab-ci.yml` template](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/ci/templates/Security/Dependency-Scanning.gitlab-ci.yml) +[`Dependency-Scanning.gitlab-ci.yml` template](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/ci/templates/Jobs/Dependency-Scanning.gitlab-ci.yml) that is provided as a part of your GitLab installation. For GitLab versions earlier than 11.9, you can copy and use the job as defined that template. @@ -534,7 +534,7 @@ Add the following to your `.gitlab-ci.yml` file: ```yaml include: - - template: Security/Dependency-Scanning.gitlab-ci.yml + - template: Jobs/Dependency-Scanning.gitlab-ci.yml ``` The included template creates dependency scanning jobs in your CI/CD @@ -624,7 +624,6 @@ The following variables allow configuration of global dependency scanning settin | ----------------------------|------------ | | `ADDITIONAL_CA_CERT_BUNDLE` | Bundle of CA certs to trust. The bundle of certificates provided here is also used by other tools during the scanning process, such as `git`, `yarn`, or `npm`. See [Using a custom SSL CA certificate authority](#using-a-custom-ssl-ca-certificate-authority) for more details. | | `DS_EXCLUDED_ANALYZERS` | Specify the analyzers (by name) to exclude from Dependency Scanning. For more information, see [Dependency Scanning Analyzers](analyzers.md). | -| `DS_DEFAULT_ANALYZERS` | This feature was [deprecated](https://gitlab.com/gitlab-org/gitlab/-/issues/287691) in GitLab 14.0 and [removed](https://gitlab.com/gitlab-org/gitlab/-/issues/333299) in 15.0. Use `DS_EXCLUDED_ANALYZERS` instead. | | `DS_EXCLUDED_PATHS` | Exclude files and directories from the scan based on the paths. A comma-separated list of patterns. Patterns can be globs (see [`doublestar.Match`](https://pkg.go.dev/github.com/bmatcuk/doublestar/v4@v4.0.2#Match) for supported patterns), or file or folder paths (for example, `doc,spec`). Parent directories also match patterns. Default: `"spec, test, tests, tmp"`. | | `DS_IMAGE_SUFFIX` | Suffix added to the image name. ([Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/354796) in GitLab 14.10.) Automatically set to `"-fips"` when FIPS mode is enabled. ([Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/357922) in GitLab 15.0.) | | `SECURE_ANALYZERS_PREFIX` | Override the name of the Docker registry providing the official default images (proxy). Read more about [customizing analyzers](analyzers.md). | @@ -641,6 +640,7 @@ The following variables are used for configuring specific analyzers (used for a | `GEMNASIUM_DB_REMOTE_URL` | `gemnasium` | `https://gitlab.com/gitlab-org/security-products/gemnasium-db.git` | Repository URL for fetching the Gemnasium database. | | `GEMNASIUM_DB_REF_NAME` | `gemnasium` | `master` | Branch name for remote repository database. `GEMNASIUM_DB_REMOTE_URL` is required. | | `DS_REMEDIATE` | `gemnasium` | `"true"`, `"false"` in FIPS mode | Enable automatic remediation of vulnerable dependencies. Not supported in FIPS mode. | +| `DS_REMEDIATE_TIMEOUT` | `gemnasium` | `5m` | Timeout for auto-remediation. | | `GEMNASIUM_LIBRARY_SCAN_ENABLED` | `gemnasium` | `"true"` | Enable detecting vulnerabilities in vendored JavaScript libraries. For now, `gemnasium` leverages [`Retire.js`](https://github.com/RetireJS/retire.js) to do this job. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/350512) in GitLab 14.8. | | `DS_JAVA_VERSION` | `gemnasium-maven` | `17` | Version of Java. Available versions: `8`, `11`, `13`, `14`, `15`, `16`, `17`. Available versions in FIPS-enabled image: `8`, `11`, `17`. | | `MAVEN_CLI_OPTS` | `gemnasium-maven` | `"-DskipTests --batch-mode"` | List of command line arguments that are passed to `maven` by the analyzer. See an example for [using private repositories](../index.md#using-private-maven-repositories). | @@ -1311,6 +1311,11 @@ gemnasium-python-dependency_scanning: - apt-get update && apt-get install -y libpq-dev ``` +### `NoSuchOptionException` when using `poetry config http-basic` with `CI_JOB_TOKEN` + +This error can occur when the automatically generated `CI_JOB_TOKEN` starts with a hyphen (`-`). +To avoid this error, follow [Poetry's configuration advice](https://python-poetry.org/docs/repositories/#configuring-credentials). + ### Error: Project has `<number>` unresolved dependencies The error message `Project has <number> unresolved dependencies` indicates a dependency resolution problem caused by your `gradle.build` or `gradle.build.kts` file. In the current release, `gemnasium-maven` cannot continue processing when an unresolved dependency is encountered. However, There is an [open issue](https://gitlab.com/gitlab-org/gitlab/-/issues/337083) to allow `gemnasium-maven` to recover from unresolved dependency errors and produce a dependency graph. Until this issue has been resolved, you'll need to consult the [Gradle dependency resolution docs](https://docs.gradle.org/current/userguide/dependency_resolution.html) for details on how to fix your `gradle.build` file. diff --git a/doc/user/application_security/get-started-security.md b/doc/user/application_security/get-started-security.md index b6213a98f91..5774bf940b0 100644 --- a/doc/user/application_security/get-started-security.md +++ b/doc/user/application_security/get-started-security.md @@ -36,7 +36,7 @@ The following steps will help you get the most from GitLab application security remediating existing vulnerabilities and preventing the introduction of new ones. 1. Enable other scan types such as [SAST](sast/index.md), [DAST](dast/index.md), [Fuzz testing](coverage_fuzzing/index.md), or [Container Scanning](container_scanning/index.md). -1. Use [Compliance Pipelines](../group/manage.md#configure-a-compliance-pipeline) +1. Use [Compliance Pipelines](../group/compliance_frameworks.md#configure-a-compliance-pipeline) or [Scan Execution Policies](policies/scan-execution-policies.md) to enforce required scan types and ensure separation of duties between security and engineering. 1. Consider enabling [Review Apps](../../development/testing_guide/review_apps.md) to allow for DAST diff --git a/doc/user/application_security/iac_scanning/index.md b/doc/user/application_security/iac_scanning/index.md index 150c2b732d8..1c14c529523 100644 --- a/doc/user/application_security/iac_scanning/index.md +++ b/doc/user/application_security/iac_scanning/index.md @@ -4,25 +4,27 @@ group: Static Analysis info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- -# Infrastructure as Code (IaC) Scanning +# Infrastructure as Code (IaC) Scanning **(FREE)** > [Introduced](https://gitlab.com/groups/gitlab-org/-/epics/6655) in GitLab 14.5. Infrastructure as Code (IaC) Scanning scans your IaC configuration files for known vulnerabilities. -Currently, IaC scanning supports configuration files for Terraform, Ansible, AWS CloudFormation, and Kubernetes. +IaC Scanning supports configuration files for Terraform, Ansible, AWS CloudFormation, and Kubernetes. ## Requirements IaC Scanning runs in the `test` stage, which is available by default. If you redefine the stages in the `.gitlab-ci.yml` file, the `test` stage is required. -To run IaC scanning jobs, by default, you need GitLab Runner with the +We recommend a minimum of 4GB RAM to ensure consistent performance. + +To run IaC Scanning jobs, by default, you need GitLab Runner with the [`docker`](https://docs.gitlab.com/runner/executors/docker.html) or [`kubernetes`](https://docs.gitlab.com/runner/install/kubernetes.html) executor. If you're using the shared runners on GitLab.com, this is enabled by default. WARNING: -Our IaC scanning jobs require a Linux/amd64 container type. Windows containers are not yet supported. +Our IaC Scanning jobs require a Linux/amd64 container type. Windows containers are not supported. WARNING: If you use your own runners, make sure the Docker version installed @@ -30,20 +32,20 @@ is **not** `19.03.0`. See [troubleshooting information](../sast/index.md#error-r ## Supported languages and frameworks -GitLab IaC scanning supports a variety of IaC configuration files. Our IaC security scanners also feature automatic language detection which works even for mixed-language projects. If any supported configuration files are detected in project source code we automatically run the appropriate IaC analyzers. +GitLab IaC Scanning supports a variety of IaC configuration files. Our IaC security scanners also feature automatic language detection which works even for mixed-language projects. If any supported configuration files are detected in project source code we automatically run the appropriate IaC analyzers. -| Configuration File Type | Scan tool | Introduced in GitLab Version | -|------------------------------------------|----------------------------------|-------------------------------| -| Ansible | [KICS](https://kics.io/) | 14.5 | -| AWS CloudFormation | [KICS](https://kics.io/) | 14.5 | -| Azure Resource Manager <sup>1</sup> | [KICS](https://kics.io/) | 14.5 | -| Dockerfile | [KICS](https://kics.io/) | 14.5 | -| Google Deployment Manager | [KICS](https://kics.io/) | 14.5 | -| Kubernetes | [KICS](https://kics.io/) | 14.5 | -| OpenAPI | [KICS](https://kics.io/) | 14.5 | -| Terraform <sup>2</sup> | [KICS](https://kics.io/) | 14.5 | +| Configuration file type | Scan tool | Introduced in GitLab version | +| ----------------------------------- | ------------------------ | ---------------------------- | +| Ansible | [KICS](https://kics.io/) | 14.5 | +| AWS CloudFormation | [KICS](https://kics.io/) | 14.5 | +| Azure Resource Manager <sup>1</sup> | [KICS](https://kics.io/) | 14.5 | +| Dockerfile | [KICS](https://kics.io/) | 14.5 | +| Google Deployment Manager | [KICS](https://kics.io/) | 14.5 | +| Kubernetes | [KICS](https://kics.io/) | 14.5 | +| OpenAPI | [KICS](https://kics.io/) | 14.5 | +| Terraform <sup>2</sup> | [KICS](https://kics.io/) | 14.5 | -1. IaC scanning can analyze Azure Resource Manager templates in JSON format. If you write templates in the [Bicep](https://learn.microsoft.com/en-us/azure/azure-resource-manager/bicep/overview) language, you must use [the bicep CLI](https://learn.microsoft.com/en-us/azure/azure-resource-manager/bicep/bicep-cli) to convert your Bicep files into JSON before GitLab IaC scanning can analyze them. +1. IaC Scanning can analyze Azure Resource Manager templates in JSON format. If you write templates in the [Bicep](https://learn.microsoft.com/en-us/azure/azure-resource-manager/bicep/overview) language, you must use [the bicep CLI](https://learn.microsoft.com/en-us/azure/azure-resource-manager/bicep/bicep-cli) to convert your Bicep files into JSON before GitLab IaC Scanning can analyze them. 1. Terraform modules in a custom registry are not scanned for vulnerabilities. You can follow [this issue](https://gitlab.com/gitlab-org/gitlab/-/issues/357004) for the proposed feature. ### Supported distributions @@ -77,7 +79,7 @@ Different features are available in different [GitLab tiers](https://about.gitla as shown in the following table: | Capability | In Free & Premium | In Ultimate | -|:----------------------------------------------------------------|:--------------------|:-------------------| +| :-------------------------------------------------------------- | :------------------ | :----------------- | | [Configure IaC scanner](#configuration) | **{check-circle}** | **{check-circle}** | | Download [JSON Report](#reports-json-format) | **{check-circle}** | **{check-circle}** | | See new findings in merge request widget | **{dotted-circle}** | **{check-circle}** | @@ -98,14 +100,14 @@ To configure IaC Scanning for a project you can: ### Configure IaC Scanning manually To enable IaC Scanning you must [include](../../../ci/yaml/index.md#includetemplate) the -[`SAST-IaC.gitlab-ci.yml template`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/ci/templates/Jobs/SAST-IaC.gitlab-ci.yml) provided as part of your GitLab installation. Here is an example of how to include it: +[`SAST-IaC.gitlab-ci.yml` template](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/ci/templates/Jobs/SAST-IaC.gitlab-ci.yml) provided as part of your GitLab installation. Here is an example of how to include it: ```yaml include: - template: Jobs/SAST-IaC.gitlab-ci.yml ``` -The included template creates IaC scanning jobs in your CI/CD pipeline and scans +The included template creates IaC Scanning jobs in your CI/CD pipeline and scans your project's configuration files for possible vulnerabilities. The results are saved as a @@ -121,7 +123,123 @@ To enable IaC Scanning in a project, you can create a merge request: 1. In the **Infrastructure as Code (IaC) Scanning** row, select **Configure with a merge request**. 1. Review and merge the merge request to enable IaC Scanning. -Pipelines now include an IaC job. +Pipelines now include an IaC Scanning job. + +## Customize rulesets **(ULTIMATE)** + +> [Added](https://gitlab.com/gitlab-org/gitlab/-/issues/235359) support for overriding rules in GitLab 14.8. + +You can customize the default IaC Scanning rules provided with GitLab. + +The following customization options can be used separately, or together: + +- [Disable predefined rules](#disable-predefined-analyzer-rules). +- [Override predefined rules](#override-predefined-analyzer-rules). + +### Disable predefined analyzer rules + +If there are specific IaC Scanning rules that you don't want active, you can disable them. + +To disable analyzer rules: + +1. Create a `.gitlab` directory at the root of your project, if one doesn't already exist. +1. Create a custom ruleset file named `sast-ruleset.toml` in the `.gitlab` directory, if + one doesn't already exist. +1. Set the `disabled` flag to `true` in the context of a `ruleset` section. +1. In one or more `ruleset` subsections, list the rules to disable. Every + `ruleset.identifier` section has: + - A `type` field for the rule. For IaC Scanning, the identifier type is `kics_id`. + - A `value` field for the rule identifier. KICS rule identifiers are alphanumeric strings. To find the rule identifier, you can: + - Find it in the [JSON report artifact](#reports-json-format). + - Search for the rule name in the [list of KICS queries](https://docs.kics.io/latest/queries/all-queries/) and copy the alphanumeric identifier that's shown. The rule name is shown on the [Vulnerability Page](../vulnerabilities/index.md) when a rule violation is detected. + +In the following example `sast-ruleset.toml` file, the disabled rules are assigned to +the `kics` analyzer by matching the `type` and `value` of identifiers: + +```toml +[kics] + [[kics.ruleset]] + disable = true + [kics.ruleset.identifier] + type = "kics_id" + value = "8212e2d7-e683-49bc-bf78-d6799075c5a7" + + [[kics.ruleset]] + disable = true + [kics.ruleset.identifier] + type = "kics_id" + value = "b03a748a-542d-44f4-bb86-9199ab4fd2d5" +``` + +### Override predefined analyzer rules + +If there are specific IaC Scanning rules you want to customize, you can override them. For +example, you might lower the severity of a rule or link to your own documentation about how to fix a finding. + +To override rules: + +1. Create a `.gitlab` directory at the root of your project, if one doesn't already exist. +1. Create a custom ruleset file named `sast-ruleset.toml` in the `.gitlab` directory, if + one doesn't already exist. +1. In one or more `ruleset.identifier` subsections, list the rules to override. Every + `ruleset.identifier` section has: + - A `type` field for the rule. For IaC Scanning, the identifier type is `kics_id`. + - A `value` field for the rule identifier. KICS rule identifiers are alphanumeric strings. To find the rule identifier, you can: + - Find it in the [JSON report artifact](#reports-json-format). + - Search for the rule name in the [list of KICS queries](https://docs.kics.io/latest/queries/all-queries/) and copy the alphanumeric identifier that's shown. The rule name is shown on the [Vulnerability Page](../vulnerabilities/index.md) when a rule violation is detected. +1. In the `ruleset.override` context of a `ruleset` section, + provide the keys to override. Any combination of keys can be + overridden. Valid keys are: + - description + - message + - name + - severity (valid options are: Critical, High, Medium, Low, Unknown, Info) + +In the following example `sast-ruleset.toml` file, rules are matched by the `type` and +`value` of identifiers and then overridden: + +```toml +[kics] + [[kics.ruleset]] + [kics.ruleset.identifier] + type = "kics_id" + value = "8212e2d7-e683-49bc-bf78-d6799075c5a7" + [kics.ruleset.override] + description = "OVERRIDDEN description" + message = "OVERRIDDEN message" + name = "OVERRIDDEN name" + severity = "Info" +``` + +## Pinning to specific analyzer version + +The GitLab-managed CI/CD template specifies a major version and automatically pulls the latest analyzer release within that major version. + +In some cases, you may need to use a specific version. +For example, you might need to avoid a regression in a later release. + +To override the automatic update behavior, set the `SAST_ANALYZER_IMAGE_TAG` CI/CD variable +in your CI/CD configuration file after you include the [`SAST-IaC.gitlab-ci.yml` template](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/ci/templates/Jobs/SAST-IaC.gitlab-ci.yml). + +Only set this variable within a specific job. +If you set it [at the top level](../../../ci/variables/index.md#create-a-custom-cicd-variable-in-the-gitlab-ciyml-file), the version you set will be used for other SAST analyzers. + +You can set the tag to: + +- A major version, like `3`. Your pipelines will use any minor or patch updates that are released within this major version. +- A minor version, like `3.7`. Your pipelines will use any patch updates that are released within this minor version. +- A patch version, like `3.7.0`. Your pipelines won't receive any updates. + +This example uses a specific minor version of the `KICS` analyzer: + +```yaml +include: + - template: Security/SAST-IaC.gitlab-ci.yml + +kics-iac-sast: + variables: + SAST_ANALYZER_IMAGE_TAG: "3.1" +``` ## Reports JSON format diff --git a/doc/user/application_security/img/secure_tools_and_cicd_stages.png b/doc/user/application_security/img/secure_tools_and_cicd_stages.png Binary files differindex 3dbfd835baf..3284b376adc 100644 --- a/doc/user/application_security/img/secure_tools_and_cicd_stages.png +++ b/doc/user/application_security/img/secure_tools_and_cicd_stages.png diff --git a/doc/user/application_security/index.md b/doc/user/application_security/index.md index 809fa5f3764..5ddfa99fc81 100644 --- a/doc/user/application_security/index.md +++ b/doc/user/application_security/index.md @@ -227,7 +227,7 @@ The artifact generated by the secure analyzer contains all findings it discovers > - [Introduced](https://gitlab.com/groups/gitlab-org/-/epics/4393) in GitLab Free 13.5. > - Made [available in all tiers](https://gitlab.com/gitlab-org/gitlab/-/issues/273205) in 13.6. -> - Report download dropdown [added](https://gitlab.com/gitlab-org/gitlab/-/issues/273418) in 13.7. +> - Report download dropdown list [added](https://gitlab.com/gitlab-org/gitlab/-/issues/273418) in 13.7. > - [Feature flag removed](https://gitlab.com/gitlab-org/gitlab/-/issues/249550) in GitLab 13.9. ### All tiers @@ -403,7 +403,7 @@ Learn more on overriding security jobs: - [Overriding Dependency Scanning jobs](dependency_scanning/index.md#overriding-dependency-scanning-jobs). - [Overriding Container Scanning jobs](container_scanning/index.md#overriding-the-container-scanning-template). - [Overriding Secret Detection jobs](secret_detection/index.md#configure-scan-settings). -- [Overriding DAST jobs](dast/index.md#customize-dast-settings). +- [Overriding DAST jobs](dast/proxy-based.md#customize-dast-settings). - [Overriding License Compliance jobs](../compliance/license_compliance/index.md#overriding-the-template). All the security scanning tools define their stage, so this error can occur with all of them. @@ -487,7 +487,7 @@ Security and compliance teams must ensure that security scans: GitLab provides two methods of accomplishing this, each with advantages and disadvantages. -- [Compliance framework pipelines](../group/manage.md#configure-a-compliance-pipeline) +- [Compliance framework pipelines](../group/compliance_frameworks.md#configure-a-compliance-pipeline) are recommended when: - Scan execution enforcement is required for any scanner that uses a GitLab template, such as SAST IaC, DAST, Dependency Scanning, @@ -499,8 +499,8 @@ GitLab provides two methods of accomplishing this, each with advantages and disa are recommended when: - Scan execution enforcement is required for DAST which uses a DAST site or scan profile. - - Scan execution enforcement is required for SAST, Secret Detection, or Container Scanning with project-specific variable - customizations. To accomplish this, users must create a separate security policy per project. + - Scan execution enforcement is required for SAST, Secret Detection, Dependency Scanning, or Container Scanning with project-specific +variable customizations. To accomplish this, users must create a separate security policy per project. - Scans are required to run on a regular, scheduled cadence. - Either solution can be used equally well when: @@ -514,7 +514,7 @@ Additional details about the differences between the two solutions are outlined | | Compliance Framework Pipelines | Scan Execution Policies | | ------ | ------ | ------ | -| **Flexibility** | Supports anything that can be done in a CI file. | Limited to only the items for which GitLab has explicitly added support. DAST, SAST, Secret Detection, and Container Scanning scans are supported. | +| **Flexibility** | Supports anything that can be done in a CI file. | Limited to only the items for which GitLab has explicitly added support. DAST, SAST, Secret Detection, Dependency Scanning, and Container Scanning scans are supported. | | **Usability** | Requires knowledge of CI YAML. | Follows a `rules` and `actions`-based YAML structure. | | **Inclusion in CI pipeline** | The compliance pipeline is executed instead of the project's `.gitlab-ci.yml` file. To include the project's `.gitlab-ci.yml` file, use an `include` statement. Defined variables aren't allowed to be overwritten by the included project's YAML file. | Forced inclusion of a new job into the CI pipeline. DAST jobs that must be customized on a per-project basis can have project-level Site Profiles and Scan Profiles defined. To ensure separation of duties, these profiles are immutable when referenced in a scan execution policy. All jobs can be customized as part of the security policy itself with the same variables that are normally available to the CI job. | | **Schedulable** | Can be scheduled through a scheduled pipeline on the group. | Can be scheduled natively through the policy configuration itself. | diff --git a/doc/user/application_security/policies/index.md b/doc/user/application_security/policies/index.md index b1315329071..f6d22ab28cd 100644 --- a/doc/user/application_security/policies/index.md +++ b/doc/user/application_security/policies/index.md @@ -57,7 +57,7 @@ project and a project that you would like to designate as the security policy pr 1. On the top bar, select **Main menu > Projects** and find your project. 1. On the left sidebar, select **Security & Compliance > Policies**. 1. Select **Edit Policy Project**, and search for and select the - project you would like to link from the dropdown menu. + project you would like to link from the dropdown list. 1. Select **Save**. To unlink a security policy project, follow the same steps but instead select the trash can icon in diff --git a/doc/user/application_security/policies/scan-execution-policies.md b/doc/user/application_security/policies/scan-execution-policies.md index 41d25dfa8c8..f950d5116b1 100644 --- a/doc/user/application_security/policies/scan-execution-policies.md +++ b/doc/user/application_security/policies/scan-execution-policies.md @@ -15,7 +15,7 @@ with a long, random job name. In the unlikely event of a job name collision, the any pre-existing job in the pipeline. If a policy is created at the group-level, it will apply to every child project or subgroup. A group-level policy cannot be edited from a child project or subgroup. -This feature has some overlap with [compliance framework pipelines](../../group/manage.md#configure-a-compliance-pipeline), +This feature has some overlap with [compliance framework pipelines](../../group/compliance_frameworks.md#configure-a-compliance-pipeline), as we have not [unified the user experience for these two features](https://gitlab.com/groups/gitlab-org/-/epics/7312). For details on the similarities and differences between these features, see [Enforce scan execution](../index.md#enforce-scan-execution). @@ -129,14 +129,14 @@ rule in the defined policy are met. | Field | Type | Possible values | Description | |-------|------|-----------------|-------------| -| `scan` | `string` | `dast`, `secret_detection`, `sast`, `container_scanning` | The action's type. | -| `site_profile` | `string` | Name of the selected [DAST site profile](../dast/index.md#site-profile). | The DAST site profile to execute the DAST scan. This field should only be set if `scan` type is `dast`. | -| `scanner_profile` | `string` or `null` | Name of the selected [DAST scanner profile](../dast/index.md#scanner-profile). | The DAST scanner profile to execute the DAST scan. This field should only be set if `scan` type is `dast`.| +| `scan` | `string` | `dast`, `secret_detection`, `sast`, `container_scanning`, `dependency_scanning` | The action's type. | +| `site_profile` | `string` | Name of the selected [DAST site profile](../dast/proxy-based.md#site-profile). | The DAST site profile to execute the DAST scan. This field should only be set if `scan` type is `dast`. | +| `scanner_profile` | `string` or `null` | Name of the selected [DAST scanner profile](../dast/proxy-based.md#scanner-profile). | The DAST scanner profile to execute the DAST scan. This field should only be set if `scan` type is `dast`.| | `variables` | `object` | | A set of CI variables, supplied as an array of `key: value` pairs, to apply and enforce for the selected scan. The `key` is the variable name, with its `value` provided as a string. This parameter supports any variable that the GitLab CI job supports for the specified scan. | Note the following: -- You must create the [site profile](../dast/index.md#site-profile) and [scanner profile](../dast/index.md#scanner-profile) +- You must create the [site profile](../dast/proxy-based.md#site-profile) and [scanner profile](../dast/proxy-based.md#scanner-profile) with selected names for each project that is assigned to the selected Security Policy Project. Otherwise, the policy is not applied and a job with an error message is created instead. - Once you associate the site profile and scanner profile by name in the policy, it is not possible @@ -152,7 +152,7 @@ Note the following: mode when executed as part of a scheduled scan. - A container scanning scan that is configured for the `pipeline` rule type ignores the agent defined in the `agents` object. The `agents` object is only considered for `schedule` rule types. An agent with a name provided in the `agents` object must be created and configured for the project. -- The SAST scan uses the default template and runs in a [child pipeline](../../../ci/pipelines/downstream_pipelines.md#parent-child-pipelines). +- The Dependency Scanning and SAST scans use the default templates and run in a [child pipeline](../../../ci/pipelines/downstream_pipelines.md#parent-child-pipelines). ## Example security policies project diff --git a/doc/user/application_security/policies/scan-result-policies.md b/doc/user/application_security/policies/scan-result-policies.md index 45b715a52b8..7482df18cc3 100644 --- a/doc/user/application_security/policies/scan-result-policies.md +++ b/doc/user/application_security/policies/scan-result-policies.md @@ -6,6 +6,8 @@ info: To determine the technical writer assigned to the Stage/Group associated w # Scan result policies **(ULTIMATE)** +> Group-level scan result policies [introduced](https://gitlab.com/groups/gitlab-org/-/epics/7622) in GitLab 15.6. + You can use scan result policies to take action based on scan results. For example, one type of scan result policy is a security approval policy that allows approval to be required based on the findings of one or more security scan jobs. Scan result policies are evaluated after a CI scanning @@ -20,7 +22,8 @@ job is fully executed. The following video gives you an overview of GitLab scan ## Scan result policy editor -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/77814) in GitLab 14.8. +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/77814) in GitLab 14.8. +> - [Enabled by default](https://gitlab.com/gitlab-org/gitlab/-/issues/369473) in GitLab 15.6. NOTE: Only project Owners have the [permissions](../../permissions.md#project-members-permissions) @@ -38,6 +41,9 @@ before the policy changes take effect. The [policy editor](index.md#policy-editor) supports YAML mode and rule mode. +NOTE: +Propagating scan result policies created for groups with a large number of projects will take a while to complete. + ## Scan result policies schema The YAML file with scan result policies consists of an array of objects matching the scan result diff --git a/doc/user/application_security/sast/analyzers.md b/doc/user/application_security/sast/analyzers.md index dfb096f14cc..e83825636bf 100644 --- a/doc/user/application_security/sast/analyzers.md +++ b/doc/user/application_security/sast/analyzers.md @@ -12,7 +12,7 @@ Static Application Security Testing (SAST) uses analyzers to detect vulnerabilities in source code. Each analyzer is a wrapper around a [scanner](../terminology/index.md#scanner), a third-party code analysis tool. The analyzers are published as Docker images that SAST uses to launch dedicated containers for each -analysis. +analysis. We recommend a minimum of 4GB RAM to ensure consistent performance of the analyzers. SAST default images are maintained by GitLab, but you can also integrate your own custom image. @@ -134,6 +134,14 @@ In GitLab 15.4, we [removed the deprecated analyzers](https://gitlab.com/gitlab- To preview the upcoming changes to the CI/CD configuration in GitLab 15.3 or earlier: 1. Open an MR to switch from the Stable CI/CD template, `SAST.gitlab-ci.yaml`, to [the Latest template](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/ci/templates/Jobs/SAST.latest.gitlab-ci.yml), `SAST.latest.gitlab-ci.yaml`. + - On GitLab.com, use the latest template directly: + + ```yaml + include: + template: 'Jobs/SAST.latest.gitlab-ci.yaml' + ``` + + - On a Self-Managed instance, download the template from GitLab.com: ```yaml include: diff --git a/doc/user/application_security/sast/index.md b/doc/user/application_security/sast/index.md index a7624db4604..b1bc9794ced 100644 --- a/doc/user/application_security/sast/index.md +++ b/doc/user/application_security/sast/index.md @@ -70,44 +70,45 @@ is **not** `19.03.0`. See [troubleshooting information](#error-response-from-dae ## Supported languages and frameworks -GitLab SAST supports a variety of languages, package managers, and frameworks. Our SAST security scanners also feature automatic language detection which works even for mixed-language projects. If any supported language is detected in project source code we automatically run the appropriate SAST analyzers. - -You can also [view our language roadmap](https://about.gitlab.com/direction/secure/static-analysis/sast/#language-support) and [request other language support by opening an issue](https://gitlab.com/groups/gitlab-org/-/epics/297). - -| Language (package managers) / framework | Scan tool | Introduced in GitLab Version | -|------------------------------------------------|-----------------------------------------------------------------------------------------------------------|-----------------------------------------------------------------------------------------| -| .NET Core | [Security Code Scan](https://security-code-scan.github.io) | 11.0 | -| .NET Framework<sup>1</sup> | [Security Code Scan](https://security-code-scan.github.io) | 13.0 | -| .NET (all versions, C# only) | [Semgrep](https://semgrep.dev) | 15.4 | -| Apex (Salesforce) | [PMD](https://pmd.github.io/pmd/index.html) | 12.1 | -| C | [Semgrep](https://semgrep.dev) | 14.2 | -| C/C++ | [Flawfinder](https://github.com/david-a-wheeler/flawfinder) | 10.7 | -| Elixir (Phoenix) | [Sobelow](https://github.com/nccgroup/sobelow) | 11.1 | -| Go<sup>3</sup> | [Gosec](https://github.com/securego/gosec) | 10.7 | -| Go | [Semgrep](https://semgrep.dev) | 14.4 | -| Groovy<sup>2</sup> | [SpotBugs](https://spotbugs.github.io/) with the [find-sec-bugs](https://find-sec-bugs.github.io/) plugin | 11.3 (Gradle) & 11.9 (Maven, SBT) | -| Helm Charts | [Kubesec](https://github.com/controlplaneio/kubesec) | 13.1 | -| Java (any build system) | [Semgrep](https://semgrep.dev) | 14.10 | -| Java<sup>2, 3</sup> | [SpotBugs](https://spotbugs.github.io/) with the [find-sec-bugs](https://find-sec-bugs.github.io/) plugin | 10.6 (Maven), 10.8 (Gradle) & 11.9 (SBT) | -| Java (Android) | [MobSF (beta)](https://github.com/MobSF/Mobile-Security-Framework-MobSF) | 13.5 | -| JavaScript<sup>3</sup> | [ESLint security plugin](https://github.com/nodesecurity/eslint-plugin-security) | 11.8 | -| JavaScript | [Semgrep](https://semgrep.dev) | 13.10 | -| Kotlin (Android) | [MobSF (beta)](https://github.com/MobSF/Mobile-Security-Framework-MobSF) | 13.5 | -| Kotlin (General)<sup>2</sup> | [SpotBugs](https://spotbugs.github.io/) with the [find-sec-bugs](https://find-sec-bugs.github.io/) plugin | 13.11 | -| Kubernetes manifests | [Kubesec](https://github.com/controlplaneio/kubesec) | 12.6 | -| Node.js | [NodeJsScan](https://github.com/ajinabraham/NodeJsScan) | 11.1 | -| Objective-C (iOS) | [MobSF (beta)](https://github.com/MobSF/Mobile-Security-Framework-MobSF) | 13.5 | -| PHP | [phpcs-security-audit](https://github.com/FloeDesignTechnologies/phpcs-security-audit) | 10.8 | -| Python<sup>3</sup> | [bandit](https://github.com/PyCQA/bandit) | 10.3 | -| Python | [Semgrep](https://semgrep.dev) | 13.9 | -| React<sup>3</sup> | [ESLint react plugin](https://github.com/yannickcr/eslint-plugin-react) | 12.5 | -| React | [Semgrep](https://semgrep.dev) | 13.10 | -| Ruby | [brakeman](https://brakemanscanner.org) | 13.9 | -| Ruby on Rails | [brakeman](https://brakemanscanner.org) | 10.3 | -| Scala<sup>2</sup> | [SpotBugs](https://spotbugs.github.io/) with the [find-sec-bugs](https://find-sec-bugs.github.io/) plugin | 11.0 (SBT) & 11.9 (Gradle, Maven) | -| Swift (iOS) | [MobSF (beta)](https://github.com/MobSF/Mobile-Security-Framework-MobSF) | 13.5 | -| TypeScript<sup>3</sup> | [ESLint security plugin](https://github.com/nodesecurity/eslint-plugin-security) | 11.9, [merged](https://gitlab.com/gitlab-org/gitlab/-/issues/36059) with ESLint in 13.2 | -| TypeScript | [Semgrep](https://semgrep.dev) | 13.10 | +GitLab SAST supports scanning a variety of programming languages and frameworks. +Once you [enable SAST](#configuration), the right set of analyzers runs automatically even if your project uses more than one language. + +Check the [SAST direction page](https://about.gitlab.com/direction/secure/static-analysis/sast/#language-support) to learn more about our plans for language support in SAST. + +| Language / framework | [Analyzer](analyzers.md) used for scanning | Minimum supported GitLab version | +|------------------------------|--------------------------------------------------------------------------------------------------------------|-----------------------------------------------------------------------------------------| +| .NET Core | [Security Code Scan](https://gitlab.com/gitlab-org/security-products/analyzers/security-code-scan) | 11.0 | +| .NET Framework<sup>1</sup> | [Security Code Scan](https://gitlab.com/gitlab-org/security-products/analyzers/security-code-scan) | 13.0 | +| .NET (all versions, C# only) | [Semgrep](https://gitlab.com/gitlab-org/security-products/analyzers/semgrep) with GitLab-managed rules | 15.4 | +| Apex (Salesforce) | [PMD](https://gitlab.com/gitlab-org/security-products/analyzers/pmd-apex) | 12.1 | +| C | [Semgrep](https://gitlab.com/gitlab-org/security-products/analyzers/semgrep) with GitLab-managed rules | 14.2 | +| C/C++ | [Flawfinder](https://gitlab.com/gitlab-org/security-products/analyzers/flawfinder) | 10.7 | +| Elixir (Phoenix) | [Sobelow](https://gitlab.com/gitlab-org/security-products/analyzers/sobelow) | 11.1 | +| Go<sup>3</sup> | [Gosec](https://gitlab.com/gitlab-org/security-products/analyzers/gosec) | 10.7 | +| Go | [Semgrep](https://gitlab.com/gitlab-org/security-products/analyzers/semgrep) with GitLab-managed rules | 14.4 | +| Groovy<sup>2</sup> | [SpotBugs](https://gitlab.com/gitlab-org/security-products/analyzers/spotbugs) with the find-sec-bugs plugin | 11.3 (Gradle) & 11.9 (Maven, SBT) | +| Helm Charts | [Kubesec](https://gitlab.com/gitlab-org/security-products/analyzers/kubesec) | 13.1 | +| Java (any build system) | [Semgrep](https://gitlab.com/gitlab-org/security-products/analyzers/semgrep) with GitLab-managed rules | 14.10 | +| Java<sup>2, 3</sup> | [SpotBugs](https://gitlab.com/gitlab-org/security-products/analyzers/spotbugs) with the find-sec-bugs plugin | 10.6 (Maven), 10.8 (Gradle) & 11.9 (SBT) | +| Java (Android) | [MobSF (beta)](https://gitlab.com/gitlab-org/security-products/analyzers/mobsf) | 13.5 | +| JavaScript<sup>3</sup> | [ESLint security plugin](https://gitlab.com/gitlab-org/security-products/analyzers/eslint) | 11.8 | +| JavaScript | [Semgrep](https://gitlab.com/gitlab-org/security-products/analyzers/semgrep) with GitLab-managed rules | 13.10 | +| Kotlin (Android) | [MobSF (beta)](https://gitlab.com/gitlab-org/security-products/analyzers/mobsf) | 13.5 | +| Kotlin (General)<sup>2</sup> | [SpotBugs](https://gitlab.com/gitlab-org/security-products/analyzers/spotbugs) with the find-sec-bugs plugin | 13.11 | +| Kubernetes manifests | [Kubesec](https://gitlab.com/gitlab-org/security-products/analyzers/kubesec) | 12.6 | +| Node.js | [NodeJsScan](https://gitlab.com/gitlab-org/security-products/analyzers/nodejs-scan) | 11.1 | +| Objective-C (iOS) | [MobSF (beta)](https://gitlab.com/gitlab-org/security-products/analyzers/mobsf) | 13.5 | +| PHP | [phpcs-security-audit](https://gitlab.com/gitlab-org/security-products/analyzers/phpcs-security-audit) | 10.8 | +| Python<sup>3</sup> | [bandit](https://gitlab.com/gitlab-org/security-products/analyzers/bandit) | 10.3 | +| Python | [Semgrep](https://gitlab.com/gitlab-org/security-products/analyzers/semgrep) with GitLab-managed rules | 13.9 | +| React<sup>3</sup> | [ESLint react plugin](https://gitlab.com/gitlab-org/security-products/analyzers/eslint) | 12.5 | +| React | [Semgrep](https://gitlab.com/gitlab-org/security-products/analyzers/semgrep) with GitLab-managed rules | 13.10 | +| Ruby | [brakeman](https://gitlab.com/gitlab-org/security-products/analyzers/brakeman) | 13.9 | +| Ruby on Rails | [brakeman](https://gitlab.com/gitlab-org/security-products/analyzers/brakeman) | 10.3 | +| Scala<sup>2</sup> | [SpotBugs](https://gitlab.com/gitlab-org/security-products/analyzers/spotbugs) with the find-sec-bugs plugin | 11.0 (SBT) & 11.9 (Gradle, Maven) | +| Swift (iOS) | [MobSF (beta)](https://gitlab.com/gitlab-org/security-products/analyzers/mobsf) | 13.5 | +| TypeScript<sup>3</sup> | [ESLint security plugin](https://gitlab.com/gitlab-org/security-products/analyzers/eslint) | 11.9, [merged](https://gitlab.com/gitlab-org/gitlab/-/issues/36059) with ESLint in 13.2 | +| TypeScript | [Semgrep](https://gitlab.com/gitlab-org/security-products/analyzers/semgrep) with GitLab-managed rules | 13.10 | 1. .NET 4 support is limited. The analyzer runs in a Linux container and does not have access to Windows-specific libraries or features. Use the Semgrep-based scanner if you need .NET 4 support. 1. The SpotBugs-based analyzer supports [Gradle](https://gradle.org/), [Maven](https://maven.apache.org/), and [SBT](https://www.scala-sbt.org/). It can also be used with variants like the @@ -116,7 +117,7 @@ You can also [view our language roadmap](https://about.gitlab.com/direction/secu and the [Maven wrapper](https://github.com/takari/maven-wrapper). However, SpotBugs has [limitations](https://gitlab.com/gitlab-org/gitlab/-/issues/350801) when used against [Ant](https://ant.apache.org/)-based projects. We recommend using the Semgrep-based analyzer for Ant-based Java projects. 1. These analyzers reached [End of Support](https://about.gitlab.com/handbook/product/gitlab-the-product/#end-of-support) status [in GitLab 15.4](https://gitlab.com/gitlab-org/gitlab/-/issues/352554). -### Multi-project support +## Multi-project support > [Introduced](https://gitlab.com/groups/gitlab-org/-/epics/4895) in GitLab 13.7. @@ -136,16 +137,52 @@ The following analyzers have multi-project support: - SpotBugs - Sobelow -#### Enable multi-project support for Security Code Scan +### Enable multi-project support for Security Code Scan Multi-project support in the Security Code Scan requires a Solution (`.sln`) file in the root of the repository. For details on the Solution format, see the Microsoft reference [Solution (`.sln`) file](https://learn.microsoft.com/en-us/visualstudio/extensibility/internals/solution-dot-sln-file?view=vs-2019). -### Supported distributions +## False positive detection **(ULTIMATE)** + +> Introduced in GitLab 14.2. + +Vulnerabilities that have been detected and are false positives will be flagged as false positives in the security dashboard. + +False positive detection is available in a subset of the [supported languages](#supported-languages-and-frameworks) and [analyzers](analyzers.md): + +- Ruby, in the Brakeman-based analyzer + + + +## Advanced vulnerability tracking **(ULTIMATE)** + +> [Introduced](https://gitlab.com/groups/gitlab-org/-/epics/5144) in GitLab 14.2. + +Source code is volatile; as developers make changes, source code may move within files or between files. +Security analyzers may have already reported vulnerabilities that are being tracked in the [Vulnerability Report](../vulnerability_report/index.md). +These vulnerabilities are linked to specific problematic code fragments so that they can be found and fixed. +If the code fragments are not tracked reliably as they move, vulnerability management is harder because the same vulnerability could be reported again. + +GitLab SAST uses an advanced vulnerability tracking algorithm to more accurately identify when the same vulnerability has moved within a file due to refactoring or unrelated changes. + +Advanced vulnerability tracking is available in a subset of the [supported languages](#supported-languages-and-frameworks) and [analyzers](analyzers.md): + +- C, in the Semgrep-based analyzer only +- Go, in the Gosec- and Semgrep-based analyzers +- Java, in the Semgrep-based analyzer only +- JavaScript, in the Semgrep-based analyzer only +- Python, in the Semgrep-based analyzer only +- Ruby, in the Brakeman-based analyzer + +Support for more languages and analyzers is tracked in [this epic](https://gitlab.com/groups/gitlab-org/-/epics/5144). + +For more information, see the confidential project `https://gitlab.com/gitlab-org/security-products/post-analyzers/tracking-calculator`. The content of this project is available only to GitLab team members. + +## Supported distributions The default scanner images are build off a base Alpine image for size and maintainability. -#### FIPS-enabled images +### FIPS-enabled images > [Introduced](https://gitlab.com/groups/gitlab-org/-/epics/6479) in GitLab 14.10. @@ -169,17 +206,14 @@ A FIPS-compliant image is only available for the Semgrep-based analyzer. To use SAST in a FIPS-compliant manner, you must [exclude other analyzers from running](analyzers.md#customize-analyzers). -### Making SAST analyzers available to all GitLab tiers - -All open source (OSS) analyzers have been moved to the GitLab Free tier as of GitLab 13.3. - -#### Summary of features per tier +## Summary of features per tier Different features are available in different [GitLab tiers](https://about.gitlab.com/pricing/), as shown in the following table: | Capability | In Free & Premium | In Ultimate | |:---------------------------------------------------------------- -|:--------------------|:-------------------| +| Automatically scan code with [appropriate analyzers](#supported-languages-and-frameworks) | **{check-circle}** | **{check-circle}** | | [Configure SAST scanners](#configuration) | **{check-circle}** | **{check-circle}** | | [Customize SAST settings](#available-cicd-variables) | **{check-circle}** | **{check-circle}** | | Download [JSON Report](#reports-json-format) | **{check-circle}** | **{check-circle}** | @@ -207,14 +241,14 @@ To configure SAST for a project you can: ### Configure SAST manually To enable SAST you must [include](../../../ci/yaml/index.md#includetemplate) -the [`SAST.gitlab-ci.yml` template](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/ci/templates/Security/SAST.gitlab-ci.yml) +the [`SAST.gitlab-ci.yml` template](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/ci/templates/Jobs/SAST.gitlab-ci.yml) provided as a part of your GitLab installation. Add the following to your `.gitlab-ci.yml` file: ```yaml include: - - template: Security/SAST.gitlab-ci.yml + - template: Jobs/SAST.gitlab-ci.yml ``` The included template creates SAST jobs in your CI/CD pipeline and scans @@ -300,14 +334,26 @@ spotbugs-sast: FAIL_NEVER: 1 ``` -#### Pinning to minor image version +### Pinning to minor image version + +The GitLab-managed CI/CD template specifies a major version and automatically pulls the latest analyzer release within that major version. -While our templates use `MAJOR` version pinning to always ensure the latest analyzer -versions are pulled, there are certain cases where it can be beneficial to pin -an analyzer to a specific release. To do so, override the `SAST_ANALYZER_IMAGE_TAG` CI/CD variable -in the job template directly. +In some cases, you may need to use a specific version. +For example, you might need to avoid a regression in a later release. -In the example below, we pin to a minor version of the `semgrep` analyzer and a specific patch version of the `brakeman` analyzer: +To override the automatic update behavior, set the `SAST_ANALYZER_IMAGE_TAG` CI/CD variable +in your CI/CD configuration file after you include the [`SAST.gitlab-ci.yml` template](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/ci/templates/Jobs/SAST.gitlab-ci.yml). + +Only set this variable within a specific job. +If you set it [at the top level](../../../ci/variables/index.md#create-a-custom-cicd-variable-in-the-gitlab-ciyml-file), the version you set will be used for other SAST analyzers. + +You can set the tag to: + +- A major version, like `3`. Your pipelines will use any minor or patch updates that are released within this major version. +- A minor version, like `3.7`. Your pipelines will use any patch updates that are released within this minor version. +- A patch version, like `3.7.0`. Your pipelines won't receive any updates. + +This example uses a specific minor version of the `semgrep` analyzer and a specific patch version of the `brakeman` analyzer: ```yaml include: @@ -315,47 +361,13 @@ include: semgrep-sast: variables: - SAST_ANALYZER_IMAGE_TAG: "2.16" + SAST_ANALYZER_IMAGE_TAG: "3.7" brakeman-sast: variables: - SAST_ANALYZER_IMAGE_TAG: "2.21.1" + SAST_ANALYZER_IMAGE_TAG: "3.1.1" ``` -### False Positive Detection **(ULTIMATE)** - -> Introduced in GitLab 14.2. - -Vulnerabilities that have been detected and are false positives will be flagged as false positives in the security dashboard. - -False positive detection is available in a subset of the [supported languages](#supported-languages-and-frameworks) and [analyzers](analyzers.md): - -- Ruby, in the Brakeman-based analyzer - - - -### Advanced vulnerability tracking **(ULTIMATE)** - -> [Introduced](https://gitlab.com/groups/gitlab-org/-/epics/5144) in GitLab 14.2. - -Source code is volatile; as developers make changes, source code may move within files or between files. -Security analyzers may have already reported vulnerabilities that are being tracked in the [Vulnerability Report](../vulnerability_report/index.md). -These vulnerabilities are linked to specific problematic code fragments so that they can be found and fixed. -If the code fragments are not tracked reliably as they move, vulnerability management is harder because the same vulnerability could be reported again. - -GitLab SAST uses an advanced vulnerability tracking algorithm to more accurately identify when the same vulnerability has moved within a file due to refactoring or unrelated changes. - -Advanced vulnerability tracking is available in a subset of the [supported languages](#supported-languages-and-frameworks) and [analyzers](analyzers.md): - -- C, in the Semgrep-based analyzer only -- Go, in the Gosec- and Semgrep-based analyzers -- Java, in the Semgrep-based analyzer only -- JavaScript, in the Semgrep-based analyzer only -- Python, in the Semgrep-based analyzer only -- Ruby, in the Brakeman-based analyzer - -Support for more languages and analyzers is tracked in [this epic](https://gitlab.com/groups/gitlab-org/-/epics/5144). - ### Using CI/CD variables to pass credentials for private repositories Some analyzers require downloading the project's dependencies to @@ -665,16 +677,16 @@ import the following default SAST analyzer images from `registry.gitlab.com` int [local Docker container registry](../../packages/container_registry/index.md): ```plaintext -registry.gitlab.com/security-products/brakeman:2 -registry.gitlab.com/security-products/flawfinder:2 -registry.gitlab.com/security-products/kubesec:2 -registry.gitlab.com/security-products/nodejs-scan:2 -registry.gitlab.com/security-products/phpcs-security-audit:2 -registry.gitlab.com/security-products/pmd-apex:2 -registry.gitlab.com/security-products/security-code-scan:2 -registry.gitlab.com/security-products/semgrep:2 -registry.gitlab.com/security-products/sobelow:2 -registry.gitlab.com/security-products/spotbugs:2 +registry.gitlab.com/security-products/brakeman:3 +registry.gitlab.com/security-products/flawfinder:3 +registry.gitlab.com/security-products/kubesec:3 +registry.gitlab.com/security-products/nodejs-scan:3 +registry.gitlab.com/security-products/phpcs-security-audit:3 +registry.gitlab.com/security-products/pmd-apex:3 +registry.gitlab.com/security-products/security-code-scan:3 +registry.gitlab.com/security-products/semgrep:3 +registry.gitlab.com/security-products/sobelow:3 +registry.gitlab.com/security-products/spotbugs:3 ``` The process for importing Docker images into a local offline Docker registry depends on diff --git a/doc/user/application_security/secret_detection/index.md b/doc/user/application_security/secret_detection/index.md index df6bb19ac25..8a066cf1be1 100644 --- a/doc/user/application_security/secret_detection/index.md +++ b/doc/user/application_security/secret_detection/index.md @@ -15,18 +15,18 @@ info: To determine the technical writer assigned to the Stage/Group associated w > `secret_detection_default_branch` and `secret_detection` were consolidated into one job, > `secret_detection`. -People may accidentally commit secrets to -remote Git repositories. Secrets include keys, passwords, API tokens, and other sensitive -information. Anyone with access to the repository could use the secrets for malicious purposes. -Exposed secrets are compromised and must be replaced, which can be costly. - -To help prevent secrets from being committed to a Git repository, you can use Secret Detection -to scan your repository for secrets. Scanning is language -and framework agnostic, but does not support scanning binary files. - -Secret Detection uses a specific analyzer containing the -[Gitleaks](https://github.com/zricethezav/gitleaks) tool to scan the repository for secrets, in a -`secret-detection` job. The results are saved as a +People may accidentally commit secrets (such as keys, passwords, and API tokens) to remote Git repositories. + +Anyone with access to the repository could use the secrets for malicious purposes. Exposed secrets +must be considered compromised and be replaced, which can be costly. + +To help prevent secrets from being committed to a Git repository, you can use Secret Detection to +scan your repository for secrets. Scanning is language and framework agnostic, but does not support +scanning binary files. + +Secret Detection uses an analyzer containing the [Gitleaks](https://github.com/zricethezav/gitleaks) +tool to scan the repository for secrets. Detection occurs in the `secret-detection` job. The results +are saved as a [Secret Detection report artifact](../../../ci/yaml/artifacts_reports.md#artifactsreportssecret_detection) that you can later download and analyze. Due to implementation limitations, we always take the latest Secret Detection artifact available. @@ -38,7 +38,7 @@ All identified secrets are reported in the: - Merge request widget - Pipelines' **Security** tab -- [Security Dashboard](../security_dashboard/index.md) +- [Vulnerability Report](../vulnerability_report/index.md)  @@ -61,7 +61,7 @@ Different features are available in different [GitLab tiers](https://about.gitla | Download [JSON Report](../sast/index.md#reports-json-format) | **{check-circle}** Yes | **{check-circle}** Yes | | See new findings in the merge request widget | **{dotted-circle}** No | **{check-circle}** Yes | | View identified secrets in the pipelines' **Security** tab | **{dotted-circle}** No | **{check-circle}** Yes | -| [Manage vulnerabilities](../vulnerabilities/index.md) | **{dotted-circle}** No | **{check-circle}** Yes | +| [Manage vulnerabilities](../vulnerability_report/index.md) | **{dotted-circle}** No | **{check-circle}** Yes | | [Access the Security Dashboard](../security_dashboard/index.md) | **{dotted-circle}** No | **{check-circle}** Yes | | [Customize Secret Detection rulesets](#custom-rulesets) | **{dotted-circle}** No | **{check-circle}** Yes | @@ -88,13 +88,13 @@ To enable Secret Detection, either: ### Enable Secret Detection by including the template -We recommend this method if you have an existing GitLab CI/CD configuration file. +You should use this method if you have an existing GitLab CI/CD configuration file. Add the following extract to your `.gitlab-ci.yml` file: ```yaml include: - - template: Security/Secret-Detection.gitlab-ci.yml + - template: Jobs/Secret-Detection.gitlab-ci.yml ``` Pipelines now include a Secret Detection job, and the results are included in the merge request @@ -122,7 +122,34 @@ widget. ## Responding to a leaked secret -If the scanner detects a secret we recommend you rotate it immediately. [Purging a file from the repository's history](../../project/repository/reducing_the_repo_size_using_git.md#purge-files-from-repository-history) may not be effective in removing all references to the file. Also, the secret remains in any forks of the repository. +If the scanner detects a secret you should rotate it immediately. [Purging a file from the repository's history](../../project/repository/reducing_the_repo_size_using_git.md#purge-files-from-repository-history) may not be effective in removing all references to the file. Also, the secret remains in any forks of the repository. + +## Pinning to specific analyzer version + +The GitLab-managed CI/CD template specifies a major version and automatically pulls the latest analyzer release within that major version. + +In some cases, you may need to use a specific version. +For example, you might need to avoid a regression in a later release. + +To override the automatic update behavior, set the `SECRETS_ANALYZER_VERSION` CI/CD variable +in your CI/CD configuration file after you include the [`Secret-Detection.gitlab-ci.yml` template](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/ci/templates/Jobs/Secret-Detection.gitlab-ci.yml). + +You can set the tag to: + +- A major version, like `4`. Your pipelines will use any minor or patch updates that are released within this major version. +- A minor version, like `4.5`. Your pipelines will use any patch updates that are released within this minor version. +- A patch version, like `4.5.0`. Your pipelines won't receive any updates. + +This example uses a specific minor version of the analyzer: + +```yaml +include: + - template: Security/Secret-Detection.gitlab-ci.yml + +secret_detection: + variables: + SECRETS_ANALYZER_VERSION: "4.5" +``` ## Configure scan settings @@ -154,11 +181,13 @@ secret_detection: SECRET_DETECTION_HISTORIC_SCAN: "true" ``` -### Ignoring Secrets +### Ignore secrets -You might want to add a fake secret to your code base. For instance, you can use a fake secret as an example in your documentation or test suite. +In some instances, you might want to ignore a secret. For example, you may have a fake secret in an +example or a test suite. In these instances, you want to ignore the secret, instead of having it +reported as a vulnerability. -In these cases, Secret Detection can ignore the fake secret and not report it as a vulnerability. To ignore a secret, add `gitleaks:allow` as a comment to the line that contains the secret. +To ignore a secret, add `gitleaks:allow` as a comment to the line that contains the secret. For example: @@ -172,7 +201,7 @@ Secret Detection can be customized by defining available CI/CD variables: | CI/CD variable | Default value | Description | |-----------------------------------|---------------|-------------| -| `SECRET_DETECTION_EXCLUDED_PATHS` | "" | Exclude vulnerabilities from output based on the paths. This is a comma-separated list of patterns. Patterns can be globs (see [`doublestar.Match`](https://pkg.go.dev/github.com/bmatcuk/doublestar/v4@v4.0.2#Match) for supported patterns), or file or folder paths (for example, `doc,spec` ). Parent directories also match patterns. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/225273) in GitLab 13.3. | +| `SECRET_DETECTION_EXCLUDED_PATHS` | "" | Exclude vulnerabilities from output based on the paths. The paths are a comma-separated list of patterns. Patterns can be globs (see [`doublestar.Match`](https://pkg.go.dev/github.com/bmatcuk/doublestar/v4@v4.0.2#Match) for supported patterns), or file or folder paths (for example, `doc,spec` ). Parent directories also match patterns. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/225273) in GitLab 13.3. | | `SECRET_DETECTION_HISTORIC_SCAN` | false | Flag to enable a historic Gitleaks scan. | | `SECRET_DETECTION_IMAGE_SUFFIX` | "" | Suffix added to the image name. If set to `-fips`, `FIPS-enabled` images are used for scan. See [Use FIPS-enabled images](#use-fips-enabled-images) for more details. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/355519) in GitLab 14.10. | | `SECRET_DETECTION_LOG_OPTIONS` | "" | [`git log`](https://git-scm.com/docs/git-log) options used to define commit ranges. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/350660) in GitLab 15.1.| @@ -214,7 +243,7 @@ By default, Secret Detection scans only the current state of the Git repository. contained in the repository's history are not detected. To address this, Secret Detection can scan the Git repository's full history. -We recommend you do a full history scan only once, after enabling Secret Detection. A full history +You should do a full history scan only once, after enabling Secret Detection. A full history can take a long time, especially for larger repositories with lengthy Git histories. After completing an initial full history scan, use only standard Secret Detection as part of your pipeline. @@ -349,15 +378,15 @@ In the `secret-detection-ruleset.toml` file, do one of the following: ## Running Secret Detection in an offline environment **(PREMIUM SELF)** -For self-managed GitLab instances in an environment with limited, restricted, or intermittent access -to external resources through the internet, some configuration changes are required for the Secret -Detection job to run successfully. The instructions in this section must be completed together with -the instructions detailed in [offline environments](../offline_deployments/index.md). +An offline environment has limited, restricted, or intermittent access to external resources through +the internet. For self-managed GitLab instances in such an environment, Secret Detection requires +some configuration changes. The instructions in this section must be completed together with the +instructions detailed in [offline environments](../offline_deployments/index.md). ### Configure GitLab Runner By default, a runner tries to pull Docker images from the GitLab container registry even if a local -copy is available. We recommend using this default setting, to ensure Docker images remain current. +copy is available. You should use this default setting, to ensure Docker images remain current. However, if no network connectivity is available, you must change the default GitLab Runner `pull_policy` variable. @@ -379,11 +408,11 @@ Prerequisites: [local Docker container registry](../../packages/container_registry/index.md): ```plaintext - registry.gitlab.com/security-products/secret-detection:3 + registry.gitlab.com/security-products/secrets:4 ``` The Secret Detection analyzer's image is [periodically updated](../index.md#vulnerability-scanner-maintenance) - so you may need to periodically update the local copy. + so you should periodically update the local copy. 1. Set the CI/CD variable `SECURE_ANALYZERS_PREFIX` to the local Docker container registry. @@ -454,14 +483,14 @@ secrets. If the number of commits in a merge request is greater than the value o [`GIT_DEPTH` CI/CD variable](../../../ci/runners/configure_runners.md#shallow-cloning), Secret Detection [fails to detect secrets](#error-couldnt-run-the-gitleaks-command-exit-status-2). -For example, if a pipeline is triggered from a merge request containing 60 commits and the -`GIT_DEPTH` variable's value is less than 60, the Secret Detection job fails as the clone is not -deep enough to contain all of the relevant commits. To veridy the current value, see +For example, you could have a pipeline triggered from a merge request containing 60 commits and the +`GIT_DEPTH` variable set to less than 60. In that case the Secret Detection job fails because the +clone is not deep enough to contain all of the relevant commits. To verify the current value, see [pipeline configuration](../../../ci/pipelines/settings.md#limit-the-number-of-changes-fetched-during-clone). -To confirm this as the cause of the error, set the [logging level](#set-the-logging-level) to `debug`, then -rerun the pipeline. The logs should look similar to the following example. The text "object not -found" is a symptom of this error. +To confirm this as the cause of the error, set the [logging level](#set-the-logging-level) to +`debug`, then rerun the pipeline. The logs should look similar to the following example. The text +"object not found" is a symptom of this error. ```plaintext ERRO[2020-11-18T18:05:52Z] object not found diff --git a/doc/user/application_security/security_dashboard/index.md b/doc/user/application_security/security_dashboard/index.md index af98fc783e7..7c44b49b78c 100644 --- a/doc/user/application_security/security_dashboard/index.md +++ b/doc/user/application_security/security_dashboard/index.md @@ -153,7 +153,7 @@ important to describe those, too. Think of things that may go wrong and include This is important to minimize requests for support, and to avoid doc comments with questions that you know someone might ask. -Each scenario can be a third-level heading, e.g. `### Getting error message X`. +Each scenario can be a third-level heading, for example `### Getting error message X`. If you have none to add when creating a doc, leave this section in place but commented out to help encourage others to add to it in the future. --> diff --git a/doc/user/application_security/terminology/index.md b/doc/user/application_security/terminology/index.md index 085a762fffa..f156d60be8f 100644 --- a/doc/user/application_security/terminology/index.md +++ b/doc/user/application_security/terminology/index.md @@ -230,7 +230,7 @@ support for cheap scan is proposed in issue [349926](https://gitlab.com/gitlab-o ### Pre-filter -An irreversible action that is done to filter out target(s) before analysis occurs. This is usually provided to allow +An irreversible action that is done to filter out targets before analysis occurs. This is usually provided to allow the user to reduce scope and noise as well as speed up the analysis. This should not be done if a record is needed as we currently do not store anything related to the skipped/excluded code or assets. diff --git a/doc/user/application_security/vulnerabilities/severities.md b/doc/user/application_security/vulnerabilities/severities.md index 36f9578f999..e75d0a45f7d 100644 --- a/doc/user/application_security/vulnerabilities/severities.md +++ b/doc/user/application_security/vulnerabilities/severities.md @@ -42,15 +42,13 @@ the following tables: | [`sobelow`](https://gitlab.com/gitlab-org/security-products/analyzers/sobelow) | **{check-circle}** Yes | Not applicable | Hardcodes all severity levels to `Unknown` | | [`nodejs-scan`](https://gitlab.com/gitlab-org/security-products/analyzers/nodejs-scan) | **{check-circle}** Yes | String | `INFO`, `WARNING`, `ERROR` | | [`flawfinder`](https://gitlab.com/gitlab-org/security-products/analyzers/flawfinder) | **{check-circle}** Yes | Integer | `0`, `1`, `2`, `3`, `4`, `5` | -| [`eslint`](https://gitlab.com/gitlab-org/security-products/analyzers/eslint) | **{check-circle}** Yes | Not applicable | Hardcodes all severity levels to `Unknown` | | [`SpotBugs`](https://gitlab.com/gitlab-org/security-products/analyzers/spotbugs) | **{check-circle}** Yes | Integer | `1`, `2`, `3`, `11`, `12`, `18` | -| [`gosec`](https://gitlab.com/gitlab-org/security-products/analyzers/gosec) | **{check-circle}** Yes | String | `HIGH`, `MEDIUM`, `LOW` | -| [`bandit`](https://gitlab.com/gitlab-org/security-products/analyzers/bandit) | **{check-circle}** Yes | String | `HIGH`, `MEDIUM`, `LOW` | | [`phpcs-security-audit`](https://gitlab.com/gitlab-org/security-products/analyzers/phpcs-security-audit) | **{check-circle}** Yes | String | `ERROR`, `WARNING` | | [`pmd-apex`](https://gitlab.com/gitlab-org/security-products/analyzers/pmd-apex) | **{check-circle}** Yes | Integer | `1`, `2`, `3`, `4`, `5` | | [`kubesec`](https://gitlab.com/gitlab-org/security-products/analyzers/kubesec) | **{check-circle}** Yes | String | `CriticalSeverity`, `InfoSeverity` | | [`secrets`](https://gitlab.com/gitlab-org/security-products/analyzers/secrets) | **{check-circle}** Yes | Not applicable | Hardcodes all severity levels to `Critical` | | [`semgrep`](https://gitlab.com/gitlab-org/security-products/analyzers/semgrep) | **{check-circle}** Yes | String | `error`, `warning`, `note`, `none` | +| [`kics`](https://gitlab.com/gitlab-org/security-products/analyzers/kics) | **{check-circle}** Yes | String | `error`, `warning`, `note`, `none` (gets mapped to `info` in [analyzer version 3.7.0 and later](https://gitlab.com/gitlab-org/security-products/analyzers/kics/-/releases/v3.7.0)) | ## Dependency Scanning diff --git a/doc/user/application_security/vulnerability_report/index.md b/doc/user/application_security/vulnerability_report/index.md index 59851fd192c..2b78dde4f63 100644 --- a/doc/user/application_security/vulnerability_report/index.md +++ b/doc/user/application_security/vulnerability_report/index.md @@ -216,6 +216,10 @@ Fields included are: - [CVE](https://cve.mitre.org/) (Common Vulnerabilities and Exposures) - [CWE](https://cwe.mitre.org/) (Common Weakness Enumeration) - Other identifiers +- Detected At +- Location +- Activity +- Comments NOTE: Full details are available through our |
