diff options
Diffstat (limited to 'doc/user')
234 files changed, 5570 insertions, 3628 deletions
diff --git a/doc/user/admin_area/appearance.md b/doc/user/admin_area/appearance.md index 5513fa3585d..fc42c7770f2 100644 --- a/doc/user/admin_area/appearance.md +++ b/doc/user/admin_area/appearance.md @@ -94,6 +94,6 @@ important to describe those, too. Think of things that may go wrong and include This is important to minimize requests for support, and to avoid doc comments with questions that you know someone might ask. -Each scenario can be a third-level heading, e.g. `### Getting error message X`. +Each scenario can be a third-level heading, for example `### Getting error message X`. If you have none to add when creating a doc, leave this section in place but commented out to help encourage others to add to it in the future. --> diff --git a/doc/user/admin_area/broadcast_messages.md b/doc/user/admin_area/broadcast_messages.md index e5d0a6e297e..acb3e92fff8 100644 --- a/doc/user/admin_area/broadcast_messages.md +++ b/doc/user/admin_area/broadcast_messages.md @@ -111,6 +111,6 @@ important to describe those, too. Think of things that may go wrong and include This is important to minimize requests for support, and to avoid doc comments with questions that you know someone might ask. -Each scenario can be a third-level heading, e.g. `### Getting error message X`. +Each scenario can be a third-level heading, for example `### Getting error message X`. If you have none to add when creating a doc, leave this section in place but commented out to help encourage others to add to it in the future. --> diff --git a/doc/user/admin_area/custom_project_templates.md b/doc/user/admin_area/custom_project_templates.md index 551ed667250..de2856c2320 100644 --- a/doc/user/admin_area/custom_project_templates.md +++ b/doc/user/admin_area/custom_project_templates.md @@ -49,6 +49,6 @@ important to describe those, too. Think of things that may go wrong and include This is important to minimize requests for support, and to avoid doc comments with questions that you know someone might ask. -Each scenario can be a third-level heading, e.g. `### Getting error message X`. +Each scenario can be a third-level heading, for example `### Getting error message X`. If you have none to add when creating a doc, leave this section in place but commented out to help encourage others to add to it in the future. --> diff --git a/doc/user/admin_area/diff_limits.md b/doc/user/admin_area/diff_limits.md index 3a1ecac6ee6..3d7c49c1f2b 100644 --- a/doc/user/admin_area/diff_limits.md +++ b/doc/user/admin_area/diff_limits.md @@ -47,6 +47,6 @@ important to describe those, too. Think of things that may go wrong and include This is important to minimize requests for support, and to avoid doc comments with questions that you know someone might ask. -Each scenario can be a third-level heading, e.g. `### Getting error message X`. +Each scenario can be a third-level heading, for example `### Getting error message X`. If you have none to add when creating a doc, leave this section in place but commented out to help encourage others to add to it in the future. --> diff --git a/doc/user/admin_area/email_from_gitlab.md b/doc/user/admin_area/email_from_gitlab.md index c1d3521d60c..ba465fbea29 100644 --- a/doc/user/admin_area/email_from_gitlab.md +++ b/doc/user/admin_area/email_from_gitlab.md @@ -55,6 +55,6 @@ important to describe those, too. Think of things that may go wrong and include This is important to minimize requests for support, and to avoid doc comments with questions that you know someone might ask. -Each scenario can be a third-level heading, e.g. `### Getting error message X`. +Each scenario can be a third-level heading, for example `### Getting error message X`. If you have none to add when creating a doc, leave this section in place but commented out to help encourage others to add to it in the future. --> diff --git a/doc/user/admin_area/geo_nodes.md b/doc/user/admin_area/geo_nodes.md deleted file mode 100644 index 710f37bb344..00000000000 --- a/doc/user/admin_area/geo_nodes.md +++ /dev/null @@ -1,11 +0,0 @@ ---- -redirect_to: 'geo_sites.md' -remove_date: '2022-10-05' ---- - -This document was moved to [another location](geo_sites.md). - -<!-- This redirect file can be deleted after <2022-10-05>. --> -<!-- Redirects that point to other docs in the same project expire in three months. --> -<!-- Redirects that point to docs in a different project or site (link is not relative and starts with `https:`) expire in one year. --> -<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/redirects.html --> diff --git a/doc/user/admin_area/geo_sites.md b/doc/user/admin_area/geo_sites.md index 093ed84f41a..f3be036fd38 100644 --- a/doc/user/admin_area/geo_sites.md +++ b/doc/user/admin_area/geo_sites.md @@ -112,6 +112,6 @@ important to describe those, too. Think of things that may go wrong and include This is important to minimize requests for support, and to avoid doc comments with questions that you know someone might ask. -Each scenario can be a third-level heading, e.g. `### Getting error message X`. +Each scenario can be a third-level heading, for example `### Getting error message X`. If you have none to add when creating a doc, leave this section in place but commented out to help encourage others to add to it in the future. --> diff --git a/doc/user/admin_area/img/minimum_password_length_settings_v12_6.png b/doc/user/admin_area/img/minimum_password_length_settings_v12_6.png Binary files differdeleted file mode 100644 index 27634a02a8a..00000000000 --- a/doc/user/admin_area/img/minimum_password_length_settings_v12_6.png +++ /dev/null diff --git a/doc/user/admin_area/index.md b/doc/user/admin_area/index.md index e6ddb810933..4c34d82dc02 100644 --- a/doc/user/admin_area/index.md +++ b/doc/user/admin_area/index.md @@ -136,15 +136,15 @@ For each user, the following are listed: 1. Date of last activity To edit a user, select the **Edit** button in that user's -row. To delete the user, or delete the user and their contributions, select the cog dropdown in +row. To delete the user, or delete the user and their contributions, select the cog dropdown list in that user's row, and select the desired option. To change the sort order: -1. Select the sort dropdown. +1. Select the sort dropdown list. 1. Select the desired order. -By default the sort dropdown shows **Name**. +By default the sort dropdown list shows **Name**. To search for users, enter your criteria in the search field. The user search is case insensitive, and applies partial matching to name and username. To search for an email address, @@ -260,7 +260,7 @@ number of members, and whether the group is private, internal, or public. To edi the **Edit** button in that group's row. To delete the group, select the **Delete** button in that group's row. -To change the sort order, select the sort dropdown and select the desired order. The default +To change the sort order, select the sort dropdown list and select the desired order. The default sort order is by **Last created**. To search for groups by name, enter your criteria in the search field. The group search is case diff --git a/doc/user/admin_area/labels.md b/doc/user/admin_area/labels.md index 524546d447c..8e1ca979707 100644 --- a/doc/user/admin_area/labels.md +++ b/doc/user/admin_area/labels.md @@ -22,6 +22,6 @@ important to describe those, too. Think of things that may go wrong and include This is important to minimize requests for support, and to avoid doc comments with questions that you know someone might ask. -Each scenario can be a third-level heading, e.g. `### Getting error message X`. +Each scenario can be a third-level heading, for example `### Getting error message X`. If you have none to add when creating a doc, leave this section in place but commented out to help encourage others to add to it in the future. --> diff --git a/doc/user/admin_area/license.md b/doc/user/admin_area/license.md index 5211a18201b..5296a918f56 100644 --- a/doc/user/admin_area/license.md +++ b/doc/user/admin_area/license.md @@ -64,6 +64,17 @@ This error occurs when you use an activation code to activate your instance, but You may have connectivity issues due to the following reasons: -- **You have an offline environment**: Configure your setup to allow connection to GitLab servers. If connection to GitLab servers is not possible, contact [GitLab support](https://about.gitlab.com/support/#contact-support) to request a license key. -- **Firewall settings**: Enable an encrypted HTTPS connection from your GitLab instance to `customers.gitlab.com` (with IP addresses 104.18.26.123 and 104.18.27.123) on port 443. -- **Customers Portal is not operational**: To check for performance or service disruptions, check the Customers Portal [status](https://status.gitlab.com/). +- **You have an offline environment**: + - Configure your setup to allow connection to GitLab servers. If connection to GitLab servers is not possible, contact your Sales Representative to request a license key. You can also contact [GitLab support](https://about.gitlab.com/support/#contact-support) if you need help finding your Sales Representative. +- **Customers Portal is not operational**: + - To check for performance or service disruptions, check the Customers Portal [status](https://status.gitlab.com/). +- **Firewall settings**: + - Check if your GitLab instance has an encrypted connection to `customers.gitlab.com` (with IP addresses 172.64.146.11 and 104.18.41.245) on port 443: + + ```shell + curl --verbose "https://customers.gitlab.com/" + ``` + + - If the curl command returns a failure, either: + - [Configure a proxy](https://docs.gitlab.com/omnibus/settings/environment-variables.html) in `gitlab.rb` to point to your server. + - Contact your network administrator to make changes to the proxy. diff --git a/doc/user/admin_area/license_file.md b/doc/user/admin_area/license_file.md index e6186fb9805..d821d9bab23 100644 --- a/doc/user/admin_area/license_file.md +++ b/doc/user/admin_area/license_file.md @@ -25,7 +25,7 @@ Otherwise, to add your license: 1. Select **Add license**. NOTE: -For GitLab versions 14.1.x or newer, you can access the **Add License** page directly from the URL, `<YourGitLabURL>/admin/license/new`. +In GitLab 14.1.x through 14.10.x, you can access the **Add License** page directly from the URL, `<YourGitLabURL>/admin/license/new`. In GitLab 15.0 and later, the path is `<YourGitLabURL>/admin/subscription`. ## Add your license file during installation @@ -147,3 +147,81 @@ the license. ### `Start GitLab Ultimate trial` still displays after adding license To fix this issue, restart [Puma or your entire GitLab instance](../../administration/restart_gitlab.md). + +### License commands in the rails console + +The following commands can be run in the [rails console](../../administration/operations/rails_console.md#starting-a-rails-console-session). + +WARNING: +Any command that changes data directly could be damaging if not run correctly, or under the right conditions. +We highly recommend running them in a test environment with a backup of the instance ready to be restored, just in case. + +#### See current license information + +```ruby +# License information (name, company, email address) +License.current.licensee + +# Plan: +License.current.plan + +# Uploaded: +License.current.created_at + +# Started: +License.current.starts_at + +# Expires at: +License.current.expires_at + +# Is this a trial license? +License.current.trial? + +# License ID for lookup on CustomersDot +License.current.license_id + +# License data in Base64-encoded ASCII format +License.current.data +``` + +#### Check if a project feature is available on the instance + +Features listed in <https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/app/models/license.rb>. + +```ruby +License.current.feature_available?(:jira_dev_panel_integration) +``` + +#### Check if a project feature is available in a project + +Features listed in [`license.rb`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/app/models/license.rb). + +```ruby +p = Project.find_by_full_path('<group>/<project>') +p.feature_available?(:jira_dev_panel_integration) +``` + +#### Add a license through the console + +```ruby +key = "<key>" +license = License.new(data: key) +license.save +License.current # check to make sure it applied +``` + +This is needed for example in a known edge-case with +[expired license and multiple LDAP servers](../../administration/auth/ldap/ldap-troubleshooting.md#expired-license-causes-errors-with-multiple-ldap-servers). + +#### Remove licenses + +To clean up the [License History table](../../user/admin_area/license_file.md#view-license-details-and-history): + +```ruby +TYPE = :trial? +# or :expired? + +License.select(&TYPE).each(&:destroy!) + +# or even License.all.each(&:destroy!) +``` diff --git a/doc/user/admin_area/moderate_users.md b/doc/user/admin_area/moderate_users.md index ace1c6be5f8..c0daf029b1f 100644 --- a/doc/user/admin_area/moderate_users.md +++ b/doc/user/admin_area/moderate_users.md @@ -54,7 +54,7 @@ To approve or reject a user sign up: 1. On the left sidebar, select **Overview > Users**. 1. Select the **Pending approval** tab. 1. Optional. Select a user. -1. Select the **{settings}** **User administration** dropdown. +1. Select the **{settings}** **User administration** dropdown list. 1. Select **Approve** or **Reject**. Approving a user: @@ -78,7 +78,7 @@ by removing them in LDAP, or directly from the Admin Area. To do this: 1. On the top bar, select **Main menu > Admin**. 1. On the left sidebar, select **Overview > Users**. 1. Optional. Select a user. -1. Select the **{settings}** **User administration** dropdown. +1. Select the **{settings}** **User administration** dropdown list. 1. Select **Block**. A blocked user: @@ -102,7 +102,7 @@ A blocked user can be unblocked from the Admin Area. To do this: 1. On the left sidebar, select **Overview > Users**. 1. Select the **Blocked** tab. 1. Optional. Select a user. -1. Select the **{settings}** **User administration** dropdown. +1. Select the **{settings}** **User administration** dropdown list. 1. Select **Unblock**. The user's state is set to active and they consume a @@ -156,13 +156,13 @@ A user can be deactivated from the Admin Area. To do this: 1. On the top bar, select **Main menu > Admin**. 1. On the left sidebar, select **Overview > Users**. 1. Optional. Select a user. -1. Select the **{settings}** **User administration** dropdown. +1. Select the **{settings}** **User administration** dropdown list. 1. Select **Deactivate**. For the deactivation option to be visible to an administrator, the user: - Must be currently active. -- Must not have signed in, or have any activity, in the last 90 days. +- Must not be [dormant](#automatically-deactivate-dormant-users). NOTE: Users can also be deactivated using the [GitLab API](../../api/users.md#deactivate-user). @@ -203,7 +203,7 @@ To do this: 1. On the left sidebar, select **Overview > Users**. 1. Select the **Deactivated** tab. 1. Optional. Select a user. -1. Select the **{settings}** **User administration** dropdown. +1. Select the **{settings}** **User administration** dropdown list. 1. Select **Activate**. The user's state is set to active and they consume a @@ -235,7 +235,7 @@ Users can be banned using the Admin Area. To do this: 1. On the top bar, select **Main menu > Admin**. 1. On the left sidebar, select **Overview > Users**. 1. Optional. Select a user. -1. Select the **{settings}** **User administration** dropdown. +1. Select the **{settings}** **User administration** dropdown list. 1. Select **Ban user**. The banned user does not consume a [seat](../../subscriptions/self_managed/index.md#billable-users). @@ -248,7 +248,7 @@ A banned user can be unbanned using the Admin Area. To do this: 1. On the left sidebar, select **Overview > Users**. 1. Select the **Banned** tab. 1. Optional. Select a user. -1. Select the **{settings}** **User administration** dropdown. +1. Select the **{settings}** **User administration** dropdown list. 1. Select **Unban user**. The user's state is set to active and they consume a @@ -283,3 +283,68 @@ You can also delete a user and their contributions, such as merge requests, issu NOTE: Before 15.1, additionally groups of which deleted user were the only owner among direct members were deleted. + +## Troubleshooting + +When moderating users, you may need to perform bulk actions on them based on certain conditions. The following rails console scripts show some examples of this. You may [start a rails console session](../../administration/operations/rails_console.md#starting-a-rails-console-session) and use scripts similar to the following: + +### Deactivate users that have no recent activity + +Administrators can deactivate users that have no recent activity. + +WARNING: +Commands that change data can cause damage if not run correctly or under the right conditions. Always run commands in a test environment first and have a backup instance ready to restore. + +```ruby +days_inactive = 90 +inactive_users = User.active.where("last_activity_on <= ?", days_inactive.days.ago) + +inactive_users.each do |user| + puts "user '#{user.username}': #{user.last_activity_on}" + user.deactivate! +end +``` + +### Block users that have no recent activity + +Administrators can block users that have no recent activity. + +WARNING: +Commands that change data can cause damage if not run correctly or under the right conditions. Always run commands in a test environment first and have a backup instance ready to restore. + +```ruby +days_inactive = 90 +inactive_users = User.active.where("last_activity_on <= ?", days_inactive.days.ago) + +inactive_users.each do |user| + puts "user '#{user.username}': #{user.last_activity_on}" + user.block! +end +``` + +### Block or delete users that have no projects or groups + +Administrators can block or delete users that have no projects or groups. + +WARNING: +Commands that change data can cause damage if not run correctly or under the right conditions. Always run commands in a test environment first and have a backup instance ready to restore. + +```ruby +users = User.where('id NOT IN (select distinct(user_id) from project_authorizations)') + +# How many users are removed? +users.count + +# If that count looks sane: + +# You can either block the users: +users.each { |user| user.blocked? ? nil : user.block! } + +# Or you can delete them: + # need 'current user' (your user) for auditing purposes +current_user = User.find_by(username: '<your username>') + +users.each do |user| + DeleteUserWorker.perform_async(current_user.id, user.id) +end +``` diff --git a/doc/user/admin_area/monitoring/health_check.md b/doc/user/admin_area/monitoring/health_check.md index e6f9c045329..2939a8b0418 100644 --- a/doc/user/admin_area/monitoring/health_check.md +++ b/doc/user/admin_area/monitoring/health_check.md @@ -143,6 +143,6 @@ important to describe those, too. Think of things that may go wrong and include This is important to minimize requests for support, and to avoid doc comments with questions that you know someone might ask. -Each scenario can be a third-level heading, e.g. `### Getting error message X`. +Each scenario can be a third-level heading, for example `### Getting error message X`. If you have none to add when creating a doc, leave this section in place but commented out to help encourage others to add to it in the future. --> diff --git a/doc/user/admin_area/reporting/git_abuse_rate_limit.md b/doc/user/admin_area/reporting/git_abuse_rate_limit.md index 432205d8fa2..f700b8b1ea3 100644 --- a/doc/user/admin_area/reporting/git_abuse_rate_limit.md +++ b/doc/user/admin_area/reporting/git_abuse_rate_limit.md @@ -4,18 +4,20 @@ group: Anti-Abuse info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- -# Git abuse rate limit **(ULTIMATE SELF)** +# Git abuse rate limit (administration) **(ULTIMATE SELF)** -> [Introduced](https://gitlab.com/groups/gitlab-org/-/epics/8066) in GitLab 15.2 [with flags](../../../administration/feature_flags.md) named `git_abuse_rate_limit_feature_flag` and `auto_ban_user_on_excessive_projects_download`. Both flags are disabled by default. +> [Introduced](https://gitlab.com/groups/gitlab-org/-/epics/8066) in GitLab 15.2 [with a flag](../../../administration/feature_flags.md) named `git_abuse_rate_limit_feature_flag`. Disabled by default. FLAG: -On self-managed GitLab, by default this feature is not available. To make it available, ask an administrator to [enable the feature flags](../../../administration/feature_flags.md) named `git_abuse_rate_limit_feature_flag` and `auto_ban_user_on_excessive_projects_download`. +On self-managed GitLab, by default this feature is not available. To make it available, ask an administrator to [enable the feature flag](../../../administration/feature_flags.md) named `git_abuse_rate_limit_feature_flag`. On GitLab.com, this feature is not available. -Git abuse rate limiting is a feature to automatically [ban users](../moderate_users.md#ban-and-unban-users) who download more than a specified number of repositories in a given time. When the `git_abuse_rate_limit_feature_flag` feature flag is enabled, the administrator receives an email when a user is about to be banned. +Git abuse rate limiting is a feature to automatically [ban users](../moderate_users.md#ban-and-unban-users) who download or clone more than a specified number of repositories in any project in the instance within a given time frame. Banned users cannot sign in to the instance and cannot access any non-public group via HTTP or SSH. -When the `auto_ban_user_on_excessive_projects_download` is not enabled, the user is not banned automatically. You can use this setup to determine the correct values of the rate limit settings. +If the `git_abuse_rate_limit_feature_flag` feature flag is enabled, all application administrators receive an email when a user is about to be banned. -When both flags are enabled, the administrator receives an email when a user is about to be banned, and the user is automatically banned from the GitLab instance. +If automatic banning is disabled, a user is not banned automatically when they exceed the limit. However, administrators are still notified. You can use this setup to determine the correct values of the rate limit settings before enabling automatic banning. + +If automatic banning is enabled, administrators receive an email when a user is about to be banned, and the user is automatically banned from the GitLab instance. ## Configure Git abuse rate limiting @@ -24,6 +26,15 @@ When both flags are enabled, the administrator receives an email when a user is 1. Expand **Git abuse rate limit**. 1. Update the Git abuse rate limit settings: 1. Enter a number in the **Number of repositories** field, greater than or equal to `0` and less than or equal to `10,000`. This number specifies the maximum amount of unique repositories a user can download in the specified time period before they're banned. When set to `0`, Git abuse rate limiting is disabled. - 1. Enter a number in the **Reporting time period (seconds)** field, greater than or equal to `0` and less than or equal to `86,400`. This number specifies the time in seconds a user can download the maximum amount of repositories before they're banned. When set to `0`, Git abuse rate limiting is disabled. - 1. Optional. Exclude users by adding them to the **Excluded users** field. Excluded users are not automatically banned. + 1. Enter a number in the **Reporting time period (seconds)** field, greater than or equal to `0` and less than or equal to `86,400` (10 days). This number specifies the time in seconds a user can download the maximum amount of repositories before they're banned. When set to `0`, Git abuse rate limiting is disabled. + 1. Optional. Exclude up to `100` users by adding them to the **Excluded users** field. Excluded users are not automatically banned. + 1. Optional. Turn on the **Automatically ban users from this namespace when they exceed the specified limits** toggle to enable automatic banning. 1. Select **Save changes**. + +## Unban a user + +1. On the top bar, select **Main menu > Admin**. +1. On the left sidebar, select **Overview > Users**. +1. Select the **Banned** tab and search for the account you want to unban. +1. From the **User administration** dropdown list select **Unban user**. +1. On the confirmation dialog, select **Unban user**. diff --git a/doc/user/admin_area/review_abuse_reports.md b/doc/user/admin_area/review_abuse_reports.md index af2f6640f8e..b8531fded18 100644 --- a/doc/user/admin_area/review_abuse_reports.md +++ b/doc/user/admin_area/review_abuse_reports.md @@ -89,6 +89,6 @@ important to describe those, too. Think of things that may go wrong and include This is important to minimize requests for support, and to avoid doc comments with questions that you know someone might ask. -Each scenario can be a third-level heading, e.g. `### Getting error message X`. +Each scenario can be a third-level heading, for example `### Getting error message X`. If you have none to add when creating a doc, leave this section in place but commented out to help encourage others to add to it in the future. --> diff --git a/doc/user/admin_area/settings/account_and_limit_settings.md b/doc/user/admin_area/settings/account_and_limit_settings.md index cd1f20a2f4e..44a6bbb9d8e 100644 --- a/doc/user/admin_area/settings/account_and_limit_settings.md +++ b/doc/user/admin_area/settings/account_and_limit_settings.md @@ -259,18 +259,6 @@ Once a lifetime for access tokens is set, GitLab: allowed lifetime. Three hours is given to allow administrators to change the allowed lifetime, or remove it, before revocation takes place. -<!-- start_remove The following content will be removed on remove_date: '2022-08-22' --> -## Allow expired access tokens to be used (removed) **(ULTIMATE SELF)** - -> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/214723) in GitLab 13.1. -> - [Feature flag removed](https://gitlab.com/gitlab-org/gitlab/-/issues/296881) in GitLab 13.9. -> - [Deprecated](https://gitlab.com/gitlab-org/gitlab/-/issues/351962) in GitLab 14.8. -> - [Removed](https://gitlab.com/gitlab-org/gitlab/-/issues/351962) in GitLab 15.0. - -This feature was [deprecated](https://gitlab.com/gitlab-org/gitlab/-/issues/351962) in GitLab 14.8. -This feature was [removed](https://gitlab.com/gitlab-org/gitlab/-/issues/351962) in GitLab 15.0. -<!-- end_remove --> - ## Disable user profile name changes **(PREMIUM SELF)** > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/24605) in GitLab 12.7. @@ -288,11 +276,11 @@ When this ability is disabled, GitLab administrators can still use the [Admin Area](../index.md#administering-users) or the [API](../../../api/users.md#user-modification) to update usernames. -## Prevent users from creating top-level groups +## Prevent new users from creating top-level groups > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/367754) in GitLab 15.5. -By default, new users can create top-level groups. GitLab administrators can prevent users from creating top-level groups: +By default, new users can create top-level groups. GitLab administrators can prevent new users from creating top-level groups: - In GitLab 15.5 and later, using either: - The GitLab UI using the steps in this section. @@ -301,7 +289,7 @@ By default, new users can create top-level groups. GitLab administrators can pre 1. On the top bar, select **Main menu > Admin**. 1. On the left sidebar, select **Settings > General**, then expand **Account and limit**. -1. Clear the **Allow users to create top-level groups** checkbox. +1. Clear the **Allow new users to create top-level groups** checkbox. ## Troubleshooting diff --git a/doc/user/admin_area/settings/continuous_integration.md b/doc/user/admin_area/settings/continuous_integration.md index afc9a006b39..adca9c85af1 100644 --- a/doc/user/admin_area/settings/continuous_integration.md +++ b/doc/user/admin_area/settings/continuous_integration.md @@ -20,10 +20,10 @@ for all projects: 1. Check (or uncheck to disable) the box that says **Default to Auto DevOps pipeline for all projects**. 1. Optionally, set up the [Auto DevOps base domain](../../../topics/autodevops/requirements.md#auto-devops-base-domain) which is used for Auto Deploy and Auto Review Apps. -1. Hit **Save changes** for the changes to take effect. +1. Select **Save changes** for the changes to take effect. From now on, every existing project and newly created ones that don't have a -`.gitlab-ci.yml`, uses the Auto DevOps pipelines. +`.gitlab-ci.yml` use the Auto DevOps pipelines. If you want to disable it for a specific project, you can do so in [its settings](../../../topics/autodevops/index.md#enable-or-disable-auto-devops). @@ -71,7 +71,7 @@ runner settings: To view the rendered details: 1. On the top bar, select **Main menu**, and: - - For a project, select ***Projects** and find your project. + - For a project, select **Projects** and find your project. - For a group, select **Groups** and find your group. 1. On the left sidebar, select **Settings > CI/CD**. 1. Expand **Runners**. @@ -132,7 +132,7 @@ NOTE: Any changes to this setting applies to new artifacts only. The expiration time is not be updated for artifacts created before this setting was changed. The administrator may need to manually search for and expire previously-created -artifacts, as described in the [troubleshooting documentation](../../../administration/troubleshooting/gitlab_rails_cheat_sheet.md#remove-artifacts-more-than-a-week-old). +artifacts, as described in the [troubleshooting documentation](../../../administration/job_artifacts.md#delete-job-artifacts-from-jobs-completed-before-a-specific-date). ## Keep the latest artifacts for all jobs in the latest successful pipelines @@ -174,7 +174,7 @@ To set the duration for which the jobs are considered as old and expired: 1. On the left sidebar, select **Settings > CI/CD**. 1. Expand the **Continuous Integration and Deployment** section. 1. Set the value of **Archive jobs**. -1. Hit **Save changes** for the changes to take effect. +1. Select **Save changes** for the changes to take effect. After that time passes, the jobs are archived in the background and no longer able to be retried. Make it empty to never expire jobs. It has to be no less than 1 day, @@ -201,7 +201,7 @@ of your GitLab instance (`.gitlab-ci.yml` if not set): 1. On the top bar, select **Main menu > Admin**. 1. On the left sidebar, select **Settings > CI/CD**. 1. Input the new file and path in the **Default CI/CD configuration file** field. -1. Hit **Save changes** for the changes to take effect. +1. Select **Save changes** for the changes to take effect. It is also possible to specify a [custom CI/CD configuration file for a specific project](../../../ci/pipelines/settings.md#specify-a-custom-cicd-configuration-file). @@ -244,7 +244,7 @@ To enable or disable the banner: > [Moved](https://gitlab.com/gitlab-org/gitlab/-/issues/352316) from GitLab Premium to GitLab Ultimate in 15.0. NOTE: -An alternative [compliance solution](../../group/manage.md#configure-a-compliance-pipeline) +An alternative [compliance solution](../../group/compliance_frameworks.md#configure-a-compliance-pipeline) is available. We recommend this alternative solution because it provides greater flexibility, allowing required pipelines to be assigned to specific compliance framework labels. @@ -272,7 +272,7 @@ To select a CI/CD template for the required pipeline configuration: 1. On the top bar, select **Main menu > Admin**. 1. On the left sidebar, select **Settings > CI/CD**. 1. Expand the **Required pipeline configuration** section. -1. Select a CI/CD template from the dropdown. +1. Select a CI/CD template from the dropdown list. 1. Select **Save changes**. ## Package Registry configuration diff --git a/doc/user/admin_area/settings/deprecated_api_rate_limits.md b/doc/user/admin_area/settings/deprecated_api_rate_limits.md index 279cac95fc9..8bf0ffd21a5 100644 --- a/doc/user/admin_area/settings/deprecated_api_rate_limits.md +++ b/doc/user/admin_area/settings/deprecated_api_rate_limits.md @@ -40,10 +40,10 @@ To override the general user and IP rate limits for requests to deprecated API e 1. Select the checkboxes for the types of rate limits you want to enable: - **Unauthenticated API request rate limit** - **Authenticated API request rate limit** -1. _If you enabled unauthenticated API request rate limits:_ +1. If you selected **unauthenticated**: 1. Select the **Maximum unauthenticated API requests per period per IP**. 1. Select the **Unauthenticated API rate limit period in seconds**. -1. _If you enabled authenticated API request rate limits:_ +1. If you selected **authenticated**: 1. Select the **Maximum authenticated API requests per period per user**. 1. Select the **Authenticated API rate limit period in seconds**. diff --git a/doc/user/admin_area/settings/email.md b/doc/user/admin_area/settings/email.md index 6a7c01ff98b..6d2a3c2cdae 100644 --- a/doc/user/admin_area/settings/email.md +++ b/doc/user/admin_area/settings/email.md @@ -92,6 +92,6 @@ important to describe those, too. Think of things that may go wrong and include This is important to minimize requests for support, and to avoid doc comments with questions that you know someone might ask. -Each scenario can be a third-level heading, e.g. `### Getting error message X`. +Each scenario can be a third-level heading, for example `### Getting error message X`. If you have none to add when creating a doc, leave this section in place but commented out to help encourage others to add to it in the future. --> diff --git a/doc/user/admin_area/settings/external_authorization.md b/doc/user/admin_area/settings/external_authorization.md index 62d3d713616..a34ceac0d95 100644 --- a/doc/user/admin_area/settings/external_authorization.md +++ b/doc/user/admin_area/settings/external_authorization.md @@ -115,6 +115,6 @@ important to describe those, too. Think of things that may go wrong and include This is important to minimize requests for support, and to avoid doc comments with questions that you know someone might ask. -Each scenario can be a third-level heading, e.g. `### Getting error message X`. +Each scenario can be a third-level heading, for example `### Getting error message X`. If you have none to add when creating a doc, leave this section in place but commented out to help encourage others to add to it in the future. --> diff --git a/doc/user/admin_area/settings/files_api_rate_limits.md b/doc/user/admin_area/settings/files_api_rate_limits.md index e5f5064304c..ef9a3674c49 100644 --- a/doc/user/admin_area/settings/files_api_rate_limits.md +++ b/doc/user/admin_area/settings/files_api_rate_limits.md @@ -36,10 +36,10 @@ To override the general user and IP rate limits for requests to the Repository f 1. Select the checkboxes for the types of rate limits you want to enable: - **Unauthenticated API request rate limit** - **Authenticated API request rate limit** -1. _If you enabled unauthenticated API request rate limits:_ +1. If you selected **unauthenticated**: 1. Select the **Max unauthenticated API requests per period per IP**. 1. Select the **Unauthenticated API rate limit period in seconds**. -1. _If you enabled authenticated API request rate limits:_ +1. If you selected **authenticated**: 1. Select the **Max authenticated API requests per period per user**. 1. Select the **Authenticated API rate limit period in seconds**. diff --git a/doc/user/admin_area/settings/floc.md b/doc/user/admin_area/settings/floc.md index 08f3e8c09b2..f8137afa40f 100644 --- a/doc/user/admin_area/settings/floc.md +++ b/doc/user/admin_area/settings/floc.md @@ -36,6 +36,6 @@ important to describe those, too. Think of things that may go wrong and include This is important to minimize requests for support, and to avoid doc comments with questions that you know someone might ask. -Each scenario can be a third-level heading, e.g. `### Getting error message X`. +Each scenario can be a third-level heading, for example `### Getting error message X`. If you have none to add when creating a doc, leave this section in place but commented out to help encourage others to add to it in the future. --> diff --git a/doc/user/admin_area/settings/help_page.md b/doc/user/admin_area/settings/help_page.md index a4072f8680b..8d0fef398af 100644 --- a/doc/user/admin_area/settings/help_page.md +++ b/doc/user/admin_area/settings/help_page.md @@ -56,7 +56,7 @@ GitLab marketing-related entries are occasionally shown on the Help page. To hid You can specify a custom URL to which users are directed when they: -- Select **Support** from the Help dropdown. +- Select **Support** from the Help dropdown list. - Select **See our website for help** on the Help page. 1. On the top bar, select **Main menu > Admin**. @@ -106,6 +106,6 @@ important to describe those, too. Think of things that may go wrong and include This is important to minimize requests for support, and to avoid doc comments with questions that you know someone might ask. -Each scenario can be a third-level heading, e.g. `### Getting error message X`. +Each scenario can be a third-level heading, for example `### Getting error message X`. If you have none to add when creating a doc, leave this section in place but commented out to help encourage others to add to it in the future. --> diff --git a/doc/user/admin_area/settings/index.md b/doc/user/admin_area/settings/index.md index 1bdacf486a2..08c4a0d0167 100644 --- a/doc/user/admin_area/settings/index.md +++ b/doc/user/admin_area/settings/index.md @@ -109,8 +109,8 @@ The **Metrics and profiling** settings contain: Enable and configure Grafana. - [Profiling - Performance bar](../../../administration/monitoring/performance/performance_bar.md#enable-the-performance-bar-for-non-administrators) - Enable access to the Performance Bar for non-administrator users in a given group. -- [Self monitoring](../../../administration/monitoring/gitlab_self_monitoring_project/index.md#create-the-self-monitoring-project) - - Enable or disable instance self monitoring. +- [Self-monitoring](../../../administration/monitoring/gitlab_self_monitoring_project/index.md#create-the-self-monitoring-project) - + Enable or disable instance self-monitoring. - [Usage statistics](usage_statistics.md) - Enable or disable version check and Service Ping. ### Network diff --git a/doc/user/admin_area/settings/instance_template_repository.md b/doc/user/admin_area/settings/instance_template_repository.md index b74f4f68230..bf07c5b2808 100644 --- a/doc/user/admin_area/settings/instance_template_repository.md +++ b/doc/user/admin_area/settings/instance_template_repository.md @@ -28,7 +28,7 @@ To select a project to serve as the custom template repository: 1. Add custom templates to the selected repository. After you add templates, you can use them for the entire instance. -They are available in the [Web Editor's dropdown](../../project/repository/web_editor.md#template-dropdowns) +They are available in the [Web Editor's dropdown list](../../project/repository/web_editor.md#template-dropdowns) and through the [API settings](../../../api/settings.md). ## Supported file types and locations @@ -67,9 +67,9 @@ extension and not be empty. So, the hierarchy should look like this: |-- another_metrics-dashboard.yml ``` -Your custom templates are displayed on the dropdown menu when a new file is added through the GitLab UI: +Your custom templates are displayed on the dropdown list when a new file is added through the GitLab UI: - + If this feature is disabled or no templates are present, no **Custom** section displays in the selection dropdown. @@ -82,6 +82,6 @@ important to describe those, too. Think of things that may go wrong and include This is important to minimize requests for support, and to avoid doc comments with questions that you know someone might ask. -Each scenario can be a third-level heading, e.g. `### Getting error message X`. +Each scenario can be a third-level heading, for example `### Getting error message X`. If you have none to add when creating a doc, leave this section in place but commented out to help encourage others to add to it in the future. --> diff --git a/doc/user/admin_area/settings/package_registry_rate_limits.md b/doc/user/admin_area/settings/package_registry_rate_limits.md index 7e0c3482e3e..a5c5536f22d 100644 --- a/doc/user/admin_area/settings/package_registry_rate_limits.md +++ b/doc/user/admin_area/settings/package_registry_rate_limits.md @@ -1,6 +1,6 @@ --- stage: Package -group: Package +group: Package Registry info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments type: reference --- diff --git a/doc/user/admin_area/settings/sign_in_restrictions.md b/doc/user/admin_area/settings/sign_in_restrictions.md index 320768e6e5a..4ea420d7ca6 100644 --- a/doc/user/admin_area/settings/sign_in_restrictions.md +++ b/doc/user/admin_area/settings/sign_in_restrictions.md @@ -32,27 +32,70 @@ In the event of an external authentication provider outage, use the [GitLab Rail > [Introduced](https://gitlab.com/groups/gitlab-org/-/epics/2158) in GitLab 13.10. -When this feature is enabled, instance administrators are limited as regular users. During that period, -they do not have access to all projects, groups, or the **Admin Area** menu. +If you are an administrator, you might want to work in GitLab without the access that +comes from being an administrator. While you could create a separate user account that +doesn't have administrator access, a more secure solution is to use *Admin Mode*. -To access potentially dangerous resources, an administrator can activate Admin Mode by: +With Admin Mode, your account does not have administrative access by default. +You can continue to access groups and projects you are a member of, but to access +administrative functionality, you must authenticate. -- Selecting the *Enable Admin Mode* button -- Trying to access any part of the UI that requires administrator access, specifically those which call `/admin` endpoints. +When Admin Mode is enabled, it applies to all administrators on the instance. -The main use case allows administrators to perform their regular tasks as a regular -user, based on their memberships, without having to set up a second account for -security reasons. +When Admin Mode is enabled for an instance, administrators: -When Admin Mode status is disabled, administrative users cannot access resources unless -they've been explicitly granted access. For example, when Admin Mode is disabled, they -get a `404` error if they try to open a private group or project, unless -they are members of that group or project. +- Are allowed to access group and projects for which they are members. +- Cannot access the **Admin Area**. -2FA should be enabled for administrators and is supported for the Admin Mode flow, as are -OmniAuth providers and LDAP auth. The Admin Mode status is stored in the active user -session and remains active until it is explicitly disabled (it will be disabled -automatically after a timeout otherwise). +### Enable Admin Mode for your instance + +Administrators can enable Admin Mode though the API, the Rails console, or the UI. + +#### Use the API to enable Admin Mode + +Make the following request to your instance endpoint: + +```shell +curl --request PUT --header "PRIVATE-TOKEN:$ADMIN_TOKEN" "<gitlab.example.com>/api/v4/application/settings?admin_mode=true" +``` + +Replace `<gitlab.example.com>` with your instance URL. + +For more information, see the [list of settings that can be accessed through API calls](../../../api/settings.md). + +#### Use the Rails console to enable Admin Mode + +Open the [Rails console](../../../administration/operations/rails_console.md) and run the following: + +```ruby +::Gitlab::CurrentSettings.update_attributes!(admin_mode: true) +``` + +#### Use the UI to enable Admin Mode + +To enable Admin Mode through the UI: + +1. On the top bar, select **Main menu > Admin**. +1. On the left sidebar, select **Settings > General**. +1. Expand **Sign-in restrictions**. +1. In the **Admin Mode** section, select the **Require additional authentication for administrative tasks** checkbox. + +### Turn on Admin Mode for your session + +To turn on Admin Mode for your current session and access potentially dangerous resources: + +1. On the top bar, select **Enable Admin Mode**. +1. Try to access any part of the UI with `/admin` in the URL (which requires administrator access). + +When Admin Mode status is disabled or turned off, administrators cannot access resources unless +they've been explicitly granted access. For example, administrators get a `404` error +if they try to open a private group or project, unless they are members of that group or project. + +2FA should be enabled for administrators. 2FA, OmniAuth providers, and LDAP +authentication are supported by Admin Mode. Admin Mode status is stored in the current user session and remains active until either: + +- It is explicitly disabled. +- It is disabled automatically after a timeout. ### Limitations of Admin Mode diff --git a/doc/user/admin_area/settings/sign_up_restrictions.md b/doc/user/admin_area/settings/sign_up_restrictions.md index 28fb188731b..76415596dce 100644 --- a/doc/user/admin_area/settings/sign_up_restrictions.md +++ b/doc/user/admin_area/settings/sign_up_restrictions.md @@ -115,7 +115,7 @@ create or update pipelines until their email address is confirmed. > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/20661) in GitLab 12.6 -You can [change](../../../security/password_length_limits.md#modify-minimum-password-length-using-gitlab-ui) +You can [change](../../../security/password_length_limits.md#modify-minimum-password-length) the minimum number of characters a user must have in their password using the GitLab UI. ### Password complexity requirements **(PREMIUM SELF)** @@ -198,6 +198,6 @@ important to describe those, too. Think of things that may go wrong and include This is important to minimize requests for support, and to avoid doc comments with questions that you know someone might ask. -Each scenario can be a third-level heading, e.g. `### Getting error message X`. +Each scenario can be a third-level heading, for example `### Getting error message X`. If you have none to add when creating a doc, leave this section in place but commented out to help encourage others to add to it in the future. --> diff --git a/doc/user/admin_area/settings/terms.md b/doc/user/admin_area/settings/terms.md index 28fe352c684..9a02e50b23f 100644 --- a/doc/user/admin_area/settings/terms.md +++ b/doc/user/admin_area/settings/terms.md @@ -43,6 +43,6 @@ important to describe those, too. Think of things that may go wrong and include This is important to minimize requests for support, and to avoid doc comments with questions that you know someone might ask. -Each scenario can be a third-level heading, e.g. `### Getting error message X`. +Each scenario can be a third-level heading, for example `### Getting error message X`. If you have none to add when creating a doc, leave this section in place but commented out to help encourage others to add to it in the future. --> diff --git a/doc/user/admin_area/settings/third_party_offers.md b/doc/user/admin_area/settings/third_party_offers.md index fbd282ed5ad..8d2ae72ba69 100644 --- a/doc/user/admin_area/settings/third_party_offers.md +++ b/doc/user/admin_area/settings/third_party_offers.md @@ -31,6 +31,6 @@ important to describe those, too. Think of things that may go wrong and include This is important to minimize requests for support, and to avoid doc comments with questions that you know someone might ask. -Each scenario can be a third-level heading, e.g. `### Getting error message X`. +Each scenario can be a third-level heading, for example `### Getting error message X`. If you have none to add when creating a doc, leave this section in place but commented out to help encourage others to add to it in the future. --> diff --git a/doc/user/admin_area/settings/usage_statistics.md b/doc/user/admin_area/settings/usage_statistics.md index fb027b462df..df60268a8bf 100644 --- a/doc/user/admin_area/settings/usage_statistics.md +++ b/doc/user/admin_area/settings/usage_statistics.md @@ -48,7 +48,7 @@ tier. Users can continue to access the features in a paid tier without sharing u ### Features available in 14.4 and later - [Repository size limit](../settings/account_and_limit_settings.md#repository-size-limit). -- [Group access restriction by IP address](../../group/access_and_permissions.md#restrict-group-access-by-ip-address). +- [Group access restriction by IP address](../../group/access_and_permissions.md#restrict-access-to-groups-by-ip-address). NOTE: Registration is not yet required for participation, but may be added in a future milestone. @@ -202,6 +202,6 @@ important to describe those, too. Think of things that may go wrong and include This is important to minimize requests for support, and to avoid doc comments with questions that you know someone might ask. -Each scenario can be a third-level heading, e.g. `### Getting error message X`. +Each scenario can be a third-level heading, for example `### Getting error message X`. If you have none to add when creating a doc, leave this section in place but commented out to help encourage others to add to it in the future. --> diff --git a/doc/user/admin_area/settings/user_and_ip_rate_limits.md b/doc/user/admin_area/settings/user_and_ip_rate_limits.md index 7431fc329d1..e285275f5bb 100644 --- a/doc/user/admin_area/settings/user_and_ip_rate_limits.md +++ b/doc/user/admin_area/settings/user_and_ip_rate_limits.md @@ -225,6 +225,6 @@ important to describe those, too. Think of things that may go wrong and include This is important to minimize requests for support, and to avoid doc comments with questions that you know someone might ask. -Each scenario can be a third-level heading, e.g. `### Getting error message X`. +Each scenario can be a third-level heading, for example `### Getting error message X`. If you have none to add when creating a doc, leave this section in place but commented out to help encourage others to add to it in the future. --> diff --git a/doc/user/admin_area/settings/visibility_and_access_controls.md b/doc/user/admin_area/settings/visibility_and_access_controls.md index d3132ae03c6..6d878bcb01c 100644 --- a/doc/user/admin_area/settings/visibility_and_access_controls.md +++ b/doc/user/admin_area/settings/visibility_and_access_controls.md @@ -45,7 +45,7 @@ By default both administrators and anyone with the **Owner** role can delete a p 1. Expand the **Visibility and access controls** section. 1. Scroll to: - (GitLab 15.1 and later) **Allowed to delete projects**, and select **Administrators**. - - (GitLab 15.0 and earlier) **Default project deletion projection** and select **Only admins can delete project**. + - (GitLab 15.0 and earlier) **Default project deletion protection** and select **Only admins can delete project**. 1. Select **Save changes**. ## Deletion protection **(PREMIUM SELF)** @@ -82,8 +82,8 @@ To configure delayed project deletion: 1. On the left sidebar, select **Settings > General**. 1. Expand the **Visibility and access controls** section. 1. Scroll to: - - (GitLab 15.1 and later) **Deletion projection** and select keep deleted groups and projects, and select a retention period. - - (GitLab 15.0 and earlier) **Default delayed project projection** and select **Enable delayed project deletion by + - (GitLab 15.1 and later) **Deletion protection** and select keep deleted groups and projects, and select a retention period. + - (GitLab 15.0 and earlier) **Default delayed project protection** and select **Enable delayed project deletion by default for newly-created groups.** Then set a retention period in **Default deletion delay**. 1. Select **Save changes**. @@ -262,7 +262,7 @@ These options specify the permitted types and lengths for SSH keys. To specify a restriction for each key type: -1. Select the desired option from the dropdown. +1. Select the desired option from the dropdown list. 1. Select **Save changes**. For more details, see [SSH key restrictions](../../../security/ssh_keys_restrictions.md). @@ -280,13 +280,13 @@ work in every repository. They can only be re-enabled by an administrator user o > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/87579) in GitLab 15.1 [with a flag](../../../administration/feature_flags.md) named `group_ip_restrictions_allow_global`. Disabled by default. > - [Generally available](https://gitlab.com/gitlab-org/gitlab/-/issues/366445) in GitLab 15.4. [Feature flag `group_ip_restrictions_allow_global`](https://gitlab.com/gitlab-org/gitlab/-/issues/366445) removed. -This setting allows you to set IP address ranges to be combined with group-level IP allowlists. -It helps administrators prevent aspects of the GitLab installation from being blocked -from working as intended when an IP allowlist is used. +Administrators can set IP address ranges to be combined with [group-level IP restrictions](../../group/access_and_permissions.md#restrict-access-to-groups-by-ip-address). +Use globally-allowed IP addresses to allow aspects of the GitLab installation to work even when group-level IP address +restrictions are set. -For example, if the GitLab Pages daemon runs on the `10.0.0.0/24` range, specify that range in this -field, as otherwise any group-level restrictions that don't include that range cause the Pages -daemon to be unable to fetch artifacts from the pipeline runs. +For example, if the GitLab Pages daemon runs on the `10.0.0.0/24` range, you can specify that range as globally-allowed. +This means GitLab Pages can still fetch artifacts from pipelines even if group-level IP address restrictions don't +include the `10.0.0.0/24` range. To add a IP address range to the group-level allowlist: @@ -294,7 +294,11 @@ To add a IP address range to the group-level allowlist: 1. On the top bar, select **Main menu > Admin**. 1. On the left sidebar, select **Settings > General**. 1. Expand the **Visibility and access controls** section. -1. In **Globally-allowed IP ranges**, provide a value. +1. In **Globally-allowed IP ranges**, provide a list of IP address ranges. This list: + - Has no limit on the number of IP address ranges. + - Has a size limit of 1 GB. + - Applies to both SSH or HTTP authorized IP address ranges. You cannot split + this list by type of authorization. 1. Select **Save changes**. <!-- ## Troubleshooting @@ -305,6 +309,6 @@ important to describe those, too. Think of things that may go wrong and include This is important to minimize requests for support, and to avoid doc comments with questions that you know someone might ask. -Each scenario can be a third-level heading, e.g. `### Getting error message X`. +Each scenario can be a third-level heading, for example `### Getting error message X`. If you have none to add when creating a doc, leave this section in place but commented out to help encourage others to add to it in the future. --> diff --git a/doc/user/analytics/ci_cd_analytics.md b/doc/user/analytics/ci_cd_analytics.md index de9275b8599..ef2d3a76990 100644 --- a/doc/user/analytics/ci_cd_analytics.md +++ b/doc/user/analytics/ci_cd_analytics.md @@ -6,7 +6,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w # CI/CD analytics **(FREE)** -Use the CI/CD analytics page to view pipeline success rates and duration, and the history of [DORA metrics](index.md#devops-research-and-assessment-dora-key-metrics) over time. +Use the CI/CD analytics page to view pipeline success rates and duration, and the history of DORA metrics over time. ## Pipeline success and duration charts @@ -35,7 +35,7 @@ To view CI/CD analytics: 1. On the top bar, select **Main menu > Projects** and find your project. 1. On the left sidebar, select **Analytics > CI/CD Analytics**. -## View deployment frequency chart **(ULTIMATE)** +## View DORA deployment frequency chart **(ULTIMATE)** > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/275991) in GitLab 13.8. @@ -44,7 +44,7 @@ frequency to the `production` environment. The environment must be part of the [production deployment tier](../../ci/environments/index.md#deployment-tier-of-environments) for its deployment information to appear on the graphs. - Deployment frequency is one of the four [DORA metrics](index.md#devops-research-and-assessment-dora-key-metrics) that DevOps teams use for measuring excellence in software delivery. + Deployment frequency is one of the four DORA metrics that DevOps teams use for measuring excellence in software delivery. The deployment frequency chart is available for groups and projects. @@ -56,7 +56,7 @@ To view the deployment frequency chart:  -## View lead time for changes chart **(ULTIMATE)** +## View DORA lead time for changes chart **(ULTIMATE)** > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/250329) in GitLab 13.11. @@ -68,7 +68,7 @@ merge requests to be deployed to a production environment. This chart is availab - For time periods in which no merge requests were deployed, the charts render a red, dashed line. - Lead time for changes is one of the four [DORA metrics](index.md#devops-research-and-assessment-dora-key-metrics) that DevOps teams use for measuring excellence in software delivery. + Lead time for changes is one of the four DORA metrics that DevOps teams use for measuring excellence in software delivery. To view the lead time for changes chart: @@ -78,13 +78,13 @@ To view the lead time for changes chart:  -## View time to restore service chart **(ULTIMATE)** +## View DORA time to restore service chart **(ULTIMATE)** > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/356959) in GitLab 15.1 The time to restore service chart shows information about the median time an incident was open in a production environment. This chart is available for groups and projects. -Time to restore service is one of the four [DORA metrics](index.md#devops-research-and-assessment-dora-key-metrics) that DevOps teams use for measuring excellence in software delivery. +Time to restore service is one of the four DORA metrics that DevOps teams use for measuring excellence in software delivery. To view the time to restore service chart: @@ -94,13 +94,13 @@ To view the time to restore service chart:  -## View change failure rate chart **(ULTIMATE)** +## View DORA change failure rate chart **(ULTIMATE)** > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/357072) in GitLab 15.2 The change failure rate chart shows information about the percentage of deployments that cause an incident in a production environment. This chart is available for groups and projects. -Change failure rate is one of the four [DORA metrics](index.md#devops-research-and-assessment-dora-key-metrics) that DevOps teams use for measuring excellence in software delivery. +Change failure rate is one of the four DORA metrics that DevOps teams use for measuring excellence in software delivery. To view the change failure rate chart: diff --git a/doc/user/analytics/dora_metrics.md b/doc/user/analytics/dora_metrics.md new file mode 100644 index 00000000000..b5f37203817 --- /dev/null +++ b/doc/user/analytics/dora_metrics.md @@ -0,0 +1,127 @@ +--- +stage: Plan +group: Optimize +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments +--- + +# DevOps Research and Assessment (DORA) metrics **(ULTIMATE)** + +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/275991) in GitLab 13.7. +> - [Added support](https://gitlab.com/gitlab-org/gitlab/-/issues/291746) for lead time for changes in GitLab 13.10. + +The [DevOps Research and Assessment (DORA)](https://cloud.google.com/blog/products/devops-sre/using-the-four-keys-to-measure-your-devops-performance) +team has identified four metrics that measure DevOps performance. +Using these metrics helps improve DevOps efficiency and communicate performance to business stakeholders, which can accelerate business results. + +DORA includes four key metrics, divided into two core areas of DevOps: + +- [Deployment Frequency](#deployment-frequency) and [Lead Time for Change](#lead-time-for-changes) measure team velocity. +- [Change Failure Rate](#change-failure-rate) and [Time to Restore Service](#time-to-restore-service) measure stability. + +For software leaders, tracking velocity alongside quality metrics ensures they're not sacrificing quality for speed. + +<div class="video-fallback"> + For an overview, see <a href="https://www.youtube.com/watch?v=1BrcMV6rCDw">GitLab Speed Run: DORA metrics in GitLab One DevOps Platform</a>. +</div> +<figure class="video-container"> + <iframe src="https://www.youtube.com/embed/1BrcMV6rCDw" frameborder="0" allowfullscreen="true"> </iframe> +</figure> + +## DORA Metrics dashboard in Value Stream Analytics + +The four DORA metrics are available out-of-the-box in the [Value Stream Analytics (VSA) overview dashboard](../group/value_stream_analytics/index.md#view-dora-metrics-and-key-metrics-for-a-group). +This helps you visualize the engineering work in the context of end-to-end value delivery. + +The One DevOps Platform [Value Stream Management](https://gitlab.com/gitlab-org/gitlab/-/value_stream_analytics) provides end-to-end visibility to the entire software delivery lifecycle. +This enables teams and managers to understand all aspects of productivity, quality, and delivery, without the ["toolchain tax"](https://about.gitlab.com/solutions/value-stream-management/). + +## Deployment frequency + +Deployment frequency is the frequency of successful deployments to production (hourly, daily, weekly, monthly, or yearly). +This measures how often you deliver value to end users. A higher deployment frequency means you can +get feedback sooner and iterate faster to deliver improvements and features. GitLab measures this as the number of +deployments to a production environment in the given time period. + +Deployment frequency displays in several charts: + +- [Group-level value stream analytics](../group/value_stream_analytics/index.md) +- [Project-level value stream analytics](value_stream_analytics.md) +- [CI/CD analytics](ci_cd_analytics.md) + +To retrieve metrics for deployment frequency, use the [GraphQL](../../api/graphql/reference/index.md) or the [REST](../../api/dora/metrics.md) APIs. + +## Lead time for changes + +DORA Lead time for changes measures the time to successfully deliver a commit into production. +This metric reflects the efficiency of CI/CD pipelines. + +In GitLab, Lead time for changes calculates the median time it takes for a merge request to get merged into production. +We measure **from** code committed **to** code successfully running in production, without adding the `coding_time` to the calculation. + +Over time, the lead time for changes should decrease, while your team's performance should increase. + +Lead time for changes displays in several charts: + +- [Group-level value stream analytics](../group/value_stream_analytics/index.md) +- [Project-level value stream analytics](value_stream_analytics.md) +- [CI/CD analytics](ci_cd_analytics.md) + +To retrieve metrics for lead time for changes, use the [GraphQL](../../api/graphql/reference/index.md) or the [REST](../../api/dora/metrics.md) APIs. + +- The definition of lead time for change can vary widely, which often creates confusion within the industry. +- "Lead time for changes" is not the same as "Lead time". In the value stream, "Lead time" measures the time it takes for work on an issue to move from the moment it's requested (Issue created) to the moment it's fulfilled and delivered (Issue closed). + +## Time to restore service + +Time to restore service measures how long it takes an organization to recover from a failure in production. +GitLab measures this as the average time required to close the incidents +in the given time period. This assumes: + +- All incidents are related to a production environment. +- Incidents and deployments have a strictly one-to-one relationship. An incident is related to only +one production deployment, and any production deployment is related to no more than one incident). + +Time to restore service displays in several charts: + +- [Group-level value stream analytics](../group/value_stream_analytics/index.md) +- [Project-level value stream analytics](value_stream_analytics.md) +- [CI/CD analytics](ci_cd_analytics.md) + +To retrieve metrics for time to restore service, use the [GraphQL](../../api/graphql/reference/index.md) or the [REST](../../api/dora/metrics.md) APIs. + +## Change failure rate + +Change failure rate measures the percentage of deployments that cause a failure in production. GitLab measures this as the number +of incidents divided by the number of deployments to a +production environment in the given time period. This assumes: + +- All incidents are related to a production environment. +- Incidents and deployments have a strictly one-to-one relationship. An incident is related to only +one production deployment, and any production deployment is related to no +more than one incident. + +To retrieve metrics for change failure rate, use the [GraphQL](../../api/graphql/reference/index.md) or the [REST](../../api/dora/metrics.md) APIs. + +### Insights: Custom DORA reporting + +Custom charts to visualize DORA data with Insights YAML-based reports. + +With this new visualization, software leaders can track metrics improvements, understand patterns in their metrics trends, and compare performance between groups and projects. + +### Measure DORA metrics without using GitLab CI/CD pipelines + +Deployment frequency is calculated based on the deployments record, which is created for typical push-based deployments. +These deployment records are not created for pull-based deployments, for example when Container Images are connected to GitLab with an agent. + +To track DORA metrics in these cases, you can [create a deployment record](../../api/deployments.md#create-a-deployment) using the Deployments API. + +### Supported DORA metrics in GitLab + +| Metric | Level | API | UI chart | Comments | +|---------------------------|-------------------|-----------------------------------------------------|------------------------|----------| +| `deployment_frequency` | Project | [GitLab 13.7 and later](../../api/dora/metrics.md) | GitLab 14.8 and later | The previous API endpoint was [deprecated](https://gitlab.com/gitlab-org/gitlab/-/issues/323713) in 13.10. | +| `deployment_frequency` | Group | [GitLab 13.10 and later](../../api/dora/metrics.md) | GitLab 13.12 and later | | +| `lead_time_for_changes` | Project | [GitLab 13.10 and later](../../api/dora/metrics.md) | GitLab 13.11 and later | Unit in seconds. Aggregation method is median. | +| `lead_time_for_changes` | Group | [GitLab 13.10 and later](../../api/dora/metrics.md) | GitLab 14.0 and later | Unit in seconds. Aggregation method is median. | +| `time_to_restore_service` | Project and group | [GitLab 14.9 and later](../../api/dora/metrics.md) | GitLab 15.1 and later | Unit in days. Aggregation method is median. | +| `change_failure_rate` | Project and group | [GitLab 14.10 and later](../../api/dora/metrics.md) | GitLab 15.2 and later | Percentage of deployments. | diff --git a/doc/user/analytics/index.md b/doc/user/analytics/index.md index 56adbff9f59..01fe466755c 100644 --- a/doc/user/analytics/index.md +++ b/doc/user/analytics/index.md @@ -34,7 +34,7 @@ GitLab provides several analytics features at the group level. Some of these fea You can use GitLab to review analytics at the project level. Some of these features require you to use a higher tier than GitLab Free. - [Application Security](../application_security/security_dashboard/index.md) -- [CI/CD](ci_cd_analytics.md) +- [CI/CD & DORA](ci_cd_analytics.md) - [Code Review](code_review_analytics.md) - [Insights](../project/insights/index.md) - [Issue](../group/issues_analytics/index.md) @@ -51,93 +51,6 @@ The following analytics features are available for users to create personalized Be sure to review the documentation page for this feature for GitLab tier requirements. -## DevOps Research and Assessment (DORA) key metrics **(ULTIMATE)** - -> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/275991) in GitLab 13.7. -> - [Added support](https://gitlab.com/gitlab-org/gitlab/-/issues/291746) for lead time for changes in GitLab 13.10. - -The [DevOps Research and Assessment (DORA)](https://cloud.google.com/blog/products/devops-sre/using-the-four-keys-to-measure-your-devops-performance) -team developed several key metrics that you can use as performance indicators for software development -teams. - -### Deployment frequency - -Deployment frequency is the frequency of successful deployments to production (hourly, daily, weekly, monthly, or yearly). -This measures how often you deliver value to end users. A higher deployment frequency means you can -get feedback sooner and iterate faster to deliver improvements and features. GitLab measures this as the number of -deployments to a production environment in the given time period. - -Deployment frequency displays in several charts: - -- [Group-level value stream analytics](../group/value_stream_analytics/index.md) -- [Project-level value stream analytics](value_stream_analytics.md) -- [CI/CD analytics](ci_cd_analytics.md) - -To retrieve metrics for deployment frequency, use the [GraphQL](../../api/graphql/reference/index.md) or the [REST](../../api/dora/metrics.md) APIs. - -### Lead time for changes - -DORA Lead time for changes measures the time to successfully deliver a feature into production. -This metric reflects the efficiency of CI/CD pipelines. - -In GitLab, Lead time for changes calculates the median time it takes for a merge request to get merged into production. -We measure "FROM code committed TO code successfully running in production" without adding the "coding_time" to the calculation. - -Over time, the lead time for changes should decrease, while your team's performance should increase. - -Lead time for changes displays in several charts: - -- [Group-level value stream analytics](../group/value_stream_analytics/index.md) -- [Project-level value stream analytics](value_stream_analytics.md) -- [CI/CD analytics](ci_cd_analytics.md) - -To retrieve metrics for lead time for changes, use the [GraphQL](../../api/graphql/reference/index.md) or the [REST](../../api/dora/metrics.md) APIs. - -- The definition of lead time for change can vary widely, which often creates confusion within the industry. -- "Lead time for changes" is not the same as "Lead time". In the value stream, "Lead time" measures the time it takes for work on issue to move from the moment it's requested (Issue created) to the time it's fulfilled and delivered (Issue closed). - -### Time to restore service - -Time to restore service measures how long it takes an organization to recover from a failure in production. -GitLab measures this as the average time required to close the incidents -in the given time period. This assumes: - -- All incidents are related to a production environment. -- Incidents and deployments have a strictly one-to-one relationship. An incident is related to only -one production deployment, and any production deployment is related to no more than one incident). - -Time to restore service displays in several charts: - -- [Group-level value stream analytics](../group/value_stream_analytics/index.md) -- [Project-level value stream analytics](value_stream_analytics.md) -- [CI/CD analytics](ci_cd_analytics.md) - -To retrieve metrics for time to restore service, use the [GraphQL](../../api/graphql/reference/index.md) or the [REST](../../api/dora/metrics.md) APIs. - -### Change failure rate - -Change failure rate measures the percentage of deployments that cause a failure in production. GitLab measures this as the number -of incidents divided by the number of deployments to a -production environment in the given time period. This assumes: - -- All incidents are related to a production environment. -- Incidents and deployments have a strictly one-to-one relationship. An incident is related to only -one production deployment, and any production deployment is related to no -more than one incident. - -To retrieve metrics for change failure rate, use the [GraphQL](../../api/graphql/reference/index.md) or the [REST](../../api/dora/metrics.md) APIs. - -### Supported DORA metrics in GitLab - -| Metric | Level | API | UI chart | Comments | -|---------------------------|-------------------|-----------------------------------------------------|------------------------|----------| -| `deployment_frequency` | Project | [GitLab 13.7 and later](../../api/dora/metrics.md) | GitLab 14.8 and later | The previous API endpoint was [deprecated](https://gitlab.com/gitlab-org/gitlab/-/issues/323713) in 13.10. | -| `deployment_frequency` | Group | [GitLab 13.10 and later](../../api/dora/metrics.md) | GitLab 13.12 and later | | -| `lead_time_for_changes` | Project | [GitLab 13.10 and later](../../api/dora/metrics.md) | GitLab 13.11 and later | Unit in seconds. Aggregation method is median. | -| `lead_time_for_changes` | Group | [GitLab 13.10 and later](../../api/dora/metrics.md) | GitLab 14.0 and later | Unit in seconds. Aggregation method is median. | -| `time_to_restore_service` | Project and group | [GitLab 14.9 and later](../../api/dora/metrics.md) | GitLab 15.1 and later | Unit in days. Aggregation method is median. | -| `change_failure_rate` | Project and group | [GitLab 14.10 and later](../../api/dora/metrics.md) | GitLab 15.2 and later | Percentage of deployments. | - ## Definitions We use the following terms to describe GitLab analytics: diff --git a/doc/user/application_security/api_fuzzing/create_har_files.md b/doc/user/application_security/api_fuzzing/create_har_files.md index 445bcfb0444..e7eaf47121b 100644 --- a/doc/user/application_security/api_fuzzing/create_har_files.md +++ b/doc/user/application_security/api_fuzzing/create_har_files.md @@ -93,7 +93,7 @@ used in [Web API Fuzz Testing](index.md#http-archive-har). 1. Select **Export Data > Current Workspace**. 1. Select requests to include in the HAR file. 1. Select **Export**. -1. In the **Select Export Type** dropdown select **HAR -- HTTP Archive Format**. +1. In the **Select Export Type** dropdown list select **HAR -- HTTP Archive Format**. 1. Select **Done**. 1. Enter a location and filename for the HAR file. @@ -109,7 +109,7 @@ responses in HAR format. have an account, first create an account. 1. Browse pages that call an API. Fiddler automatically captures the requests. 1. Select one or more requests, then from the context menu, select **Export > Selected Sessions**. -1. In the **Choose Format** dropdown select **HTTPArchive v1.2**. +1. In the **Choose Format** dropdown list select **HTTPArchive v1.2**. 1. Enter a filename and select **Save**. Fiddler shows a popup message confirming the export has succeeded. diff --git a/doc/user/application_security/api_fuzzing/index.md b/doc/user/application_security/api_fuzzing/index.md index d542c2d4be5..03eed6fdbf8 100644 --- a/doc/user/application_security/api_fuzzing/index.md +++ b/doc/user/application_security/api_fuzzing/index.md @@ -1211,7 +1211,7 @@ Example usage for setting a `body-json` override: } ``` -Note that each JSON property name in the object `body-json` is set to a [JSON Path](https://goessner.net/articles/JsonPath/) +Each JSON property name in the object `body-json` is set to a [JSON Path](https://goessner.net/articles/JsonPath/) expression. The JSON Path expression `$.credentials.access-token` identifies the node to be overridden with the value `iddqd!42.$`. The override engine uses `body-json` when the request body has only [JSON](https://www.json.org/json-en.html) content. @@ -1250,7 +1250,7 @@ the second entry overrides an XML element: } ``` -Note that each JSON property name in the object `body-xml` is set to an +Each JSON property name in the object `body-xml` is set to an [XPath v2](https://www.w3.org/TR/xpath20/) expression. The XPath expression `/credentials/@isEnabled` identifies the attribute node to override with the value `true`. The XPath expression `/credentials/access-token/text()` identifies the @@ -1392,7 +1392,7 @@ It is also possible to write messages from your script to a log file that is col Adding some basic logging to your overrides script is useful in case the script fails unexpectedly during normal running of the job. The log file is automatically included as an artifact of the job, allowing you to download it after the job has finished. -Following our example, we provided `renew_token.py` in the environmental variable `FUZZAPI_OVERRIDES_CMD`. Please notice two things in the script: +Following our example, we provided `renew_token.py` in the environmental variable `FUZZAPI_OVERRIDES_CMD`. Notice two things in the script: - Log file is saved in the location indicated by the environment variable `CI_PROJECT_DIR`. - Log filename should match `gl-*.log`. @@ -1871,7 +1871,7 @@ variables: ##### Excluding two URLs and their child resources -In order to exclude the URLs: `http://target/api/buy` and `http://target/api/sell`, and their child resources. To provide multiple URLs we use the `,` character as follows: +To exclude the URLs: `http://target/api/buy` and `http://target/api/sell`, and their child resources. To provide multiple URLs we use the `,` character as follows: ```yaml stages: @@ -1888,7 +1888,11 @@ variables: ##### Excluding URL using regular expressions -In order to exclude exactly `https://target/api/v1/user/create` and `https://target/api/v2/user/create` or any other version (`v3`,`v4`, and more). We could use `https://target/api/v.*/user/create$`, in the previous regular expression `.` indicates any character and `*` indicates zero or more times, additionally `$` indicates that the URL should end there. +To exclude exactly `https://target/api/v1/user/create` and `https://target/api/v2/user/create` or any other version (`v3`,`v4`, and more), we could use `https://target/api/v.*/user/create$`. In the previous regular expression: + +- `.` indicates any character. +- `*` indicates zero or more times. +- `$` indicates that the URL should end there. ```yaml stages: @@ -2211,9 +2215,235 @@ Setting `SECURE_ANALYZERS_PREFIX` changes the Docker image registry location for For more information, see [Offline environments](../offline_deployments/index.md). +## Performance tuning and testing speed + +Security tools that perform API fuzz testing, such as API Fuzzing, perform testing by sending requests to an instance of your running application. The requests are mutated by our fuzzing engine to trigger unexpected behavior that might exist in your application. The speed of an API fuzzing test depends on the following: + +- How many requests per second can be sent to your application by our tooling +- How fast your application responds to requests +- How many requests must be sent to test the application + - How many operations your API is comprised of + - How many fields are in each operation (think JSON bodies, headers, query string, cookies, etc.) + +If API Fuzzing testing job still takes longer than expected after following the advice in this performance guide, reach out to support for further assistance. + +### Diagnosing performance issues + +The first step to resolving performance issues is to understand what is contributing to the slower-than-expected testing time. Some common issues we see are: + +- API Fuzzing is running on a slow or single-CPU GitLab Runner (GitLab Shared Runners are single-CPU) +- The application deployed to a slow/single-CPU instance and is not able to keep up with the testing load +- The application contains a slow operation that impacts the overall test speed (> 1/2 second) +- The application contains an operation that returns a large amount of data (> 500K+) +- The application contains a large number of operations (> 40) + +#### The application contains a slow operation that impacts the overall test speed (> 1/2 second) + +The API Fuzzing job output contains helpful information about how fast we are testing, how fast each operation being tested responds, and summary information. Let's take a look at some sample output to see how it can be used in tracking down performance issues: + +```shell +API Security: Loaded 10 operations from: assets/har-large-response/large_responses.har +API Security: +API Security: Testing operation [1/10]: 'GET http://target:7777/api/large_response_json'. +API Security: - Parameters: (Headers: 4, Query: 0, Body: 0) +API Security: - Request body size: 0 Bytes (0 bytes) +API Security: +API Security: Finished testing operation 'GET http://target:7777/api/large_response_json'. +API Security: - Excluded Parameters: (Headers: 0, Query: 0, Body: 0) +API Security: - Performed 767 requests +API Security: - Average response body size: 130 MB +API Security: - Average call time: 2 seconds and 82.69 milliseconds (2.082693 seconds) +API Security: - Time to complete: 14 minutes, 8 seconds and 788.36 milliseconds (848.788358 seconds) +``` + +This job console output snippet starts by telling us how many operations were found (10), followed by notifications that testing has started on a specific operation and a summary of the operation has been completed. The summary is the most interesting part of this log output. In the summary, we can see that it took API Fuzzing 767 requests to fully test this operation and its related fields. We can also see that the average response time was 2 seconds and the time to complete was 14 minutes for this one operation. + +An average response time of 2 seconds is a good initial indicator that this specific operation takes a long time to test. Further, we can see that the response body size is quite large. The large body size is the culprit here, transferring that much data on each request is what takes the majority of that 2 seconds. + +For this issue, the team might decide to: + +- Use a multi-CPU runner. Using a multi-CPU runner allows API Fuzzing to parallelize the work being performed. This helps lower the test time, but getting the test down under 10 minutes might still be problematic without moving to a high CPU machine due to how long the operation takes to test. + - Trade off between how many CPUs and cost. +- [Exclude this operation](#excluding-slow-operations) from the API Fuzzing test. While this is the simplest, it has the downside of a gap in security test coverage. +- [Exclude the operation from feature branch API Fuzzing tests, but include it in the default branch test](#excluding-operations-in-feature-branches-but-not-default-branch). +- [Split up the API Fuzzing testing into multiple jobs](#splitting-a-test-into-multiple-jobs). + +The likely solution is to use a combination of these solutions to reach an acceptable test time, assuming your team's requirements are in the 5-7 minute range. + +### Addressing performance issues + +The following sections document various options for addressing performance issues for API Fuzzing: + +- [Using a multi-CPU Runner](#using-a-multi-cpu-runner) +- [Excluding slow operations](#excluding-slow-operations) +- [Splitting a test into multiple jobs](#splitting-a-test-into-multiple-jobs) +- [Excluding operations in feature branches, but not default branch](#excluding-operations-in-feature-branches-but-not-default-branch) + +#### Using a multi-CPU Runner + +One of the easiest performance boosts can be achieved using a multi-CPU runner with API Fuzzing. This table shows statistics collected during benchmarking of a Java Spring Boot REST API. In this benchmark, the target and API Fuzzing share a single runner instance. + +| CPU Count | Request per Second | +|----------------------|--------------------| +| 1 CPU (Shared Runner)| 75 | +| 4 CPU | 255 | +| 8 CPU | 400 | + +As we can see from this table, increasing the CPU count of the runner can have a large impact on testing speed/performance. + +To use a multi-CPU typically requires deploying a self-managed GitLab Runner onto a multi-CPU machine or cloud compute instance. + +When multiple types of GitLab Runners are available for use, the various instances are commonly set up with tags that can be used in the job definition to select a type of runner. + +Here is an example job definition for API Fuzzing that adds a `tags` section with the tag `multi-cpu`. The job automatically extends the job definition included through the API Fuzzing template. + +```yaml +apifuzzer_fuzz: + tags: + - multi-cpu +``` + +To verify that API Fuzzing can detect multiple CPUs in the runner, download the `gl-api-security-scanner.log` file from a completed job's artifacts. Search the file for the string `Starting work item processor` and inspect the reported max DOP (degree of parallelism). The max DOP should be greater than or equal to the number of CPUs assigned to the runner. The value is never lower than 2, even on single CPU runners, unless forced through a configuration variable. If the value reported is less than the number of CPUs assigned to the runner, then something is wrong with the runner deployment. If unable to identify the problem, open a ticket with support to assist. + +Example log entry: + +`17:00:01.084 [INF] <Peach.Web.Core.Services.WebRunnerMachine> Starting work item processor with 2 max DOP` + +#### Excluding slow operations + +In the case of one or two slow operations, the team might decide to skip testing the operations. Excluding the operation is done using the `FUZZAPI_EXCLUDE_PATHS` configuration [variable as explained in this section.](#exclude-paths) + +In this example, we have an operation that returns a large amount of data. The operation is `GET http://target:7777/api/large_response_json`. To exclude it we provide the `FUZZAPI_EXCLUDE_PATHS` configuration variable with the path portion of our operation URL `/api/large_response_json`. + +To verify the operation is excluded, run the API Fuzzing job and review the job console output. It includes a list of included and excluded operations at the end of the test. + +```yaml +apifuzzer_fuzz: + variables: + FUZZAPI_EXCLUDE_PATHS: /api/large_response_json +``` + +Excluding operations from testing could allow some vulnerabilities to go undetected. +{: .alert .alert-warning} + +#### Splitting a test into multiple jobs + +Splitting a test into multiple jobs is supported by API Fuzzing through the use of [`FUZZAPI_EXCLUDE_PATHS`](#exclude-paths) and [`FUZZAPI_EXCLUDE_URLS`](#exclude-urls). When splitting a test up, a good pattern is to disable the `apifuzzer_fuzz` job and replace it with two jobs with identifying names. In this example we have two jobs, each job is testing a version of the API, so our names reflect that. However, this technique can be applied to any situation, not just with versions of an API. + +The rules we are using in the `apifuzzer_v1` and `apifuzzer_v2` jobs are copied from the [API Fuzzing template](https://gitlab.com/gitlab-org/gitlab/blob/master/lib/gitlab/ci/templates/Security/DAST-API.gitlab-ci.yml). + +```yaml +# Disable the main apifuzzer_fuzz job +apifuzzer_fuzz: + rules: + - if: $CI_COMMIT_BRANCH + when: never + +apifuzzer_v1: + extends: apifuzzer_fuzz + variables: + FUZZAPI_EXCLUDE_PATHS: /api/v1/** + rules: + rules: + - if: $API_FUZZING_DISABLED + when: never + - if: $API_FUZZING_DISABLED_FOR_DEFAULT_BRANCH && + $CI_DEFAULT_BRANCH == $CI_COMMIT_REF_NAME + when: never + - if: $CI_COMMIT_BRANCH && + $CI_GITLAB_FIPS_MODE == "true" + variables: + FUZZAPI_IMAGE_SUFFIX: "-fips" + - if: $CI_COMMIT_BRANCH + +apifuzzer_v2: + variables: + FUZZAPI_EXCLUDE_PATHS: /api/v2/** + rules: + rules: + - if: $API_FUZZING_DISABLED + when: never + - if: $API_FUZZING_DISABLED_FOR_DEFAULT_BRANCH && + $CI_DEFAULT_BRANCH == $CI_COMMIT_REF_NAME + when: never + - if: $CI_COMMIT_BRANCH && + $CI_GITLAB_FIPS_MODE == "true" + variables: + FUZZAPI_IMAGE_SUFFIX: "-fips" + - if: $CI_COMMIT_BRANCH +``` + +#### Excluding operations in feature branches, but not default branch + +In the case of one or two slow operations, the team might decide to skip testing the operations, or exclude them from feature branch tests, but include them for default branch tests. Excluding the operation is done using the `FUZZAPI_EXCLUDE_PATHS` configuration [variable as explained in this section.](#exclude-paths) + +In this example, we have an operation that returns a large amount of data. The operation is `GET http://target:7777/api/large_response_json`. To exclude it we provide the `FUZZAPI_EXCLUDE_PATHS` configuration variable with the path portion of our operation URL `/api/large_response_json`. Our configuration disables the main `apifuzzer_fuzz` job and creates two new jobs `apifuzzer_main` and `apifuzzer_branch`. The `apifuzzer_branch` is set up to exclude the long operation and only run on non-default branches (e.g. feature branches). The `apifuzzer_main` branch is set up to only execute on the default branch (`main` in this example). The `apifuzzer_branch` jobs run faster, allowing for quick development cycles, while the `apifuzzer_main` job which only runs on default branch builds, takes longer to run. + +To verify the operation is excluded, run the API Fuzzing job and review the job console output. It includes a list of included and excluded operations at the end of the test. + +```yaml +# Disable the main job so we can create two jobs with +# different names +apifuzzer_fuzz: + rules: + - if: $CI_COMMIT_BRANCH + when: never + +# API Fuzzing for feature branch work, excludes /api/large_response_json +apifuzzer_branch: + extends: apifuzzer_fuzz + variables: + FUZZAPI_EXCLUDE_PATHS: /api/large_response_json + rules: + rules: + - if: $API_FUZZING_DISABLED + when: never + - if: $API_FUZZING_DISABLED_FOR_DEFAULT_BRANCH && + $CI_DEFAULT_BRANCH == $CI_COMMIT_REF_NAME + when: never + - if: $CI_COMMIT_BRANCH && + $CI_GITLAB_FIPS_MODE == "true" + variables: + FUZZAPI_IMAGE_SUFFIX: "-fips" + - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH + when: never + - if: $CI_COMMIT_BRANCH + +# API Fuzzing for default branch (main in our case) +# Includes the long running operations +apifuzzer_main: + extends: apifuzzer_fuzz + rules: + - if: $API_FUZZING_DISABLED + when: never + - if: $API_FUZZING_DISABLED_FOR_DEFAULT_BRANCH && + $CI_DEFAULT_BRANCH == $CI_COMMIT_REF_NAME + when: never + - if: $CI_COMMIT_BRANCH && + $CI_GITLAB_FIPS_MODE == "true" + variables: + FUZZAPI_IMAGE_SUFFIX: "-fips" + - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH +``` + ## Troubleshooting -### `Error waiting for API Security 'http://127.0.0.1:5000' to become available` +### API Fuzzing job times out after N hours + +The top two reasons for the API Fuzzing job timing out are slow operations (> 1 second) and using a single-CPU runner for API Fuzzing (GitLab shared runners are single-CPU). Before you can diagnose the problem further, the job must complete so the output can be analyzed. We recommend to start with a multi-CPU runner first, then exclude portions of your API operations until the job completes and the output can be further reviewed. + +See the following documentation sections for assistance: + +- [Performance tuning and testing speed](#performance-tuning-and-testing-speed) +- [Using a multi-CPU Runner](#using-a-multi-cpu-runner) +- [Excluding operations by path](#exclude-paths) +- [Excluding slow operations](#excluding-slow-operations) + +### API Fuzzing job takes too long to complete + +See [Performance Tuning and Testing Speed](#performance-tuning-and-testing-speed) + +### Error waiting for API Security 'http://127.0.0.1:5000' to become available A bug exists in versions of the API Fuzzing analyzer prior to v1.6.196 that can cause a background process to fail under certain conditions. The solution is to update to a newer version of the API Fuzzing analyzer. @@ -2281,16 +2511,16 @@ For OpenAPI Specifications that are generated automatically validation errors ar 1. Identify the validation errors. 1. Use the [Swagger Editor](https://editor.swagger.io/) to identify validation problems in your specification. The visual nature of the Swagger Editor makes it easier to understand what needs to change. - 1. Alternatively, you can check the log output and look for schema validation warnings. They are prefixed with messages such as `OpenAPI 2.0 schema validation error` or `OpenAPI 3.0.x schema validation error`. Each failed validation provides extra information about `location` and `description`. Note that JSON Schema validation messages might not be easy to understand. This is why we recommend the use of editors to validate schema documents. + 1. Alternatively, you can check the log output and look for schema validation warnings. They are prefixed with messages such as `OpenAPI 2.0 schema validation error` or `OpenAPI 3.0.x schema validation error`. Each failed validation provides extra information about `location` and `description`. JSON Schema validation messages can be complex, and editors can help you validate schema documents. 1. Review the documentation for the OpenAPI generation your framework/tech stack is using. Identify the changes needed to produce a correct OpenAPI document. -1. Once the validation issues are resolved, re-run your pipeline. +1. After the validation issues are resolved, re-run your pipeline. **For manually created OpenAPI Specifications** 1. Identify the validation errors. 1. The simplest solution is to use a visual tool to edit and validate the OpenAPI document. For example the [Swagger Editor](https://editor.swagger.io/) highlights schema errors and possible solutions. - 1. Alternatively, you can check the log output and look for schema validation warnings. They are prefixed with messages such as `OpenAPI 2.0 schema validation error` or `OpenAPI 3.0.x schema validation error`. Each failed validation provides extra information about `location` and `description`. Correct each of the validation failures and then resubmit the OpenAPI doc. Note that JSON Schema validation message might not be easy to understand. This is why we recommend the use of editors to validate document. -1. Once the validation issues are resolved, re-run your pipeline. + 1. Alternatively, you can check the log output and look for schema validation warnings. They are prefixed with messages such as `OpenAPI 2.0 schema validation error` or `OpenAPI 3.0.x schema validation error`. Each failed validation provides extra information about `location` and `description`. Correct each of the validation failures and then resubmit the OpenAPI doc. JSON Schema validation messages can be complex, and editors can help you validate schema documents. +1. After the validation issues are resolved, re-run your pipeline. ### `Failed to start scanner session (version header not found)` @@ -2312,7 +2542,10 @@ The API Fuzzing analyzer outputs an error message when it cannot determine the t There is an order of precedence in which the API Fuzzing analyzer tries to get the target API when checking the different sources. First, it will try to use the `FUZZAPI_TARGET_URL`. If the environment variable has not been set, then the API Fuzzing analyzer will attempt to use the `environment_url.txt` file. If there is no file `environment_url.txt`, the API Fuzzing analyzer will then use the OpenAPI document contents and the URL provided in `FUZZAPI_OPENAPI` (if a URL is provided) to try to compute the target API. -The best-suited solution will depend on whether or not your target API changes for each deployment. In static environments, the target API is the same for each deployment, in this case please refer to the [static environment solution](#static-environment-solution). If the target API changes for each deployment a [dynamic environment solution](#dynamic-environment-solutions) should be applied. +The best-suited solution depends on whether or not your target API changes for each deployment: + +- If the target API is the same for each deployment (a static environment), use the [static environment solution](#static-environment-solution). +- If the target API changes for each deployment, use a [dynamic environment solution](#dynamic-environment-solutions). #### Static environment solution @@ -2417,10 +2650,10 @@ API Fuzzing uses the specified media types in the OpenAPI document to generate r ## Get support or request an improvement -To get support for your particular problem please use the [getting help channels](https://about.gitlab.com/get-help/). +To get support for your particular problem use the [getting help channels](https://about.gitlab.com/get-help/). The [GitLab issue tracker on GitLab.com](https://gitlab.com/gitlab-org/gitlab/-/issues) is the right place for bugs and feature proposals about API Security and API Fuzzing. -Please use `~"Category:API Security"` [label](../../../development/contributing/issue_workflow.md#labels) when opening a new issue regarding API fuzzing to ensure it is quickly reviewed by the right people. Please refer to our [review response SLO](https://about.gitlab.com/handbook/engineering/workflow/code-review/#review-response-slo) to understand when you should receive a response. +Use `~"Category:API Security"` [label](../../../development/contributing/issue_workflow.md#labels) when opening a new issue regarding API fuzzing to ensure it is quickly reviewed by the right people. Refer to our [review response SLO](https://about.gitlab.com/handbook/engineering/workflow/code-review/#review-response-slo) to understand when you should receive a response. [Search the issue tracker](https://gitlab.com/gitlab-org/gitlab/-/issues) for similar entries before submitting your own, there's a good chance somebody else had the same issue or feature proposal. Show your support with an award emoji and or join the discussion. @@ -2432,7 +2665,7 @@ When experiencing a behavior not working as expected, consider providing context - Scanner log file available as a job artifact named `gl-api-security-scanner.log`. WARNING: -**Sanitize data attached to a support issue**. Please remove sensitive information, including: credentials, passwords, tokens, keys, and secrets. +**Sanitize data attached to a support issue**. Remove sensitive information, including: credentials, passwords, tokens, keys, and secrets. ## Glossary diff --git a/doc/user/application_security/configuration/index.md b/doc/user/application_security/configuration/index.md index 5eb1b93eb76..d9ba4640855 100644 --- a/doc/user/application_security/configuration/index.md +++ b/doc/user/application_security/configuration/index.md @@ -55,7 +55,7 @@ You can configure the following security controls: - [Dynamic Application Security Testing](../dast/index.md) (DAST) - Select **Enable DAST** to configure DAST for the current project. - Select **Manage scans** to manage the saved DAST scans, site profiles, and scanner profiles. - For more details, read [DAST on-demand scans](../dast/index.md#on-demand-scans). + For more details, read [DAST on-demand scans](../dast/proxy-based.md#on-demand-scans). - [Dependency Scanning](../dependency_scanning/index.md) - Select **Configure with a merge request** to create a merge request with the changes required to enable Dependency Scanning. For more details, see [Enable Dependency Scanning via an automatic merge request](../dependency_scanning/index.md#enable-dependency-scanning-via-an-automatic-merge-request). diff --git a/doc/user/application_security/container_scanning/index.md b/doc/user/application_security/container_scanning/index.md index f6bd6157a28..6fc01a716b2 100644 --- a/doc/user/application_security/container_scanning/index.md +++ b/doc/user/application_security/container_scanning/index.md @@ -173,7 +173,7 @@ container_scanning: before_script: - ruby -r open-uri -e "IO.copy_stream(URI.open('https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip'), 'awscliv2.zip')" - unzip awscliv2.zip - - ./aws/install + - sudo ./aws/install - aws --version - export AWS_ECR_PASSWORD=$(aws ecr get-login-password --region region) @@ -259,7 +259,7 @@ including a large number of false positives. | `CS_IMAGE_SUFFIX` | `""` | Suffix added to `CS_ANALYZER_IMAGE`. If set to `-fips`, `FIPS-enabled` image is used for scan. See [FIPS-enabled images](#fips-enabled-images) for more details. [Introduced](https://gitlab.com/groups/gitlab-org/-/epics/7630) in GitLab 14.10. | All | | `CS_IGNORE_UNFIXED` | `"false"` | Ignore vulnerabilities that are not fixed. | All | | `CS_REGISTRY_INSECURE` | `"false"` | Allow access to insecure registries (HTTP only). Should only be set to `true` when testing the image locally. Works with all scanners, but the registry must listen on port `80/tcp` for Trivy to work. | All | -| `CS_SEVERITY_THRESHOLD` | `UNKNOWN` | Severity level threshold. The scanner outputs vulnerabilities with severity level higher than or equal to this threshold. Supported levels are Unknown, Low, Medium, High, and Critical. | Trivy | +| `CS_SEVERITY_THRESHOLD` | `UNKNOWN` | Severity level threshold. The scanner outputs vulnerabilities with severity level higher than or equal to this threshold. Supported levels are `UNKNOWN`, `LOW`, `MEDIUM`, `HIGH`, and `CRITICAL`. | Trivy | | <!-- start_remove The following content will be removed on remove_date: '2023-08-22' --> `DOCKER_IMAGE` | `$CI_APPLICATION_REPOSITORY:$CI_APPLICATION_TAG` | **Deprecated** will be removed in GitLab 16.0. Replaced by `CS_IMAGE`. The Docker image to be scanned. If set, this variable overrides the `$CI_APPLICATION_REPOSITORY` and `$CI_APPLICATION_TAG` variables. | All | | `DOCKER_PASSWORD` | `$CI_REGISTRY_PASSWORD` | **Deprecated** will be removed in GitLab 16.0. Replaced by `CS_REGISTRY_PASSWORD`. Password for accessing a Docker registry requiring authentication. The default is only set if `$DOCKER_IMAGE` resides at [`$CI_REGISTRY`](../../../ci/variables/predefined_variables.md). Not supported when [FIPS mode](../../../development/fips_compliance.md#enable-fips-mode) is enabled. | All | | `DOCKER_USER` | `$CI_REGISTRY_USER` | **Deprecated** will be removed in GitLab 16.0. Replaced by `CS_REGISTRY_USER`. Username for accessing a Docker registry requiring authentication. The default is only set if `$DOCKER_IMAGE` resides at [`$CI_REGISTRY`](../../../ci/variables/predefined_variables.md). Not supported when [FIPS mode](../../../development/fips_compliance.md#enable-fips-mode) is enabled. | All | @@ -359,46 +359,12 @@ The container-scanning analyzer can use different scanners, depending on the val The following options are available: -| Scanner name | `CS_ANALYZER_IMAGE` | -| ------------ | ------------------- | -| Default ([Trivy](https://github.com/aquasecurity/trivy)) | `registry.gitlab.com/security-products/container-scanning:5` | +| Scanner name | `CS_ANALYZER_IMAGE` | +|----------------------------------------------------------|--------------------------------------------------------------------| +| Default ([Trivy](https://github.com/aquasecurity/trivy)) | `registry.gitlab.com/security-products/container-scanning:5` | | [Grype](https://github.com/anchore/grype) | `registry.gitlab.com/security-products/container-scanning/grype:5` | | Trivy | `registry.gitlab.com/security-products/container-scanning/trivy:5` | -If you're migrating from a GitLab 13.x release to a GitLab 14.x release and have customized the -`container_scanning` job or its CI variables, you might need to perform these migration steps in -your CI file: - -1. Remove these variables: - - - `CS_MAJOR_VERSION` - - `CS_PROJECT` - - `SECURE_ANALYZERS_PREFIX` - -1. Review the `CS_ANALYZER_IMAGE` variable. It no longer depends on the variables above and its new - default value is `registry.gitlab.com/security-products/container-scanning:5`. If you have an - offline environment, see - [Running container scanning in an offline environment](#running-container-scanning-in-an-offline-environment). - -1. If present, remove the `.cs_common` and `container_scanning_new` configuration sections. - -1. If the `container_scanning` section is present, it's safer to create one from scratch based on - the new version of the [`Container-Scanning.gitlab-ci.yml` template](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/ci/templates/Security/Container-Scanning.gitlab-ci.yml). - Once finished, it should not have any variables that are only applicable to Klar or Clair. For a - complete list of supported variables, see [available variables](#available-cicd-variables). - -1. Make any [necessary customizations](#customizing-the-container-scanning-settings) - to the chosen scanner. We recommend that you minimize such customizations, as they might require - changes in future GitLab major releases. - -1. Trigger a new run of a pipeline that includes the `container_scanning` job. Inspect the job - output and ensure that the log messages do not mention Clair. - -NOTE: -Prior to the GitLab 14.0 release, any variable defined under the scope `container_scanning` is not -considered for scanners other than Clair. In GitLab 14.0 and later, all variables can be defined -either as a global variable or under `container_scanning`. - ### Setting the default branch image > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/338877) in GitLab 14.5. diff --git a/doc/user/application_security/dast/browser_based.md b/doc/user/application_security/dast/browser_based.md index 5a4acc78728..c0a97c0ff92 100644 --- a/doc/user/application_security/dast/browser_based.md +++ b/doc/user/application_security/dast/browser_based.md @@ -36,8 +36,8 @@ current DAST scanner is much more effective at finding and testing every page in To enable the browser-based analyzer: 1. Ensure the DAST [prerequisites](index.md#prerequisites) are met. -1. Include the [DAST CI/CD template](index.md#include-the-dast-template). -1. Set the target website using the [`DAST_WEBSITE` CI/CD variable](index.md#available-cicd-variables). +1. Include the [DAST CI/CD template](proxy-based.md#include-the-dast-template). +1. Set the target website using the [`DAST_WEBSITE` CI/CD variable](proxy-based.md#available-cicd-variables). 1. Set the CI/CD variable `DAST_BROWSER_SCAN` to `true`. Example extract of `.gitlab-ci.yml` file: @@ -79,7 +79,7 @@ The browser-based crawler can be configured using CI/CD variables. | `DAST_BROWSER_ELEMENT_TIMEOUT` | [Duration string](https://pkg.go.dev/time#ParseDuration) | `600ms` | The maximum amount of time to wait for an element before determining it is ready for analysis. | | `DAST_BROWSER_PAGE_READY_SELECTOR` | selector | `css:#page-is-ready` | Selector that when detected as visible on the page, indicates to the analyzer that the page has finished loading and the scan can continue. Note: When this selector is set, but the element is not found, the scanner waits for the period defined in `DAST_BROWSER_STABILITY_TIMEOUT` before continuing the scan. This can significantly increase scanning time if the element is not present on multiple pages within the site. | -The [DAST variables](index.md#available-cicd-variables) `SECURE_ANALYZERS_PREFIX`, `DAST_FULL_SCAN_ENABLED`, `DAST_AUTO_UPDATE_ADDONS`, `DAST_EXCLUDE_RULES`, `DAST_REQUEST_HEADERS`, `DAST_HTML_REPORT`, `DAST_MARKDOWN_REPORT`, `DAST_XML_REPORT`, +The [DAST variables](proxy-based.md#available-cicd-variables) `SECURE_ANALYZERS_PREFIX`, `DAST_FULL_SCAN_ENABLED`, `DAST_AUTO_UPDATE_ADDONS`, `DAST_EXCLUDE_RULES`, `DAST_REQUEST_HEADERS`, `DAST_HTML_REPORT`, `DAST_MARKDOWN_REPORT`, `DAST_XML_REPORT`, `DAST_AUTH_URL`, `DAST_USERNAME`, `DAST_PASSWORD`, `DAST_USERNAME_FIELD`, `DAST_PASSWORD_FIELD`, `DAST_FIRST_SUBMIT_FIELD`, `DAST_SUBMIT_FIELD`, `DAST_EXCLUDE_URLS`, `DAST_AUTH_VERIFICATION_URL`, `DAST_BROWSER_AUTH_VERIFICATION_SELECTOR`, `DAST_BROWSER_AUTH_VERIFICATION_LOGIN_FORM`, `DAST_BROWSER_AUTH_REPORT`, `DAST_INCLUDE_ALPHA_VULNERABILITIES`, `DAST_PATHS_FILE`, `DAST_PATHS`, `DAST_ZAP_CLI_OPTIONS`, and `DAST_ZAP_LOG_CONFIGURATION` are also compatible with browser-based crawler scans. diff --git a/doc/user/application_security/dast/dast_troubleshooting.md b/doc/user/application_security/dast/dast_troubleshooting.md index 194761797de..61a7520bf7c 100644 --- a/doc/user/application_security/dast/dast_troubleshooting.md +++ b/doc/user/application_security/dast/dast_troubleshooting.md @@ -20,7 +20,7 @@ A DAST job has two executing processes: Enable the `DAST_DEBUG` CI/CD variable to debug scripts. This can help when troubleshooting the job, and outputs statements indicating what percentage of the scan is complete. -For details on using variables, see [Overriding the DAST template](index.md#customize-dast-settings). +For details on using variables, see [Overriding the DAST template](proxy-based.md#customize-dast-settings). Debug mode of the ZAP server can be enabled using the `DAST_ZAP_LOG_CONFIGURATION` variable. The following table outlines examples of values that can be set and the effect that they have on the output that is logged. diff --git a/doc/user/application_security/dast/index.md b/doc/user/application_security/dast/index.md index 6d1b7beefc7..d78a8fca98f 100644 --- a/doc/user/application_security/dast/index.md +++ b/doc/user/application_security/dast/index.md @@ -13,16 +13,7 @@ application server or incorrect assumptions about security controls may not be visible from the source code. Dynamic Application Security Testing (DAST) examines applications for -vulnerabilities like these in deployed environments. DAST uses the open source -tool [OWASP Zed Attack Proxy](https://www.zaproxy.org/) for analysis. - -After DAST creates its report, GitLab evaluates it for discovered -vulnerabilities between the source and target branches. Relevant -findings are noted in the merge request. - -The comparison logic uses only the latest pipeline executed for the target -branch's base commit. Running the pipeline on other commits has no effect on -the merge request. +vulnerabilities like these in deployed environments. NOTE: To learn how four of the top six attacks were application-based and how @@ -30,16 +21,88 @@ to protect your organization, download our ["A Seismic Shift in Application Security"](https://about.gitlab.com/resources/whitepaper-seismic-shift-application-security/) whitepaper. -## DAST application analysis +## GitLab DAST + +GitLab provides the following DAST analyzers, one or more of which may be useful depending on the kind of application you're testing. + +For scanning websites, use one of: + +- The [DAST proxy-based analyzer](proxy-based.md) for scanning traditional applications serving simple HTML. The proxy-based analyzer can be run automatically or on-demand. +- The [DAST browser-based analyzer](browser_based.md) for scanning applications that make heavy use of JavaScript. This includes single page web applications. + +For scanning APIs, use: + +- The [DAST API analyzer](../dast_api/index.md) for scanning web APIs. Web API technologies such as GraphQL, REST, and SOAP are supported. + +Analyzers follow the architectural patterns described in [Secure your application](../index.md). +Each analyzer can be configured in the pipeline using a CI template and runs the scan in a Docker container. Scans output a [DAST report artifact](../../../ci/yaml/artifacts_reports.md#artifactsreportsdast) +which GitLab uses to determine discovered vulnerabilities based on differences between scan results on the source and target branches. + +### Getting started + +#### Prerequisites + +- [GitLab Runner](../../../ci/runners/index.md) available, with the + [`docker` executor](https://docs.gitlab.com/runner/executors/docker.html) on Linux/amd64. +- Target application deployed. For more details, read [Deployment options](#application-deployment-options). +- `dast` stage added to the CI/CD pipeline definition. This should be added after the deploy step, for example: + + ```yaml + stages: + - build + - test + - deploy + - dast + ``` + +#### Recommendations + +- Take care if your pipeline is configured to deploy to the same web server in each run. Running a DAST scan while a server is being updated will lead to inaccurate and non-deterministic results. +- Configure runners to use the [always pull policy](https://docs.gitlab.com/runner/executors/docker.html#using-the-always-pull-policy) to run the latest versions of the analyzers. +- By default, DAST downloads all artifacts defined by previous jobs in the pipeline. If + your DAST job does not rely on `environment_url.txt` to define the URL under test or any other files created + in previous jobs, we recommend you don't download artifacts. To avoid downloading + artifacts, extend the analyzer CI/CD job to specify no dependencies. For example, for the DAST proxy-based analyzer add the following to your `.gitlab-ci.yml` file: + + ```yaml + dast: + dependencies: [] + ``` + +#### Analyzer configuration + +Please see [DAST proxy-based analyzer](proxy-based.md), [DAST browser-based analyzer](browser_based.md) or [DAST API analyzer](../dast_api/index.md) for +analyzer-specific configuration instructions. + +### View scan results -DAST can analyze applications in two ways: +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/36332) in GitLab 13.1. + +Detected vulnerabilities are shown in [Merge requests](../index.md#view-security-scan-information-in-merge-requests), the [Pipeline security tab](../index.md#view-security-scan-information-in-the-pipeline-security-tab), +and the [Vulnerability report](../index.md#view-security-scan-information-in-the-vulnerability-report). -- Passive scan only (DAST default). DAST executes - [ZAP's Baseline Scan](https://www.zaproxy.org/docs/docker/baseline-scan/) and doesn't - actively attack your application. -- Passive and active scan. DAST can be [configured](#full-scan) to also perform an active scan - to attack your application and produce a more extensive security report. It can be very - useful when combined with [Review Apps](../../../ci/review_apps/index.md). +1. To see all vulnerabilities detected, either: + - From your project, select **Security & Compliance**, then **Vulnerability report**. + - From your pipeline, click on the **Security** tab. + - From the merge request, go to the **Security scanning** widget and click **Full report** tab. + +1. Select a DAST vulnerability's description. The following fields are examples of what a DAST analyzer may produce to aid investigation and rectification of the underlying cause. Each analyzer may output different fields. + + | Field | Description | + |:------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|:------------------------------------------------------------------ | + | Description | Description of the vulnerability. | + | Evidence | Evidence of the data found that verified the vulnerability. Often a snippet of the request or response, this can be used to help verify that the finding is a vulnerability. | + | Identifiers | Identifiers of the vulnerability. | + | Links | Links to further details of the detected vulnerability. | + | Method | HTTP method used to detect the vulnerability. | + | Project | Namespace and project in which the vulnerability was detected. | + | Request Headers | Headers of the request. | + | Response Headers | Headers of the response received from the application. | + | Response Status | Response status received from the application. | + | Scanner Type | Type of vulnerability report. | + | Severity | Severity of the vulnerability. | + | Solution | Details of a recommended solution to the vulnerability. | + | URL | URL at which the vulnerability was detected. | NOTE: A pipeline may consist of multiple jobs, including SAST and DAST scanning. If any job @@ -48,17 +111,19 @@ example, if the DAST job finishes but the SAST job fails, the security dashboard results. On failure, the analyzer outputs an [exit code](../../../development/integrations/secure.md#exit-code). -## Prerequisites +#### List URLs scanned -- [GitLab Runner](../../../ci/runners/index.md) available, with the -[`docker` executor](https://docs.gitlab.com/runner/executors/docker.html) on Linux/amd64. -- Target application deployed. For more details, read [Deployment options](#deployment-options). -- DAST runs in the `dast` stage, which must be added manually to your `.gitlab-ci.yml`. +When DAST completes scanning, the merge request page states the number of URLs scanned. +Select **View details** to view the web console output which includes the list of scanned URLs. + + -### Deployment options +### Application deployment options + +DAST requires a deployed application to be available to scan. Depending on the complexity of the target application, there are a few options as to how to deploy and configure -the DAST template. We provided a set of example applications with their configurations in our +the DAST template. A set of example applications have been provided with their configurations in the [DAST demonstrations](https://gitlab.com/gitlab-org/security-products/demos/dast/) project. #### Review Apps @@ -77,6 +142,8 @@ After your Docker build job completes and your image is added to your container By using service definitions in your `.gitlab-ci.yml`, you can scan services with the DAST analyzer. +When adding a `services` section to the job, the `alias` is used to define the hostname that can be used to access the service. In the following example, the `alias: yourapp` portion of the `dast` job definition means that the URL to the deployed application will use `yourapp` as the hostname (`https://yourapp/`). + ```yaml stages: - build @@ -105,6 +172,7 @@ dast: alias: yourapp variables: + DAST_WEBSITE: https://yourapp DAST_FULL_SCAN_ENABLED: "true" # do a full scan DAST_BROWSER_SCAN: "true" # use the browser-based GitLab DAST crawler ``` @@ -122,1323 +190,3 @@ services: # use services to link the container to the dast job - name: $CI_REGISTRY_IMAGE:$CI_COMMIT_SHA alias: yourapp ``` - -### DAST job order - -When using the `DAST.gitlab-ci.yml` template, the `dast` stage is run last as shown in -the example below. To ensure DAST scans the latest code, deploy your application -in a stage before the `dast` stage. - -```yaml - stages: - - build - - test - - deploy - - dast -``` - -Take care if your pipeline is configured to deploy to the same web server in each run. Running a -pipeline while another is still running could result in one pipeline overwriting the code from -another pipeline. The site to be scanned should be excluded from changes for the duration of a DAST -scan. The only changes to the site should be from the DAST scanner. - -Changes to the site during a scan from any of the following could lead to inaccurate results: - -- Users. -- Scheduled tasks. -- Database changes. -- Code changes. -- Other pipelines. -- Other scanners. - -## DAST run options - -You can use DAST to examine your web application: - -- Automatically, initiated by a merge request. -- Manually, initiated on demand. - -Some of the differences between these run options: - -| Automatic scan | On-demand scan | -|:-----------------------------------------------------------------|:------------------------------| -| DAST scan is initiated by a merge request. | DAST scan is initiated manually, outside the DevOps life cycle. | -| CI/CD variables are sourced from `.gitlab-ci.yml`. | CI/CD variables are provided in the UI. | -| All [DAST CI/CD variables](#available-cicd-variables) available. | Subset of [DAST CI/CD variables](#available-cicd-variables) available. | -| `DAST.gitlab-ci.yml` template. | `DAST-On-Demand-Scan.gitlab-ci.yml` template. | - -### Enable automatic DAST run - -To enable DAST to run automatically, either: - -- Enable [Auto DAST](../../../topics/autodevops/stages.md#auto-dast) (provided - by [Auto DevOps](../../../topics/autodevops/index.md)). -- [Include the DAST template](#include-the-dast-template) in your existing - `.gitlab-ci.yml` file. -- [Configure DAST using the UI](#configure-dast-using-the-ui). - -#### Include the DAST template - -> - This template was [updated](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/62597) to DAST_VERSION: 2 in GitLab 14.0. -> - This template was [updated](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/87183) to DAST_VERSION: 3 in GitLab 15.0. - -If you want to manually add DAST to your application, the DAST job is defined -in a CI/CD template file. Updates to the template are provided with GitLab -upgrades, allowing you to benefit from any improvements and additions. - -To include the DAST template: - -1. Select the CI/CD template you want to use: - - - [`DAST.gitlab-ci.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/ci/templates/Security/DAST.gitlab-ci.yml): - Stable version of the DAST CI/CD template. - - [`DAST.latest.gitlab-ci.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/ci/templates/Security/DAST.latest.gitlab-ci.yml): - Latest version of the DAST template. ([Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/254325) - in GitLab 13.8). - - WARNING: - The latest version of the template may include breaking changes. Use the - stable template unless you need a feature provided only in the latest template. - - For more information about template versioning, see the - [CI/CD documentation](../../../development/cicd/templates.md#latest-version). - -1. Add a `dast` stage to your GitLab CI stages configuration: - - ```yaml - stages: - - dast - ``` - -1. Add the template to GitLab, based on your version of GitLab: - - - In GitLab 11.9 and later, [include](../../../ci/yaml/index.md#includetemplate) - the template by adding the following to your `.gitlab-ci.yml` file: - - ```yaml - include: - - template: <template_file.yml> - - variables: - DAST_WEBSITE: https://example.com - ``` - - - In GitLab 11.8 and earlier, add the contents of the template to your - `.gitlab_ci.yml` file. - -1. Define the URL to be scanned by DAST by using one of these methods: - - - Set the `DAST_WEBSITE` [CI/CD variable](../../../ci/yaml/index.md#variables). - If set, this value takes precedence. - - - Add the URL in an `environment_url.txt` file at the root of your project. This is - useful for testing in dynamic environments. To run DAST against an application - dynamically created during a GitLab CI/CD pipeline, a job that runs prior to - the DAST scan must persist the application's domain in an `environment_url.txt` - file. DAST automatically parses the `environment_url.txt` file to find its - scan target. - - For example, in a job that runs prior to DAST, you could include code that - looks similar to: - - ```yaml - script: - - echo http://${CI_PROJECT_ID}-${CI_ENVIRONMENT_SLUG}.domain.com > environment_url.txt - artifacts: - paths: [environment_url.txt] - when: always - ``` - - You can see an example of this in our - [Auto DevOps CI YAML](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/ci/templates/Jobs/Deploy.gitlab-ci.yml) - file. - -The included template creates a `dast` job in your CI/CD pipeline and scans -your project's running application for possible vulnerabilities. - -The results are saved as a -[DAST report artifact](../../../ci/yaml/artifacts_reports.md#artifactsreportsdast) -that you can later download and analyze. Due to implementation limitations, we -always take the latest DAST artifact available. Behind the scenes, the -[GitLab DAST Docker image](https://gitlab.com/security-products/dast) -is used to run the tests on the specified URL and scan it for possible -vulnerabilities. - -By default, the DAST template uses the latest major version of the DAST Docker -image. Using the `DAST_VERSION` variable, you can choose how DAST updates: - -- Automatically update DAST with new features and fixes by pinning to a major - version (such as `1`). -- Only update fixes by pinning to a minor version (such as `1.6`). -- Prevent all updates by pinning to a specific version (such as `1.6.4`). - -Find the latest DAST versions on the [Releases](https://gitlab.com/gitlab-org/security-products/dast/-/releases) -page. - -#### Configure DAST using the UI - -You can enable or configure DAST settings using the UI. The generated settings are formatted so they -can be conveniently pasted into the `.gitlab-ci.yml` file. - -1. On the top bar, select **Main menu > Projects** and find your project. -1. On the left sidebar, select **Security & Compliance > Configuration**. -1. In the **Dynamic Application Security Testing (DAST)** section, select **Enable DAST** or - **Configure DAST**. -1. Select the desired **Scanner profile**, or select **Create scanner profile** and save a - scanner profile. For more details, see [scanner profiles](#scanner-profile). -1. Select the desired **Site profile**, or select **Create site profile** and save a site - profile. For more details, see [site profiles](#site-profile). -1. Select **Generate code snippet**. A modal opens with the YAML snippet corresponding to the - options you selected. -1. Do one of the following: - 1. To copy the snippet to your clipboard, select **Copy code only**. - 1. To add the snippet to your project's `.gitlab-ci.yml` file, select - **Copy code and open `.gitlab-ci.yml` file**. The Pipeline Editor opens. - 1. Paste the snippet into the `.gitlab-ci.yml` file. - 1. Select the **Lint** tab to confirm the edited `.gitlab-ci.yml` file is valid. - 1. Select the **Edit** tab, then select **Commit changes**. - -When the snippet is committed to the `.gitlab-ci.yml` file, pipelines include a DAST job. - -#### Crawling web applications dependent on JavaScript - -GitLab has released a new browser-based crawler, an add-on to DAST that uses a browser to crawl web applications for content. This crawler replaces the standard DAST Spider and Ajax Crawler, and uses the same authentication mechanisms as a normal DAST scan. - -The browser-based crawler crawls websites by browsing web pages as a user would. This approach works well with web applications that make heavy use of JavaScript, such as Single Page Applications. - -For more details, including setup instructions, see [DAST browser-based crawler](browser_based.md). - -### Full scan - -DAST can be configured to perform [ZAP Full Scan](https://www.zaproxy.org/docs/docker/full-scan/), which -includes both passive and active scanning against the same target website: - -```yaml -include: - - template: DAST.gitlab-ci.yml - -variables: - DAST_FULL_SCAN_ENABLED: "true" - DAST_BROWSER_SCAN: "true" # use the browser-based GitLab DAST crawler -``` - -If your DAST job exceeds the job timeout and you need to reduce the scan duration, we shared some -tips for optimizing DAST scans in a [blog post](https://about.gitlab.com/blog/2020/08/31/how-to-configure-dast-full-scans-for-complex-web-applications/). - -### API scan - -> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/10928) in GitLab 12.10. -> - A new DAST API scanning engine was introduced in GitLab 13.10. - -Using an API specification as a scan's target is a useful way to seed URLs for scanning an API. -Vulnerability rules in an API scan are different than those in a normal website scan. - -A new DAST API scanning engine is available in GitLab 13.12 and later. For more details, see [DAST API scanning engine](../dast_api). The new scanning engine supports REST, SOAP, GraphQL, and generic APIs using forms, XML, and JSON. Testing can be performed using OpenAPI, Postman Collections, and HTTP Archive (HAR) documents. - -The target API instance's base URL is provided by using the `DAST_API_TARGET_URL` variable or an `environment_url.txt` file. - -#### Specification format - -API scans support OpenAPI V2 and OpenAPI V3 specifications. You can define these specifications using `JSON` or `YAML`. - -#### Import API specification from a URL - -If your API specification is accessible at a URL, you can pass that URL in directly as the target. -The specification does not have to be hosted on the same host as the API being tested. - -```yaml -include: - - template: DAST-API.gitlab-ci.yml - -variables: - DAST_API_SPECIFICATION: http://my.api/api-specification.yml -``` - -#### Import API specification from a file - -If your API specification file is in your repository, you can provide its filename as the target. - -```yaml -dast: - variables: - GIT_STRATEGY: fetch - DAST_API_SPECIFICATION: api-specification.yml -``` - -#### Full API scan - -API scans support full scanning, which can be enabled by using the `DAST_FULL_SCAN_ENABLED` -CI/CD variable. Domain validation is not supported for full API scans. - -#### Host override - -Specifications often define a host, which contains a domain name and a port. The -host referenced may be different than the host of the API's review instance. -This can cause incorrect URLs to be imported, or a scan on an incorrect host. -Use the `DAST_API_HOST_OVERRIDE` CI/CD variable to override these values. - -WARNING: -When using the API host override feature, you cannot use the `$DAST_WEBSITE` variable to override the hostname. -A host override is _only_ supported when importing the API specification from a URL. Attempts to override the -host throw an error when the API specification is imported from a file. This is due to a limitation in the -ZAP OpenAPI extension. - -For example, with a OpenAPI V3 specification containing: - -```yaml -servers: - - url: https://api.host.com -``` - -If the test version of the API is running at `https://api-test.host.com`, then -the following DAST configuration can be used: - -```yaml -include: - - template: DAST-API.gitlab-ci.yml - -variables: - DAST_API_SPECIFICATION: http://api-test.host.com/api-specification.yml - DAST_API_HOST_OVERRIDE: api-test.host.com -``` - -#### Authentication using headers - -Tokens in request headers are often used as a way to authenticate API requests. -You can achieve this by using the `DAST_REQUEST_HEADERS` CI/CD variable. -Headers are applied to every request DAST makes. - -```yaml -include: - - template: DAST-API.gitlab-ci.yml - -variables: - DAST_API_SPECIFICATION: http://api-test.api.com/api-specification.yml - DAST_REQUEST_HEADERS: "Authorization: Bearer my.token" -``` - -### URL scan - -> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/214120) in GitLab 13.4. -> - [Improved](https://gitlab.com/gitlab-org/gitlab/-/issues/273141) in GitLab 13.11. - -A URL scan allows you to specify which parts of a website are scanned by DAST. - -#### Define the URLs to scan - -URLs to scan can be specified by either of the following methods: - -- Use `DAST_PATHS_FILE` CI/CD variable to specify the name of a file containing the paths. -- Use `DAST_PATHS` variable to list the paths. - -##### Use `DAST_PATHS_FILE` CI/CD variable - -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/258825) in GitLab 13.6. - -To define the URLs to scan in a file, create a plain text file with one path per line. - -```plaintext -page1.html -/page2.html -category/shoes/page1.html -``` - -To scan the URLs in that file, set the CI/CD variable `DAST_PATHS_FILE` to the path of that file. -The file can be checked into the project repository or generated as an artifact by a job that -runs before DAST. - -By default, DAST scans do not clone the project repository. Instruct the DAST job to clone -the project by setting `GIT_STRATEGY` to fetch. Give a file path relative to `CI_PROJECT_DIR` to `DAST_PATHS_FILE`. - -```yaml -include: - - template: DAST.gitlab-ci.yml - -variables: - GIT_STRATEGY: fetch - DAST_PATHS_FILE: url_file.txt # url_file.txt lives in the root directory of the project - DAST_BROWSER_SCAN: "true" # use the browser-based GitLab DAST crawler -``` - -##### Use `DAST_PATHS` CI/CD variable - -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/214120) in GitLab 13.4. - -To specify the paths to scan in a CI/CD variable, add a comma-separated list of the paths to the `DAST_PATHS` -variable. Note that you can only scan paths of a single host. - -```yaml -include: - - template: DAST.gitlab-ci.yml - -variables: - DAST_PATHS: "/page1.html,/category1/page1.html,/page3.html" - DAST_BROWSER_SCAN: "true" # use the browser-based GitLab DAST crawler -``` - -When using `DAST_PATHS` and `DAST_PATHS_FILE`, note the following: - -- `DAST_WEBSITE` must be defined when using either `DAST_PATHS_FILE` or `DAST_PATHS`. The paths listed in either use `DAST_WEBSITE` to build the URLs to scan -- Spidering is disabled when `DAST_PATHS` or `DAST_PATHS_FILE` are defined -- `DAST_PATHS_FILE` and `DAST_PATHS` cannot be used together -- The `DAST_PATHS` variable has a limit of about 130kb. If you have a list or paths - greater than this, use `DAST_PATHS_FILE`. - -#### Full Scan - -To perform a [full scan](#full-scan) on the listed paths, use the `DAST_FULL_SCAN_ENABLED` CI/CD variable. - -### List URLs scanned - -When DAST completes scanning, the merge request page states the number of URLs scanned. -Select **View details** to view the web console output which includes the list of scanned URLs. - - - -### View details of a vulnerability detected by DAST - -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/36332) in GitLab 13.1. - -Vulnerabilities detected by DAST occur in the live web application. Addressing these types of -vulnerabilities requires specific information. DAST provides the information required to -investigate and rectify the underlying cause. - -To view details of vulnerabilities detected by DAST: - -1. To see all vulnerabilities detected, either: - - Go to your project and select **Security & Compliance**. - - Go to the merge request and select the **Security** tab. - -1. Select a vulnerability's description. The following details are provided: - - | Field | Description | - |:-----------------|:------------------------------------------------------------------ | - | Description | Description of the vulnerability. | - | Project | Namespace and project in which the vulnerability was detected. | - | Method | HTTP method used to detect the vulnerability. | - | URL | URL at which the vulnerability was detected. | - | Request Headers | Headers of the request. | - | Response Status | Response status received from the application. | - | Response Headers | Headers of the response received from the application. | - | Evidence | Evidence of the data found that verified the vulnerability. Often a snippet of the request or response, this can be used to help verify that the finding is a vulnerability. | - | Identifiers | Identifiers of the vulnerability. | - | Severity | Severity of the vulnerability. | - | Scanner Type | Type of vulnerability report. | - | Links | Links to further details of the detected vulnerability. | - | Solution | Details of a recommended solution to the vulnerability (optional). | - -## Customize DAST settings - -You can customize the behavior of DAST using both CI/CD variables and command-line options. Use of CI/CD -variables overrides the values contained in the DAST template. - -### Customize DAST using CI/CD variables - -WARNING: -Beginning in GitLab 13.0, the use of [`only` and `except`](../../../ci/yaml/index.md#only--except) -is no longer supported. You must use [`rules`](../../../ci/yaml/index.md#rules) instead. - -The DAST settings can be changed through CI/CD variables by using the -[`variables`](../../../ci/yaml/index.md#variables) parameter in `.gitlab-ci.yml`. For details of -all DAST CI/CD variables, read [Available CI/CD variables](#available-cicd-variables). - -For example: - -```yaml -include: - - template: DAST.gitlab-ci.yml - -variables: - DAST_WEBSITE: https://example.com - DAST_SPIDER_MINS: 120 - DAST_BROWSER_SCAN: "true" # use the browser-based GitLab DAST crawler -``` - -Because the template is [evaluated before](../../../ci/yaml/index.md#include) the pipeline -configuration, the last mention of the variable takes precedence. - -#### Enable or disable rules - -A complete list of the rules that DAST uses to scan for vulnerabilities can be -found in the [ZAP documentation](https://www.zaproxy.org/docs/alerts/). - -`DAST_EXCLUDE_RULES` disables the rules with the given IDs. - -`DAST_ONLY_INCLUDE_RULES` restricts the set of rules used in the scan to -those with the given IDs. - -`DAST_EXCLUDE_RULES` and `DAST_ONLY_INCLUDE_RULES` are mutually exclusive and a -DAST scan with both configured exits with an error. - -By default, several rules are disabled because they either take a long time to -run or frequently generate false positives. The complete list of disabled rules -can be found in [`exclude_rules.yml`](https://gitlab.com/gitlab-org/security-products/dast/-/blob/main/src/config/exclude_rules.yml). - -The lists for `DAST_EXCLUDE_RULES` and `DAST_ONLY_INCLUDE_RULES` **must** be enclosed in double -quotes (`"`), otherwise they are interpreted as numeric values. - -#### Hide sensitive information - -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/36332) in GitLab 13.1. - -HTTP request and response headers may contain sensitive information, including cookies and -authorization credentials. By default, the following headers are masked: - -- `Authorization`. -- `Proxy-Authorization`. -- `Set-Cookie` (values only). -- `Cookie` (values only). - -Using the [`DAST_MASK_HTTP_HEADERS` CI/CD variable](#available-cicd-variables), you can list the -headers whose values you want masked. For details on how to mask headers, see -[Customizing the DAST settings](#customize-dast-settings). - -#### Use Mutual TLS - -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/299596) in GitLab 14.8. - -Mutual TLS allows a target application server to verify that requests are from a known source. Browser-based scans do not support Mutual TLS. - -**Requirements** - -- Base64-encoded PKCS12 certificate -- Password of the base64-encoded PKCS12 certificate - -To enable Mutual TLS: - -1. If the PKCS12 certificate is not already base64-encoded, convert it to base64 encoding. For security reasons, we recommend encoding the certificate locally, **not** using a web-hosted conversion service. For example, to encode the certificate on either macOS or Linux: - - ```shell - base64 <path-to-pkcs12-certificate-file> - ``` - -1. Create a [masked variable](../../../ci/variables/index.md) named `DAST_PKCS12_CERTIFICATE_BASE64` and store the base64-encoded PKCS12 certificate's value in that variable. -1. Create a masked variable `DAST_PKCS12_PASSWORD` and store the PKCS12 certificate's password in that variable. - -#### Available CI/CD variables - -These CI/CD variables are specific to DAST. They can be used to customize the behavior of DAST to your requirements. - -WARNING: -All customization of GitLab security scanning tools should be tested in a merge request before -merging these changes to the default branch. Failure to do so can give unexpected results, -including a large number of false positives. - -| CI/CD variable | Type | Description | -|:-------------------------------------------------|:--------------|:------------------------------| -| `DAST_ADVERTISE_SCAN` | boolean | Set to `true` to add a `Via` header to every request sent, advertising that the request was sent as part of a GitLab DAST scan. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/334947) in GitLab 14.1. | -| `DAST_AGGREGATE_VULNERABILITIES` | boolean | Vulnerability aggregation is set to `true` by default. To disable this feature and see each vulnerability individually set to `false`. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/254043) in GitLab 14.0. | -| `DAST_API_HOST_OVERRIDE` <sup>1</sup> | string | Used to override domains defined in API specification files. Only supported when importing the API specification from a URL. Example: `example.com:8080`. | -| `DAST_API_SPECIFICATION` <sup>1</sup> | URL or string | The API specification to import. The specification can be hosted at a URL, or the name of a file present in the `/zap/wrk` directory. The variable `DAST_WEBSITE` must be specified if this is omitted. | -| `DAST_AUTH_REPORT` <sup>2</sup> | boolean | Used in combination with exporting the `gl-dast-debug-auth-report.html` artifact to aid in debugging authentication issues. | -| `DAST_AUTH_EXCLUDE_URLS` <sup>2</sup> | URLs | **{warning}** **[Removed](https://gitlab.com/gitlab-org/gitlab/-/issues/289959)** in GitLab 14.0. Replaced by `DAST_EXCLUDE_URLS`. The URLs to skip during the authenticated scan; comma-separated. Regular expression syntax can be used to match multiple URLs. For example, `.*` matches an arbitrary character sequence. Not supported for API scans. | -| `DAST_AUTH_URL` <sup>1,2</sup> | URL | The URL of the page containing the sign-in HTML form on the target website. `DAST_USERNAME` and `DAST_PASSWORD` are submitted with the login form to create an authenticated scan. Not supported for API scans. Example: `https://login.example.com`. | -| `DAST_AUTH_VERIFICATION_LOGIN_FORM` <sup>2</sup> | boolean | Verifies successful authentication by checking for the lack of a login form once the login form has been submitted. | -| `DAST_AUTH_VERIFICATION_SELECTOR` <sup>2</sup> | selector | Verifies successful authentication by checking for presence of a selector once the login form has been submitted. Example: `css:.user-photo`. | -| `DAST_AUTH_VERIFICATION_URL` <sup>1,2</sup> | URL | A URL only accessible to logged in users that DAST can use to confirm successful authentication. If provided, DAST exits if it cannot access the URL. Example: `"http://example.com/loggedin_page"`. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/207335) in GitLab 13.8. | -| `DAST_AUTO_UPDATE_ADDONS` | boolean | ZAP add-ons are pinned to specific versions in the DAST Docker image. Set to `true` to download the latest versions when the scan starts. Default: `false`. | -| `DAST_BROWSER_PATH_TO_LOGIN_FORM` <sup>1,2</sup> | selector | Comma-separated list of selectors that are selected prior to attempting to enter `DAST_USERNAME` and `DAST_PASSWORD` into the login form. Example: `"css:.navigation-menu,css:.login-menu-item"`. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/326633) in GitLab 14.1. | -| `DAST_DEBUG` <sup>1</sup> | boolean | Enable debug message output. Default: `false`. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/12652) in GitLab 13.1. | -| `DAST_EXCLUDE_RULES` | string | Set to a comma-separated list of Vulnerability Rule IDs to exclude them from running during the scan. Rule IDs are numbers and can be found from the DAST log or on the [ZAP project](https://www.zaproxy.org/docs/alerts/). For example, `HTTP Parameter Override` has a rule ID of `10026`. Cannot be used when `DAST_ONLY_INCLUDE_RULES` is set. **Note:** In earlier versions of GitLab the excluded rules were executed but vulnerabilities they generated were suppressed. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/118641) in GitLab 12.10. | -| `DAST_EXCLUDE_URLS` <sup>1,2</sup> | URLs | The URLs to skip during the authenticated scan; comma-separated. Regular expression syntax can be used to match multiple URLs. For example, `.*` matches an arbitrary character sequence. Not supported for API scans. Example, `http://example.com/sign-out`. | -| `DAST_FIRST_SUBMIT_FIELD` <sup>2</sup> | string | The `id` or `name` of the element that when selected submits the username form of a multi-page login process. For example, `css:button[type='user-submit']`. [Introduced](https://gitlab.com/gitlab-org/gitlab-ee/issues/9894) in GitLab 12.4. | -| `DAST_FULL_SCAN_DOMAIN_VALIDATION_REQUIRED` | boolean | **{warning}** **[Removed](https://gitlab.com/gitlab-org/gitlab/-/issues/293595)** in GitLab 14.0. Set to `true` to require domain validation when running DAST full scans. Not supported for API scans. Default: `false` | -| `DAST_FULL_SCAN_ENABLED` <sup>1</sup> | boolean | Set to `true` to run a [ZAP Full Scan](https://github.com/zaproxy/zaproxy/wiki/ZAP-Full-Scan) instead of a [ZAP Baseline Scan](https://github.com/zaproxy/zaproxy/wiki/ZAP-Baseline-Scan). Default: `false` | -| `DAST_HTML_REPORT` | string | The filename of the HTML report written at the end of a scan. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/12652) in GitLab 13.1. | -| `DAST_INCLUDE_ALPHA_VULNERABILITIES` | boolean | Set to `true` to include alpha passive and active scan rules. Default: `false`. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/12652) in GitLab 13.1. | -| `DAST_MARKDOWN_REPORT` | string | The filename of the Markdown report written at the end of a scan. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/12652) in GitLab 13.1. | -| `DAST_MASK_HTTP_HEADERS` | string | Comma-separated list of request and response headers to be masked (GitLab 13.1). Must contain **all** headers to be masked. Refer to [list of headers that are masked by default](#hide-sensitive-information). | -| `DAST_MAX_URLS_PER_VULNERABILITY` | number | The maximum number of URLs reported for a single vulnerability. `DAST_MAX_URLS_PER_VULNERABILITY` is set to `50` by default. To list all the URLs set to `0`. [Introduced](https://gitlab.com/gitlab-org/security-products/dast/-/merge_requests/433) in GitLab 13.12. | -| `DAST_ONLY_INCLUDE_RULES` | string | Set to a comma-separated list of Vulnerability Rule IDs to configure the scan to run only them. Rule IDs are numbers and can be found from the DAST log or on the [ZAP project](https://www.zaproxy.org/docs/alerts/). Cannot be used when `DAST_EXCLUDE_RULES` is set. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/250651) in GitLab 13.12. | -| `DAST_PASSWORD` <sup>1,2</sup> | string | The password to authenticate to in the website. Example: `P@55w0rd!` | -| `DAST_PASSWORD_FIELD` <sup>1,2</sup> | string | The selector of password field at the sign-in HTML form. Example: `id:password` | -| `DAST_PATHS` | string | Set to a comma-separated list of URLs for DAST to scan. For example, `/page1.html,/category1/page3.html,/page2.html`. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/214120) in GitLab 13.4. | -| `DAST_PATHS_FILE` | string | The file path containing the paths within `DAST_WEBSITE` to scan. The file must be plain text with one path per line. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/258825) in GitLab 13.6. | -| `DAST_PKCS12_CERTIFICATE_BASE64` | string | The PKCS12 certificate used for sites that require Mutual TLS. Must be encoded as base64 text. | -| `DAST_PKCS12_PASSWORD` | string | The password of the certificate used in `DAST_PKCS12_CERTIFICATE_BASE64`. | -| `DAST_REQUEST_HEADERS` <sup>1</sup> | string | Set to a comma-separated list of request header names and values. Headers are added to every request made by DAST. For example, `Cache-control: no-cache,User-Agent: DAST/1.0` | -| `DAST_SKIP_TARGET_CHECK` | boolean | Set to `true` to prevent DAST from checking that the target is available before scanning. Default: `false`. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/229067) in GitLab 13.8. | -| `DAST_SPIDER_MINS` <sup>1</sup> | number | The maximum duration of the spider scan in minutes. Set to `0` for unlimited. Default: One minute, or unlimited when the scan is a full scan. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/12652) in GitLab 13.1. | -| `DAST_SPIDER_START_AT_HOST` | boolean | Set to `false` to prevent DAST from resetting the target to its host before scanning. When `true`, non-host targets `http://test.site/some_path` is reset to `http://test.site` before scan. Default: `true`. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/258805) in GitLab 13.6. | -| `DAST_SUBMIT_FIELD` <sup>2</sup> | string | The `id` or `name` of the element that when selected submits the login form or the password form of a multi-page login process. For example, `css:button[type='submit']`. [Introduced](https://gitlab.com/gitlab-org/gitlab-ee/issues/9894) in GitLab 12.4. | -| `DAST_TARGET_AVAILABILITY_TIMEOUT` <sup>1</sup> | number | Time limit in seconds to wait for target availability. | -| `DAST_USE_AJAX_SPIDER` <sup>1</sup> | boolean | Set to `true` to use the AJAX spider in addition to the traditional spider, useful for crawling sites that require JavaScript. Default: `false`. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/12652) in GitLab 13.1. | -| `DAST_USERNAME` <sup>1,2</sup> | string | The username to authenticate to in the website. Example: `admin` | -| `DAST_USERNAME_FIELD` <sup>1,2</sup> | string | The selector of username field at the sign-in HTML form. Example: `name:username` | -| `DAST_XML_REPORT` | string | The filename of the XML report written at the end of a scan. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/12652) in GitLab 13.1. | -| `DAST_WEBSITE` <sup>1</sup> | URL | The URL of the website to scan. The variable `DAST_API_SPECIFICATION` must be specified if this is omitted. | -| `DAST_ZAP_CLI_OPTIONS` | string | ZAP server command-line options. For example, `-Xmx3072m` would set the Java maximum memory allocation pool size. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/12652) in GitLab 13.1. | -| `DAST_ZAP_LOG_CONFIGURATION` | string | Set to a semicolon-separated list of additional log4j properties for the ZAP Server. Example: `logger.httpsender.name=org.parosproxy.paros.network.HttpSender;logger.httpsender.level=debug;logger.sitemap.name=org.parosproxy.paros.model.SiteMap;logger.sitemap.level=debug;` | -| `SECURE_ANALYZERS_PREFIX` | URL | Set the Docker registry base address from which to download the analyzer. | - -1. Available to an on-demand DAST scan. -1. Used for authentication. - -### Customize DAST using command-line options - -Not all DAST configuration is available via CI/CD variables. To find out all -possible options, run the following configuration. -Available command-line options are printed to the job log: - -```yaml -include: - template: DAST.gitlab-ci.yml - -dast: - script: - - /analyze --help -``` - -You must then overwrite the `script` command to pass in the appropriate -argument. For example, vulnerability definitions in alpha can be included with -`-a`. The following configuration includes those definitions: - -```yaml -include: - template: DAST.gitlab-ci.yml - -dast: - script: - - export DAST_WEBSITE=${DAST_WEBSITE:-$(cat environment_url.txt)} - - /analyze -a -t $DAST_WEBSITE -``` - -### Custom ZAProxy configuration - -The ZAProxy server contains many [useful configurable values](https://gitlab.com/gitlab-org/gitlab/-/issues/36437#note_245801885). -Many key/values for `-config` remain undocumented, but there is an untested list of -[possible keys](https://gitlab.com/gitlab-org/gitlab/-/issues/36437#note_244981023). -Note that these options are not supported by DAST, and may break the DAST scan -when used. An example of how to rewrite the Authorization header value with `TOKEN` follows: - -```yaml -include: - template: DAST.gitlab-ci.yml - -variables: - DAST_ZAP_CLI_OPTIONS: "-config replacer.full_list(0).description=auth -config replacer.full_list(0).enabled=true -config replacer.full_list(0).matchtype=REQ_HEADER -config replacer.full_list(0).matchstr=Authorization -config replacer.full_list(0).regex=false -config replacer.full_list(0).replacement=TOKEN" -``` - -## Authentication - -NOTE: -We highly recommend you configure the scanner to authenticate to the application. If you don't, it cannot check most of the application for security risks, as most -of your application is likely not accessible without authentication. We also recommend -you periodically confirm the scanner's authentication is still working, as this tends to break over -time due to authentication changes to the application. - -Create masked CI/CD variables to pass the credentials that DAST uses. -To create masked variables for the username and password, see [Create a custom variable in the UI](../../../ci/variables/index.md#custom-cicd-variables). -The key of the username variable must be `DAST_USERNAME`, -and the key of the password variable must be `DAST_PASSWORD`. - -After DAST has authenticated with the application, all cookies are collected from the web browser. -For each cookie a matching session token is created for use by ZAP. This ensures ZAP is recognized -by the application as correctly authenticated. - -Authentication supports single form logins, multi-step login forms, and authenticating to URLs outside of the configured target URL. - -WARNING: -**Never** run an authenticated scan against a production server. When an authenticated -scan is run, it may perform *any* function that the authenticated user can. This -includes actions like modifying and deleting data, submitting forms, and following links. -Only run an authenticated scan against a test server. - -### SSO - -DAST can authenticate to websites making use of SSO, with the following restrictions: - -- DAST cannot bypass a CAPTCHA if the authentication flow includes one. -- DAST cannot handle multi-factor authentication like one-time passwords (OTP) by using SMS or authenticator apps. -- DAST must get a cookie, or a local or session storage, with a sufficiently random value. - -The [authentication debug output](index.md#configure-the-authentication-debug-output) can be helpful for troubleshooting SSO authentication -with DAST. - -### Log in using automatic detection of the login form - -By providing a `DAST_USERNAME`, `DAST_PASSWORD`, and `DAST_AUTH_URL`, DAST attempts to authenticate to the -target application by locating the login form based on a determination about whether or not the form contains username or password fields. - -Automatic detection is "best-effort", and depending on the application being scanned may provide either a resilient login experience or one that fails to authenticate the user. - -Login process: - -1. The `DAST_AUTH_URL` is loaded into the browser, and any forms on the page are located. - 1. If a form contains a username and password field, `DAST_USERNAME` and `DAST_PASSWORD` is inputted into the respective fields, the form submit button is selected and the user is logged in. - 1. If a form contains only a username field, it is assumed that the login form is multi-step. - 1. The `DAST_USERNAME` is inputted into the username field and the form submit button is selected. - 1. The subsequent pages loads where it is expected that a form exists and contains a password field. If found, `DAST_PASSWORD` is inputted, form submit button is selected and the user is logged in. - -### Log in using explicit selection of the login form - -By providing a `DAST_USERNAME_FIELD`, `DAST_PASSWORD_FIELD`, and `DAST_SUBMIT_FIELD`, in addition to the fields required for automatic login, -DAST attempts to authenticate to the target application by locating the login form based on the selectors provided. -Most applications benefit from this approach to authentication. - -Login process: - -1. The `DAST_AUTH_URL` is loaded into the browser, and any forms on the page are located. - 1. If the `DAST_FIRST_SUBMIT_FIELD` is not defined, then `DAST_USERNAME` is inputted into `DAST_USERNAME_FIELD`, `DAST_PASSWORD` is inputted into `DAST_PASSWORD_FIELD`, `DAST_SUBMIT_FIELD` is selected and the user is logged in. - 1. If the `DAST_FIRST_SUBMIT_FIELD` is defined, then it is assumed that the login form is multi-step. - 1. The `DAST_USERNAME` is inputted into the `DAST_USERNAME_FIELD` field and the `DAST_FIRST_SUBMIT_FIELD` is selected. - 1. The subsequent pages loads where the `DAST_PASSWORD` is inputted into the `DAST_PASSWORD_FIELD` field, the `DAST_SUBMIT_FIELD` is selected and the user is logged in. - -### Verifying successful login - -Once the login form has been submitted, DAST determines if the login was successful. Unsuccessful attempts at authentication cause the scan to halt. - -Following the submission of the login form, authentication is determined to be unsuccessful when: - -- A `400` or `500` series HTTP response status code is returned. -- A new cookie/browser storage value determined to be sufficiently random has not been set. - -In addition to these checks, the user can configure their own verification checks. -Each of the following checks can be used in conjunction with one another, if none are configured by default the presence of a login form is checked. - -#### Verifying based on the URL - -When `DAST_AUTH_VERIFICATION_URL` is configured, the URL displayed in the browser tab post login form submission is directly compared to the URL in the CI/CD variable. -If these are not exactly the same, authentication is deemed to be unsuccessful. - -For example: - -```yaml -include: - - template: DAST.gitlab-ci.yml - -dast: - variables: - DAST_WEBSITE: "https://example.com" - DAST_BROWSER_SCAN: "true" # use the browser-based GitLab DAST crawler - ... - DAST_AUTH_VERIFICATION_URL: "https://example.com/user/welcome" -``` - -#### Verify based on presence of an element - -When `DAST_AUTH_VERIFICATION_SELECTOR` is configured, the page displayed in the browser tab is searched for an element described by the selector in the CI/CD variable. -If no element is found, authentication is deemed to be unsuccessful. - -For example: - -```yaml -include: - - template: DAST.gitlab-ci.yml - -dast: - variables: - DAST_WEBSITE: "https://example.com" - DAST_BROWSER_SCAN: "true" # use the browser-based GitLab DAST crawler - ... - DAST_AUTH_VERIFICATION_SELECTOR: "css:.welcome-user" -``` - -#### Verify based on presence of a login form - -When `DAST_AUTH_VERIFICATION_LOGIN_FORM` is configured, the page displayed in the browser tab is searched for a form that is detected to be a login form. -If any such form is found, authentication is deemed to be unsuccessful. - -For example: - -```yaml -include: - - template: DAST.gitlab-ci.yml - -dast: - variables: - DAST_WEBSITE: "https://example.com" - DAST_BROWSER_SCAN: "true" # use the browser-based GitLab DAST crawler - ... - DAST_AUTH_VERIFICATION_LOGIN_FORM: "true" -``` - -### View the login form - -Many web applications show the user the login form in a pop-up (modal) window. -For these applications, navigating to the form requires both: - -- A starting URL. -- A list of elements to select to display the modal window. - -When `DAST_BROWSER_PATH_TO_LOGIN_FORM` is present, like in this example: - -```yaml -include: - - template: DAST.gitlab-ci.yml - -dast: - variables: - DAST_WEBSITE: "https://my.site.com" - DAST_BROWSER_SCAN: "true" # use the browser-based GitLab DAST crawler - ... - DAST_AUTH_URL: "https://my.site.com/admin" - DAST_BROWSER_PATH_TO_LOGIN_FORM: "css:.navigation-menu,css:.login-menu-item" -``` - -DAST performs these actions: - -1. Load the `DAST_AUTH_URL` page, such as `https://my.site.com/admin`. -1. After the page loads, DAST selects elements found by the selectors described - in `DAST_BROWSER_PATH_TO_LOGIN_FORM`. This example opens the navigation menu - and selects the login menu, to display the login modal window. -1. To continue the authentication process, DAST fills in the username and password - on the login form. - -### Configure the authentication debug output - -It is often difficult to understand the cause of an authentication failure when running DAST in a CI/CD pipeline. -To assist users in debugging authentication issues, a debug report can be generated and saved as a job artifact. -This HTML report contains all steps made during the login process, along with HTTP requests and responses, the Document Object Model (DOM) and screenshots. - - - -An example configuration where the authentication debug report is exported may look like the following: - -```yaml -dast: - variables: - DAST_WEBSITE: "https://example.com" - DAST_BROWSER_SCAN: "true" # use the browser-based GitLab DAST crawler - ... - DAST_AUTH_REPORT: "true" - artifacts: - paths: [gl-dast-debug-auth-report.html] - when: always -``` - -### Selectors - -Selectors are used by CI/CD variables to specify the location of an element displayed on a page in a browser. -Selectors have the format `type`:`search string`. The crawler searches for the selector using the search string based on the type. - -| Selector type | Example | Description | -| ------------- | ---------------------------------- | ----------- | -| `css` | `css:.password-field` | Searches for a HTML element having the supplied CSS selector. Selectors should be as specific as possible for performance reasons. | -| `id` | `id:element` | Searches for an HTML element with the provided element ID. | -| `name` | `name:element` | Searches for an HTML element with the provided element name. | -| `xpath` | `xpath://input[@id="my-button"]/a` | Searches for a HTML element with the provided XPath. Note that XPath searches are expected to be less performant than other searches. | -| None provided | `a.click-me` | Defaults to searching using a CSS selector. | - -#### Find selectors with Google Chrome - -Chrome DevTools element selector tool is an effective way to find a selector. - -1. Open Chrome and navigate to the page where you would like to find a selector, for example, the login page for your site. -1. Open the `Elements` tab in Chrome DevTools with the keyboard shortcut `Command + Shift + c` in macOS or `Ctrl + Shift + c` in Windows. -1. Select the `Select an element in the page to select it` tool. -  -1. Select the field on your page that you would like to know the selector for. -1. Once the tool is active, highlight a field you wish to view the details of. -  -1. Once highlighted, you can see the element's details, including attributes that would make a good candidate for a selector. - -In this example, the `id="user_login"` appears to be a good candidate. You can use this as a selector as the DAST username field by setting -`DAST_USERNAME_FIELD: "id:user_login"`. - -#### Choose the right selector - -Judicious choice of selector leads to a scan that is resilient to the application changing. - -In order of preference, it is recommended to choose as selectors: - -- `id` fields. These are generally unique on a page, and rarely change. -- `name` fields. These are generally unique on a page, and rarely change. -- `class` values specific to the field, such as the selector `"css:.username"` for the `username` class on the username field. -- Presence of field specific data attributes, such as the selector, `"css:[data-username]"` when the `data-username` field has any value on the username field. -- Multiple `class` hierarchy values, such as the selector `"css:.login-form .username"` when there are multiple elements with class `username` but only one nested inside the element with the class `login-form`. - -When using selectors to locate specific fields we recommend you avoid searching on: - -- Any `id`, `name`, `attribute`, `class` or `value` that is dynamically generated. -- Generic class names, such as `column-10` and `dark-grey`. -- XPath searches as they are less performant than other selector searches. -- Unscoped searches, such as those beginning with `css:*` and `xpath://*`. - -### Bleeding-edge vulnerability definitions - -ZAP first creates rules in the `alpha` class. After a testing period with -the community, they are promoted to `beta`. DAST uses `beta` definitions by -default. To request `alpha` definitions, use the -`DAST_INCLUDE_ALPHA_VULNERABILITIES` CI/CD variable as shown in the -following configuration: - -```yaml -include: - template: DAST.gitlab-ci.yml - -variables: - DAST_INCLUDE_ALPHA_VULNERABILITIES: "true" -``` - -### Cloning the project's repository - -The DAST job does not require the project's repository to be present when running, so by default -[`GIT_STRATEGY`](../../../ci/runners/configure_runners.md#git-strategy) is set to `none`. - -## On-demand scans - -> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/218465) in GitLab 13.2. -> - [Improved](https://gitlab.com/gitlab-org/gitlab/-/issues/218465) in GitLab 13.3. -> - The saved scans feature was [introduced](https://gitlab.com/groups/gitlab-org/-/epics/5100) in GitLab 13.9. -> - The option to select a branch was [introduced](https://gitlab.com/groups/gitlab-org/-/epics/4847) in GitLab 13.10. -> - DAST branch selection [feature flag removed](https://gitlab.com/gitlab-org/gitlab/-/issues/322672) in GitLab 13.11. -> - Auditing for DAST profile management was [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/217872) in GitLab 14.1. - -An on-demand DAST scan runs outside the DevOps life cycle. Changes in your repository don't trigger -the scan. You must either start it manually, or schedule it to run. - -An on-demand DAST scan: - -- Can run a specific combination of a [site profile](#site-profile) and a - [scanner profile](#scanner-profile). -- Is associated with your project's default branch. -- Is saved on creation so it can be run later. - -### On-demand scan modes - -An on-demand scan can be run in active or passive mode: - -- _Passive mode_ is the default and runs a ZAP Baseline Scan. -- _Active mode_ runs a ZAP Full Scan which is potentially harmful to the site being scanned. To - minimize the risk of accidental damage, running an active scan requires a [validated site profile](#site-profile-validation). - -### View on-demand DAST scans - -To view running completed and scheduled on-demand DAST scans for a project, go to -**Security & Compliance > On-demand Scans** in the left sidebar. - -- To view both running and completed scans, select **All**. -- To view running scans only, select **Running**. -- To view finished scans, select **Finished**. A finished scan is a scan that either succeeded, - failed, or was canceled. -- To view scheduled scans, select **Scheduled**. It shows on-demand scans that have a schedule - set up. Those are _not_ included in the **All** tab. -- To view saved on-demand scan profiles, select **Scan library**. - Those are _not_ included in the **All** tab. - -#### Cancel an on-demand scan - -To cancel a pending or running on-demand scan, select **Cancel** (**{cancel}**) in the -on-demand scans list. - -#### Retry an on-demand scan - -To retry a scan that failed or succeeded with warnings, select **Retry** (**{retry}**) in the -on-demand scans list. - -#### View an on-demand scan's results - -To view a finished scan's results, select **View results** in the on-demand scans list. - -#### Edit an on-demand scan - -To edit an on-demand scan's settings, select **Edit** (**{pencil}**) in the **Scheduled** tab. - -### Run an on-demand DAST scan - -Prerequisites: - -- You must have permission to run an on-demand DAST scan against a protected branch. The default - branch is automatically protected. For more information, read - [Pipeline security on protected branches](../../../ci/pipelines/index.md#pipeline-security-on-protected-branches). -- A [scanner profile](#create-a-scanner-profile). -- A [site profile](#create-a-site-profile). -- If you are running an active scan the site profile must have been [validated](#validate-a-site-profile). - -You can run an on-demand scan immediately, once at a scheduled date and time or at a specified -frequency: - -- Every day -- Every week -- Every month -- Every 3 months -- Every 6 months -- Every year - -To run an on-demand scan immediately, either: - -- [Create and run an on-demand scan immediately](#create-and-run-an-on-demand-scan-immediately). -- [Run a previously saved on-demand scan](#run-a-saved-on-demand-scan). - -To run an on-demand scan either at a scheduled date or frequency, read -[Schedule an on-demand scan](#schedule-an-on-demand-scan). - -#### Create and run an on-demand scan immediately - -1. From your project's home page, go to **Security & Compliance > On-demand Scans** in the left - sidebar. -1. Select **New scan**. -1. Complete the **Scan name** and **Description** fields. -1. In GitLab 13.10 and later, select the desired branch from the **Branch** dropdown. -1. In **Scanner profile**, select a scanner profile from the dropdown. -1. In **Site profile**, select a site profile from the dropdown. -1. To run the on-demand scan immediately, select **Save and run scan**. Otherwise, select - **Save scan** to [run](#run-a-saved-on-demand-scan) it later. - -The on-demand DAST scan runs and the project's dashboard shows the results. - -#### Run a saved on-demand scan - -To run a saved on-demand scan: - -1. On the top bar, select **Main menu > Projects** and find your project. -1. On the left sidebar, select **Security & Compliance > On-demand Scans**. -1. Select the **Scan library** tab. -1. In the scan's row, select **Run scan**. - - If the branch saved in the scan no longer exists, you must first - [edit the scan](#edit-an-on-demand-scan), select a new branch, and save the edited scan. - -The on-demand DAST scan runs, and the project's dashboard shows the results. - -#### Schedule an on-demand scan - -> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/328749) in GitLab 14.3. [Deployed behind the `dast_on_demand_scans_scheduler` flag](../../../administration/feature_flags.md), disabled by default. -> - [Enabled on GitLab.com](https://gitlab.com/gitlab-org/gitlab/-/issues/328749) in GitLab 14.4. -> - [Enabled on self-managed](https://gitlab.com/gitlab-org/gitlab/-/issues/328749) in GitLab 14.4. -> - [Feature flag `dast_on_demand_scans_scheduler` removed](https://gitlab.com/gitlab-org/gitlab/-/issues/328749) in GitLab 14.5. - -To schedule a scan: - -1. On the top bar, select **Main menu > Projects** and find your project. -1. On the left sidebar, select **Security & Compliance > On-demand Scans**. -1. Select **New scan**. -1. Complete the **Scan name** and **Description** text boxes. -1. In GitLab 13.10 and later, from the **Branch** dropdown list, select the desired branch. -1. In the **Scanner profile** section, from the dropdown list, select a scanner profile. -1. In the **Site profile** section, from the dropdown list, select a site profile. -1. Select **Schedule scan**. -1. In the **Start time** section, select a time zone, date, and time. -1. From the **Repeats** dropdown list, select your desired frequency: - - To run the scan once, select **Never**. - - For a recurring scan, select any other option. -1. To run the on-demand scan immediately, select **Save and run scan**. To [run](#run-a-saved-on-demand-scan) it according to the schedule you set, select - **Save scan**. - -#### List saved on-demand scans - -To list saved on-demand scans: - -1. From your project's home page, go to **Security & Compliance > On-demand Scans**. -1. Select the **Scan library** tab. - -#### View details of an on-demand scan - -To view details of an on-demand scan: - -1. From your project's home page, go to **Security & Compliance > On-demand Scans**. -1. Select the **Scan library** tab. -1. In the saved scan's row select **More actions** (**{ellipsis_v}**), then select **Edit**. - -#### Edit an on-demand scan - -To edit an on-demand scan: - -1. From your project's home page, go to **Security & Compliance > On-demand Scans**. -1. Select the **Scan library** tab. -1. In the saved scan's row select **More actions** (**{ellipsis_v}**), then select **Edit**. -1. Edit the form. -1. Select **Save scan**. - -#### Delete an on-demand scan - -To delete an on-demand scan: - -1. From your project's home page, go to **Security & Compliance > On-demand Scans**. -1. Select the **Scan library** tab. -1. In the saved scan's row select **More actions** (**{ellipsis_v}**), then select **Delete**. -1. Select **Delete** to confirm the deletion. - -## Site profile - -A site profile defines the attributes and configuration details of the deployed application, -website, or API to be scanned by DAST. A site profile can be referenced in `.gitlab-ci.yml` and -on-demand scans. - -A site profile contains: - -- **Profile name**: A name you assign to the site to be scanned. While a site profile is referenced - in either `.gitlab-ci.yml` or an on-demand scan, it **cannot** be renamed. -- **Site type**: The type of target to be scanned, either website or API scan. -- **Target URL**: The URL that DAST runs against. -- **Excluded URLs**: A comma-separated list of URLs to exclude from the scan. -- **Request headers**: A comma-separated list of HTTP request headers, including names and values. These headers are added to every request made by DAST. -- **Authentication**: - - **Authenticated URL**: The URL of the page containing the sign-in HTML form on the target website. The username and password are submitted with the login form to create an authenticated scan. - - **Username**: The username used to authenticate to the website. - - **Password**: The password used to authenticate to the website. - - **Username form field**: The name of username field at the sign-in HTML form. - - **Password form field**: The name of password field at the sign-in HTML form. - - **Submit form field**: The `id` or `name` of the element that when selected submits the sign-in HTML form. - -When an API site type is selected, a [host override](#host-override) is used to ensure the API being scanned is on the same host as the target. This is done to reduce the risk of running an active scan against the wrong API. - -When configured, request headers and password fields are encrypted using [`aes-256-gcm`](https://en.wikipedia.org/wiki/Advanced_Encryption_Standard) before being stored in the database. -This data can only be read and decrypted with a valid secrets file. - -### Site profile validation - -> - Site profile validation [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/233020) in GitLab 13.8. -> - Meta tag validation [introduced](https://gitlab.com/groups/gitlab-org/-/epics/6460) in GitLab 14.2. - -Site profile validation reduces the risk of running an active scan against the wrong website. A site -must be validated before an active scan can run against it. The site validation methods are as -follows: - -- _Text file validation_ requires a text file be uploaded to the target site. The text file is - allocated a name and content that is unique to the project. The validation process checks the - file's content. -- _Header validation_ requires the header `Gitlab-On-Demand-DAST` be added to the target site, - with a value unique to the project. The validation process checks that the header is present, and - checks its value. -- _Meta tag validation_ requires the meta tag named `gitlab-dast-validation` be added to the target site, - with a value unique to the project. Make sure it's added to the `<head>` section of the page. The validation process checks that the meta tag is present, and - checks its value. - -All these methods are equivalent in functionality. Use whichever is feasible. - -In [GitLab 14.2 and later](https://gitlab.com/gitlab-org/gitlab/-/issues/324990), site profile -validation happens in a CI job using the [GitLab Runner](../../../ci/runners/index.md). - -### Create a site profile - -To create a site profile: - -1. From your project's home page, go to **Security & Compliance > Configuration**. -1. Select **Manage** in the **DAST Profiles** row. -1. Select **New > Site Profile**. -1. Complete the fields then select **Save profile**. - -The site profile is created. - -### Edit a site profile - -If a site profile is linked to a security policy, a user cannot edit the profile from this page. See -[Scan execution policies](../policies/scan-execution-policies.md) -for more information. - -When a validated site profile's file, header, or meta tag is edited, the site's -[validation status](#site-profile-validation) is revoked. - -To edit a site profile: - -1. From your project's home page, go to **Security & Compliance > Configuration**. -1. In the **DAST Profiles** row select **Manage**. -1. Select the **Site Profiles** tab. -1. In the profile's row select the **More actions** (**{ellipsis_v}**) menu, then select **Edit**. -1. Edit the fields then select **Save profile**. - -### Delete a site profile - -If a site profile is linked to a security policy, a user cannot delete the profile from this page. -See [Scan execution policies](../policies/scan-execution-policies.md) -for more information. - -To delete a site profile: - -1. From your project's home page, go to **Security & Compliance > Configuration**. -1. In the **DAST Profiles** row select **Manage**. -1. Select the **Site Profiles** tab. -1. In the profile's row, select the **More actions** (**{ellipsis_v}**) menu, then select **Delete**. -1. Select **Delete** to confirm the deletion. - -### Validate a site profile - -Validating a site is required to run an active scan. - -To validate a site profile: - -1. On the top bar, select **Main menu > Projects** and find your project. -1. On the left sidebar, select **Security & Compliance > Configuration**. -1. In the **Dynamic Application Security Testing (DAST)** section, select **Manage profiles**. -1. Select the **Site Profiles** tab. -1. In the profile's row, select **Validate**. -1. Select the validation method. - 1. For **Text file validation**: - 1. Download the validation file listed in **Step 2**. - 1. Upload the validation file to the host, to the location in **Step 3** or any location you - prefer. - 1. If required, edit the file location in **Step 3**. - 1. Select **Validate**. - 1. For **Header validation**: - 1. Select the clipboard icon in **Step 2**. - 1. Edit the header of the site to validate, and paste the clipboard content. - 1. Select the input field in **Step 3** and enter the location of the header. - 1. Select **Validate**. - 1. For **Meta tag validation**: - 1. Select the clipboard icon in **Step 2**. - 1. Edit the content of the site to validate, and paste the clipboard content. - 1. Select the input field in **Step 3** and enter the location of the meta tag. - 1. Select **Validate**. - -The site is validated and an active scan can run against it. A site profile's validation status is -revoked only when it's revoked manually, or its file, header, or meta tag is edited. - -### Retry a failed validation - -> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/322609) in GitLab 14.3. -> - [Deployed behind the `dast_failed_site_validations` flag](../../../administration/feature_flags.md), enabled by default. -> - [Feature flag `dast_failed_site_validations` removed](https://gitlab.com/gitlab-org/gitlab/-/issues/323961) in GitLab 14.4. - -Failed site validation attempts are listed on the **Site profiles** tab of the **Manage profiles** -page. - -To retry a site profile's failed validation: - -1. On the top bar, select **Main menu > Projects** and find your project. -1. On the left sidebar, select **Security & Compliance > Configuration**. -1. In the **Dynamic Application Security Testing (DAST)** section, select **Manage profiles**. -1. Select the **Site Profiles** tab. -1. In the profile's row, select **Retry validation**. - -### Revoke a site profile's validation status - -WARNING: -When a site profile's validation status is revoked, all site profiles that share the same URL also -have their validation status revoked. - -To revoke a site profile's validation status: - -1. From your project's home page, go to **Security & Compliance > Configuration**. -1. In the **DAST Profiles** row select **Manage**. -1. Beside the validated profile, select **Revoke validation**. - -The site profile's validation status is revoked. - -### Validated site profile headers - -The following are code samples of how you can provide the required site profile header in your -application. - -#### Ruby on Rails example for on-demand scan - -Here's how you can add a custom header in a Ruby on Rails application: - -```ruby -class DastWebsiteTargetController < ActionController::Base - def dast_website_target - response.headers['Gitlab-On-Demand-DAST'] = '0dd79c9a-7b29-4e26-a815-eaaf53fcab1c' - head :ok - end -end -``` - -#### Django example for on-demand scan - -Here's how you can add a -[custom header in Django](https://docs.djangoproject.com/en/2.2/ref/request-response/#setting-header-fields): - -```python -class DastWebsiteTargetView(View): - def head(self, *args, **kwargs): - response = HttpResponse() - response['Gitlab-On-Demand-DAST'] = '0dd79c9a-7b29-4e26-a815-eaaf53fcab1c' - - return response -``` - -#### Node (with Express) example for on-demand scan - -Here's how you can add a -[custom header in Node (with Express)](https://expressjs.com/en/5x/api.html#res.append): - -```javascript -app.get('/dast-website-target', function(req, res) { - res.append('Gitlab-On-Demand-DAST', '0dd79c9a-7b29-4e26-a815-eaaf53fcab1c') - res.send('Respond to DAST ping') -}) -``` - -## Scanner profile - -> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/222767) in GitLab 13.4. -> - [Added](https://gitlab.com/gitlab-org/gitlab/-/issues/225804) in GitLab 13.5: scan mode, AJAX spider, debug messages. - -A scanner profile defines the configuration details of a security scanner. A scanner profile can be -referenced in `.gitlab-ci.yml` and on-demand scans. - -A scanner profile contains: - -- **Profile name:** A name you give the scanner profile. For example, "Spider_15". While a scanner - profile is referenced in either `.gitlab-ci.yml` or an on-demand scan, it **cannot** be renamed. -- **Scan mode:** A passive scan monitors all HTTP messages (requests and responses) sent to the target. An active scan attacks the target to find potential vulnerabilities. -- **Spider timeout:** The maximum number of minutes allowed for the spider to traverse the site. -- **Target timeout:** The maximum number of seconds DAST waits for the site to be available before - starting the scan. -- **AJAX spider:** Run the AJAX spider, in addition to the traditional spider, to crawl the target site. -- **Debug messages:** Include debug messages in the DAST console output. - -### Create a scanner profile - -To create a scanner profile: - -1. From your project's home page, go to **Security & Compliance > Configuration**. -1. In the **DAST Profiles** row, select **Manage**. -1. Select **New > Scanner Profile**. -1. Complete the form. For details of each field, see [Scanner profile](#scanner-profile). -1. Select **Save profile**. - -### Edit a scanner profile - -If a scanner profile is linked to a security policy, a user cannot edit the profile from this page. -See [Scan execution policies](../policies/scan-execution-policies.md) -for more information. - -To edit a scanner profile: - -1. From your project's home page, go to **Security & Compliance > Configuration**. -1. In the **DAST Profiles** row, select **Manage**. -1. Select the **Scanner Profiles** tab. -1. In the scanner's row, select the **More actions** (**{ellipsis_v}**) menu, then select **Edit**. -1. Edit the form. -1. Select **Save profile**. - -### Delete a scanner profile - -If a scanner profile is linked to a security policy, a user cannot delete the profile from this -page. See [Scan execution policies](../policies/scan-execution-policies.md) -for more information. - -To delete a scanner profile: - -1. From your project's home page, go to **Security & Compliance > Configuration**. -1. In the **DAST Profiles** row, select **Manage**. -1. Select the **Scanner Profiles** tab. -1. In the scanner's row, select the **More actions** (**{ellipsis_v}**) menu, then select **Delete**. -1. Select **Delete**. - -## Auditing - -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/217872) in GitLab 14.1. - -The creation, updating, and deletion of DAST profiles, DAST scanner profiles, -and DAST site profiles are included in the [audit log](../../../administration/audit_events.md). - -## Reports - -The DAST tool outputs a `gl-dast-report.json` report file containing details of the scan and its results. -This file is included in the job's artifacts. JSON is the default format, but -you can output the report in Markdown, HTML, and XML formats. To specify an alternative -format, use a [CI/CD variable](#available-cicd-variables). You can also use a CI/CD variable -to configure the job to output the `gl-dast-debug-auth-report.html` file which helps when debugging -authentication issues. - -For details of the report's schema, see the [schema for DAST reports](https://gitlab.com/gitlab-org/security-products/security-report-schemas/-/blob/master/dist/dast-report-format.json). Example reports can be found in the -[DAST repository](https://gitlab.com/gitlab-org/security-products/dast/-/tree/main/test/end-to-end/expect). - -WARNING: -The JSON report artifacts are not a public API of DAST and their format is expected to change in the -future. - -## Optimizing DAST - -By default, DAST downloads all artifacts defined by previous jobs in the pipeline. If -your DAST job does not rely on `environment_url.txt` to define the URL under test or any other files created -in previous jobs, we recommend you don't download artifacts. To avoid downloading -artifacts, add the following to your `.gitlab-ci.yml` file: - -```yaml -dast: - dependencies: [] -``` diff --git a/doc/user/application_security/dast/proxy-based.md b/doc/user/application_security/dast/proxy-based.md new file mode 100644 index 00000000000..ec98b809fb7 --- /dev/null +++ b/doc/user/application_security/dast/proxy-based.md @@ -0,0 +1,1247 @@ +--- +stage: Secure +group: Dynamic Analysis +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments +type: reference, howto +--- + +# DAST proxy-based analyzer **(ULTIMATE)** + +The DAST proxy-based analyzer can be added to your [GitLab CI/CD](../../../ci/index.md) pipeline. +This helps you discover vulnerabilities in web applications that do not use JavaScript heavily. For applications that do, +please see the [DAST browser-based analyzer](browser_based.md). + +WARNING: +Do not run DAST scans against a production server. Not only can it perform *any* function that +a user can, such as clicking buttons or submitting forms, but it may also trigger bugs, leading to modification or loss of production data. Only run DAST scans against a test server. + +The analyzer uses the [OWASP Zed Attack Proxy](https://www.zaproxy.org/) (ZAP) to scan in two different ways: + +- Passive scan only (default). DAST executes + [ZAP's Baseline Scan](https://www.zaproxy.org/docs/docker/baseline-scan/) and doesn't + actively attack your application. +- Passive and active (or full) scan. DAST can be [configured](#full-scan) to also perform an active scan + to attack your application and produce a more extensive security report. It can be very + useful when combined with [Review Apps](../../../ci/review_apps/index.md). + +## DAST run options + +You can use DAST to examine your web application: + +- Automatically, initiated by a merge request. +- Manually, initiated on demand. + +Some of the differences between these run options: + +| Automatic scan | On-demand scan | +|:-----------------------------------------------------------------|:------------------------------| +| DAST scan is initiated by a merge request. | DAST scan is initiated manually, outside the DevOps life cycle. | +| CI/CD variables are sourced from `.gitlab-ci.yml`. | CI/CD variables are provided in the UI. | +| All [DAST CI/CD variables](#available-cicd-variables) available. | Subset of [DAST CI/CD variables](#available-cicd-variables) available. | +| `DAST.gitlab-ci.yml` template. | `DAST-On-Demand-Scan.gitlab-ci.yml` template. | + +### Enable automatic DAST run + +To enable DAST to run automatically, either: + +- Enable [Auto DAST](../../../topics/autodevops/stages.md#auto-dast) (provided + by [Auto DevOps](../../../topics/autodevops/index.md)). +- [Include the DAST template](#include-the-dast-template) in your existing + `.gitlab-ci.yml` file. +- [Configure DAST using the UI](#configure-dast-using-the-ui). + +#### Include the DAST template + +> - This template was [updated](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/62597) to DAST_VERSION: 2 in GitLab 14.0. +> - This template was [updated](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/87183) to DAST_VERSION: 3 in GitLab 15.0. + +If you want to manually add DAST to your application, the DAST job is defined +in a CI/CD template file. Updates to the template are provided with GitLab +upgrades, allowing you to benefit from any improvements and additions. + +To include the DAST template: + +1. Select the CI/CD template you want to use: + + - [`DAST.gitlab-ci.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/ci/templates/Security/DAST.gitlab-ci.yml): + Stable version of the DAST CI/CD template. + - [`DAST.latest.gitlab-ci.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/ci/templates/Security/DAST.latest.gitlab-ci.yml): + Latest version of the DAST template. ([Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/254325) + in GitLab 13.8). + + WARNING: + The latest version of the template may include breaking changes. Use the + stable template unless you need a feature provided only in the latest template. + + For more information about template versioning, see the + [CI/CD documentation](../../../development/cicd/templates.md#latest-version). + +1. Add a `dast` stage to your GitLab CI stages configuration: + + ```yaml + stages: + - dast + ``` + +1. Add the template to GitLab, based on your version of GitLab: + + - In GitLab 11.9 and later, [include](../../../ci/yaml/index.md#includetemplate) + the template by adding the following to your `.gitlab-ci.yml` file: + + ```yaml + include: + - template: <template_file.yml> + + variables: + DAST_WEBSITE: https://example.com + ``` + + - In GitLab 11.8 and earlier, add the contents of the template to your + `.gitlab_ci.yml` file. + +1. Define the URL to be scanned by DAST by using one of these methods: + + - Set the `DAST_WEBSITE` [CI/CD variable](../../../ci/yaml/index.md#variables). + If set, this value takes precedence. + + - Add the URL in an `environment_url.txt` file at the root of your project. This is + useful for testing in dynamic environments. To run DAST against an application + dynamically created during a GitLab CI/CD pipeline, a job that runs prior to + the DAST scan must persist the application's domain in an `environment_url.txt` + file. DAST automatically parses the `environment_url.txt` file to find its + scan target. + + For example, in a job that runs prior to DAST, you could include code that + looks similar to: + + ```yaml + script: + - echo http://${CI_PROJECT_ID}-${CI_ENVIRONMENT_SLUG}.domain.com > environment_url.txt + artifacts: + paths: [environment_url.txt] + when: always + ``` + + You can see an example of this in our + [Auto DevOps CI YAML](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/ci/templates/Jobs/Deploy.gitlab-ci.yml) + file. + +The included template creates a `dast` job in your CI/CD pipeline and scans +your project's running application for possible vulnerabilities. + +The results are saved as a +[DAST report artifact](../../../ci/yaml/artifacts_reports.md#artifactsreportsdast) +that you can later download and analyze. Due to implementation limitations, we +always take the latest DAST artifact available. Behind the scenes, the +[GitLab DAST Docker image](https://gitlab.com/security-products/dast) +is used to run the tests on the specified URL and scan it for possible +vulnerabilities. + +By default, the DAST template uses the latest major version of the DAST Docker +image. Using the `DAST_VERSION` variable, you can choose how DAST updates: + +- Automatically update DAST with new features and fixes by pinning to a major + version (such as `1`). +- Only update fixes by pinning to a minor version (such as `1.6`). +- Prevent all updates by pinning to a specific version (such as `1.6.4`). + +Find the latest DAST versions on the [Releases](https://gitlab.com/gitlab-org/security-products/dast/-/releases) +page. + +#### Configure DAST using the UI + +You can enable or configure DAST settings using the UI. The generated settings are formatted so they +can be conveniently pasted into the `.gitlab-ci.yml` file. + +1. On the top bar, select **Main menu > Projects** and find your project. +1. On the left sidebar, select **Security & Compliance > Configuration**. +1. In the **Dynamic Application Security Testing (DAST)** section, select **Enable DAST** or + **Configure DAST**. +1. Select the desired **Scanner profile**, or select **Create scanner profile** and save a + scanner profile. For more details, see [scanner profiles](#scanner-profile). +1. Select the desired **Site profile**, or select **Create site profile** and save a site + profile. For more details, see [site profiles](#site-profile). +1. Select **Generate code snippet**. A modal opens with the YAML snippet corresponding to the + options you selected. +1. Do one of the following: + 1. To copy the snippet to your clipboard, select **Copy code only**. + 1. To add the snippet to your project's `.gitlab-ci.yml` file, select + **Copy code and open `.gitlab-ci.yml` file**. The Pipeline Editor opens. + 1. Paste the snippet into the `.gitlab-ci.yml` file. + 1. Select the **Lint** tab to confirm the edited `.gitlab-ci.yml` file is valid. + 1. Select the **Edit** tab, then select **Commit changes**. + +When the snippet is committed to the `.gitlab-ci.yml` file, pipelines include a DAST job. + +### API scan + +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/10928) in GitLab 12.10. +> - A new DAST API scanning engine was introduced in GitLab 13.10. + +Using an API specification as a scan's target is a useful way to seed URLs for scanning an API. +Vulnerability rules in an API scan are different than those in a normal website scan. + +A new DAST API scanning engine is available in GitLab 13.12 and later. For more details, see [DAST API scanning engine](../dast_api). The new scanning engine supports REST, SOAP, GraphQL, and generic APIs using forms, XML, and JSON. Testing can be performed using OpenAPI, Postman Collections, and HTTP Archive (HAR) documents. + +The target API instance's base URL is provided by using the `DAST_API_TARGET_URL` variable or an `environment_url.txt` file. + +#### Specification format + +API scans support OpenAPI V2 and OpenAPI V3 specifications. You can define these specifications using `JSON` or `YAML`. + +#### Import API specification from a URL + +If your API specification is accessible at a URL, you can pass that URL in directly as the target. +The specification does not have to be hosted on the same host as the API being tested. + +```yaml +include: + - template: DAST-API.gitlab-ci.yml + +variables: + DAST_API_SPECIFICATION: http://my.api/api-specification.yml +``` + +#### Import API specification from a file + +If your API specification file is in your repository, you can provide its filename as the target. + +```yaml +dast: + variables: + GIT_STRATEGY: fetch + DAST_API_SPECIFICATION: api-specification.yml +``` + +#### Full API scan + +API scans support full scanning, which can be enabled by using the `DAST_FULL_SCAN_ENABLED` +CI/CD variable. Domain validation is not supported for full API scans. + +#### Host override + +Specifications often define a host, which contains a domain name and a port. The +host referenced may be different than the host of the API's review instance. +This can cause incorrect URLs to be imported, or a scan on an incorrect host. +Use the `DAST_API_HOST_OVERRIDE` CI/CD variable to override these values. + +WARNING: +When using the API host override feature, you cannot use the `$DAST_WEBSITE` variable to override the hostname. +A host override is _only_ supported when importing the API specification from a URL. Attempts to override the +host throw an error when the API specification is imported from a file. This is due to a limitation in the +ZAP OpenAPI extension. + +For example, with a OpenAPI V3 specification containing: + +```yaml +servers: + - url: https://api.host.com +``` + +If the test version of the API is running at `https://api-test.host.com`, then +the following DAST configuration can be used: + +```yaml +include: + - template: DAST-API.gitlab-ci.yml + +variables: + DAST_API_SPECIFICATION: http://api-test.host.com/api-specification.yml + DAST_API_HOST_OVERRIDE: api-test.host.com +``` + +#### Authentication using headers + +Tokens in request headers are often used as a way to authenticate API requests. +You can achieve this by using the `DAST_REQUEST_HEADERS` CI/CD variable. +Headers are applied to every request DAST makes. + +```yaml +include: + - template: DAST-API.gitlab-ci.yml + +variables: + DAST_API_SPECIFICATION: http://api-test.api.com/api-specification.yml + DAST_REQUEST_HEADERS: "Authorization: Bearer my.token" +``` + +### URL scan + +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/214120) in GitLab 13.4. +> - [Improved](https://gitlab.com/gitlab-org/gitlab/-/issues/273141) in GitLab 13.11. + +A URL scan allows you to specify which parts of a website are scanned by DAST. + +#### Define the URLs to scan + +URLs to scan can be specified by either of the following methods: + +- Use `DAST_PATHS_FILE` CI/CD variable to specify the name of a file containing the paths. +- Use `DAST_PATHS` variable to list the paths. + +##### Use `DAST_PATHS_FILE` CI/CD variable + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/258825) in GitLab 13.6. + +To define the URLs to scan in a file, create a plain text file with one path per line. + +```plaintext +page1.html +/page2.html +category/shoes/page1.html +``` + +To scan the URLs in that file, set the CI/CD variable `DAST_PATHS_FILE` to the path of that file. +The file can be checked into the project repository or generated as an artifact by a job that +runs before DAST. + +By default, DAST scans do not clone the project repository. Instruct the DAST job to clone +the project by setting `GIT_STRATEGY` to fetch. Give a file path relative to `CI_PROJECT_DIR` to `DAST_PATHS_FILE`. + +```yaml +include: + - template: DAST.gitlab-ci.yml + +variables: + GIT_STRATEGY: fetch + DAST_PATHS_FILE: url_file.txt # url_file.txt lives in the root directory of the project + DAST_BROWSER_SCAN: "true" # use the browser-based GitLab DAST crawler +``` + +##### Use `DAST_PATHS` CI/CD variable + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/214120) in GitLab 13.4. + +To specify the paths to scan in a CI/CD variable, add a comma-separated list of the paths to the `DAST_PATHS` +variable. Note that you can only scan paths of a single host. + +```yaml +include: + - template: DAST.gitlab-ci.yml + +variables: + DAST_PATHS: "/page1.html,/category1/page1.html,/page3.html" + DAST_BROWSER_SCAN: "true" # use the browser-based GitLab DAST crawler +``` + +When using `DAST_PATHS` and `DAST_PATHS_FILE`, note the following: + +- `DAST_WEBSITE` must be defined when using either `DAST_PATHS_FILE` or `DAST_PATHS`. The paths listed in either use `DAST_WEBSITE` to build the URLs to scan +- Spidering is disabled when `DAST_PATHS` or `DAST_PATHS_FILE` are defined +- `DAST_PATHS_FILE` and `DAST_PATHS` cannot be used together +- The `DAST_PATHS` variable has a limit of about 130kb. If you have a list or paths + greater than this, use `DAST_PATHS_FILE`. + +#### Full Scan + +To perform a [full scan](#full-scan) on the listed paths, use the `DAST_FULL_SCAN_ENABLED` CI/CD variable. + +## Customize DAST settings + +You can customize the behavior of DAST using both CI/CD variables and command-line options. Use of CI/CD +variables overrides the values contained in the DAST template. + +### Customize DAST using CI/CD variables + +WARNING: +Beginning in GitLab 13.0, the use of [`only` and `except`](../../../ci/yaml/index.md#only--except) +is no longer supported. You must use [`rules`](../../../ci/yaml/index.md#rules) instead. + +The DAST settings can be changed through CI/CD variables by using the +[`variables`](../../../ci/yaml/index.md#variables) parameter in `.gitlab-ci.yml`. For details of +all DAST CI/CD variables, read [Available CI/CD variables](#available-cicd-variables). + +For example: + +```yaml +include: + - template: DAST.gitlab-ci.yml + +variables: + DAST_WEBSITE: https://example.com + DAST_SPIDER_MINS: 120 + DAST_BROWSER_SCAN: "true" # use the browser-based GitLab DAST crawler +``` + +Because the template is [evaluated before](../../../ci/yaml/index.md#include) the pipeline +configuration, the last mention of the variable takes precedence. + +#### Enable or disable rules + +A complete list of the rules that DAST uses to scan for vulnerabilities can be +found in the [ZAP documentation](https://www.zaproxy.org/docs/alerts/). + +`DAST_EXCLUDE_RULES` disables the rules with the given IDs. + +`DAST_ONLY_INCLUDE_RULES` restricts the set of rules used in the scan to +those with the given IDs. + +`DAST_EXCLUDE_RULES` and `DAST_ONLY_INCLUDE_RULES` are mutually exclusive and a +DAST scan with both configured exits with an error. + +By default, several rules are disabled because they either take a long time to +run or frequently generate false positives. The complete list of disabled rules +can be found in [`exclude_rules.yml`](https://gitlab.com/gitlab-org/security-products/dast/-/blob/main/src/config/exclude_rules.yml). + +The lists for `DAST_EXCLUDE_RULES` and `DAST_ONLY_INCLUDE_RULES` **must** be enclosed in double +quotes (`"`), otherwise they are interpreted as numeric values. + +#### Hide sensitive information + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/36332) in GitLab 13.1. + +HTTP request and response headers may contain sensitive information, including cookies and +authorization credentials. By default, the following headers are masked: + +- `Authorization`. +- `Proxy-Authorization`. +- `Set-Cookie` (values only). +- `Cookie` (values only). + +Using the [`DAST_MASK_HTTP_HEADERS` CI/CD variable](#available-cicd-variables), you can list the +headers whose values you want masked. For details on how to mask headers, see +[Customizing the DAST settings](#customize-dast-settings). + +#### Use Mutual TLS + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/299596) in GitLab 14.8. + +Mutual TLS allows a target application server to verify that requests are from a known source. Browser-based scans do not support Mutual TLS. + +**Requirements** + +- Base64-encoded PKCS12 certificate +- Password of the base64-encoded PKCS12 certificate + +To enable Mutual TLS: + +1. If the PKCS12 certificate is not already base64-encoded, convert it to base64 encoding. For security reasons, we recommend encoding the certificate locally, **not** using a web-hosted conversion service. For example, to encode the certificate on either macOS or Linux: + + ```shell + base64 <path-to-pkcs12-certificate-file> + ``` + +1. Create a [masked variable](../../../ci/variables/index.md) named `DAST_PKCS12_CERTIFICATE_BASE64` and store the base64-encoded PKCS12 certificate's value in that variable. +1. Create a masked variable `DAST_PKCS12_PASSWORD` and store the PKCS12 certificate's password in that variable. + +#### Available CI/CD variables + +These CI/CD variables are specific to DAST. They can be used to customize the behavior of DAST to your requirements. + +WARNING: +All customization of GitLab security scanning tools should be tested in a merge request before +merging these changes to the default branch. Failure to do so can give unexpected results, +including a large number of false positives. + +| CI/CD variable | Type | Description | +|:-------------------------------------------------|:--------------|:------------------------------| +| `DAST_ADVERTISE_SCAN` | boolean | Set to `true` to add a `Via` header to every request sent, advertising that the request was sent as part of a GitLab DAST scan. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/334947) in GitLab 14.1. | +| `DAST_AGGREGATE_VULNERABILITIES` | boolean | Vulnerability aggregation is set to `true` by default. To disable this feature and see each vulnerability individually set to `false`. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/254043) in GitLab 14.0. | +| `DAST_API_HOST_OVERRIDE` <sup>1</sup> | string | Used to override domains defined in API specification files. Only supported when importing the API specification from a URL. Example: `example.com:8080`. | +| `DAST_API_SPECIFICATION` <sup>1</sup> | URL or string | The API specification to import. The specification can be hosted at a URL, or the name of a file present in the `/zap/wrk` directory. The variable `DAST_WEBSITE` must be specified if this is omitted. | +| `DAST_AUTH_REPORT` <sup>2</sup> | boolean | Used in combination with exporting the `gl-dast-debug-auth-report.html` artifact to aid in debugging authentication issues. | +| `DAST_AUTH_EXCLUDE_URLS` <sup>2</sup> | URLs | **{warning}** **[Removed](https://gitlab.com/gitlab-org/gitlab/-/issues/289959)** in GitLab 14.0. Replaced by `DAST_EXCLUDE_URLS`. The URLs to skip during the authenticated scan; comma-separated. Regular expression syntax can be used to match multiple URLs. For example, `.*` matches an arbitrary character sequence. Not supported for API scans. | +| `DAST_AUTH_URL` <sup>1,2</sup> | URL | The URL of the page containing the sign-in HTML form on the target website. `DAST_USERNAME` and `DAST_PASSWORD` are submitted with the login form to create an authenticated scan. Not supported for API scans. Example: `https://login.example.com`. | +| `DAST_AUTH_VERIFICATION_LOGIN_FORM` <sup>2</sup> | boolean | Verifies successful authentication by checking for the lack of a login form once the login form has been submitted. | +| `DAST_AUTH_VERIFICATION_SELECTOR` <sup>2</sup> | selector | Verifies successful authentication by checking for presence of a selector once the login form has been submitted. Example: `css:.user-photo`. | +| `DAST_AUTH_VERIFICATION_URL` <sup>1,2</sup> | URL | A URL only accessible to logged in users that DAST can use to confirm successful authentication. If provided, DAST exits if it cannot access the URL. Example: `"http://example.com/loggedin_page"`. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/207335) in GitLab 13.8. | +| `DAST_AUTO_UPDATE_ADDONS` | boolean | ZAP add-ons are pinned to specific versions in the DAST Docker image. Set to `true` to download the latest versions when the scan starts. Default: `false`. | +| `DAST_BROWSER_PATH_TO_LOGIN_FORM` <sup>1,2</sup> | selector | Comma-separated list of selectors that are selected prior to attempting to enter `DAST_USERNAME` and `DAST_PASSWORD` into the login form. Example: `"css:.navigation-menu,css:.login-menu-item"`. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/326633) in GitLab 14.1. | +| `DAST_DEBUG` <sup>1</sup> | boolean | Enable debug message output. Default: `false`. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/12652) in GitLab 13.1. | +| `DAST_EXCLUDE_RULES` | string | Set to a comma-separated list of Vulnerability Rule IDs to exclude them from running during the scan. Rule IDs are numbers and can be found from the DAST log or on the [ZAP project](https://www.zaproxy.org/docs/alerts/). For example, `HTTP Parameter Override` has a rule ID of `10026`. Cannot be used when `DAST_ONLY_INCLUDE_RULES` is set. **Note:** In earlier versions of GitLab the excluded rules were executed but vulnerabilities they generated were suppressed. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/118641) in GitLab 12.10. | +| `DAST_EXCLUDE_URLS` <sup>1,2</sup> | URLs | The URLs to skip during the authenticated scan; comma-separated. Regular expression syntax can be used to match multiple URLs. For example, `.*` matches an arbitrary character sequence. Not supported for API scans. Example, `http://example.com/sign-out`. | +| `DAST_FIRST_SUBMIT_FIELD` <sup>2</sup> | string | The `id` or `name` of the element that when selected submits the username form of a multi-page login process. For example, `css:button[type='user-submit']`. [Introduced](https://gitlab.com/gitlab-org/gitlab-ee/issues/9894) in GitLab 12.4. | +| `DAST_FULL_SCAN_DOMAIN_VALIDATION_REQUIRED` | boolean | **{warning}** **[Removed](https://gitlab.com/gitlab-org/gitlab/-/issues/293595)** in GitLab 14.0. Set to `true` to require domain validation when running DAST full scans. Not supported for API scans. Default: `false` | +| `DAST_FULL_SCAN_ENABLED` <sup>1</sup> | boolean | Set to `true` to run a [ZAP Full Scan](https://github.com/zaproxy/zaproxy/wiki/ZAP-Full-Scan) instead of a [ZAP Baseline Scan](https://github.com/zaproxy/zaproxy/wiki/ZAP-Baseline-Scan). Default: `false` | +| `DAST_HTML_REPORT` | string | The filename of the HTML report written at the end of a scan. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/12652) in GitLab 13.1. | +| `DAST_INCLUDE_ALPHA_VULNERABILITIES` | boolean | Set to `true` to include alpha passive and active scan rules. Default: `false`. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/12652) in GitLab 13.1. | +| `DAST_MARKDOWN_REPORT` | string | The filename of the Markdown report written at the end of a scan. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/12652) in GitLab 13.1. | +| `DAST_MASK_HTTP_HEADERS` | string | Comma-separated list of request and response headers to be masked (GitLab 13.1). Must contain **all** headers to be masked. Refer to [list of headers that are masked by default](#hide-sensitive-information). | +| `DAST_MAX_URLS_PER_VULNERABILITY` | number | The maximum number of URLs reported for a single vulnerability. `DAST_MAX_URLS_PER_VULNERABILITY` is set to `50` by default. To list all the URLs set to `0`. [Introduced](https://gitlab.com/gitlab-org/security-products/dast/-/merge_requests/433) in GitLab 13.12. | +| `DAST_ONLY_INCLUDE_RULES` | string | Set to a comma-separated list of Vulnerability Rule IDs to configure the scan to run only them. Rule IDs are numbers and can be found from the DAST log or on the [ZAP project](https://www.zaproxy.org/docs/alerts/). Cannot be used when `DAST_EXCLUDE_RULES` is set. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/250651) in GitLab 13.12. | +| `DAST_PASSWORD` <sup>1,2</sup> | string | The password to authenticate to in the website. Example: `P@55w0rd!` | +| `DAST_PASSWORD_FIELD` <sup>1,2</sup> | string | The selector of password field at the sign-in HTML form. Example: `id:password` | +| `DAST_PATHS` | string | Set to a comma-separated list of URLs for DAST to scan. For example, `/page1.html,/category1/page3.html,/page2.html`. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/214120) in GitLab 13.4. | +| `DAST_PATHS_FILE` | string | The file path containing the paths within `DAST_WEBSITE` to scan. The file must be plain text with one path per line. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/258825) in GitLab 13.6. | +| `DAST_PKCS12_CERTIFICATE_BASE64` | string | The PKCS12 certificate used for sites that require Mutual TLS. Must be encoded as base64 text. | +| `DAST_PKCS12_PASSWORD` | string | The password of the certificate used in `DAST_PKCS12_CERTIFICATE_BASE64`. | +| `DAST_REQUEST_HEADERS` <sup>1</sup> | string | Set to a comma-separated list of request header names and values. Headers are added to every request made by DAST. For example, `Cache-control: no-cache,User-Agent: DAST/1.0` | +| `DAST_SKIP_TARGET_CHECK` | boolean | Set to `true` to prevent DAST from checking that the target is available before scanning. Default: `false`. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/229067) in GitLab 13.8. | +| `DAST_SPIDER_MINS` <sup>1</sup> | number | The maximum duration of the spider scan in minutes. Set to `0` for unlimited. Default: One minute, or unlimited when the scan is a full scan. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/12652) in GitLab 13.1. | +| `DAST_SPIDER_START_AT_HOST` | boolean | Set to `false` to prevent DAST from resetting the target to its host before scanning. When `true`, non-host targets `http://test.site/some_path` is reset to `http://test.site` before scan. Default: `true`. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/258805) in GitLab 13.6. | +| `DAST_SUBMIT_FIELD` <sup>2</sup> | string | The `id` or `name` of the element that when selected submits the login form or the password form of a multi-page login process. For example, `css:button[type='submit']`. [Introduced](https://gitlab.com/gitlab-org/gitlab-ee/issues/9894) in GitLab 12.4. | +| `DAST_TARGET_AVAILABILITY_TIMEOUT` <sup>1</sup> | number | Time limit in seconds to wait for target availability. | +| `DAST_USE_AJAX_SPIDER` <sup>1</sup> | boolean | Set to `true` to use the AJAX spider in addition to the traditional spider, useful for crawling sites that require JavaScript. Default: `false`. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/12652) in GitLab 13.1. | +| `DAST_USERNAME` <sup>1,2</sup> | string | The username to authenticate to in the website. Example: `admin` | +| `DAST_USERNAME_FIELD` <sup>1,2</sup> | string | The selector of username field at the sign-in HTML form. Example: `name:username` | +| `DAST_XML_REPORT` | string | The filename of the XML report written at the end of a scan. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/12652) in GitLab 13.1. | +| `DAST_WEBSITE` <sup>1</sup> | URL | The URL of the website to scan. The variable `DAST_API_SPECIFICATION` must be specified if this is omitted. | +| `DAST_ZAP_CLI_OPTIONS` | string | ZAP server command-line options. For example, `-Xmx3072m` would set the Java maximum memory allocation pool size. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/12652) in GitLab 13.1. | +| `DAST_ZAP_LOG_CONFIGURATION` | string | Set to a semicolon-separated list of additional log4j properties for the ZAP Server. Example: `logger.httpsender.name=org.parosproxy.paros.network.HttpSender;logger.httpsender.level=debug;logger.sitemap.name=org.parosproxy.paros.model.SiteMap;logger.sitemap.level=debug;` | +| `SECURE_ANALYZERS_PREFIX` | URL | Set the Docker registry base address from which to download the analyzer. | + +1. Available to an on-demand DAST scan. +1. Used for authentication. + +### Customize DAST using command-line options + +Not all DAST configuration is available via CI/CD variables. To find out all +possible options, run the following configuration. +Available command-line options are printed to the job log: + +```yaml +include: + template: DAST.gitlab-ci.yml + +dast: + script: + - /analyze --help +``` + +You must then overwrite the `script` command to pass in the appropriate +argument. For example, vulnerability definitions in alpha can be included with +`-a`. The following configuration includes those definitions: + +```yaml +include: + template: DAST.gitlab-ci.yml + +dast: + script: + - export DAST_WEBSITE=${DAST_WEBSITE:-$(cat environment_url.txt)} + - /analyze -a -t $DAST_WEBSITE +``` + +### Custom ZAProxy configuration + +The ZAProxy server contains many [useful configurable values](https://gitlab.com/gitlab-org/gitlab/-/issues/36437#note_245801885). +Many key/values for `-config` remain undocumented, but there is an untested list of +[possible keys](https://gitlab.com/gitlab-org/gitlab/-/issues/36437#note_244981023). +Note that these options are not supported by DAST, and may break the DAST scan +when used. An example of how to rewrite the Authorization header value with `TOKEN` follows: + +```yaml +include: + template: DAST.gitlab-ci.yml + +variables: + DAST_ZAP_CLI_OPTIONS: "-config replacer.full_list(0).description=auth -config replacer.full_list(0).enabled=true -config replacer.full_list(0).matchtype=REQ_HEADER -config replacer.full_list(0).matchstr=Authorization -config replacer.full_list(0).regex=false -config replacer.full_list(0).replacement=TOKEN" +``` + +## Authentication + +NOTE: +We highly recommend you configure the scanner to authenticate to the application. If you don't, it cannot check most of the application for security risks, as most +of your application is likely not accessible without authentication. We also recommend +you periodically confirm the scanner's authentication is still working, as this tends to break over +time due to authentication changes to the application. + +Create masked CI/CD variables to pass the credentials that DAST uses. +To create masked variables for the username and password, see [Create a custom variable in the UI](../../../ci/variables/index.md#custom-cicd-variables). +The key of the username variable must be `DAST_USERNAME`, +and the key of the password variable must be `DAST_PASSWORD`. + +After DAST has authenticated with the application, all cookies are collected from the web browser. +For each cookie a matching session token is created for use by ZAP. This ensures ZAP is recognized +by the application as correctly authenticated. + +Authentication supports single form logins, multi-step login forms, and authenticating to URLs outside of the configured target URL. + +WARNING: +**Never** run an authenticated scan against a production server. When an authenticated +scan is run, it may perform *any* function that the authenticated user can. This +includes actions like modifying and deleting data, submitting forms, and following links. +Only run an authenticated scan against a test server. + +### SSO + +DAST can authenticate to websites making use of SSO, with the following restrictions: + +- DAST cannot bypass a CAPTCHA if the authentication flow includes one. +- DAST cannot handle multi-factor authentication like one-time passwords (OTP) by using SMS or authenticator apps. +- DAST must get a cookie, or a local or session storage, with a sufficiently random value. + +The [authentication debug output](#configure-the-authentication-debug-output) can be helpful for troubleshooting SSO authentication +with DAST. + +### Log in using automatic detection of the login form + +By providing a `DAST_USERNAME`, `DAST_PASSWORD`, and `DAST_AUTH_URL`, DAST attempts to authenticate to the +target application by locating the login form based on a determination about whether or not the form contains username or password fields. + +Automatic detection is "best-effort", and depending on the application being scanned may provide either a resilient login experience or one that fails to authenticate the user. + +Login process: + +1. The `DAST_AUTH_URL` is loaded into the browser, and any forms on the page are located. + 1. If a form contains a username and password field, `DAST_USERNAME` and `DAST_PASSWORD` is inputted into the respective fields, the form submit button is selected and the user is logged in. + 1. If a form contains only a username field, it is assumed that the login form is multi-step. + 1. The `DAST_USERNAME` is inputted into the username field and the form submit button is selected. + 1. The subsequent pages loads where it is expected that a form exists and contains a password field. If found, `DAST_PASSWORD` is inputted, form submit button is selected and the user is logged in. + +### Log in using explicit selection of the login form + +By providing a `DAST_USERNAME_FIELD`, `DAST_PASSWORD_FIELD`, and `DAST_SUBMIT_FIELD`, in addition to the fields required for automatic login, +DAST attempts to authenticate to the target application by locating the login form based on the selectors provided. +Most applications benefit from this approach to authentication. + +Login process: + +1. The `DAST_AUTH_URL` is loaded into the browser, and any forms on the page are located. + 1. If the `DAST_FIRST_SUBMIT_FIELD` is not defined, then `DAST_USERNAME` is inputted into `DAST_USERNAME_FIELD`, `DAST_PASSWORD` is inputted into `DAST_PASSWORD_FIELD`, `DAST_SUBMIT_FIELD` is selected and the user is logged in. + 1. If the `DAST_FIRST_SUBMIT_FIELD` is defined, then it is assumed that the login form is multi-step. + 1. The `DAST_USERNAME` is inputted into the `DAST_USERNAME_FIELD` field and the `DAST_FIRST_SUBMIT_FIELD` is selected. + 1. The subsequent pages loads where the `DAST_PASSWORD` is inputted into the `DAST_PASSWORD_FIELD` field, the `DAST_SUBMIT_FIELD` is selected and the user is logged in. + +### Verifying successful login + +Once the login form has been submitted, DAST determines if the login was successful. Unsuccessful attempts at authentication cause the scan to halt. + +Following the submission of the login form, authentication is determined to be unsuccessful when: + +- A `400` or `500` series HTTP response status code is returned. +- A new cookie/browser storage value determined to be sufficiently random has not been set. + +In addition to these checks, the user can configure their own verification checks. +Each of the following checks can be used in conjunction with one another, if none are configured by default the presence of a login form is checked. + +#### Verifying based on the URL + +When `DAST_AUTH_VERIFICATION_URL` is configured, the URL displayed in the browser tab post login form submission is directly compared to the URL in the CI/CD variable. +If these are not exactly the same, authentication is deemed to be unsuccessful. + +For example: + +```yaml +include: + - template: DAST.gitlab-ci.yml + +dast: + variables: + DAST_WEBSITE: "https://example.com" + DAST_BROWSER_SCAN: "true" # use the browser-based GitLab DAST crawler + ... + DAST_AUTH_VERIFICATION_URL: "https://example.com/user/welcome" +``` + +#### Verify based on presence of an element + +When `DAST_AUTH_VERIFICATION_SELECTOR` is configured, the page displayed in the browser tab is searched for an element described by the selector in the CI/CD variable. +If no element is found, authentication is deemed to be unsuccessful. + +For example: + +```yaml +include: + - template: DAST.gitlab-ci.yml + +dast: + variables: + DAST_WEBSITE: "https://example.com" + DAST_BROWSER_SCAN: "true" # use the browser-based GitLab DAST crawler + ... + DAST_AUTH_VERIFICATION_SELECTOR: "css:.welcome-user" +``` + +#### Verify based on presence of a login form + +When `DAST_AUTH_VERIFICATION_LOGIN_FORM` is configured, the page displayed in the browser tab is searched for a form that is detected to be a login form. +If any such form is found, authentication is deemed to be unsuccessful. + +For example: + +```yaml +include: + - template: DAST.gitlab-ci.yml + +dast: + variables: + DAST_WEBSITE: "https://example.com" + DAST_BROWSER_SCAN: "true" # use the browser-based GitLab DAST crawler + ... + DAST_AUTH_VERIFICATION_LOGIN_FORM: "true" +``` + +### View the login form + +Many web applications show the user the login form in a pop-up (modal) window. +For these applications, navigating to the form requires both: + +- A starting URL. +- A list of elements to select to display the modal window. + +When `DAST_BROWSER_PATH_TO_LOGIN_FORM` is present, like in this example: + +```yaml +include: + - template: DAST.gitlab-ci.yml + +dast: + variables: + DAST_WEBSITE: "https://my.site.com" + DAST_BROWSER_SCAN: "true" # use the browser-based GitLab DAST crawler + ... + DAST_AUTH_URL: "https://my.site.com/admin" + DAST_BROWSER_PATH_TO_LOGIN_FORM: "css:.navigation-menu,css:.login-menu-item" +``` + +DAST performs these actions: + +1. Load the `DAST_AUTH_URL` page, such as `https://my.site.com/admin`. +1. After the page loads, DAST selects elements found by the selectors described + in `DAST_BROWSER_PATH_TO_LOGIN_FORM`. This example opens the navigation menu + and selects the login menu, to display the login modal window. +1. To continue the authentication process, DAST fills in the username and password + on the login form. + +### Configure the authentication debug output + +It is often difficult to understand the cause of an authentication failure when running DAST in a CI/CD pipeline. +To assist users in debugging authentication issues, a debug report can be generated and saved as a job artifact. +This HTML report contains all steps made during the login process, along with HTTP requests and responses, the Document Object Model (DOM) and screenshots. + + + +An example configuration where the authentication debug report is exported may look like the following: + +```yaml +dast: + variables: + DAST_WEBSITE: "https://example.com" + DAST_BROWSER_SCAN: "true" # use the browser-based GitLab DAST crawler + ... + DAST_AUTH_REPORT: "true" + artifacts: + paths: [gl-dast-debug-auth-report.html] + when: always +``` + +### Selectors + +Selectors are used by CI/CD variables to specify the location of an element displayed on a page in a browser. +Selectors have the format `type`:`search string`. The crawler searches for the selector using the search string based on the type. + +| Selector type | Example | Description | +| ------------- | ---------------------------------- | ----------- | +| `css` | `css:.password-field` | Searches for a HTML element having the supplied CSS selector. Selectors should be as specific as possible for performance reasons. | +| `id` | `id:element` | Searches for an HTML element with the provided element ID. | +| `name` | `name:element` | Searches for an HTML element with the provided element name. | +| `xpath` | `xpath://input[@id="my-button"]/a` | Searches for a HTML element with the provided XPath. Note that XPath searches are expected to be less performant than other searches. | +| None provided | `a.click-me` | Defaults to searching using a CSS selector. | + +#### Find selectors with Google Chrome + +Chrome DevTools element selector tool is an effective way to find a selector. + +1. Open Chrome and navigate to the page where you would like to find a selector, for example, the login page for your site. +1. Open the `Elements` tab in Chrome DevTools with the keyboard shortcut `Command + Shift + c` in macOS or `Ctrl + Shift + c` in Windows. +1. Select the `Select an element in the page to select it` tool. +  +1. Select the field on your page that you would like to know the selector for. +1. Once the tool is active, highlight a field you wish to view the details of. +  +1. Once highlighted, you can see the element's details, including attributes that would make a good candidate for a selector. + +In this example, the `id="user_login"` appears to be a good candidate. You can use this as a selector as the DAST username field by setting +`DAST_USERNAME_FIELD: "id:user_login"`. + +#### Choose the right selector + +Judicious choice of selector leads to a scan that is resilient to the application changing. + +In order of preference, it is recommended to choose as selectors: + +- `id` fields. These are generally unique on a page, and rarely change. +- `name` fields. These are generally unique on a page, and rarely change. +- `class` values specific to the field, such as the selector `"css:.username"` for the `username` class on the username field. +- Presence of field specific data attributes, such as the selector, `"css:[data-username]"` when the `data-username` field has any value on the username field. +- Multiple `class` hierarchy values, such as the selector `"css:.login-form .username"` when there are multiple elements with class `username` but only one nested inside the element with the class `login-form`. + +When using selectors to locate specific fields we recommend you avoid searching on: + +- Any `id`, `name`, `attribute`, `class` or `value` that is dynamically generated. +- Generic class names, such as `column-10` and `dark-grey`. +- XPath searches as they are less performant than other selector searches. +- Unscoped searches, such as those beginning with `css:*` and `xpath://*`. + +### Bleeding-edge vulnerability definitions + +ZAP first creates rules in the `alpha` class. After a testing period with +the community, they are promoted to `beta`. DAST uses `beta` definitions by +default. To request `alpha` definitions, use the +`DAST_INCLUDE_ALPHA_VULNERABILITIES` CI/CD variable as shown in the +following configuration: + +```yaml +include: + template: DAST.gitlab-ci.yml + +variables: + DAST_INCLUDE_ALPHA_VULNERABILITIES: "true" +``` + +### Cloning the project's repository + +The DAST job does not require the project's repository to be present when running, so by default +[`GIT_STRATEGY`](../../../ci/runners/configure_runners.md#git-strategy) is set to `none`. + +## On-demand scans + +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/218465) in GitLab 13.2. +> - [Improved](https://gitlab.com/gitlab-org/gitlab/-/issues/218465) in GitLab 13.3. +> - The saved scans feature was [introduced](https://gitlab.com/groups/gitlab-org/-/epics/5100) in GitLab 13.9. +> - The option to select a branch was [introduced](https://gitlab.com/groups/gitlab-org/-/epics/4847) in GitLab 13.10. +> - DAST branch selection [feature flag removed](https://gitlab.com/gitlab-org/gitlab/-/issues/322672) in GitLab 13.11. +> - Auditing for DAST profile management was [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/217872) in GitLab 14.1. + +An on-demand DAST scan runs outside the DevOps life cycle. Changes in your repository don't trigger +the scan. You must either start it manually, or schedule it to run. + +An on-demand DAST scan: + +- Can run a specific combination of a [site profile](#site-profile) and a + [scanner profile](#scanner-profile). +- Is associated with your project's default branch. +- Is saved on creation so it can be run later. + +### On-demand scan modes + +An on-demand scan can be run in active or passive mode: + +- _Passive mode_ is the default and runs a ZAP Baseline Scan. +- _Active mode_ runs a ZAP Full Scan which is potentially harmful to the site being scanned. To + minimize the risk of accidental damage, running an active scan requires a [validated site profile](#site-profile-validation). + +### View on-demand DAST scans + +To view running completed and scheduled on-demand DAST scans for a project, go to +**Security & Compliance > On-demand Scans** in the left sidebar. + +- To view both running and completed scans, select **All**. +- To view running scans only, select **Running**. +- To view finished scans, select **Finished**. A finished scan is a scan that either succeeded, + failed, or was canceled. +- To view scheduled scans, select **Scheduled**. It shows on-demand scans that have a schedule + set up. Those are _not_ included in the **All** tab. +- To view saved on-demand scan profiles, select **Scan library**. + Those are _not_ included in the **All** tab. + +#### Cancel an on-demand scan + +To cancel a pending or running on-demand scan, select **Cancel** (**{cancel}**) in the +on-demand scans list. + +#### Retry an on-demand scan + +To retry a scan that failed or succeeded with warnings, select **Retry** (**{retry}**) in the +on-demand scans list. + +#### View an on-demand scan's results + +To view a finished scan's results, select **View results** in the on-demand scans list. + +#### Edit an on-demand scan + +To edit an on-demand scan's settings, select **Edit** (**{pencil}**) in the **Scheduled** tab. + +### Run an on-demand DAST scan + +Prerequisites: + +- You must have permission to run an on-demand DAST scan against a protected branch. The default + branch is automatically protected. For more information, read + [Pipeline security on protected branches](../../../ci/pipelines/index.md#pipeline-security-on-protected-branches). +- A [scanner profile](#create-a-scanner-profile). +- A [site profile](#create-a-site-profile). +- If you are running an active scan the site profile must have been [validated](#validate-a-site-profile). + +You can run an on-demand scan immediately, once at a scheduled date and time or at a specified +frequency: + +- Every day +- Every week +- Every month +- Every 3 months +- Every 6 months +- Every year + +To run an on-demand scan immediately, either: + +- [Create and run an on-demand scan immediately](#create-and-run-an-on-demand-scan-immediately). +- [Run a previously saved on-demand scan](#run-a-saved-on-demand-scan). + +To run an on-demand scan either at a scheduled date or frequency, read +[Schedule an on-demand scan](#schedule-an-on-demand-scan). + +#### Create and run an on-demand scan immediately + +1. From your project's home page, go to **Security & Compliance > On-demand Scans** in the left + sidebar. +1. Select **New scan**. +1. Complete the **Scan name** and **Description** fields. +1. In GitLab 13.10 and later, select the desired branch from the **Branch** dropdown list. +1. In **Scanner profile**, select a scanner profile from the dropdown list. +1. In **Site profile**, select a site profile from the dropdown list. +1. To run the on-demand scan immediately, select **Save and run scan**. Otherwise, select + **Save scan** to [run](#run-a-saved-on-demand-scan) it later. + +The on-demand DAST scan runs and the project's dashboard shows the results. + +#### Run a saved on-demand scan + +To run a saved on-demand scan: + +1. On the top bar, select **Main menu > Projects** and find your project. +1. On the left sidebar, select **Security & Compliance > On-demand Scans**. +1. Select the **Scan library** tab. +1. In the scan's row, select **Run scan**. + + If the branch saved in the scan no longer exists, you must first + [edit the scan](#edit-an-on-demand-scan), select a new branch, and save the edited scan. + +The on-demand DAST scan runs, and the project's dashboard shows the results. + +#### Schedule an on-demand scan + +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/328749) in GitLab 14.3. [Deployed behind the `dast_on_demand_scans_scheduler` flag](../../../administration/feature_flags.md), disabled by default. +> - [Enabled on GitLab.com](https://gitlab.com/gitlab-org/gitlab/-/issues/328749) in GitLab 14.4. +> - [Enabled on self-managed](https://gitlab.com/gitlab-org/gitlab/-/issues/328749) in GitLab 14.4. +> - [Feature flag `dast_on_demand_scans_scheduler` removed](https://gitlab.com/gitlab-org/gitlab/-/issues/328749) in GitLab 14.5. + +To schedule a scan: + +1. On the top bar, select **Main menu > Projects** and find your project. +1. On the left sidebar, select **Security & Compliance > On-demand Scans**. +1. Select **New scan**. +1. Complete the **Scan name** and **Description** text boxes. +1. In GitLab 13.10 and later, from the **Branch** dropdown list, select the desired branch. +1. In the **Scanner profile** section, from the dropdown list, select a scanner profile. +1. In the **Site profile** section, from the dropdown list, select a site profile. +1. Select **Schedule scan**. +1. In the **Start time** section, select a time zone, date, and time. +1. From the **Repeats** dropdown list, select your desired frequency: + - To run the scan once, select **Never**. + - For a recurring scan, select any other option. +1. To run the on-demand scan immediately, select **Save and run scan**. To [run](#run-a-saved-on-demand-scan) it according to the schedule you set, select + **Save scan**. + +#### List saved on-demand scans + +To list saved on-demand scans: + +1. From your project's home page, go to **Security & Compliance > On-demand Scans**. +1. Select the **Scan library** tab. + +#### View details of an on-demand scan + +To view details of an on-demand scan: + +1. From your project's home page, go to **Security & Compliance > On-demand Scans**. +1. Select the **Scan library** tab. +1. In the saved scan's row select **More actions** (**{ellipsis_v}**), then select **Edit**. + +#### Edit an on-demand scan + +To edit an on-demand scan: + +1. From your project's home page, go to **Security & Compliance > On-demand Scans**. +1. Select the **Scan library** tab. +1. In the saved scan's row select **More actions** (**{ellipsis_v}**), then select **Edit**. +1. Edit the form. +1. Select **Save scan**. + +#### Delete an on-demand scan + +To delete an on-demand scan: + +1. From your project's home page, go to **Security & Compliance > On-demand Scans**. +1. Select the **Scan library** tab. +1. In the saved scan's row select **More actions** (**{ellipsis_v}**), then select **Delete**. +1. Select **Delete** to confirm the deletion. + +## Site profile + +> - Scan method [Enabled on GitLab.com and self-managed](https://gitlab.com/gitlab-org/gitlab/-/issues/345837) in GitLab 15.6. +> - File URL [Enabled on GitLab.com and self-managed](https://gitlab.com/gitlab-org/gitlab/-/issues/345837) in GitLab 15.6. + +A site profile defines the attributes and configuration details of the deployed application, +website, or API to be scanned by DAST. A site profile can be referenced in `.gitlab-ci.yml` and +on-demand scans. + +A site profile contains: + +- **Profile name**: A name you assign to the site to be scanned. While a site profile is referenced + in either `.gitlab-ci.yml` or an on-demand scan, it **cannot** be renamed. +- **Site type**: The type of target to be scanned, either website or API scan. +- **Target URL**: The URL that DAST runs against. +- **Excluded URLs**: A comma-separated list of URLs to exclude from the scan. +- **Request headers**: A comma-separated list of HTTP request headers, including names and values. These headers are added to every request made by DAST. +- **Authentication**: + - **Authenticated URL**: The URL of the page containing the sign-in HTML form on the target website. The username and password are submitted with the login form to create an authenticated scan. + - **Username**: The username used to authenticate to the website. + - **Password**: The password used to authenticate to the website. + - **Username form field**: The name of username field at the sign-in HTML form. + - **Password form field**: The name of password field at the sign-in HTML form. + - **Submit form field**: The `id` or `name` of the element that when selected submits the sign-in HTML form. + +- **Scan method**: A type of method to perform API testing. The supported methods are OpenAPI, Postman Collections, and HTTP Archive (HAR) documents. +- **File URL**: The URL of the OpenAPI, Postman Collection, or HTTP Archive file. + +When an API site type is selected, a [host override](#host-override) is used to ensure the API being scanned is on the same host as the target. This is done to reduce the risk of running an active scan against the wrong API. + +When configured, request headers and password fields are encrypted using [`aes-256-gcm`](https://en.wikipedia.org/wiki/Advanced_Encryption_Standard) before being stored in the database. +This data can only be read and decrypted with a valid secrets file. + +### Site profile validation + +> - Site profile validation [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/233020) in GitLab 13.8. +> - Meta tag validation [introduced](https://gitlab.com/groups/gitlab-org/-/epics/6460) in GitLab 14.2. + +Site profile validation reduces the risk of running an active scan against the wrong website. A site +must be validated before an active scan can run against it. The site validation methods are as +follows: + +- _Text file validation_ requires a text file be uploaded to the target site. The text file is + allocated a name and content that is unique to the project. The validation process checks the + file's content. +- _Header validation_ requires the header `Gitlab-On-Demand-DAST` be added to the target site, + with a value unique to the project. The validation process checks that the header is present, and + checks its value. +- _Meta tag validation_ requires the meta tag named `gitlab-dast-validation` be added to the target site, + with a value unique to the project. Make sure it's added to the `<head>` section of the page. The validation process checks that the meta tag is present, and + checks its value. + +All these methods are equivalent in functionality. Use whichever is feasible. + +In [GitLab 14.2 and later](https://gitlab.com/gitlab-org/gitlab/-/issues/324990), site profile +validation happens in a CI job using the [GitLab Runner](../../../ci/runners/index.md). + +### Create a site profile + +To create a site profile: + +1. From your project's home page, go to **Security & Compliance > Configuration**. +1. Select **Manage** in the **DAST Profiles** row. +1. Select **New > Site Profile**. +1. Complete the fields then select **Save profile**. + +The site profile is created. + +### Edit a site profile + +If a site profile is linked to a security policy, a user cannot edit the profile from this page. See +[Scan execution policies](../policies/scan-execution-policies.md) +for more information. + +When a validated site profile's file, header, or meta tag is edited, the site's +[validation status](#site-profile-validation) is revoked. + +To edit a site profile: + +1. From your project's home page, go to **Security & Compliance > Configuration**. +1. In the **DAST Profiles** row select **Manage**. +1. Select the **Site Profiles** tab. +1. In the profile's row select the **More actions** (**{ellipsis_v}**) menu, then select **Edit**. +1. Edit the fields then select **Save profile**. + +### Delete a site profile + +If a site profile is linked to a security policy, a user cannot delete the profile from this page. +See [Scan execution policies](../policies/scan-execution-policies.md) +for more information. + +To delete a site profile: + +1. From your project's home page, go to **Security & Compliance > Configuration**. +1. In the **DAST Profiles** row select **Manage**. +1. Select the **Site Profiles** tab. +1. In the profile's row, select the **More actions** (**{ellipsis_v}**) menu, then select **Delete**. +1. Select **Delete** to confirm the deletion. + +### Validate a site profile + +Validating a site is required to run an active scan. + +To validate a site profile: + +1. On the top bar, select **Main menu > Projects** and find your project. +1. On the left sidebar, select **Security & Compliance > Configuration**. +1. In the **Dynamic Application Security Testing (DAST)** section, select **Manage profiles**. +1. Select the **Site Profiles** tab. +1. In the profile's row, select **Validate**. +1. Select the validation method. + 1. For **Text file validation**: + 1. Download the validation file listed in **Step 2**. + 1. Upload the validation file to the host, to the location in **Step 3** or any location you + prefer. + 1. If required, edit the file location in **Step 3**. + 1. Select **Validate**. + 1. For **Header validation**: + 1. Select the clipboard icon in **Step 2**. + 1. Edit the header of the site to validate, and paste the clipboard content. + 1. Select the input field in **Step 3** and enter the location of the header. + 1. Select **Validate**. + 1. For **Meta tag validation**: + 1. Select the clipboard icon in **Step 2**. + 1. Edit the content of the site to validate, and paste the clipboard content. + 1. Select the input field in **Step 3** and enter the location of the meta tag. + 1. Select **Validate**. + +The site is validated and an active scan can run against it. A site profile's validation status is +revoked only when it's revoked manually, or its file, header, or meta tag is edited. + +### Retry a failed validation + +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/322609) in GitLab 14.3. +> - [Deployed behind the `dast_failed_site_validations` flag](../../../administration/feature_flags.md), enabled by default. +> - [Feature flag `dast_failed_site_validations` removed](https://gitlab.com/gitlab-org/gitlab/-/issues/323961) in GitLab 14.4. + +Failed site validation attempts are listed on the **Site profiles** tab of the **Manage profiles** +page. + +To retry a site profile's failed validation: + +1. On the top bar, select **Main menu > Projects** and find your project. +1. On the left sidebar, select **Security & Compliance > Configuration**. +1. In the **Dynamic Application Security Testing (DAST)** section, select **Manage profiles**. +1. Select the **Site Profiles** tab. +1. In the profile's row, select **Retry validation**. + +### Revoke a site profile's validation status + +WARNING: +When a site profile's validation status is revoked, all site profiles that share the same URL also +have their validation status revoked. + +To revoke a site profile's validation status: + +1. From your project's home page, go to **Security & Compliance > Configuration**. +1. In the **DAST Profiles** row select **Manage**. +1. Beside the validated profile, select **Revoke validation**. + +The site profile's validation status is revoked. + +### Validated site profile headers + +The following are code samples of how you can provide the required site profile header in your +application. + +#### Ruby on Rails example for on-demand scan + +Here's how you can add a custom header in a Ruby on Rails application: + +```ruby +class DastWebsiteTargetController < ActionController::Base + def dast_website_target + response.headers['Gitlab-On-Demand-DAST'] = '0dd79c9a-7b29-4e26-a815-eaaf53fcab1c' + head :ok + end +end +``` + +#### Django example for on-demand scan + +Here's how you can add a +[custom header in Django](https://docs.djangoproject.com/en/2.2/ref/request-response/#setting-header-fields): + +```python +class DastWebsiteTargetView(View): + def head(self, *args, **kwargs): + response = HttpResponse() + response['Gitlab-On-Demand-DAST'] = '0dd79c9a-7b29-4e26-a815-eaaf53fcab1c' + + return response +``` + +#### Node (with Express) example for on-demand scan + +Here's how you can add a +[custom header in Node (with Express)](https://expressjs.com/en/5x/api.html#res.append): + +```javascript +app.get('/dast-website-target', function(req, res) { + res.append('Gitlab-On-Demand-DAST', '0dd79c9a-7b29-4e26-a815-eaaf53fcab1c') + res.send('Respond to DAST ping') +}) +``` + +## Scanner profile + +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/222767) in GitLab 13.4. +> - [Added](https://gitlab.com/gitlab-org/gitlab/-/issues/225804) in GitLab 13.5: scan mode, AJAX spider, debug messages. + +A scanner profile defines the configuration details of a security scanner. A scanner profile can be +referenced in `.gitlab-ci.yml` and on-demand scans. + +A scanner profile contains: + +- **Profile name:** A name you give the scanner profile. For example, "Spider_15". While a scanner + profile is referenced in either `.gitlab-ci.yml` or an on-demand scan, it **cannot** be renamed. +- **Scan mode:** A passive scan monitors all HTTP messages (requests and responses) sent to the target. An active scan attacks the target to find potential vulnerabilities. +- **Spider timeout:** The maximum number of minutes allowed for the spider to traverse the site. +- **Target timeout:** The maximum number of seconds DAST waits for the site to be available before + starting the scan. +- **AJAX spider:** Run the AJAX spider, in addition to the traditional spider, to crawl the target site. +- **Debug messages:** Include debug messages in the DAST console output. + +### Create a scanner profile + +To create a scanner profile: + +1. From your project's home page, go to **Security & Compliance > Configuration**. +1. In the **DAST Profiles** row, select **Manage**. +1. Select **New > Scanner Profile**. +1. Complete the form. For details of each field, see [Scanner profile](#scanner-profile). +1. Select **Save profile**. + +### Edit a scanner profile + +If a scanner profile is linked to a security policy, a user cannot edit the profile from this page. +See [Scan execution policies](../policies/scan-execution-policies.md) +for more information. + +To edit a scanner profile: + +1. From your project's home page, go to **Security & Compliance > Configuration**. +1. In the **DAST Profiles** row, select **Manage**. +1. Select the **Scanner Profiles** tab. +1. In the scanner's row, select the **More actions** (**{ellipsis_v}**) menu, then select **Edit**. +1. Edit the form. +1. Select **Save profile**. + +### Delete a scanner profile + +If a scanner profile is linked to a security policy, a user cannot delete the profile from this +page. See [Scan execution policies](../policies/scan-execution-policies.md) +for more information. + +To delete a scanner profile: + +1. From your project's home page, go to **Security & Compliance > Configuration**. +1. In the **DAST Profiles** row, select **Manage**. +1. Select the **Scanner Profiles** tab. +1. In the scanner's row, select the **More actions** (**{ellipsis_v}**) menu, then select **Delete**. +1. Select **Delete**. + +## Auditing + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/217872) in GitLab 14.1. + +The creation, updating, and deletion of DAST profiles, DAST scanner profiles, +and DAST site profiles are included in the [audit log](../../../administration/audit_events.md). + +## Reports + +The DAST tool outputs a `gl-dast-report.json` report file containing details of the scan and its results. +This file is included in the job's artifacts. JSON is the default format, but +you can output the report in Markdown, HTML, and XML formats. To specify an alternative +format, use a [CI/CD variable](#available-cicd-variables). You can also use a CI/CD variable +to configure the job to output the `gl-dast-debug-auth-report.html` file which helps when debugging +authentication issues. + +For details of the report's schema, see the [schema for DAST reports](https://gitlab.com/gitlab-org/security-products/security-report-schemas/-/blob/master/dist/dast-report-format.json). Example reports can be found in the +[DAST repository](https://gitlab.com/gitlab-org/security-products/dast/-/tree/main/test/end-to-end/expect). + +WARNING: +The JSON report artifacts are not a public API of DAST and their format is expected to change in the +future. diff --git a/doc/user/application_security/dast_api/index.md b/doc/user/application_security/dast_api/index.md index eae32f789b8..d77be0f0ca9 100644 --- a/doc/user/application_security/dast_api/index.md +++ b/doc/user/application_security/dast_api/index.md @@ -7,20 +7,19 @@ type: reference, howto # DAST API **(ULTIMATE)** -You can add dynamic application security testing (DAST) of web APIs to your -[GitLab CI/CD](../../../ci/index.md) pipelines. This helps you discover bugs and potential security -issues that other QA processes may miss. +> DAST API analyzer [became the default analyzer for on-demand DAST API scans](https://gitlab.com/groups/gitlab-org/-/epics/4254) in GitLab 15.6. -We recommend that you use DAST API testing in addition to [GitLab Secure](../index.md)'s -other security scanners and your own test processes. If you're using [GitLab CI/CD](../../../ci/index.md), -you can run DAST API tests as part your CI/CD workflow. +Perform Dynamic Application Security Testing (DAST) of web APIs to help discover bugs and potential +security issues that other QA processes may miss. Use DAST API tests in addition to +[GitLab Secure](../index.md)'s other security scanners and your own test processes. You can run DAST +API tests either as part your CI/CD workflow, [on-demand](../dast/proxy-based.md#on-demand-scans), or both. WARNING: -Do not run DAST API testing against a production server. Not only can it perform *any* function that +Do not run DAST API testing against a production server. Not only can it perform _any_ function that the API can, it may also trigger bugs in the API. This includes actions like modifying and deleting data. Only run DAST API against a test server. -You can run DAST API scanning against the following web API types: +DAST API can test the following web API types: - REST API - SOAP @@ -29,9 +28,9 @@ You can run DAST API scanning against the following web API types: ## When DAST API scans run -DAST API scanning runs in the `dast` stage by default. To ensure DAST API scanning examines the latest -code, ensure your CI/CD pipeline deploys changes to a test environment in a stage before the `dast` -stage. +When run in your CI/CD pipeline, DAST API scanning runs in the `dast` stage by default. To ensure +DAST API scanning examines the latest code, ensure your CI/CD pipeline deploys changes to a test +environment in a stage before the `dast` stage. If your pipeline is configured to deploy to the same web server on each run, running a pipeline while another is still running could cause a race condition in which one pipeline overwrites the @@ -2057,9 +2056,232 @@ Setting `SECURE_ANALYZERS_PREFIX` changes the Docker image registry location for For more information, see [Offline environments](../offline_deployments/index.md). +## Performance tuning and testing speed + +Security tools that perform dynamic analysis testing, such as DAST API, perform testing by sending requests to an instance of your running application. The requests are engineered to test for specific vulnerabilities that might exist in your application. The speed of a dynamic analysis test depends on the following: + +- How many requests per second can be sent to your application by our tooling +- How fast your application responds to requests +- How many requests must be sent to test the application + - How many operations your API is comprised of + - How many fields are in each operation (think JSON bodies, headers, query string, cookies, etc.) + +If the DAST API testing job still takes longer than expected reach after following the advice in this performance guide, reach out to support for further assistance. + +### Diagnosing performance issues + +The first step to resolving performance issues is to understand what is contributing to the slower-than-expected testing time. Some common issues we see are: + +- DAST API is running on a slow or single-CPU GitLab Runner (GitLab Shared Runners are single-CPU) +- The application deployed to a slow/single-CPU instance and is not able to keep up with the testing load +- The application contains a slow operation that impacts the overall test speed (> 1/2 second) +- The application contains an operation that returns a large amount of data (> 500K+) +- The application contains a large number of operations (> 40) + +#### The application contains a slow operation that impacts the overall test speed (> 1/2 second) + +The DAST API job output contains helpful information about how fast we are testing, how fast each operation being tested responds, and summary information. Let's take a look at some sample output to see how it can be used in tracking down performance issues: + +```shell +API Security: Loaded 10 operations from: assets/har-large-response/large_responses.har +API Security: +API Security: Testing operation [1/10]: 'GET http://target:7777/api/large_response_json'. +API Security: - Parameters: (Headers: 4, Query: 0, Body: 0) +API Security: - Request body size: 0 Bytes (0 bytes) +API Security: +API Security: Finished testing operation 'GET http://target:7777/api/large_response_json'. +API Security: - Excluded Parameters: (Headers: 0, Query: 0, Body: 0) +API Security: - Performed 767 requests +API Security: - Average response body size: 130 MB +API Security: - Average call time: 2 seconds and 82.69 milliseconds (2.082693 seconds) +API Security: - Time to complete: 14 minutes, 8 seconds and 788.36 milliseconds (848.788358 seconds) +``` + +This job console output snippet starts by telling us how many operations were found (10), followed by notifications that testing has started on a specific operation and a summary of the operation has been completed. The summary is the most interesting part of this log output. In the summary, we can see that it took DAST API 767 requests to fully test this operation and its related fields. We can also see that the average response time was 2 seconds and the time to complete was 14 minutes for this one operation. + +An average response time of 2 seconds is a good initial indicator that this specific operation takes a long time to test. Further, we can see that the response body size is quite large. The large body size is the culprit here, transferring that much data on each request is what takes the majority of that 2 seconds. + +For this issue, the team might decide to: + +- Use a multi-CPU runner. Using a multi-CPU runner allows DAST API to parallelize the work being performed. This helps lower the test time, but getting the test down under 10 minutes might still be problematic without moving to a high CPU machine due to how long the operation takes to test. + - Trade off between how many CPUs and cost. +- [Exclude this operation](#excluding-slow-operations) from the DAST API test. While this is the simplest, it has the downside of a gap in security test coverage. +- [Exclude the operation from feature branch DAST API tests, but include it in the default branch test](#excluding-operations-in-feature-branches-but-not-default-branch). +- [Split up the DAST API testing into multiple jobs](#splitting-a-test-into-multiple-jobs). + +The likely solution is to use a combination of these solutions to reach an acceptable test time, assuming your team's requirements are in the 5-7 minute range. + +### Addressing performance issues + +The following sections document various options for addressing performance issues for DAST API: + +- [Using a multi-CPU Runner](#using-a-multi-cpu-runner) +- [Excluding slow operations](#excluding-slow-operations) +- [Splitting a test into multiple jobs](#splitting-a-test-into-multiple-jobs) +- [Excluding operations in feature branches, but not default branch](#excluding-operations-in-feature-branches-but-not-default-branch) + +#### Using a multi-CPU Runner + +One of the easiest performance boosts can be achieved using a multi-CPU runner with DAST API. This table shows statistics collected during benchmarking of a Java Spring Boot REST API. In this benchmark, the target and DAST API share a single runner instance. + +| CPU Count | Request per Second | +|----------------------|--------------------| +| 1 CPU (Shared Runner)| 75 | +| 4 CPU | 255 | +| 8 CPU | 400 | + +As we can see from this table, increasing the CPU count of the runner can have a large impact on testing speed/performance. + +To use a multi-CPU typically requires deploying a self-managed GitLab Runner onto a multi-CPU machine or cloud compute instance. + +When multiple types of GitLab Runners are available for use, the various instances are commonly set up with tags that can be used in the job definition to select a type of runner. + +Here is an example job definition for DAST API that adds a `tags` section with the tag `multi-cpu`. The job automatically extends the job definition included through the DAST API template. + +```yaml +dast_api: + tags: + - multi-cpu +``` + +To verify that DAST API can detect multiple CPUs in the runner, download the `gl-api-security-scanner.log` file from a completed job's artifacts. Search the file for the string `Starting work item processor` and inspect the reported max DOP (degree of parallelism). The max DOP should be greater than or equal to the number of CPUs assigned to the runner. The value is never lower than 2, even on single CPU runners, unless forced through a configuration variable. If the value reported is less than the number of CPUs assigned to the runner, then something is wrong with the runner deployment. If unable to identify the problem, open a ticket with support to assist. + +Example log entry: + +`17:00:01.084 [INF] <Peach.Web.Core.Services.WebRunnerMachine> Starting work item processor with 2 max DOP` + +#### Excluding slow operations + +In the case of one or two slow operations, the team might decide to skip testing the operations. Excluding the operation is done using the `DAST_API_EXCLUDE_PATHS` configuration [variable as explained in this section.](#exclude-paths) + +In this example, we have an operation that returns a large amount of data. The operation is `GET http://target:7777/api/large_response_json`. To exclude it we provide the `DAST_API_EXCLUDE_PATHS` configuration variable with the path portion of our operation URL `/api/large_response_json`. + +To verify the operation is excluded, run the DAST API job and review the job console output. It includes a list of included and excluded operations at the end of the test. + +```yaml +dast_api: + variables: + DAST_API_EXCLUDE_PATHS: /api/large_response_json +``` + +Excluding operations from testing could allow some vulnerabilities to go undetected. +{: .alert .alert-warning} + +#### Splitting a test into multiple jobs + +Splitting a test into multiple jobs is supported by DAST API through the use of [`DAST_API_EXCLUDE_PATHS`](#exclude-paths) and [`DAST_API_EXCLUDE_URLS`](#exclude-urls). When splitting a test up, a good pattern is to disable the `dast_api` job and replace it with two jobs with identifying names. In this example we have two jobs, each job is testing a version of the API, so our names reflect that. However, this technique can be applied to any situation, not just with versions of an API. + +The rules we are using in the `dast_api_v1` and `dast_api_v2` jobs are copied from the [DAST API template](https://gitlab.com/gitlab-org/gitlab/blob/master/lib/gitlab/ci/templates/Security/DAST-API.gitlab-ci.yml). + +```yaml +# Disable the main dast_api job +dast_api: + rules: + - if: $CI_COMMIT_BRANCH + when: never + +dast_api_v1: + extends: dast_api + variables: + DAST_API_EXCLUDE_PATHS: /api/v1/** + rules: + - if: $DAST_API_DISABLED + when: never + - if: $DAST_API_DISABLED_FOR_DEFAULT_BRANCH && + $CI_DEFAULT_BRANCH == $CI_COMMIT_REF_NAME + when: never + - if: $CI_COMMIT_BRANCH && + $CI_GITLAB_FIPS_MODE == "true" + variables: + DAST_API_IMAGE_SUFFIX: "-fips" + - if: $CI_COMMIT_BRANCH + +dast_api_v2: + variables: + DAST_API_EXCLUDE_PATHS: /api/v2/** + rules: + - if: $DAST_API_DISABLED + when: never + - if: $DAST_API_DISABLED_FOR_DEFAULT_BRANCH && + $CI_DEFAULT_BRANCH == $CI_COMMIT_REF_NAME + when: never + - if: $CI_COMMIT_BRANCH && + $CI_GITLAB_FIPS_MODE == "true" + variables: + DAST_API_IMAGE_SUFFIX: "-fips" + - if: $CI_COMMIT_BRANCH +``` + +#### Excluding operations in feature branches, but not default branch + +In the case of one or two slow operations, the team might decide to skip testing the operations, or exclude them from feature branch tests, but include them for default branch tests. Excluding the operation is done using the `DAST_API_EXCLUDE_PATHS` configuration [variable as explained in this section.](#exclude-paths) + +In this example, we have an operation that returns a large amount of data. The operation is `GET http://target:7777/api/large_response_json`. To exclude it we provide the `DAST_API_EXCLUDE_PATHS` configuration variable with the path portion of our operation URL `/api/large_response_json`. Our configuration disables the main `dast_api` job and creates two new jobs `dast_api_main` and `dast_api_branch`. The `dast_api_branch` is set up to exclude the long operation and only run on non-default branches (e.g. feature branches). The `dast_api_main` branch is set up to only execute on the default branch (`main` in this example). The `dast_api_branch` jobs run faster, allowing for quick development cycles, while the `dast_api_main` job which only runs on default branch builds, takes longer to run. + +To verify the operation is excluded, run the DAST API job and review the job console output. It includes a list of included and excluded operations at the end of the test. + +```yaml +# Disable the main job so we can create two jobs with +# different names +dast_api: + rules: + - if: $CI_COMMIT_BRANCH + when: never + +# DAST API for feature branch work, excludes /api/large_response_json +dast_api_branch: + extends: dast_api + variables: + DAST_API_EXCLUDE_PATHS: /api/large_response_json + rules: + - if: $DAST_API_DISABLED + when: never + - if: $DAST_API_DISABLED_FOR_DEFAULT_BRANCH && + $CI_DEFAULT_BRANCH == $CI_COMMIT_REF_NAME + when: never + - if: $CI_COMMIT_BRANCH && + $CI_GITLAB_FIPS_MODE == "true" + variables: + DAST_API_IMAGE_SUFFIX: "-fips" + - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH + when: never + - if: $CI_COMMIT_BRANCH + +# DAST API for default branch (main in our case) +# Includes the long running operations +dast_api_main: + extends: dast_api + rules: + - if: $DAST_API_DISABLED + when: never + - if: $DAST_API_DISABLED_FOR_DEFAULT_BRANCH && + $CI_DEFAULT_BRANCH == $CI_COMMIT_REF_NAME + when: never + - if: $CI_COMMIT_BRANCH && + $CI_GITLAB_FIPS_MODE == "true" + variables: + DAST_API_IMAGE_SUFFIX: "-fips" + - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH +``` + ## Troubleshooting -### `Error waiting for API Security 'http://127.0.0.1:5000' to become available` +### DAST API job times out after N hours + +The top two reasons for the DAST API job timing out are slow operations (> 1 second) and using a single-CPU runner for DAST API (GitLab shared runners are single-CPU). Before you can diagnose the problem further, the job must complete so the output can be analyzed. We recommend to start with a multi-CPU runner first, then exclude portions of your API operations until the job completes and the output can be further reviewed. + +See the following documentation sections for assistance: + +- [Performance tuning and testing speed](#performance-tuning-and-testing-speed) +- [Using a multi-CPU Runner](#using-a-multi-cpu-runner) +- [Excluding operations by path](#exclude-paths) +- [Excluding slow operations](#excluding-slow-operations) + +### DAST API job takes too long to complete + +See [Performance Tuning and Testing Speed](#performance-tuning-and-testing-speed) + +### Error waiting for API Security 'http://127.0.0.1:5000' to become available A bug exists in versions of the DAST API analyzer prior to v1.6.196 that can cause a background process to fail under certain conditions. The solution is to update to a newer version of the DAST API analyzer. diff --git a/doc/user/application_security/dependency_scanning/index.md b/doc/user/application_security/dependency_scanning/index.md index a3c6c46b081..4ed9ceedb4d 100644 --- a/doc/user/application_security/dependency_scanning/index.md +++ b/doc/user/application_security/dependency_scanning/index.md @@ -221,7 +221,7 @@ table.supported-languages ul { <td>N</td> </tr> <tr> - <td rowspan="2">JavaScript</td> + <td rowspan="2">JavaScript and TypeScript</td> <td>All versions</td> <td><a href="https://www.npmjs.com/">npm</a></td> <td> @@ -501,11 +501,11 @@ From GitLab 14.8 the `gemnasium` analyzer scans supported JavaScript projects fo #### Go -When scanning a Go project, gemnasium invokes a builder and attempts to generate a [build list](https://go.dev/ref/mod#glos-build-list) using -[Minimal Version Selection](https://go.dev/ref/mod#glos-minimal-version-selection). If a non-fatal error is encountered, the build process signals -that the execution should proceed and falls back to parsing the available `go.sum` file. +Multiple files are supported. When a `go.mod` file is detected, the analyzer attempts to generate a [build list](https://go.dev/ref/mod#glos-build-list) using +[Minimal Version Selection](https://go.dev/ref/mod#glos-minimal-version-selection). If a non-fatal error is encountered, the analyzer falls back to parsing the +available `go.sum` file. The process is repeated for every detected `go.mod` and `go.sum` file. -#### PHP, Go, C, C++, .NET, C#, Ruby, JavaScript +#### PHP, C, C++, .NET, C#, Ruby, JavaScript The analyzer for these languages supports multiple lockfiles. @@ -525,7 +525,7 @@ The [Security Scanner Integration](../../../development/integrations/secure.md) To enable dependency scanning for GitLab 11.9 and later, you must [include](../../../ci/yaml/index.md#includetemplate) the -[`Dependency-Scanning.gitlab-ci.yml` template](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/ci/templates/Security/Dependency-Scanning.gitlab-ci.yml) +[`Dependency-Scanning.gitlab-ci.yml` template](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/ci/templates/Jobs/Dependency-Scanning.gitlab-ci.yml) that is provided as a part of your GitLab installation. For GitLab versions earlier than 11.9, you can copy and use the job as defined that template. @@ -534,7 +534,7 @@ Add the following to your `.gitlab-ci.yml` file: ```yaml include: - - template: Security/Dependency-Scanning.gitlab-ci.yml + - template: Jobs/Dependency-Scanning.gitlab-ci.yml ``` The included template creates dependency scanning jobs in your CI/CD @@ -624,7 +624,6 @@ The following variables allow configuration of global dependency scanning settin | ----------------------------|------------ | | `ADDITIONAL_CA_CERT_BUNDLE` | Bundle of CA certs to trust. The bundle of certificates provided here is also used by other tools during the scanning process, such as `git`, `yarn`, or `npm`. See [Using a custom SSL CA certificate authority](#using-a-custom-ssl-ca-certificate-authority) for more details. | | `DS_EXCLUDED_ANALYZERS` | Specify the analyzers (by name) to exclude from Dependency Scanning. For more information, see [Dependency Scanning Analyzers](analyzers.md). | -| `DS_DEFAULT_ANALYZERS` | This feature was [deprecated](https://gitlab.com/gitlab-org/gitlab/-/issues/287691) in GitLab 14.0 and [removed](https://gitlab.com/gitlab-org/gitlab/-/issues/333299) in 15.0. Use `DS_EXCLUDED_ANALYZERS` instead. | | `DS_EXCLUDED_PATHS` | Exclude files and directories from the scan based on the paths. A comma-separated list of patterns. Patterns can be globs (see [`doublestar.Match`](https://pkg.go.dev/github.com/bmatcuk/doublestar/v4@v4.0.2#Match) for supported patterns), or file or folder paths (for example, `doc,spec`). Parent directories also match patterns. Default: `"spec, test, tests, tmp"`. | | `DS_IMAGE_SUFFIX` | Suffix added to the image name. ([Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/354796) in GitLab 14.10.) Automatically set to `"-fips"` when FIPS mode is enabled. ([Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/357922) in GitLab 15.0.) | | `SECURE_ANALYZERS_PREFIX` | Override the name of the Docker registry providing the official default images (proxy). Read more about [customizing analyzers](analyzers.md). | @@ -641,6 +640,7 @@ The following variables are used for configuring specific analyzers (used for a | `GEMNASIUM_DB_REMOTE_URL` | `gemnasium` | `https://gitlab.com/gitlab-org/security-products/gemnasium-db.git` | Repository URL for fetching the Gemnasium database. | | `GEMNASIUM_DB_REF_NAME` | `gemnasium` | `master` | Branch name for remote repository database. `GEMNASIUM_DB_REMOTE_URL` is required. | | `DS_REMEDIATE` | `gemnasium` | `"true"`, `"false"` in FIPS mode | Enable automatic remediation of vulnerable dependencies. Not supported in FIPS mode. | +| `DS_REMEDIATE_TIMEOUT` | `gemnasium` | `5m` | Timeout for auto-remediation. | | `GEMNASIUM_LIBRARY_SCAN_ENABLED` | `gemnasium` | `"true"` | Enable detecting vulnerabilities in vendored JavaScript libraries. For now, `gemnasium` leverages [`Retire.js`](https://github.com/RetireJS/retire.js) to do this job. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/350512) in GitLab 14.8. | | `DS_JAVA_VERSION` | `gemnasium-maven` | `17` | Version of Java. Available versions: `8`, `11`, `13`, `14`, `15`, `16`, `17`. Available versions in FIPS-enabled image: `8`, `11`, `17`. | | `MAVEN_CLI_OPTS` | `gemnasium-maven` | `"-DskipTests --batch-mode"` | List of command line arguments that are passed to `maven` by the analyzer. See an example for [using private repositories](../index.md#using-private-maven-repositories). | @@ -1311,6 +1311,11 @@ gemnasium-python-dependency_scanning: - apt-get update && apt-get install -y libpq-dev ``` +### `NoSuchOptionException` when using `poetry config http-basic` with `CI_JOB_TOKEN` + +This error can occur when the automatically generated `CI_JOB_TOKEN` starts with a hyphen (`-`). +To avoid this error, follow [Poetry's configuration advice](https://python-poetry.org/docs/repositories/#configuring-credentials). + ### Error: Project has `<number>` unresolved dependencies The error message `Project has <number> unresolved dependencies` indicates a dependency resolution problem caused by your `gradle.build` or `gradle.build.kts` file. In the current release, `gemnasium-maven` cannot continue processing when an unresolved dependency is encountered. However, There is an [open issue](https://gitlab.com/gitlab-org/gitlab/-/issues/337083) to allow `gemnasium-maven` to recover from unresolved dependency errors and produce a dependency graph. Until this issue has been resolved, you'll need to consult the [Gradle dependency resolution docs](https://docs.gradle.org/current/userguide/dependency_resolution.html) for details on how to fix your `gradle.build` file. diff --git a/doc/user/application_security/get-started-security.md b/doc/user/application_security/get-started-security.md index b6213a98f91..5774bf940b0 100644 --- a/doc/user/application_security/get-started-security.md +++ b/doc/user/application_security/get-started-security.md @@ -36,7 +36,7 @@ The following steps will help you get the most from GitLab application security remediating existing vulnerabilities and preventing the introduction of new ones. 1. Enable other scan types such as [SAST](sast/index.md), [DAST](dast/index.md), [Fuzz testing](coverage_fuzzing/index.md), or [Container Scanning](container_scanning/index.md). -1. Use [Compliance Pipelines](../group/manage.md#configure-a-compliance-pipeline) +1. Use [Compliance Pipelines](../group/compliance_frameworks.md#configure-a-compliance-pipeline) or [Scan Execution Policies](policies/scan-execution-policies.md) to enforce required scan types and ensure separation of duties between security and engineering. 1. Consider enabling [Review Apps](../../development/testing_guide/review_apps.md) to allow for DAST diff --git a/doc/user/application_security/iac_scanning/index.md b/doc/user/application_security/iac_scanning/index.md index 150c2b732d8..1c14c529523 100644 --- a/doc/user/application_security/iac_scanning/index.md +++ b/doc/user/application_security/iac_scanning/index.md @@ -4,25 +4,27 @@ group: Static Analysis info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- -# Infrastructure as Code (IaC) Scanning +# Infrastructure as Code (IaC) Scanning **(FREE)** > [Introduced](https://gitlab.com/groups/gitlab-org/-/epics/6655) in GitLab 14.5. Infrastructure as Code (IaC) Scanning scans your IaC configuration files for known vulnerabilities. -Currently, IaC scanning supports configuration files for Terraform, Ansible, AWS CloudFormation, and Kubernetes. +IaC Scanning supports configuration files for Terraform, Ansible, AWS CloudFormation, and Kubernetes. ## Requirements IaC Scanning runs in the `test` stage, which is available by default. If you redefine the stages in the `.gitlab-ci.yml` file, the `test` stage is required. -To run IaC scanning jobs, by default, you need GitLab Runner with the +We recommend a minimum of 4GB RAM to ensure consistent performance. + +To run IaC Scanning jobs, by default, you need GitLab Runner with the [`docker`](https://docs.gitlab.com/runner/executors/docker.html) or [`kubernetes`](https://docs.gitlab.com/runner/install/kubernetes.html) executor. If you're using the shared runners on GitLab.com, this is enabled by default. WARNING: -Our IaC scanning jobs require a Linux/amd64 container type. Windows containers are not yet supported. +Our IaC Scanning jobs require a Linux/amd64 container type. Windows containers are not supported. WARNING: If you use your own runners, make sure the Docker version installed @@ -30,20 +32,20 @@ is **not** `19.03.0`. See [troubleshooting information](../sast/index.md#error-r ## Supported languages and frameworks -GitLab IaC scanning supports a variety of IaC configuration files. Our IaC security scanners also feature automatic language detection which works even for mixed-language projects. If any supported configuration files are detected in project source code we automatically run the appropriate IaC analyzers. +GitLab IaC Scanning supports a variety of IaC configuration files. Our IaC security scanners also feature automatic language detection which works even for mixed-language projects. If any supported configuration files are detected in project source code we automatically run the appropriate IaC analyzers. -| Configuration File Type | Scan tool | Introduced in GitLab Version | -|------------------------------------------|----------------------------------|-------------------------------| -| Ansible | [KICS](https://kics.io/) | 14.5 | -| AWS CloudFormation | [KICS](https://kics.io/) | 14.5 | -| Azure Resource Manager <sup>1</sup> | [KICS](https://kics.io/) | 14.5 | -| Dockerfile | [KICS](https://kics.io/) | 14.5 | -| Google Deployment Manager | [KICS](https://kics.io/) | 14.5 | -| Kubernetes | [KICS](https://kics.io/) | 14.5 | -| OpenAPI | [KICS](https://kics.io/) | 14.5 | -| Terraform <sup>2</sup> | [KICS](https://kics.io/) | 14.5 | +| Configuration file type | Scan tool | Introduced in GitLab version | +| ----------------------------------- | ------------------------ | ---------------------------- | +| Ansible | [KICS](https://kics.io/) | 14.5 | +| AWS CloudFormation | [KICS](https://kics.io/) | 14.5 | +| Azure Resource Manager <sup>1</sup> | [KICS](https://kics.io/) | 14.5 | +| Dockerfile | [KICS](https://kics.io/) | 14.5 | +| Google Deployment Manager | [KICS](https://kics.io/) | 14.5 | +| Kubernetes | [KICS](https://kics.io/) | 14.5 | +| OpenAPI | [KICS](https://kics.io/) | 14.5 | +| Terraform <sup>2</sup> | [KICS](https://kics.io/) | 14.5 | -1. IaC scanning can analyze Azure Resource Manager templates in JSON format. If you write templates in the [Bicep](https://learn.microsoft.com/en-us/azure/azure-resource-manager/bicep/overview) language, you must use [the bicep CLI](https://learn.microsoft.com/en-us/azure/azure-resource-manager/bicep/bicep-cli) to convert your Bicep files into JSON before GitLab IaC scanning can analyze them. +1. IaC Scanning can analyze Azure Resource Manager templates in JSON format. If you write templates in the [Bicep](https://learn.microsoft.com/en-us/azure/azure-resource-manager/bicep/overview) language, you must use [the bicep CLI](https://learn.microsoft.com/en-us/azure/azure-resource-manager/bicep/bicep-cli) to convert your Bicep files into JSON before GitLab IaC Scanning can analyze them. 1. Terraform modules in a custom registry are not scanned for vulnerabilities. You can follow [this issue](https://gitlab.com/gitlab-org/gitlab/-/issues/357004) for the proposed feature. ### Supported distributions @@ -77,7 +79,7 @@ Different features are available in different [GitLab tiers](https://about.gitla as shown in the following table: | Capability | In Free & Premium | In Ultimate | -|:----------------------------------------------------------------|:--------------------|:-------------------| +| :-------------------------------------------------------------- | :------------------ | :----------------- | | [Configure IaC scanner](#configuration) | **{check-circle}** | **{check-circle}** | | Download [JSON Report](#reports-json-format) | **{check-circle}** | **{check-circle}** | | See new findings in merge request widget | **{dotted-circle}** | **{check-circle}** | @@ -98,14 +100,14 @@ To configure IaC Scanning for a project you can: ### Configure IaC Scanning manually To enable IaC Scanning you must [include](../../../ci/yaml/index.md#includetemplate) the -[`SAST-IaC.gitlab-ci.yml template`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/ci/templates/Jobs/SAST-IaC.gitlab-ci.yml) provided as part of your GitLab installation. Here is an example of how to include it: +[`SAST-IaC.gitlab-ci.yml` template](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/ci/templates/Jobs/SAST-IaC.gitlab-ci.yml) provided as part of your GitLab installation. Here is an example of how to include it: ```yaml include: - template: Jobs/SAST-IaC.gitlab-ci.yml ``` -The included template creates IaC scanning jobs in your CI/CD pipeline and scans +The included template creates IaC Scanning jobs in your CI/CD pipeline and scans your project's configuration files for possible vulnerabilities. The results are saved as a @@ -121,7 +123,123 @@ To enable IaC Scanning in a project, you can create a merge request: 1. In the **Infrastructure as Code (IaC) Scanning** row, select **Configure with a merge request**. 1. Review and merge the merge request to enable IaC Scanning. -Pipelines now include an IaC job. +Pipelines now include an IaC Scanning job. + +## Customize rulesets **(ULTIMATE)** + +> [Added](https://gitlab.com/gitlab-org/gitlab/-/issues/235359) support for overriding rules in GitLab 14.8. + +You can customize the default IaC Scanning rules provided with GitLab. + +The following customization options can be used separately, or together: + +- [Disable predefined rules](#disable-predefined-analyzer-rules). +- [Override predefined rules](#override-predefined-analyzer-rules). + +### Disable predefined analyzer rules + +If there are specific IaC Scanning rules that you don't want active, you can disable them. + +To disable analyzer rules: + +1. Create a `.gitlab` directory at the root of your project, if one doesn't already exist. +1. Create a custom ruleset file named `sast-ruleset.toml` in the `.gitlab` directory, if + one doesn't already exist. +1. Set the `disabled` flag to `true` in the context of a `ruleset` section. +1. In one or more `ruleset` subsections, list the rules to disable. Every + `ruleset.identifier` section has: + - A `type` field for the rule. For IaC Scanning, the identifier type is `kics_id`. + - A `value` field for the rule identifier. KICS rule identifiers are alphanumeric strings. To find the rule identifier, you can: + - Find it in the [JSON report artifact](#reports-json-format). + - Search for the rule name in the [list of KICS queries](https://docs.kics.io/latest/queries/all-queries/) and copy the alphanumeric identifier that's shown. The rule name is shown on the [Vulnerability Page](../vulnerabilities/index.md) when a rule violation is detected. + +In the following example `sast-ruleset.toml` file, the disabled rules are assigned to +the `kics` analyzer by matching the `type` and `value` of identifiers: + +```toml +[kics] + [[kics.ruleset]] + disable = true + [kics.ruleset.identifier] + type = "kics_id" + value = "8212e2d7-e683-49bc-bf78-d6799075c5a7" + + [[kics.ruleset]] + disable = true + [kics.ruleset.identifier] + type = "kics_id" + value = "b03a748a-542d-44f4-bb86-9199ab4fd2d5" +``` + +### Override predefined analyzer rules + +If there are specific IaC Scanning rules you want to customize, you can override them. For +example, you might lower the severity of a rule or link to your own documentation about how to fix a finding. + +To override rules: + +1. Create a `.gitlab` directory at the root of your project, if one doesn't already exist. +1. Create a custom ruleset file named `sast-ruleset.toml` in the `.gitlab` directory, if + one doesn't already exist. +1. In one or more `ruleset.identifier` subsections, list the rules to override. Every + `ruleset.identifier` section has: + - A `type` field for the rule. For IaC Scanning, the identifier type is `kics_id`. + - A `value` field for the rule identifier. KICS rule identifiers are alphanumeric strings. To find the rule identifier, you can: + - Find it in the [JSON report artifact](#reports-json-format). + - Search for the rule name in the [list of KICS queries](https://docs.kics.io/latest/queries/all-queries/) and copy the alphanumeric identifier that's shown. The rule name is shown on the [Vulnerability Page](../vulnerabilities/index.md) when a rule violation is detected. +1. In the `ruleset.override` context of a `ruleset` section, + provide the keys to override. Any combination of keys can be + overridden. Valid keys are: + - description + - message + - name + - severity (valid options are: Critical, High, Medium, Low, Unknown, Info) + +In the following example `sast-ruleset.toml` file, rules are matched by the `type` and +`value` of identifiers and then overridden: + +```toml +[kics] + [[kics.ruleset]] + [kics.ruleset.identifier] + type = "kics_id" + value = "8212e2d7-e683-49bc-bf78-d6799075c5a7" + [kics.ruleset.override] + description = "OVERRIDDEN description" + message = "OVERRIDDEN message" + name = "OVERRIDDEN name" + severity = "Info" +``` + +## Pinning to specific analyzer version + +The GitLab-managed CI/CD template specifies a major version and automatically pulls the latest analyzer release within that major version. + +In some cases, you may need to use a specific version. +For example, you might need to avoid a regression in a later release. + +To override the automatic update behavior, set the `SAST_ANALYZER_IMAGE_TAG` CI/CD variable +in your CI/CD configuration file after you include the [`SAST-IaC.gitlab-ci.yml` template](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/ci/templates/Jobs/SAST-IaC.gitlab-ci.yml). + +Only set this variable within a specific job. +If you set it [at the top level](../../../ci/variables/index.md#create-a-custom-cicd-variable-in-the-gitlab-ciyml-file), the version you set will be used for other SAST analyzers. + +You can set the tag to: + +- A major version, like `3`. Your pipelines will use any minor or patch updates that are released within this major version. +- A minor version, like `3.7`. Your pipelines will use any patch updates that are released within this minor version. +- A patch version, like `3.7.0`. Your pipelines won't receive any updates. + +This example uses a specific minor version of the `KICS` analyzer: + +```yaml +include: + - template: Security/SAST-IaC.gitlab-ci.yml + +kics-iac-sast: + variables: + SAST_ANALYZER_IMAGE_TAG: "3.1" +``` ## Reports JSON format diff --git a/doc/user/application_security/img/secure_tools_and_cicd_stages.png b/doc/user/application_security/img/secure_tools_and_cicd_stages.png Binary files differindex 3dbfd835baf..3284b376adc 100644 --- a/doc/user/application_security/img/secure_tools_and_cicd_stages.png +++ b/doc/user/application_security/img/secure_tools_and_cicd_stages.png diff --git a/doc/user/application_security/index.md b/doc/user/application_security/index.md index 809fa5f3764..5ddfa99fc81 100644 --- a/doc/user/application_security/index.md +++ b/doc/user/application_security/index.md @@ -227,7 +227,7 @@ The artifact generated by the secure analyzer contains all findings it discovers > - [Introduced](https://gitlab.com/groups/gitlab-org/-/epics/4393) in GitLab Free 13.5. > - Made [available in all tiers](https://gitlab.com/gitlab-org/gitlab/-/issues/273205) in 13.6. -> - Report download dropdown [added](https://gitlab.com/gitlab-org/gitlab/-/issues/273418) in 13.7. +> - Report download dropdown list [added](https://gitlab.com/gitlab-org/gitlab/-/issues/273418) in 13.7. > - [Feature flag removed](https://gitlab.com/gitlab-org/gitlab/-/issues/249550) in GitLab 13.9. ### All tiers @@ -403,7 +403,7 @@ Learn more on overriding security jobs: - [Overriding Dependency Scanning jobs](dependency_scanning/index.md#overriding-dependency-scanning-jobs). - [Overriding Container Scanning jobs](container_scanning/index.md#overriding-the-container-scanning-template). - [Overriding Secret Detection jobs](secret_detection/index.md#configure-scan-settings). -- [Overriding DAST jobs](dast/index.md#customize-dast-settings). +- [Overriding DAST jobs](dast/proxy-based.md#customize-dast-settings). - [Overriding License Compliance jobs](../compliance/license_compliance/index.md#overriding-the-template). All the security scanning tools define their stage, so this error can occur with all of them. @@ -487,7 +487,7 @@ Security and compliance teams must ensure that security scans: GitLab provides two methods of accomplishing this, each with advantages and disadvantages. -- [Compliance framework pipelines](../group/manage.md#configure-a-compliance-pipeline) +- [Compliance framework pipelines](../group/compliance_frameworks.md#configure-a-compliance-pipeline) are recommended when: - Scan execution enforcement is required for any scanner that uses a GitLab template, such as SAST IaC, DAST, Dependency Scanning, @@ -499,8 +499,8 @@ GitLab provides two methods of accomplishing this, each with advantages and disa are recommended when: - Scan execution enforcement is required for DAST which uses a DAST site or scan profile. - - Scan execution enforcement is required for SAST, Secret Detection, or Container Scanning with project-specific variable - customizations. To accomplish this, users must create a separate security policy per project. + - Scan execution enforcement is required for SAST, Secret Detection, Dependency Scanning, or Container Scanning with project-specific +variable customizations. To accomplish this, users must create a separate security policy per project. - Scans are required to run on a regular, scheduled cadence. - Either solution can be used equally well when: @@ -514,7 +514,7 @@ Additional details about the differences between the two solutions are outlined | | Compliance Framework Pipelines | Scan Execution Policies | | ------ | ------ | ------ | -| **Flexibility** | Supports anything that can be done in a CI file. | Limited to only the items for which GitLab has explicitly added support. DAST, SAST, Secret Detection, and Container Scanning scans are supported. | +| **Flexibility** | Supports anything that can be done in a CI file. | Limited to only the items for which GitLab has explicitly added support. DAST, SAST, Secret Detection, Dependency Scanning, and Container Scanning scans are supported. | | **Usability** | Requires knowledge of CI YAML. | Follows a `rules` and `actions`-based YAML structure. | | **Inclusion in CI pipeline** | The compliance pipeline is executed instead of the project's `.gitlab-ci.yml` file. To include the project's `.gitlab-ci.yml` file, use an `include` statement. Defined variables aren't allowed to be overwritten by the included project's YAML file. | Forced inclusion of a new job into the CI pipeline. DAST jobs that must be customized on a per-project basis can have project-level Site Profiles and Scan Profiles defined. To ensure separation of duties, these profiles are immutable when referenced in a scan execution policy. All jobs can be customized as part of the security policy itself with the same variables that are normally available to the CI job. | | **Schedulable** | Can be scheduled through a scheduled pipeline on the group. | Can be scheduled natively through the policy configuration itself. | diff --git a/doc/user/application_security/policies/index.md b/doc/user/application_security/policies/index.md index b1315329071..f6d22ab28cd 100644 --- a/doc/user/application_security/policies/index.md +++ b/doc/user/application_security/policies/index.md @@ -57,7 +57,7 @@ project and a project that you would like to designate as the security policy pr 1. On the top bar, select **Main menu > Projects** and find your project. 1. On the left sidebar, select **Security & Compliance > Policies**. 1. Select **Edit Policy Project**, and search for and select the - project you would like to link from the dropdown menu. + project you would like to link from the dropdown list. 1. Select **Save**. To unlink a security policy project, follow the same steps but instead select the trash can icon in diff --git a/doc/user/application_security/policies/scan-execution-policies.md b/doc/user/application_security/policies/scan-execution-policies.md index 41d25dfa8c8..f950d5116b1 100644 --- a/doc/user/application_security/policies/scan-execution-policies.md +++ b/doc/user/application_security/policies/scan-execution-policies.md @@ -15,7 +15,7 @@ with a long, random job name. In the unlikely event of a job name collision, the any pre-existing job in the pipeline. If a policy is created at the group-level, it will apply to every child project or subgroup. A group-level policy cannot be edited from a child project or subgroup. -This feature has some overlap with [compliance framework pipelines](../../group/manage.md#configure-a-compliance-pipeline), +This feature has some overlap with [compliance framework pipelines](../../group/compliance_frameworks.md#configure-a-compliance-pipeline), as we have not [unified the user experience for these two features](https://gitlab.com/groups/gitlab-org/-/epics/7312). For details on the similarities and differences between these features, see [Enforce scan execution](../index.md#enforce-scan-execution). @@ -129,14 +129,14 @@ rule in the defined policy are met. | Field | Type | Possible values | Description | |-------|------|-----------------|-------------| -| `scan` | `string` | `dast`, `secret_detection`, `sast`, `container_scanning` | The action's type. | -| `site_profile` | `string` | Name of the selected [DAST site profile](../dast/index.md#site-profile). | The DAST site profile to execute the DAST scan. This field should only be set if `scan` type is `dast`. | -| `scanner_profile` | `string` or `null` | Name of the selected [DAST scanner profile](../dast/index.md#scanner-profile). | The DAST scanner profile to execute the DAST scan. This field should only be set if `scan` type is `dast`.| +| `scan` | `string` | `dast`, `secret_detection`, `sast`, `container_scanning`, `dependency_scanning` | The action's type. | +| `site_profile` | `string` | Name of the selected [DAST site profile](../dast/proxy-based.md#site-profile). | The DAST site profile to execute the DAST scan. This field should only be set if `scan` type is `dast`. | +| `scanner_profile` | `string` or `null` | Name of the selected [DAST scanner profile](../dast/proxy-based.md#scanner-profile). | The DAST scanner profile to execute the DAST scan. This field should only be set if `scan` type is `dast`.| | `variables` | `object` | | A set of CI variables, supplied as an array of `key: value` pairs, to apply and enforce for the selected scan. The `key` is the variable name, with its `value` provided as a string. This parameter supports any variable that the GitLab CI job supports for the specified scan. | Note the following: -- You must create the [site profile](../dast/index.md#site-profile) and [scanner profile](../dast/index.md#scanner-profile) +- You must create the [site profile](../dast/proxy-based.md#site-profile) and [scanner profile](../dast/proxy-based.md#scanner-profile) with selected names for each project that is assigned to the selected Security Policy Project. Otherwise, the policy is not applied and a job with an error message is created instead. - Once you associate the site profile and scanner profile by name in the policy, it is not possible @@ -152,7 +152,7 @@ Note the following: mode when executed as part of a scheduled scan. - A container scanning scan that is configured for the `pipeline` rule type ignores the agent defined in the `agents` object. The `agents` object is only considered for `schedule` rule types. An agent with a name provided in the `agents` object must be created and configured for the project. -- The SAST scan uses the default template and runs in a [child pipeline](../../../ci/pipelines/downstream_pipelines.md#parent-child-pipelines). +- The Dependency Scanning and SAST scans use the default templates and run in a [child pipeline](../../../ci/pipelines/downstream_pipelines.md#parent-child-pipelines). ## Example security policies project diff --git a/doc/user/application_security/policies/scan-result-policies.md b/doc/user/application_security/policies/scan-result-policies.md index 45b715a52b8..7482df18cc3 100644 --- a/doc/user/application_security/policies/scan-result-policies.md +++ b/doc/user/application_security/policies/scan-result-policies.md @@ -6,6 +6,8 @@ info: To determine the technical writer assigned to the Stage/Group associated w # Scan result policies **(ULTIMATE)** +> Group-level scan result policies [introduced](https://gitlab.com/groups/gitlab-org/-/epics/7622) in GitLab 15.6. + You can use scan result policies to take action based on scan results. For example, one type of scan result policy is a security approval policy that allows approval to be required based on the findings of one or more security scan jobs. Scan result policies are evaluated after a CI scanning @@ -20,7 +22,8 @@ job is fully executed. The following video gives you an overview of GitLab scan ## Scan result policy editor -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/77814) in GitLab 14.8. +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/77814) in GitLab 14.8. +> - [Enabled by default](https://gitlab.com/gitlab-org/gitlab/-/issues/369473) in GitLab 15.6. NOTE: Only project Owners have the [permissions](../../permissions.md#project-members-permissions) @@ -38,6 +41,9 @@ before the policy changes take effect. The [policy editor](index.md#policy-editor) supports YAML mode and rule mode. +NOTE: +Propagating scan result policies created for groups with a large number of projects will take a while to complete. + ## Scan result policies schema The YAML file with scan result policies consists of an array of objects matching the scan result diff --git a/doc/user/application_security/sast/analyzers.md b/doc/user/application_security/sast/analyzers.md index dfb096f14cc..e83825636bf 100644 --- a/doc/user/application_security/sast/analyzers.md +++ b/doc/user/application_security/sast/analyzers.md @@ -12,7 +12,7 @@ Static Application Security Testing (SAST) uses analyzers to detect vulnerabilities in source code. Each analyzer is a wrapper around a [scanner](../terminology/index.md#scanner), a third-party code analysis tool. The analyzers are published as Docker images that SAST uses to launch dedicated containers for each -analysis. +analysis. We recommend a minimum of 4GB RAM to ensure consistent performance of the analyzers. SAST default images are maintained by GitLab, but you can also integrate your own custom image. @@ -134,6 +134,14 @@ In GitLab 15.4, we [removed the deprecated analyzers](https://gitlab.com/gitlab- To preview the upcoming changes to the CI/CD configuration in GitLab 15.3 or earlier: 1. Open an MR to switch from the Stable CI/CD template, `SAST.gitlab-ci.yaml`, to [the Latest template](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/ci/templates/Jobs/SAST.latest.gitlab-ci.yml), `SAST.latest.gitlab-ci.yaml`. + - On GitLab.com, use the latest template directly: + + ```yaml + include: + template: 'Jobs/SAST.latest.gitlab-ci.yaml' + ``` + + - On a Self-Managed instance, download the template from GitLab.com: ```yaml include: diff --git a/doc/user/application_security/sast/index.md b/doc/user/application_security/sast/index.md index a7624db4604..b1bc9794ced 100644 --- a/doc/user/application_security/sast/index.md +++ b/doc/user/application_security/sast/index.md @@ -70,44 +70,45 @@ is **not** `19.03.0`. See [troubleshooting information](#error-response-from-dae ## Supported languages and frameworks -GitLab SAST supports a variety of languages, package managers, and frameworks. Our SAST security scanners also feature automatic language detection which works even for mixed-language projects. If any supported language is detected in project source code we automatically run the appropriate SAST analyzers. - -You can also [view our language roadmap](https://about.gitlab.com/direction/secure/static-analysis/sast/#language-support) and [request other language support by opening an issue](https://gitlab.com/groups/gitlab-org/-/epics/297). - -| Language (package managers) / framework | Scan tool | Introduced in GitLab Version | -|------------------------------------------------|-----------------------------------------------------------------------------------------------------------|-----------------------------------------------------------------------------------------| -| .NET Core | [Security Code Scan](https://security-code-scan.github.io) | 11.0 | -| .NET Framework<sup>1</sup> | [Security Code Scan](https://security-code-scan.github.io) | 13.0 | -| .NET (all versions, C# only) | [Semgrep](https://semgrep.dev) | 15.4 | -| Apex (Salesforce) | [PMD](https://pmd.github.io/pmd/index.html) | 12.1 | -| C | [Semgrep](https://semgrep.dev) | 14.2 | -| C/C++ | [Flawfinder](https://github.com/david-a-wheeler/flawfinder) | 10.7 | -| Elixir (Phoenix) | [Sobelow](https://github.com/nccgroup/sobelow) | 11.1 | -| Go<sup>3</sup> | [Gosec](https://github.com/securego/gosec) | 10.7 | -| Go | [Semgrep](https://semgrep.dev) | 14.4 | -| Groovy<sup>2</sup> | [SpotBugs](https://spotbugs.github.io/) with the [find-sec-bugs](https://find-sec-bugs.github.io/) plugin | 11.3 (Gradle) & 11.9 (Maven, SBT) | -| Helm Charts | [Kubesec](https://github.com/controlplaneio/kubesec) | 13.1 | -| Java (any build system) | [Semgrep](https://semgrep.dev) | 14.10 | -| Java<sup>2, 3</sup> | [SpotBugs](https://spotbugs.github.io/) with the [find-sec-bugs](https://find-sec-bugs.github.io/) plugin | 10.6 (Maven), 10.8 (Gradle) & 11.9 (SBT) | -| Java (Android) | [MobSF (beta)](https://github.com/MobSF/Mobile-Security-Framework-MobSF) | 13.5 | -| JavaScript<sup>3</sup> | [ESLint security plugin](https://github.com/nodesecurity/eslint-plugin-security) | 11.8 | -| JavaScript | [Semgrep](https://semgrep.dev) | 13.10 | -| Kotlin (Android) | [MobSF (beta)](https://github.com/MobSF/Mobile-Security-Framework-MobSF) | 13.5 | -| Kotlin (General)<sup>2</sup> | [SpotBugs](https://spotbugs.github.io/) with the [find-sec-bugs](https://find-sec-bugs.github.io/) plugin | 13.11 | -| Kubernetes manifests | [Kubesec](https://github.com/controlplaneio/kubesec) | 12.6 | -| Node.js | [NodeJsScan](https://github.com/ajinabraham/NodeJsScan) | 11.1 | -| Objective-C (iOS) | [MobSF (beta)](https://github.com/MobSF/Mobile-Security-Framework-MobSF) | 13.5 | -| PHP | [phpcs-security-audit](https://github.com/FloeDesignTechnologies/phpcs-security-audit) | 10.8 | -| Python<sup>3</sup> | [bandit](https://github.com/PyCQA/bandit) | 10.3 | -| Python | [Semgrep](https://semgrep.dev) | 13.9 | -| React<sup>3</sup> | [ESLint react plugin](https://github.com/yannickcr/eslint-plugin-react) | 12.5 | -| React | [Semgrep](https://semgrep.dev) | 13.10 | -| Ruby | [brakeman](https://brakemanscanner.org) | 13.9 | -| Ruby on Rails | [brakeman](https://brakemanscanner.org) | 10.3 | -| Scala<sup>2</sup> | [SpotBugs](https://spotbugs.github.io/) with the [find-sec-bugs](https://find-sec-bugs.github.io/) plugin | 11.0 (SBT) & 11.9 (Gradle, Maven) | -| Swift (iOS) | [MobSF (beta)](https://github.com/MobSF/Mobile-Security-Framework-MobSF) | 13.5 | -| TypeScript<sup>3</sup> | [ESLint security plugin](https://github.com/nodesecurity/eslint-plugin-security) | 11.9, [merged](https://gitlab.com/gitlab-org/gitlab/-/issues/36059) with ESLint in 13.2 | -| TypeScript | [Semgrep](https://semgrep.dev) | 13.10 | +GitLab SAST supports scanning a variety of programming languages and frameworks. +Once you [enable SAST](#configuration), the right set of analyzers runs automatically even if your project uses more than one language. + +Check the [SAST direction page](https://about.gitlab.com/direction/secure/static-analysis/sast/#language-support) to learn more about our plans for language support in SAST. + +| Language / framework | [Analyzer](analyzers.md) used for scanning | Minimum supported GitLab version | +|------------------------------|--------------------------------------------------------------------------------------------------------------|-----------------------------------------------------------------------------------------| +| .NET Core | [Security Code Scan](https://gitlab.com/gitlab-org/security-products/analyzers/security-code-scan) | 11.0 | +| .NET Framework<sup>1</sup> | [Security Code Scan](https://gitlab.com/gitlab-org/security-products/analyzers/security-code-scan) | 13.0 | +| .NET (all versions, C# only) | [Semgrep](https://gitlab.com/gitlab-org/security-products/analyzers/semgrep) with GitLab-managed rules | 15.4 | +| Apex (Salesforce) | [PMD](https://gitlab.com/gitlab-org/security-products/analyzers/pmd-apex) | 12.1 | +| C | [Semgrep](https://gitlab.com/gitlab-org/security-products/analyzers/semgrep) with GitLab-managed rules | 14.2 | +| C/C++ | [Flawfinder](https://gitlab.com/gitlab-org/security-products/analyzers/flawfinder) | 10.7 | +| Elixir (Phoenix) | [Sobelow](https://gitlab.com/gitlab-org/security-products/analyzers/sobelow) | 11.1 | +| Go<sup>3</sup> | [Gosec](https://gitlab.com/gitlab-org/security-products/analyzers/gosec) | 10.7 | +| Go | [Semgrep](https://gitlab.com/gitlab-org/security-products/analyzers/semgrep) with GitLab-managed rules | 14.4 | +| Groovy<sup>2</sup> | [SpotBugs](https://gitlab.com/gitlab-org/security-products/analyzers/spotbugs) with the find-sec-bugs plugin | 11.3 (Gradle) & 11.9 (Maven, SBT) | +| Helm Charts | [Kubesec](https://gitlab.com/gitlab-org/security-products/analyzers/kubesec) | 13.1 | +| Java (any build system) | [Semgrep](https://gitlab.com/gitlab-org/security-products/analyzers/semgrep) with GitLab-managed rules | 14.10 | +| Java<sup>2, 3</sup> | [SpotBugs](https://gitlab.com/gitlab-org/security-products/analyzers/spotbugs) with the find-sec-bugs plugin | 10.6 (Maven), 10.8 (Gradle) & 11.9 (SBT) | +| Java (Android) | [MobSF (beta)](https://gitlab.com/gitlab-org/security-products/analyzers/mobsf) | 13.5 | +| JavaScript<sup>3</sup> | [ESLint security plugin](https://gitlab.com/gitlab-org/security-products/analyzers/eslint) | 11.8 | +| JavaScript | [Semgrep](https://gitlab.com/gitlab-org/security-products/analyzers/semgrep) with GitLab-managed rules | 13.10 | +| Kotlin (Android) | [MobSF (beta)](https://gitlab.com/gitlab-org/security-products/analyzers/mobsf) | 13.5 | +| Kotlin (General)<sup>2</sup> | [SpotBugs](https://gitlab.com/gitlab-org/security-products/analyzers/spotbugs) with the find-sec-bugs plugin | 13.11 | +| Kubernetes manifests | [Kubesec](https://gitlab.com/gitlab-org/security-products/analyzers/kubesec) | 12.6 | +| Node.js | [NodeJsScan](https://gitlab.com/gitlab-org/security-products/analyzers/nodejs-scan) | 11.1 | +| Objective-C (iOS) | [MobSF (beta)](https://gitlab.com/gitlab-org/security-products/analyzers/mobsf) | 13.5 | +| PHP | [phpcs-security-audit](https://gitlab.com/gitlab-org/security-products/analyzers/phpcs-security-audit) | 10.8 | +| Python<sup>3</sup> | [bandit](https://gitlab.com/gitlab-org/security-products/analyzers/bandit) | 10.3 | +| Python | [Semgrep](https://gitlab.com/gitlab-org/security-products/analyzers/semgrep) with GitLab-managed rules | 13.9 | +| React<sup>3</sup> | [ESLint react plugin](https://gitlab.com/gitlab-org/security-products/analyzers/eslint) | 12.5 | +| React | [Semgrep](https://gitlab.com/gitlab-org/security-products/analyzers/semgrep) with GitLab-managed rules | 13.10 | +| Ruby | [brakeman](https://gitlab.com/gitlab-org/security-products/analyzers/brakeman) | 13.9 | +| Ruby on Rails | [brakeman](https://gitlab.com/gitlab-org/security-products/analyzers/brakeman) | 10.3 | +| Scala<sup>2</sup> | [SpotBugs](https://gitlab.com/gitlab-org/security-products/analyzers/spotbugs) with the find-sec-bugs plugin | 11.0 (SBT) & 11.9 (Gradle, Maven) | +| Swift (iOS) | [MobSF (beta)](https://gitlab.com/gitlab-org/security-products/analyzers/mobsf) | 13.5 | +| TypeScript<sup>3</sup> | [ESLint security plugin](https://gitlab.com/gitlab-org/security-products/analyzers/eslint) | 11.9, [merged](https://gitlab.com/gitlab-org/gitlab/-/issues/36059) with ESLint in 13.2 | +| TypeScript | [Semgrep](https://gitlab.com/gitlab-org/security-products/analyzers/semgrep) with GitLab-managed rules | 13.10 | 1. .NET 4 support is limited. The analyzer runs in a Linux container and does not have access to Windows-specific libraries or features. Use the Semgrep-based scanner if you need .NET 4 support. 1. The SpotBugs-based analyzer supports [Gradle](https://gradle.org/), [Maven](https://maven.apache.org/), and [SBT](https://www.scala-sbt.org/). It can also be used with variants like the @@ -116,7 +117,7 @@ You can also [view our language roadmap](https://about.gitlab.com/direction/secu and the [Maven wrapper](https://github.com/takari/maven-wrapper). However, SpotBugs has [limitations](https://gitlab.com/gitlab-org/gitlab/-/issues/350801) when used against [Ant](https://ant.apache.org/)-based projects. We recommend using the Semgrep-based analyzer for Ant-based Java projects. 1. These analyzers reached [End of Support](https://about.gitlab.com/handbook/product/gitlab-the-product/#end-of-support) status [in GitLab 15.4](https://gitlab.com/gitlab-org/gitlab/-/issues/352554). -### Multi-project support +## Multi-project support > [Introduced](https://gitlab.com/groups/gitlab-org/-/epics/4895) in GitLab 13.7. @@ -136,16 +137,52 @@ The following analyzers have multi-project support: - SpotBugs - Sobelow -#### Enable multi-project support for Security Code Scan +### Enable multi-project support for Security Code Scan Multi-project support in the Security Code Scan requires a Solution (`.sln`) file in the root of the repository. For details on the Solution format, see the Microsoft reference [Solution (`.sln`) file](https://learn.microsoft.com/en-us/visualstudio/extensibility/internals/solution-dot-sln-file?view=vs-2019). -### Supported distributions +## False positive detection **(ULTIMATE)** + +> Introduced in GitLab 14.2. + +Vulnerabilities that have been detected and are false positives will be flagged as false positives in the security dashboard. + +False positive detection is available in a subset of the [supported languages](#supported-languages-and-frameworks) and [analyzers](analyzers.md): + +- Ruby, in the Brakeman-based analyzer + + + +## Advanced vulnerability tracking **(ULTIMATE)** + +> [Introduced](https://gitlab.com/groups/gitlab-org/-/epics/5144) in GitLab 14.2. + +Source code is volatile; as developers make changes, source code may move within files or between files. +Security analyzers may have already reported vulnerabilities that are being tracked in the [Vulnerability Report](../vulnerability_report/index.md). +These vulnerabilities are linked to specific problematic code fragments so that they can be found and fixed. +If the code fragments are not tracked reliably as they move, vulnerability management is harder because the same vulnerability could be reported again. + +GitLab SAST uses an advanced vulnerability tracking algorithm to more accurately identify when the same vulnerability has moved within a file due to refactoring or unrelated changes. + +Advanced vulnerability tracking is available in a subset of the [supported languages](#supported-languages-and-frameworks) and [analyzers](analyzers.md): + +- C, in the Semgrep-based analyzer only +- Go, in the Gosec- and Semgrep-based analyzers +- Java, in the Semgrep-based analyzer only +- JavaScript, in the Semgrep-based analyzer only +- Python, in the Semgrep-based analyzer only +- Ruby, in the Brakeman-based analyzer + +Support for more languages and analyzers is tracked in [this epic](https://gitlab.com/groups/gitlab-org/-/epics/5144). + +For more information, see the confidential project `https://gitlab.com/gitlab-org/security-products/post-analyzers/tracking-calculator`. The content of this project is available only to GitLab team members. + +## Supported distributions The default scanner images are build off a base Alpine image for size and maintainability. -#### FIPS-enabled images +### FIPS-enabled images > [Introduced](https://gitlab.com/groups/gitlab-org/-/epics/6479) in GitLab 14.10. @@ -169,17 +206,14 @@ A FIPS-compliant image is only available for the Semgrep-based analyzer. To use SAST in a FIPS-compliant manner, you must [exclude other analyzers from running](analyzers.md#customize-analyzers). -### Making SAST analyzers available to all GitLab tiers - -All open source (OSS) analyzers have been moved to the GitLab Free tier as of GitLab 13.3. - -#### Summary of features per tier +## Summary of features per tier Different features are available in different [GitLab tiers](https://about.gitlab.com/pricing/), as shown in the following table: | Capability | In Free & Premium | In Ultimate | |:---------------------------------------------------------------- -|:--------------------|:-------------------| +| Automatically scan code with [appropriate analyzers](#supported-languages-and-frameworks) | **{check-circle}** | **{check-circle}** | | [Configure SAST scanners](#configuration) | **{check-circle}** | **{check-circle}** | | [Customize SAST settings](#available-cicd-variables) | **{check-circle}** | **{check-circle}** | | Download [JSON Report](#reports-json-format) | **{check-circle}** | **{check-circle}** | @@ -207,14 +241,14 @@ To configure SAST for a project you can: ### Configure SAST manually To enable SAST you must [include](../../../ci/yaml/index.md#includetemplate) -the [`SAST.gitlab-ci.yml` template](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/ci/templates/Security/SAST.gitlab-ci.yml) +the [`SAST.gitlab-ci.yml` template](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/ci/templates/Jobs/SAST.gitlab-ci.yml) provided as a part of your GitLab installation. Add the following to your `.gitlab-ci.yml` file: ```yaml include: - - template: Security/SAST.gitlab-ci.yml + - template: Jobs/SAST.gitlab-ci.yml ``` The included template creates SAST jobs in your CI/CD pipeline and scans @@ -300,14 +334,26 @@ spotbugs-sast: FAIL_NEVER: 1 ``` -#### Pinning to minor image version +### Pinning to minor image version + +The GitLab-managed CI/CD template specifies a major version and automatically pulls the latest analyzer release within that major version. -While our templates use `MAJOR` version pinning to always ensure the latest analyzer -versions are pulled, there are certain cases where it can be beneficial to pin -an analyzer to a specific release. To do so, override the `SAST_ANALYZER_IMAGE_TAG` CI/CD variable -in the job template directly. +In some cases, you may need to use a specific version. +For example, you might need to avoid a regression in a later release. -In the example below, we pin to a minor version of the `semgrep` analyzer and a specific patch version of the `brakeman` analyzer: +To override the automatic update behavior, set the `SAST_ANALYZER_IMAGE_TAG` CI/CD variable +in your CI/CD configuration file after you include the [`SAST.gitlab-ci.yml` template](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/ci/templates/Jobs/SAST.gitlab-ci.yml). + +Only set this variable within a specific job. +If you set it [at the top level](../../../ci/variables/index.md#create-a-custom-cicd-variable-in-the-gitlab-ciyml-file), the version you set will be used for other SAST analyzers. + +You can set the tag to: + +- A major version, like `3`. Your pipelines will use any minor or patch updates that are released within this major version. +- A minor version, like `3.7`. Your pipelines will use any patch updates that are released within this minor version. +- A patch version, like `3.7.0`. Your pipelines won't receive any updates. + +This example uses a specific minor version of the `semgrep` analyzer and a specific patch version of the `brakeman` analyzer: ```yaml include: @@ -315,47 +361,13 @@ include: semgrep-sast: variables: - SAST_ANALYZER_IMAGE_TAG: "2.16" + SAST_ANALYZER_IMAGE_TAG: "3.7" brakeman-sast: variables: - SAST_ANALYZER_IMAGE_TAG: "2.21.1" + SAST_ANALYZER_IMAGE_TAG: "3.1.1" ``` -### False Positive Detection **(ULTIMATE)** - -> Introduced in GitLab 14.2. - -Vulnerabilities that have been detected and are false positives will be flagged as false positives in the security dashboard. - -False positive detection is available in a subset of the [supported languages](#supported-languages-and-frameworks) and [analyzers](analyzers.md): - -- Ruby, in the Brakeman-based analyzer - - - -### Advanced vulnerability tracking **(ULTIMATE)** - -> [Introduced](https://gitlab.com/groups/gitlab-org/-/epics/5144) in GitLab 14.2. - -Source code is volatile; as developers make changes, source code may move within files or between files. -Security analyzers may have already reported vulnerabilities that are being tracked in the [Vulnerability Report](../vulnerability_report/index.md). -These vulnerabilities are linked to specific problematic code fragments so that they can be found and fixed. -If the code fragments are not tracked reliably as they move, vulnerability management is harder because the same vulnerability could be reported again. - -GitLab SAST uses an advanced vulnerability tracking algorithm to more accurately identify when the same vulnerability has moved within a file due to refactoring or unrelated changes. - -Advanced vulnerability tracking is available in a subset of the [supported languages](#supported-languages-and-frameworks) and [analyzers](analyzers.md): - -- C, in the Semgrep-based analyzer only -- Go, in the Gosec- and Semgrep-based analyzers -- Java, in the Semgrep-based analyzer only -- JavaScript, in the Semgrep-based analyzer only -- Python, in the Semgrep-based analyzer only -- Ruby, in the Brakeman-based analyzer - -Support for more languages and analyzers is tracked in [this epic](https://gitlab.com/groups/gitlab-org/-/epics/5144). - ### Using CI/CD variables to pass credentials for private repositories Some analyzers require downloading the project's dependencies to @@ -665,16 +677,16 @@ import the following default SAST analyzer images from `registry.gitlab.com` int [local Docker container registry](../../packages/container_registry/index.md): ```plaintext -registry.gitlab.com/security-products/brakeman:2 -registry.gitlab.com/security-products/flawfinder:2 -registry.gitlab.com/security-products/kubesec:2 -registry.gitlab.com/security-products/nodejs-scan:2 -registry.gitlab.com/security-products/phpcs-security-audit:2 -registry.gitlab.com/security-products/pmd-apex:2 -registry.gitlab.com/security-products/security-code-scan:2 -registry.gitlab.com/security-products/semgrep:2 -registry.gitlab.com/security-products/sobelow:2 -registry.gitlab.com/security-products/spotbugs:2 +registry.gitlab.com/security-products/brakeman:3 +registry.gitlab.com/security-products/flawfinder:3 +registry.gitlab.com/security-products/kubesec:3 +registry.gitlab.com/security-products/nodejs-scan:3 +registry.gitlab.com/security-products/phpcs-security-audit:3 +registry.gitlab.com/security-products/pmd-apex:3 +registry.gitlab.com/security-products/security-code-scan:3 +registry.gitlab.com/security-products/semgrep:3 +registry.gitlab.com/security-products/sobelow:3 +registry.gitlab.com/security-products/spotbugs:3 ``` The process for importing Docker images into a local offline Docker registry depends on diff --git a/doc/user/application_security/secret_detection/index.md b/doc/user/application_security/secret_detection/index.md index df6bb19ac25..8a066cf1be1 100644 --- a/doc/user/application_security/secret_detection/index.md +++ b/doc/user/application_security/secret_detection/index.md @@ -15,18 +15,18 @@ info: To determine the technical writer assigned to the Stage/Group associated w > `secret_detection_default_branch` and `secret_detection` were consolidated into one job, > `secret_detection`. -People may accidentally commit secrets to -remote Git repositories. Secrets include keys, passwords, API tokens, and other sensitive -information. Anyone with access to the repository could use the secrets for malicious purposes. -Exposed secrets are compromised and must be replaced, which can be costly. - -To help prevent secrets from being committed to a Git repository, you can use Secret Detection -to scan your repository for secrets. Scanning is language -and framework agnostic, but does not support scanning binary files. - -Secret Detection uses a specific analyzer containing the -[Gitleaks](https://github.com/zricethezav/gitleaks) tool to scan the repository for secrets, in a -`secret-detection` job. The results are saved as a +People may accidentally commit secrets (such as keys, passwords, and API tokens) to remote Git repositories. + +Anyone with access to the repository could use the secrets for malicious purposes. Exposed secrets +must be considered compromised and be replaced, which can be costly. + +To help prevent secrets from being committed to a Git repository, you can use Secret Detection to +scan your repository for secrets. Scanning is language and framework agnostic, but does not support +scanning binary files. + +Secret Detection uses an analyzer containing the [Gitleaks](https://github.com/zricethezav/gitleaks) +tool to scan the repository for secrets. Detection occurs in the `secret-detection` job. The results +are saved as a [Secret Detection report artifact](../../../ci/yaml/artifacts_reports.md#artifactsreportssecret_detection) that you can later download and analyze. Due to implementation limitations, we always take the latest Secret Detection artifact available. @@ -38,7 +38,7 @@ All identified secrets are reported in the: - Merge request widget - Pipelines' **Security** tab -- [Security Dashboard](../security_dashboard/index.md) +- [Vulnerability Report](../vulnerability_report/index.md)  @@ -61,7 +61,7 @@ Different features are available in different [GitLab tiers](https://about.gitla | Download [JSON Report](../sast/index.md#reports-json-format) | **{check-circle}** Yes | **{check-circle}** Yes | | See new findings in the merge request widget | **{dotted-circle}** No | **{check-circle}** Yes | | View identified secrets in the pipelines' **Security** tab | **{dotted-circle}** No | **{check-circle}** Yes | -| [Manage vulnerabilities](../vulnerabilities/index.md) | **{dotted-circle}** No | **{check-circle}** Yes | +| [Manage vulnerabilities](../vulnerability_report/index.md) | **{dotted-circle}** No | **{check-circle}** Yes | | [Access the Security Dashboard](../security_dashboard/index.md) | **{dotted-circle}** No | **{check-circle}** Yes | | [Customize Secret Detection rulesets](#custom-rulesets) | **{dotted-circle}** No | **{check-circle}** Yes | @@ -88,13 +88,13 @@ To enable Secret Detection, either: ### Enable Secret Detection by including the template -We recommend this method if you have an existing GitLab CI/CD configuration file. +You should use this method if you have an existing GitLab CI/CD configuration file. Add the following extract to your `.gitlab-ci.yml` file: ```yaml include: - - template: Security/Secret-Detection.gitlab-ci.yml + - template: Jobs/Secret-Detection.gitlab-ci.yml ``` Pipelines now include a Secret Detection job, and the results are included in the merge request @@ -122,7 +122,34 @@ widget. ## Responding to a leaked secret -If the scanner detects a secret we recommend you rotate it immediately. [Purging a file from the repository's history](../../project/repository/reducing_the_repo_size_using_git.md#purge-files-from-repository-history) may not be effective in removing all references to the file. Also, the secret remains in any forks of the repository. +If the scanner detects a secret you should rotate it immediately. [Purging a file from the repository's history](../../project/repository/reducing_the_repo_size_using_git.md#purge-files-from-repository-history) may not be effective in removing all references to the file. Also, the secret remains in any forks of the repository. + +## Pinning to specific analyzer version + +The GitLab-managed CI/CD template specifies a major version and automatically pulls the latest analyzer release within that major version. + +In some cases, you may need to use a specific version. +For example, you might need to avoid a regression in a later release. + +To override the automatic update behavior, set the `SECRETS_ANALYZER_VERSION` CI/CD variable +in your CI/CD configuration file after you include the [`Secret-Detection.gitlab-ci.yml` template](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/ci/templates/Jobs/Secret-Detection.gitlab-ci.yml). + +You can set the tag to: + +- A major version, like `4`. Your pipelines will use any minor or patch updates that are released within this major version. +- A minor version, like `4.5`. Your pipelines will use any patch updates that are released within this minor version. +- A patch version, like `4.5.0`. Your pipelines won't receive any updates. + +This example uses a specific minor version of the analyzer: + +```yaml +include: + - template: Security/Secret-Detection.gitlab-ci.yml + +secret_detection: + variables: + SECRETS_ANALYZER_VERSION: "4.5" +``` ## Configure scan settings @@ -154,11 +181,13 @@ secret_detection: SECRET_DETECTION_HISTORIC_SCAN: "true" ``` -### Ignoring Secrets +### Ignore secrets -You might want to add a fake secret to your code base. For instance, you can use a fake secret as an example in your documentation or test suite. +In some instances, you might want to ignore a secret. For example, you may have a fake secret in an +example or a test suite. In these instances, you want to ignore the secret, instead of having it +reported as a vulnerability. -In these cases, Secret Detection can ignore the fake secret and not report it as a vulnerability. To ignore a secret, add `gitleaks:allow` as a comment to the line that contains the secret. +To ignore a secret, add `gitleaks:allow` as a comment to the line that contains the secret. For example: @@ -172,7 +201,7 @@ Secret Detection can be customized by defining available CI/CD variables: | CI/CD variable | Default value | Description | |-----------------------------------|---------------|-------------| -| `SECRET_DETECTION_EXCLUDED_PATHS` | "" | Exclude vulnerabilities from output based on the paths. This is a comma-separated list of patterns. Patterns can be globs (see [`doublestar.Match`](https://pkg.go.dev/github.com/bmatcuk/doublestar/v4@v4.0.2#Match) for supported patterns), or file or folder paths (for example, `doc,spec` ). Parent directories also match patterns. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/225273) in GitLab 13.3. | +| `SECRET_DETECTION_EXCLUDED_PATHS` | "" | Exclude vulnerabilities from output based on the paths. The paths are a comma-separated list of patterns. Patterns can be globs (see [`doublestar.Match`](https://pkg.go.dev/github.com/bmatcuk/doublestar/v4@v4.0.2#Match) for supported patterns), or file or folder paths (for example, `doc,spec` ). Parent directories also match patterns. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/225273) in GitLab 13.3. | | `SECRET_DETECTION_HISTORIC_SCAN` | false | Flag to enable a historic Gitleaks scan. | | `SECRET_DETECTION_IMAGE_SUFFIX` | "" | Suffix added to the image name. If set to `-fips`, `FIPS-enabled` images are used for scan. See [Use FIPS-enabled images](#use-fips-enabled-images) for more details. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/355519) in GitLab 14.10. | | `SECRET_DETECTION_LOG_OPTIONS` | "" | [`git log`](https://git-scm.com/docs/git-log) options used to define commit ranges. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/350660) in GitLab 15.1.| @@ -214,7 +243,7 @@ By default, Secret Detection scans only the current state of the Git repository. contained in the repository's history are not detected. To address this, Secret Detection can scan the Git repository's full history. -We recommend you do a full history scan only once, after enabling Secret Detection. A full history +You should do a full history scan only once, after enabling Secret Detection. A full history can take a long time, especially for larger repositories with lengthy Git histories. After completing an initial full history scan, use only standard Secret Detection as part of your pipeline. @@ -349,15 +378,15 @@ In the `secret-detection-ruleset.toml` file, do one of the following: ## Running Secret Detection in an offline environment **(PREMIUM SELF)** -For self-managed GitLab instances in an environment with limited, restricted, or intermittent access -to external resources through the internet, some configuration changes are required for the Secret -Detection job to run successfully. The instructions in this section must be completed together with -the instructions detailed in [offline environments](../offline_deployments/index.md). +An offline environment has limited, restricted, or intermittent access to external resources through +the internet. For self-managed GitLab instances in such an environment, Secret Detection requires +some configuration changes. The instructions in this section must be completed together with the +instructions detailed in [offline environments](../offline_deployments/index.md). ### Configure GitLab Runner By default, a runner tries to pull Docker images from the GitLab container registry even if a local -copy is available. We recommend using this default setting, to ensure Docker images remain current. +copy is available. You should use this default setting, to ensure Docker images remain current. However, if no network connectivity is available, you must change the default GitLab Runner `pull_policy` variable. @@ -379,11 +408,11 @@ Prerequisites: [local Docker container registry](../../packages/container_registry/index.md): ```plaintext - registry.gitlab.com/security-products/secret-detection:3 + registry.gitlab.com/security-products/secrets:4 ``` The Secret Detection analyzer's image is [periodically updated](../index.md#vulnerability-scanner-maintenance) - so you may need to periodically update the local copy. + so you should periodically update the local copy. 1. Set the CI/CD variable `SECURE_ANALYZERS_PREFIX` to the local Docker container registry. @@ -454,14 +483,14 @@ secrets. If the number of commits in a merge request is greater than the value o [`GIT_DEPTH` CI/CD variable](../../../ci/runners/configure_runners.md#shallow-cloning), Secret Detection [fails to detect secrets](#error-couldnt-run-the-gitleaks-command-exit-status-2). -For example, if a pipeline is triggered from a merge request containing 60 commits and the -`GIT_DEPTH` variable's value is less than 60, the Secret Detection job fails as the clone is not -deep enough to contain all of the relevant commits. To veridy the current value, see +For example, you could have a pipeline triggered from a merge request containing 60 commits and the +`GIT_DEPTH` variable set to less than 60. In that case the Secret Detection job fails because the +clone is not deep enough to contain all of the relevant commits. To verify the current value, see [pipeline configuration](../../../ci/pipelines/settings.md#limit-the-number-of-changes-fetched-during-clone). -To confirm this as the cause of the error, set the [logging level](#set-the-logging-level) to `debug`, then -rerun the pipeline. The logs should look similar to the following example. The text "object not -found" is a symptom of this error. +To confirm this as the cause of the error, set the [logging level](#set-the-logging-level) to +`debug`, then rerun the pipeline. The logs should look similar to the following example. The text +"object not found" is a symptom of this error. ```plaintext ERRO[2020-11-18T18:05:52Z] object not found diff --git a/doc/user/application_security/security_dashboard/index.md b/doc/user/application_security/security_dashboard/index.md index af98fc783e7..7c44b49b78c 100644 --- a/doc/user/application_security/security_dashboard/index.md +++ b/doc/user/application_security/security_dashboard/index.md @@ -153,7 +153,7 @@ important to describe those, too. Think of things that may go wrong and include This is important to minimize requests for support, and to avoid doc comments with questions that you know someone might ask. -Each scenario can be a third-level heading, e.g. `### Getting error message X`. +Each scenario can be a third-level heading, for example `### Getting error message X`. If you have none to add when creating a doc, leave this section in place but commented out to help encourage others to add to it in the future. --> diff --git a/doc/user/application_security/terminology/index.md b/doc/user/application_security/terminology/index.md index 085a762fffa..f156d60be8f 100644 --- a/doc/user/application_security/terminology/index.md +++ b/doc/user/application_security/terminology/index.md @@ -230,7 +230,7 @@ support for cheap scan is proposed in issue [349926](https://gitlab.com/gitlab-o ### Pre-filter -An irreversible action that is done to filter out target(s) before analysis occurs. This is usually provided to allow +An irreversible action that is done to filter out targets before analysis occurs. This is usually provided to allow the user to reduce scope and noise as well as speed up the analysis. This should not be done if a record is needed as we currently do not store anything related to the skipped/excluded code or assets. diff --git a/doc/user/application_security/vulnerabilities/severities.md b/doc/user/application_security/vulnerabilities/severities.md index 36f9578f999..e75d0a45f7d 100644 --- a/doc/user/application_security/vulnerabilities/severities.md +++ b/doc/user/application_security/vulnerabilities/severities.md @@ -42,15 +42,13 @@ the following tables: | [`sobelow`](https://gitlab.com/gitlab-org/security-products/analyzers/sobelow) | **{check-circle}** Yes | Not applicable | Hardcodes all severity levels to `Unknown` | | [`nodejs-scan`](https://gitlab.com/gitlab-org/security-products/analyzers/nodejs-scan) | **{check-circle}** Yes | String | `INFO`, `WARNING`, `ERROR` | | [`flawfinder`](https://gitlab.com/gitlab-org/security-products/analyzers/flawfinder) | **{check-circle}** Yes | Integer | `0`, `1`, `2`, `3`, `4`, `5` | -| [`eslint`](https://gitlab.com/gitlab-org/security-products/analyzers/eslint) | **{check-circle}** Yes | Not applicable | Hardcodes all severity levels to `Unknown` | | [`SpotBugs`](https://gitlab.com/gitlab-org/security-products/analyzers/spotbugs) | **{check-circle}** Yes | Integer | `1`, `2`, `3`, `11`, `12`, `18` | -| [`gosec`](https://gitlab.com/gitlab-org/security-products/analyzers/gosec) | **{check-circle}** Yes | String | `HIGH`, `MEDIUM`, `LOW` | -| [`bandit`](https://gitlab.com/gitlab-org/security-products/analyzers/bandit) | **{check-circle}** Yes | String | `HIGH`, `MEDIUM`, `LOW` | | [`phpcs-security-audit`](https://gitlab.com/gitlab-org/security-products/analyzers/phpcs-security-audit) | **{check-circle}** Yes | String | `ERROR`, `WARNING` | | [`pmd-apex`](https://gitlab.com/gitlab-org/security-products/analyzers/pmd-apex) | **{check-circle}** Yes | Integer | `1`, `2`, `3`, `4`, `5` | | [`kubesec`](https://gitlab.com/gitlab-org/security-products/analyzers/kubesec) | **{check-circle}** Yes | String | `CriticalSeverity`, `InfoSeverity` | | [`secrets`](https://gitlab.com/gitlab-org/security-products/analyzers/secrets) | **{check-circle}** Yes | Not applicable | Hardcodes all severity levels to `Critical` | | [`semgrep`](https://gitlab.com/gitlab-org/security-products/analyzers/semgrep) | **{check-circle}** Yes | String | `error`, `warning`, `note`, `none` | +| [`kics`](https://gitlab.com/gitlab-org/security-products/analyzers/kics) | **{check-circle}** Yes | String | `error`, `warning`, `note`, `none` (gets mapped to `info` in [analyzer version 3.7.0 and later](https://gitlab.com/gitlab-org/security-products/analyzers/kics/-/releases/v3.7.0)) | ## Dependency Scanning diff --git a/doc/user/application_security/vulnerability_report/index.md b/doc/user/application_security/vulnerability_report/index.md index 59851fd192c..2b78dde4f63 100644 --- a/doc/user/application_security/vulnerability_report/index.md +++ b/doc/user/application_security/vulnerability_report/index.md @@ -216,6 +216,10 @@ Fields included are: - [CVE](https://cve.mitre.org/) (Common Vulnerabilities and Exposures) - [CWE](https://cwe.mitre.org/) (Common Weakness Enumeration) - Other identifiers +- Detected At +- Location +- Activity +- Comments NOTE: Full details are available through our diff --git a/doc/user/award_emojis.md b/doc/user/award_emojis.md index d0aa9fa6119..5df38550c40 100644 --- a/doc/user/award_emojis.md +++ b/doc/user/award_emojis.md @@ -4,23 +4,23 @@ group: Project Management info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- -# Award emoji **(FREE)** +# Award emojis **(FREE)** When you're collaborating online, you get fewer opportunities for high-fives -and thumbs-ups. Emoji can be awarded to [issues](project/issues/index.md), [merge requests](project/merge_requests/index.md), +and thumbs-ups. Emojis can be awarded to [issues](project/issues/index.md), [merge requests](project/merge_requests/index.md), [snippets](snippets.md), and anywhere you can have a thread.  -Award emoji make it much easier to give and receive feedback without a long +Award emojis make it much easier to give and receive feedback without a long comment thread. For information on the relevant API, see [Award Emoji API](../api/award_emoji.md). ## Sort issues and merge requests on vote count -You can quickly sort issues and merge requests by the number of votes they -have received. The sort options can be found in the dropdown menu as "Most +You can quickly sort issues and merge requests by the number of votes ("thumbs up" and "thumbs down" emoji) they +have received. The sort options can be found in the dropdown list as "Most popular" and "Least popular".  @@ -29,9 +29,9 @@ The total number of votes is not summed up. An issue with 18 upvotes and 5 downvotes is considered more popular than an issue with 17 upvotes and no downvotes. -## Award emoji for comments +## Award emojis for comments -Award emoji can also be applied to individual comments when you want to +Award emojis can also be applied to individual comments when you want to celebrate an accomplishment or agree with an opinion. To add an award emoji: @@ -40,3 +40,15 @@ To add an award emoji: 1. Select an emoji from the dropdown list. To remove an award emoji, select the emoji again. + +## Custom emojis + +You can upload custom emojis to a GitLab instance with the GraphQL API. +For more, visit [Use custom emojis with GraphQL](../api/graphql/custom_emoji.md). + +Custom emojis don't show in the emoji picker. +To use them in a text box, type the filename without the extension and surrounded by colons. +For example, for a file named `thank-you.png`, type `:thank-you:`. + +For the list of custom emojis available for GitLab.com, visit +[the `custom_emoji` project](https://gitlab.com/custom_emoji/custom_emoji/-/tree/main/img). diff --git a/doc/user/clusters/agent/ci_cd_workflow.md b/doc/user/clusters/agent/ci_cd_workflow.md index 7a3c09687a5..454be3c53c7 100644 --- a/doc/user/clusters/agent/ci_cd_workflow.md +++ b/doc/user/clusters/agent/ci_cd_workflow.md @@ -58,7 +58,8 @@ Authorization configuration can take one or two minutes to propagate. ### Authorize the agent to access your projects -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/327850) in GitLab 14.4. +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/327850) in GitLab 14.4. +> - [Changed](https://gitlab.com/gitlab-org/gitlab/-/issues/346566) to remove hierarchy restrictions in GitLab 15.6. To authorize the agent to access the GitLab project where you keep Kubernetes manifests: @@ -72,7 +73,7 @@ To authorize the agent to access the GitLab project where you keep Kubernetes ma - id: path/to/project ``` - - The Kubernetes projects must be in the same group hierarchy as the project where the agent's configuration is. + - Authorized projects must have the same root group as the agent's configuration project. - You can install additional agents into the same cluster to accommodate additional hierarchies. - You can authorize up to 100 projects. @@ -81,7 +82,8 @@ Choose the context to run `kubectl` commands from your CI/CD scripts. ### Authorize the agent to access projects in your groups -> [Introduced](https://gitlab.com/groups/gitlab-org/-/epics/5784) in GitLab 14.3. +> - [Introduced](https://gitlab.com/groups/gitlab-org/-/epics/5784) in GitLab 14.3. +> - [Changed](https://gitlab.com/gitlab-org/gitlab/-/issues/346566) to remove hierarchy restrictions in GitLab 15.6. To authorize the agent to access all of the GitLab projects in a group or subgroup: @@ -95,7 +97,7 @@ To authorize the agent to access all of the GitLab projects in a group or subgro - id: path/to/group/subgroup ``` - - The Kubernetes projects must be in the same group hierarchy as the project where the agent's configuration is. + - Authorized groups must have the same root group as the agent's configuration project. - You can install additional agents into the same cluster to accommodate additional hierarchies. - All of the subgroups of an authorized group also have access to the same agent (without being specified individually). - You can authorize up to 100 groups. @@ -125,6 +127,30 @@ deploy: If you are not sure what your agent's context is, open a terminal and connect to your cluster. Run `kubectl config get-contexts`. +### Environments that use Auto DevOps + +If Auto DevOps is enabled, you must define the CI/CD variable `KUBE_CONTEXT`. +Set the value of `KUBE_CONTEXT` to the context of the agent you want Auto DevOps to use: + +```yaml +deploy: + variables: + KUBE_CONTEXT: <path_to_agent_config_repository>:<agent_name> +``` + +You can assign different agents to separate Auto DevOps jobs. For instance, +Auto DevOps can use one agent for `staging` jobs, and another agent for `production` jobs. +To use multiple agents, define an [environment-scoped CI/CD variable](../../../ci/variables/index.md#limit-the-environment-scope-of-a-cicd-variable) +for each agent. For example: + +1. Define two variables named `KUBE_CONTEXT`. +1. For the first variable: + 1. Set the `environment` to `staging`. + 1. Set the value to the context of your staging agent. +1. For the second variable: + 1. Set the `environment` to `production`. + 1. Set the value to the context of your production agent. + ### Environments with both certificate-based and agent-based connections When you deploy to an environment that has both a @@ -156,20 +182,6 @@ deploy: # ... rest of your job configuration ``` -### Using the agent with Auto DevOps - -If Auto DevOps is enabled, you must define the `KUBE_CONTEXT` CI/CD variable. Set the value of `KUBE_CONTEXT` to the context of the agent you want to use in your Auto DevOps pipeline jobs (`<PATH_TO_AGENT_CONFIG_REPOSITORY>:<AGENT_NAME>`). - -You can also use different agents for different Auto DevOps jobs. For instance, you can use one agent for `staging` jobs and a different agent for `production` jobs. To use multiple agents, define a unique CI/CD variable for each agent. - -For example: - -1. Add two [environment-scoped CI/CD variables](../../../ci/variables/index.md#limit-the-environment-scope-of-a-cicd-variable) and name both `KUBE_CONTEXT`. -1. Set the `environment` of the first variable to `staging`. Set the value of the variable to `<PATH_TO_AGENT_CONFIGURATION_PROJECT>:<STAGING_AGENT_NAME>`. -1. Set the `environment` of the second variable to `production`. Set the value of the variable to `<PATH_TO_AGENT_CONFIGURATION_PROJECT>:<PRODUCTION_AGENT_NAME>`. - -When the `staging` job runs, it will connect to the cluster via the agent named `<STAGING_AGENT_NAME>`, and when the `production` job runs it will connect to the cluster via the agent named `<PRODUCTION_AGENT_NAME>`. - ## Restrict project and group access by using impersonation **(PREMIUM)** > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/345014) in GitLab 14.5. @@ -281,13 +293,6 @@ See the [official Kubernetes documentation for details](https://kubernetes.io/do ## Troubleshooting -### `kubectl` commands not supported - -The commands `kubectl exec`, `kubectl cp`, `kubectl attach`, `kubectl run --attach=true` and `kubectl port-forward` are not supported. -Anything that uses these API endpoints does not work, because they use the deprecated -SPDY protocol. -[An issue exists](https://gitlab.com/gitlab-org/gitlab/-/issues/346248) to add support for these commands. - ### Grant write permissions to `~/.kube/cache` Tools like `kubectl`, Helm, `kpt`, and `kustomize` cache information about diff --git a/doc/user/clusters/agent/gitops/helm.md b/doc/user/clusters/agent/gitops/helm.md index af9f80618b5..0ec87376636 100644 --- a/doc/user/clusters/agent/gitops/helm.md +++ b/doc/user/clusters/agent/gitops/helm.md @@ -54,16 +54,50 @@ gitops: path: dir-in-project/with/charts namespace: my-ns max_history: 1 + values: + - inline: + someKey: example value ``` | Keyword | Description | |--|--| | `charts` | List of charts you want to be applied in your cluster. Charts are applied concurrently. | | `release_name` | Required. Name of the release to use when applying the chart. | -| `id` | Required. ID of the project where Helm chart is committed. No authentication mechanisms are currently supported. | -| `path` | Optional. Path of the chart in the project repository. Root of the repository is used by default. This is the directory with the `Chart.yaml` file. | +| `values` | Optional. [Custom values](#custom-values) for the release. An array of objects. Only supports `inline` values. | | `namespace` | Optional. Namespace to use when applying the chart. Defaults to `default`. | | `max_history` | Optional. Maximum number of release [revisions to store in the cluster](https://helm.sh/docs/helm/helm_history/). | +| `source` | Required. From where the chart should get installed. Only supports project sources. | +| `source.project.id` | Required. ID of the project where Helm chart is committed. Authentication is not supported. | +| `source.project.path` | Optional. Path of the chart in the project repository. Root of the repository is used by default. Should be the directory with the `Chart.yaml` file. | + +## Custom values + +> [Introduced](https://gitlab.com/gitlab-org/cluster-integration/gitlab-agent/-/merge_requests/766) in GitLab 15.6. Requires both GitLab and the installed agent to be version 15.6 or later. + +To customize the values for a release, set the `values` key. It must be +an array of objects. Each object must have exactly one top-level key that describes +where the values come from. The supported top-level keys are: + +- `inline`: Specify the values inline in YAML format, similar to a Helm values + file. + +When installing a chart with custom values: + +- Custom values get merged on top of the chart's default `values.yaml` file. +- Values from subsequent entries in the `values` array overwrite values from + previous entries. + +Example: + +```yaml +gitops: + charts: + - release_name: some-release + values: + - inline: + someKey: example value + # ... +``` ## Automatic drift remediation @@ -98,7 +132,7 @@ The following are known issues: [this epic](https://gitlab.com/groups/gitlab-org/-/epics/7704). - Values for the chart must be in a `values.yaml` file. This file must be with the chart, in the same project and path. -- Because of drift detection and remediation, release history, stored in the cluster, is not useful. +- Because of drift detection and remediation, the release history stored in the cluster is not useful. A new release is created every five minutes and the oldest release is discarded. Eventually history consists only of the same information. View [this issue](https://gitlab.com/gitlab-org/gitlab/-/issues/372023) for details. diff --git a/doc/user/clusters/agent/index.md b/doc/user/clusters/agent/index.md index 7fdf0bb2bf0..8f4855a7f93 100644 --- a/doc/user/clusters/agent/index.md +++ b/doc/user/clusters/agent/index.md @@ -69,14 +69,15 @@ This workflow has a weaker security model and is not recommended for production GitLab supports the following Kubernetes versions. You can upgrade your Kubernetes version to a supported version at any time: -- 1.24 (support ends on September 22, 2023 or when 1.27 becomes supported) +- 1.25 (support ends on October 22, 2023 or when 1.28 becomes supported) +- 1.24 (support ends on July 22, 2023 or when 1.27 becomes supported) - 1.23 (support ends on February 22, 2023 or when 1.26 becomes supported) -- 1.22 (support ends on October 22, 2022) -- 1.21 (support ends on August 22, 2022) GitLab aims to support a new minor Kubernetes version three months after its initial release. GitLab supports at least three production-ready Kubernetes minor versions at any given time. +When installing the agent, use a Helm version compatible with your Kubernetes version. Other versions of Helm might not work. For a list of compatible versions, see the [Helm version support policy](https://helm.sh/docs/topics/version_skew/). + Support for deprecated APIs can be removed from the GitLab codebase when we drop support for the Kubernetes version that only supports the deprecated API. Some GitLab features might work on versions not listed here. [This epic](https://gitlab.com/groups/gitlab-org/-/epics/4827) tracks support for Kubernetes versions. diff --git a/doc/user/clusters/agent/install/index.md b/doc/user/clusters/agent/install/index.md index 19628419784..2030052e3b0 100644 --- a/doc/user/clusters/agent/install/index.md +++ b/doc/user/clusters/agent/install/index.md @@ -45,6 +45,8 @@ For configuration settings, the agent uses a YAML file in the GitLab project. Yo - You use [a GitOps workflow](../gitops.md#gitops-workflow-steps). - You use [a GitLab CI/CD workflow](../ci_cd_workflow.md#gitlab-cicd-workflow-steps) and want to authorize a different project to use the agent. +Otherwise it is optional. + To create an agent configuration file: 1. Choose a name for your agent. The agent name follows the @@ -90,7 +92,7 @@ You must register an agent before you can install the agent in your cluster. To - If you already have an [agent configuration file](#create-an-agent-configuration-file), select it from the list. 1. Select **Register an agent**. 1. GitLab generates an access token for the agent. You need this token to install the agent - in your cluster and to [update the agent](#update-the-agent-version) to another version. + in your cluster. WARNING: Securely store the agent access token. A bad actor can use this token to access source code in the agent's configuration project, access source code in any public project on the GitLab instance, or even, under very specific conditions, obtain a Kubernetes manifest. diff --git a/doc/user/clusters/agent/troubleshooting.md b/doc/user/clusters/agent/troubleshooting.md index 0c26c533cc3..6b5f58f8f76 100644 --- a/doc/user/clusters/agent/troubleshooting.md +++ b/doc/user/clusters/agent/troubleshooting.md @@ -208,3 +208,24 @@ kubectl delete jobs -l app.kubernetes.io/managed-by=starboard -n gitlab-agent ``` [We're working on making the cleanup of these jobs more robust.](https://gitlab.com/gitlab-org/gitlab/-/issues/362016) + +## Inventory policy prevented actuation (strategy: Apply, status: Empty, policy: MustMatch) + +```json +{ + "error":"inventory policy prevented actuation (strategy: Apply, status: Empty, policy: MustMatch)", + "group":"networking.k8s.io", + "kind":"Deployment", + "name":"resource-name", + "namespace":"namespace", + "status":"Skipped", + "timestamp":"2022-10-29T15:34:21Z", + "type":"apply" +} +``` + +This error occurs when the GitLab agent tries to update an object and the object doesn't have the required annotations. To fix this error, you can: + +- Add the required annotations manually. +- Delete the object and let the agent recreate it. +- Change your [`inventory_policy`](../../infrastructure/clusters/deploy/inventory_object.md#inventory_policy-options) setting. diff --git a/doc/user/clusters/management_project.md b/doc/user/clusters/management_project.md index df338e8fcee..18317ae09ed 100644 --- a/doc/user/clusters/management_project.md +++ b/doc/user/clusters/management_project.md @@ -63,7 +63,7 @@ To associate a cluster management project with your cluster: page. - [Instance-level cluster](../instance/clusters/index.md), on the top bar, select **Main menu > Admin > Kubernetes**. 1. Expand **Advanced settings**. -1. From the **Cluster management project** dropdown, select the cluster management project +1. From the **Cluster management project** dropdown list, select the cluster management project you created in the previous step. ### Configuring your pipeline diff --git a/doc/user/clusters/management_project_template.md b/doc/user/clusters/management_project_template.md index bdd11f11f9c..cbe577b9b74 100644 --- a/doc/user/clusters/management_project_template.md +++ b/doc/user/clusters/management_project_template.md @@ -103,7 +103,6 @@ The [built-in supported applications](https://gitlab.com/gitlab-org/project-temp - [GitLab Runner](../infrastructure/clusters/manage/management_project_applications/runner.md) - [Ingress](../infrastructure/clusters/manage/management_project_applications/ingress.md) - [Prometheus](../infrastructure/clusters/manage/management_project_applications/prometheus.md) -- [Sentry](../infrastructure/clusters/manage/management_project_applications/sentry.md) - [Vault](../infrastructure/clusters/manage/management_project_applications/vault.md) Each application has an `applications/{app}/values.yaml` file. diff --git a/doc/user/compliance/compliance_report/index.md b/doc/user/compliance/compliance_report/index.md index ac4b20b5166..0d33dfce30b 100644 --- a/doc/user/compliance/compliance_report/index.md +++ b/doc/user/compliance/compliance_report/index.md @@ -94,9 +94,7 @@ Our criteria for the separation of duties is as follows: > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/213364) in GitLab 13.3. > - Chain of Custody reports sent using email [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/342594) in GitLab 15.3 with a flag named `async_chain_of_custody_report`. Disabled by default. - -FLAG: -On self-managed GitLab, by default sending Chain of Custody reports through email is not available. To make it available, ask an administrator to [enable the feature flag](../../../administration/feature_flags.md) named `async_chain_of_custody_report`. On GitLab.com, this feature is not available. +> - [Generally available](https://gitlab.com/gitlab-org/gitlab/-/issues/370100) in GitLab 15.5. Feature flag `async_chain_of_custody_report` removed. The Chain of Custody report allows customers to export a list of merge commits within the group. The data provides a comprehensive view with respect to merge commits. It includes the merge commit SHA, @@ -112,7 +110,7 @@ To generate the Chain of Custody report: The Chain of Custody report is either: - Available for download. -- Sent through email. Requires GitLab 15.3 and later with `async_chain_of_custody_report` feature flag enabled. +- Sent through email. Requires GitLab 15.5 and later. ### Commit-specific Chain of Custody report @@ -131,7 +129,7 @@ Authenticated group owners can generate a commit-specific Chain of Custody repor The Chain of Custody report is either: - Available for download. -- Sent through email. Requires GitLab 15.3 and later with `async_chain_of_custody_report` feature flag enabled. +- Sent through email. Requires GitLab 15.5 and later. - Using a direct link: `https://gitlab.com/groups/<group-name>/-/security/merge_commit_reports.csv?commit_sha={optional_commit_sha}`, passing in an optional value to the `commit_sha` query parameter. diff --git a/doc/user/crm/index.md b/doc/user/crm/index.md index d7ab2195e2d..2d1aeaa1c07 100644 --- a/doc/user/crm/index.md +++ b/doc/user/crm/index.md @@ -79,6 +79,21 @@ To edit an existing contact: You can also [edit](../../api/graphql/reference/index.md#mutationcustomerrelationscontactupdate) contacts using the GraphQL API. +#### Change the state of a contact + +Each contact can be in one of two states: + +- **Active**: contacts in this state can be added to an issue. +- **Inactive**: contacts in this state cannot be added to an issue. + +To change the state of a contact: + +1. On the top bar, select **Main menu > Groups** and find your group. +1. On the left sidebar, select **Customer relations > Contacts**. +1. Next to the contact you wish to edit, select **Edit** (**{pencil}**). +1. Select or clear the **Active** checkbox. +1. Select **Save changes**. + ## Organizations ### View organizations @@ -153,7 +168,7 @@ API. ### Add contacts to an issue -To add contacts to an issue use the `/add_contacts [contact:address@example.com]` +To add [active](#change-the-state-of-a-contact) contacts to an issue use the `/add_contacts [contact:address@example.com]` [quick action](../project/quick_actions.md). You can also add, remove, or replace issue contacts using the @@ -175,10 +190,15 @@ API. > - [Enabled on GitLab.com and self-managed](https://gitlab.com/gitlab-org/gitlab/-/issues/352123) in GitLab 15.0. > - [Generally available](https://gitlab.com/gitlab-org/gitlab/-/issues/352123) in GitLab 15.2. [Feature flag `contacts_autocomplete`](https://gitlab.com/gitlab-org/gitlab/-/issues/352123) removed. -When you use the `/add_contacts` or `/remove_contacts` quick actions, follow them with `[contact:` and an autocomplete list appears: +When you use the `/add_contacts` quick action, follow it with `[contact:` and an autocomplete list with the [active](#change-the-state-of-a-contact) contacts appears: ```plaintext /add_contacts [contact: +``` + +When you use the `/remove_contacts` quick action, follow it with `[contact:` and an autocomplete list with the contacts added to the issue appears: + +```plaintext /remove_contacts [contact: ``` diff --git a/doc/user/discussions/index.md b/doc/user/discussions/index.md index 13d5b27ad41..d9cacb6395d 100644 --- a/doc/user/discussions/index.md +++ b/doc/user/discussions/index.md @@ -44,8 +44,9 @@ You can quickly see which comments involve you, because mentions for yourself (the user currently signed in) are highlighted in a different color. -Avoid mentioning `@all` in issues and merge requests, because it sends an email notification -to all the members of that project's group. This might be interpreted as spam. +Avoid mentioning `@all` in issues and merge requests. It sends an email notification +to all members of that project's parent group, not only the participants of the project, +and may be interpreted as spam. Notifications and mentions can be disabled in [a group's settings](../group/manage.md#disable-email-notifications). @@ -77,7 +78,7 @@ To add a commit diff comment: You can select multiple lines by dragging the **Comment** (**{comment}**) icon. 1. Enter your comment and select **Start a review** or **Add comment now**. -The comment is displayed on the merge request's **Discussions** tab. +The comment is displayed on the merge request's **Overview** tab. The comment is not displayed on your project's **Repository > Commits** page. @@ -143,7 +144,7 @@ If you edit an existing comment to add a user mention that wasn't there before, - Creates a to-do item for the mentioned user. - Does not send a notification email. -## Prevent comments by locking an issue +## Prevent comments by locking the discussion You can prevent public comments in an issue or merge request. When you do, only project members can add and edit comments. @@ -153,6 +154,8 @@ Prerequisite: - In merge requests, you must have at least the Developer role. - In issues, you must have at least the Reporter role. +To lock an issue or merge request: + 1. On the right sidebar, next to **Lock issue** or **Lock merge request**, select **Edit**. 1. On the confirmation dialog, select **Lock**. @@ -160,6 +163,9 @@ Notes are added to the page details. If an issue or merge request is locked and closed, you cannot reopen it. +<!-- Delete when the `moved_mr_sidebar` feature flag is removed --> +If you don't see this action on the right sidebar, your project or instance might have [moved sidebar actions](../project/merge_requests/index.md#move-sidebar-actions) enabled. + ## Add an internal note > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/207473) in GitLab 13.9 [with a flag](../../administration/feature_flags.md) named `confidential_notes`. Disabled by default. @@ -167,12 +173,9 @@ If an issue or merge request is locked and closed, you cannot reopen it. > - [Renamed](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/87403) from "confidential comments" to "internal notes" in GitLab 15.0. > - [Enabled on GitLab.com and self-managed](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/87383) in GitLab 15.0. > - [Feature flag `confidential_notes`](https://gitlab.com/gitlab-org/gitlab/-/issues/362712) removed in GitLab 15.2. +> - [Changed](https://gitlab.com/gitlab-org/gitlab/-/issues/363045) permissions in GitLab 15.5 to at least the Reporter role. In GitLab 15.4 and earlier, issue or epic authors and assignees could also read and create internal notes. -You can add an internal note **to an issue or an epic**. It's then visible only to the following people: - -- Project members who have at least the Reporter role -- Issue or epic author -- Users assigned to the issue or epic +You can add an internal note **to an issue or an epic**. It's then visible only to project members who have at least the Reporter role. Keep in mind: @@ -181,10 +184,7 @@ Keep in mind: Prerequisites: -- You must either: - - Have at least the Reporter role for the project. - - Be the issue or epic assignee. - - Be the issue or epic author. +- You must have at least the Reporter role for the project. To add an internal note: @@ -200,14 +200,14 @@ You can also mark an [issue as confidential](../project/issues/confidential_issu For issues and merge requests with many comments, you can filter the page to show comments only. -1. Open a merge request's **Discussion** tab, or epic or issue's **Overview** tab. +1. Open the **Overview** tab in a merge request, issue, or epic. 1. On the right side of the page, select from the filter: - **Show all activity**: Display all user comments and system notes. (issue updates, mentions from other issues, changes to the description, and so on). - **Show comments only**: Display only user comments. - **Show history only**: Display only activity notes. - + GitLab saves your preference, so it persists when you visit the same page again from any device you're logged into. diff --git a/doc/user/feature_highlight.md b/doc/user/feature_highlight.md deleted file mode 100644 index ef96d2524a6..00000000000 --- a/doc/user/feature_highlight.md +++ /dev/null @@ -1,11 +0,0 @@ ---- -redirect_to: 'index.md' -remove_date: '2022-10-29' ---- - -This document was moved to [another location](index.md). - -<!-- This redirect file can be deleted after <2022-10-29>. --> -<!-- Redirects that point to other docs in the same project expire in three months. --> -<!-- Redirects that point to docs in a different project or site (for example, link is not relative and starts with `https:`) expire in one year. --> -<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/redirects.html --> diff --git a/doc/user/free_user_limit.md b/doc/user/free_user_limit.md index 3fbfb2e1aa7..35777847947 100644 --- a/doc/user/free_user_limit.md +++ b/doc/user/free_user_limit.md @@ -29,6 +29,34 @@ If you need more time to manage your members, or to try GitLab features with a team of more than five members, you can [start a trial](https://gitlab.com/-/trial_registrations/new?glm_source=docs.gitlab.com&glm_content=free-user-limit). A trial lasts for 30 days and includes an unlimited number of members. +## Determining namespace user counts + +Every unique user of a top-level namespace with private visibility counts towards the five-user limit. This includes every user of a group, subgroup, and project within a namespace. + +For example: + +The group `example-1` has: + +- One group owner, `A`. +- One subgroup called `subgroup-1` with one member, `B`. + - `subgroup-1` inherits `A` as a member from `example-1`. +- One project in `subgroup-1` called `project-1` with two members, `C` and `D`. + - `project-1` inherits `A` and `B` as members from `subgroup-1`. + +The namespace `example-1` has four unique members: `A`, `B`, `C`, and `D`. Because `example-1` has only four unique members, it is not impacted by the five-user limit. + +The group `example-2` has: + +- One group owner, `A`. +- One subgroup called `subgroup-2` with one member, `B`. + - `subgroup-2` inherits `A` as a member from `example-2`. +- One project in `subgroup-2` called `project-2a` with two members, `C` and `D`. + - `project-2a` inherits `A` and `B` as members from `subgroup-2`. +- One project in `subgroup-2` called `project-2b` with two members, `E` and `F`. + - `project-2b` inherits `A` and `B` as members from `subgroup-2`. + +The namespace `example-2` has six unique members: `A`, `B`, `C`, `D`, `E`, and `F`. Because `example-2` has six unique users, it is impacted by the five-user limit. + ## Related topics - [GitLab SaaS Free tier frequently asked questions](https://about.gitlab.com/pricing/faq-efficient-free-tier/) diff --git a/doc/user/group/access_and_permissions.md b/doc/user/group/access_and_permissions.md index 395ed3c91c7..13a1fd31ee4 100644 --- a/doc/user/group/access_and_permissions.md +++ b/doc/user/group/access_and_permissions.md @@ -56,18 +56,20 @@ To change the permitted Git access protocols for a group: 1. Choose the permitted protocols from **Enabled Git access protocols**. 1. Select **Save changes**. -## Restrict group access by IP address **(PREMIUM)** +## Restrict access to groups by IP address **(PREMIUM)** > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/1985) in GitLab 12.0. > - [Moved](https://gitlab.com/gitlab-org/gitlab/-/issues/215410) from GitLab Ultimate to GitLab Premium in 13.1. -To ensure only people from your organization can access particular -resources, you can restrict access to groups by IP address. This group-level setting -applies to: +To ensure only people from your organization can access particular resources, you can restrict access to groups by IP +address. This group-level setting applies to: - The GitLab UI, including subgroups, projects, and issues. - [In GitLab 12.3 and later](https://gitlab.com/gitlab-org/gitlab/-/issues/12874), the API. +Administrators can combine restricted access by IP address with +[globally-allowed IP addresses](../admin_area/settings/visibility_and_access_controls.md#configure-globally-allowed-ip-address-ranges). + ### Security implications You should consider some security implications before configuring IP address restrictions. @@ -94,7 +96,12 @@ To restrict group access by IP address: 1. On the top bar, select **Main menu > Groups** and find your group. 1. On the left sidebar, select **Settings > General**. 1. Expand the **Permissions and group features** section. -1. In the **Restrict access by IP address** field, enter IPv4 or IPv6 address ranges in CIDR notation. +1. In the **Restrict access by IP address** field, enter a list of IPv4 or IPv6 + address ranges in CIDR notation. This list: + - Has no limit on the number of IP address ranges. + - Has a size limit of 1 GB. + - Applies to both SSH or HTTP authorized IP address ranges. You cannot split + this list by type of authorization. 1. Select **Save changes**. In self-managed installations of GitLab 15.1 and later, you can also configure @@ -290,5 +297,4 @@ If a user sees a 404 when they would normally expect access, and the problem is - `json.message`: `'Attempting to access IP restricted group'` - `json.allowed`: `false` -In viewing the log entries, compare the `remote.ip` with the list of -[allowed IPs](#restrict-group-access-by-ip-address) for the group. +In viewing the log entries, compare the `remote.ip` with the list of [allowed IP addresses](#restrict-access-to-groups-by-ip-address) for the group. diff --git a/doc/user/group/clusters/index.md b/doc/user/group/clusters/index.md index c6cc828302f..62f5a3ba54f 100644 --- a/doc/user/group/clusters/index.md +++ b/doc/user/group/clusters/index.md @@ -187,6 +187,6 @@ important to describe those, too. Think of things that may go wrong and include This is important to minimize requests for support, and to avoid doc comments with questions that you know someone might ask. -Each scenario can be a third-level heading, e.g. `### Getting error message X`. +Each scenario can be a third-level heading, for example `### Getting error message X`. If you have none to add when creating a doc, leave this section in place but commented out to help encourage others to add to it in the future. --> diff --git a/doc/user/group/compliance_frameworks.md b/doc/user/group/compliance_frameworks.md new file mode 100644 index 00000000000..7bd545003db --- /dev/null +++ b/doc/user/group/compliance_frameworks.md @@ -0,0 +1,262 @@ +--- +stage: Govern +group: Compliance +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments +--- + +# Compliance frameworks **(PREMIUM)** + +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/276221) in GitLab 13.9. +> - [Feature flag removed](https://gitlab.com/gitlab-org/gitlab/-/issues/287779) in GitLab 13.12. + +You can create a compliance framework that is a label to identify that your project has certain compliance +requirements or needs additional oversight. The label can optionally enforce +[compliance pipeline configuration](#configure-a-compliance-pipeline) to the projects on which it is +[applied](../project/settings/index.md#add-a-compliance-framework-to-a-project). + +Group owners can create, edit, and delete compliance frameworks: + +1. On the top bar, select **Main menu > Groups > View all groups** and find your group. +1. On the left sidebar, select **Settings** > **General**. +1. Expand the **Compliance frameworks** section. +1. Create, edit, or delete compliance frameworks. + +## Set a default compliance framework + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/375036) in GitLab 15.6. + +Group owners can set a default compliance framework. The default framework is applied to all the new projects +that are created within that group. It does not affect the framework applied to the existing projects. The default +framework cannot be deleted. + +### Example GraphQL mutations for setting a default compliance framework + +Creating a new compliance framework and setting it as the default framework for the group. + +```graphql +mutation { + createComplianceFramework( + input: {params: {name: "SOX", description: "Sarbanes-Oxley Act", color: "#87CEEB", default: true}, namespacePath: "gitlab-org"} + ) { + framework { + id + name + default + description + color + pipelineConfigurationFullPath + } + errors + } +} +``` + +Setting an existing compliance framework as the default framework the group. + +```graphql +mutation { + updateComplianceFramework( + input: {id: "gid://gitlab/ComplianceManagement::Framework/<id>", params: {default: true}} + ) { + complianceFramework { + id + name + default + description + color + pipelineConfigurationFullPath + } + } +} +``` + +## Configure a compliance pipeline **(ULTIMATE)** + +> - [Introduced](https://gitlab.com/groups/gitlab-org/-/epics/3156) in GitLab 13.9, disabled behind `ff_evaluate_group_level_compliance_pipeline` [feature flag](../../administration/feature_flags.md). +> - [Enabled by default](https://gitlab.com/gitlab-org/gitlab/-/issues/300324) in GitLab 13.11. +> - [Feature flag removed](https://gitlab.com/gitlab-org/gitlab/-/issues/331231) in GitLab 14.2. + +Group owners can configure a compliance pipeline in a project separate to other projects. By default, the compliance +pipeline configuration (`.gitlab-ci.yml` file) is run instead of the pipeline configuration of labeled projects. + +However, the compliance pipeline configuration can reference the `.gitlab-ci.yml` file of the labeled projects so that: + +- The compliance pipeline can also run jobs of labeled project pipelines. This allows for centralized control of + pipeline configuration. +- Jobs and variables defined in the compliance pipeline can't be changed by variables in the labeled project's + `.gitlab-ci.yml` file. + +See [example configuration](#example-configuration) for help configuring a compliance pipeline that runs jobs from +labeled project pipeline configuration. + +To configure a compliance pipeline: + +1. On the top bar, select **Main menu > Groups > View all groups** and find your group. +1. On the left sidebar, select **Settings** > **General**. +1. Expand the **Compliance frameworks** section. +1. In **Compliance pipeline configuration (optional)**, add the path to the compliance framework configuration. Use the + `path/file.y[a]ml@group-name/project-name` format. For example: + + - `.compliance-ci.yml@gitlab-org/gitlab`. + - `.compliance-ci.yaml@gitlab-org/gitlab`. + +This configuration is inherited by projects where the compliance framework label is +[applied](../project/settings/index.md#add-a-compliance-framework-to-a-project). In projects with the applied compliance +framework label, the compliance pipeline configuration is run instead of the labeled project's own pipeline configuration. + +The user running the pipeline in the labeled project must at least have the Reporter role on the compliance project. + +When used to enforce scan execution, this feature has some overlap with +[scan execution policies](../application_security/policies/scan-execution-policies.md). We have not +[unified the user experience for these two features](https://gitlab.com/groups/gitlab-org/-/epics/7312). For details on +the similarities and differences between these features, see [Enforce scan execution](../application_security/index.md#enforce-scan-execution). + +### Example configuration + +The following example `.compliance-gitlab-ci.yml` includes the `include` keyword to ensure labeled project pipeline +configuration is also executed. + +```yaml +# Allows compliance team to control the ordering and interweaving of stages/jobs. +# Stages without jobs defined will remain hidden. +stages: + - pre-compliance + - build + - test + - pre-deploy-compliance + - deploy + - post-compliance + +variables: # Can be overridden by setting a job-specific variable in project's local .gitlab-ci.yml + FOO: sast + +sast: # None of these attributes can be overridden by a project's local .gitlab-ci.yml + variables: + FOO: sast + image: ruby:2.6 + stage: pre-compliance + rules: + - if: $CI_COMMIT_BRANCH && $CI_OPEN_MERGE_REQUESTS && $CI_PIPELINE_SOURCE == "push" + when: never + - when: always # or when: on_success + allow_failure: false + before_script: + - "# No before scripts." + script: + - echo "running $FOO" + after_script: + - "# No after scripts." + +sanity check: + image: ruby:2.6 + stage: pre-deploy-compliance + rules: + - if: $CI_COMMIT_BRANCH && $CI_OPEN_MERGE_REQUESTS && $CI_PIPELINE_SOURCE == "push" + when: never + - when: always # or when: on_success + allow_failure: false + before_script: + - "# No before scripts." + script: + - echo "running $FOO" + after_script: + - "# No after scripts." + +audit trail: + image: ruby:2.7 + stage: post-compliance + rules: + - if: $CI_COMMIT_BRANCH && $CI_OPEN_MERGE_REQUESTS && $CI_PIPELINE_SOURCE == "push" + when: never + - when: always # or when: on_success + allow_failure: false + before_script: + - "# No before scripts." + script: + - echo "running $FOO" + after_script: + - "# No after scripts." + +include: # Execute individual project's configuration (if project contains .gitlab-ci.yml) + project: '$CI_PROJECT_PATH' + file: '$CI_CONFIG_PATH' + ref: '$CI_COMMIT_REF_NAME' # Must be defined or MR pipelines always use the use default branch +``` + +#### CF pipelines in Merge Requests originating in project forks + +When an MR originates in a fork, the branch to be merged usually only exists in the fork. +When creating such an MR against a project with CF pipelines, the above snippet will fail with a +`Project <project-name> reference <branch-name> does not exist!` error message. +This is because in the context of the target project, `$CI_COMMIT_REF_NAME` evaluates to a non-existing branch name. + +To get the correct context, use `$CI_MERGE_REQUEST_SOURCE_PROJECT_PATH` instead of `$CI_PROJECT_PATH`. +This variable is only availabe in +[merge request pipelines](../../ci/pipelines/merge_request_pipelines.md). + +For example, for a configuration that supports both merge request pipelines originating in project forks and branch pipelines, +you need to [combine both `include` directives with `rules:if`](../../ci/yaml/includes.md#use-rules-with-include): + +```yaml +include: # Execute individual project's configuration (if project contains .gitlab-ci.yml) + - project: '$CI_MERGE_REQUEST_SOURCE_PROJECT_PATH' + file: '$CI_CONFIG_PATH' + ref: '$CI_COMMIT_REF_NAME' + rules: + - if: $CI_PIPELINE_SOURCE == 'merge_request_event' + - project: '$CI_PROJECT_PATH' + file: '$CI_CONFIG_PATH' + ref: '$CI_COMMIT_REF_NAME' + rules: + - if: $CI_PIPELINE_SOURCE != 'merge_request_event' +``` + +## Ensure compliance jobs are always run + +Compliance pipelines [use GitLab CI/CD](../../ci/index.md) to give you an incredible amount of flexibility +for defining any sort of compliance jobs you like. Depending on your goals, these jobs +can be configured to be: + +- Modified by users. +- Non-modifiable. + +Generally, if a value in a compliance job: + +- Is set, it cannot be changed or overridden by project-level configurations. +- Is not set, a project-level configuration may set. + +Either might be wanted or not depending on your use case. + +There are a few best practices for ensuring that these jobs are always run exactly +as you define them and that downstream, project-level pipeline configurations +cannot change them: + +- Add [a `rules:when:always` block](../../ci/yaml/index.md#when) to each of your compliance jobs. This ensures they are + non-modifiable and are always run. +- Explicitly set any [variables](../../ci/yaml/index.md#variables) the job references. This: + - Ensures that project-level pipeline configurations do not set them and alter their + behavior. + - Includes any jobs that drive the logic of your job. +- Explicitly set the [container image](../../ci/yaml/index.md#image) to run the job in. This ensures that your script + steps execute in the correct environment. +- Explicitly set any relevant GitLab pre-defined [job keywords](../../ci/yaml/index.md#job-keywords). + This ensures that your job uses the settings you intend and that they are not overridden by + project-level pipelines. + +## Avoid parent and child pipelines in GitLab 14.7 and earlier + +NOTE: +This advice does not apply to GitLab 14.8 and later because [a fix](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/78878) added +compatibility for combining compliance pipelines, and parent and child pipelines. + +Compliance pipelines start on the run of _every_ pipeline in a labeled project. This means that if a pipeline in the labeled project +triggers a child pipeline, the compliance pipeline runs first. This can trigger the parent pipeline, instead of the child pipeline. + +Therefore, in projects with compliance frameworks, we recommend replacing +[parent-child pipelines](../../ci/pipelines/downstream_pipelines.md#parent-child-pipelines) with the following: + +- Direct [`include`](../../ci/yaml/index.md#include) statements that provide the parent pipeline with child pipeline configuration. +- Child pipelines placed in another project that are run using the [trigger API](../../ci/triggers/index.md) rather than the parent-child + pipeline feature. + +This alternative ensures the compliance pipeline does not re-start the parent pipeline. diff --git a/doc/user/group/contribution_analytics/index.md b/doc/user/group/contribution_analytics/index.md index 280781a4cad..b1efd2e9251 100644 --- a/doc/user/group/contribution_analytics/index.md +++ b/doc/user/group/contribution_analytics/index.md @@ -8,7 +8,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/3090) in GitLab 12.2 for subgroups. -With Contribution Analytics, you can get an overview of the [contribution events](../../profile/index.md#user-contribution-events) in your +With Contribution Analytics, you can get an overview of the [contribution events](../../profile/contributions_calendar.md#user-contribution-events) in your group. - Analyze your team's contributions over a period of time. @@ -43,7 +43,7 @@ You can choose from the following three periods: - Last month - Last three months -Select the desired period from the calendar dropdown. +Select the desired period from the calendar dropdown list.  @@ -62,6 +62,10 @@ Contributions per group member are also presented in tabular format. Select a co  +## Contribution analytics GraphQL API + +To retrieve metrics for user contributions, use the [GraphQL](../../../api/graphql/reference/index.md#groupcontributions) API. + <!-- ## Troubleshooting Include any troubleshooting steps that you can foresee. If you know beforehand what issues @@ -70,6 +74,6 @@ important to describe those, too. Think of things that may go wrong and include This is important to minimize requests for support, and to avoid doc comments with questions that you know someone might ask. -Each scenario can be a third-level heading, e.g. `### Getting error message X`. +Each scenario can be a third-level heading, for example `### Getting error message X`. If you have none to add when creating a doc, leave this section in place but commented out to help encourage others to add to it in the future. --> diff --git a/doc/user/group/custom_project_templates.md b/doc/user/group/custom_project_templates.md index 7f77c2147e1..547e64df7c5 100644 --- a/doc/user/group/custom_project_templates.md +++ b/doc/user/group/custom_project_templates.md @@ -78,6 +78,6 @@ important to describe those, too. Think of things that may go wrong and include This is important to minimize requests for support, and to avoid doc comments with questions that you know someone might ask. -Each scenario can be a third-level heading, e.g. `### Getting error message X`. +Each scenario can be a third-level heading, for example `### Getting error message X`. If you have none to add when creating a doc, leave this section in place but commented out to help encourage others to add to it in the future. --> diff --git a/doc/user/group/epics/epic_boards.md b/doc/user/group/epics/epic_boards.md index 78cc65f1923..64addd524ad 100644 --- a/doc/user/group/epics/epic_boards.md +++ b/doc/user/group/epics/epic_boards.md @@ -30,7 +30,7 @@ To create a new epic board: 1. On the top bar, select **Main menu > Groups** and find your group. 1. On the left sidebar, select **Epics > Boards**. -1. In the upper left corner, select the dropdown with the current board name. +1. In the upper left corner, select the dropdown list with the current board name. 1. Select **Create new board**. 1. Enter the new board's title. 1. Optional. To hide the Open or Closed lists, clear the **Show the Open list** and @@ -54,7 +54,7 @@ Prerequisites: To delete the active epic board: -1. Select the dropdown with the current board name in the upper left corner of the epic boards page. +1. Select the dropdown list with the current board name in the upper left corner of the epic boards page. 1. Select **Delete board**. 1. Select **Delete**. @@ -80,7 +80,7 @@ To create a new list: 1. On the top bar, select **Main menu > Groups** and find your group. 1. On the left sidebar, select **Epics > Boards**. 1. In the upper-right corner, select **Create list**. -1. In the **New list** column expand the **Select a label** dropdown and select the label to use as +1. In the **New list** column expand the **Select a label** dropdown list and select the label to use as list scope. 1. Select **Add to board**. diff --git a/doc/user/group/epics/index.md b/doc/user/group/epics/index.md index da6e675f0eb..21c95f37aeb 100644 --- a/doc/user/group/epics/index.md +++ b/doc/user/group/epics/index.md @@ -37,15 +37,9 @@ Also, read more about possible [planning hierarchies](../planning_hierarchy/inde ### Child issues from different group hierarchies -<!-- When feature flag is removed, integrate this info as a sentence in -https://docs.gitlab.com/ee/user/group/epics/manage_epics.html#add-an-existing-issue-to-an-epic --> - > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/371081) in GitLab 15.5 [with a flag](../../../administration/feature_flags.md) named `epic_issues_from_different_hierarchies`. Disabled by default. > - [Enabled on GitLab.com](https://gitlab.com/gitlab-org/gitlab/-/issues/373304) in GitLab 15.5. - -FLAG: -On self-managed GitLab, by default this feature is unavailable. To make it available, ask an administrator to [enable the feature flag](../../../administration/feature_flags.md) named `epic_issues_from_different_hierarchies`. -On GitLab.com, this feature is available. +> - Feature flag `epic_issues_from_different_hierarchies` removed in GitLab 15.6. You can add issues from a different group hierarchy to an epic. To do it, paste the issue URL when @@ -77,6 +71,6 @@ important to describe those, too. Think of things that may go wrong and include This is important to minimize requests for support, and to avoid doc comments with questions that you know someone might ask. -Each scenario can be a third-level heading, e.g. `### Getting error message X`. +Each scenario can be a third-level heading, for example `### Getting error message X`. If you have none to add when creating a doc, leave this section in place but commented out to help encourage others to add to it in the future. --> diff --git a/doc/user/group/epics/manage_epics.md b/doc/user/group/epics/manage_epics.md index 0a12f551588..61aa5c5fe02 100644 --- a/doc/user/group/epics/manage_epics.md +++ b/doc/user/group/epics/manage_epics.md @@ -24,8 +24,8 @@ To create an epic in the group you're in: 1. Get to the New Epic form: - Go to your group and from the left sidebar select **Epics**. Then select **New epic**. - - From an epic in your group, select **New epic**. - - From anywhere, in the top menu, select **New...** (**{plus-square}**) **> New epic**. + - From an epic in your group, select the vertical ellipsis (**{ellipsis_v}**). Then select **New epic**. + - From anywhere, in the top menu, select **New...** (**{plus-square}**). Then select **New epic**. - In an empty [roadmap](../roadmap/index.md), select **New epic**. 1. Enter a title. @@ -257,7 +257,7 @@ To filter: 1. On the top bar, select **Main menu > Groups** and find your group. 1. On the left sidebar, select **Epics**. 1. Select the field **Search or filter results**. -1. From the dropdown menu, select the scope or enter plain text to search by epic title or description. +1. From the dropdown list, select the scope or enter plain text to search by epic title or description. 1. Press <kbd>Enter</kbd> on your keyboard. The list is filtered. ## Sort the list of epics @@ -282,10 +282,10 @@ You can reverse the default order and interact with the activity feed sorted by at the top. Your preference is saved via local storage and automatically applied to every epic and issue you view. -To change the activity sort order, select the **Oldest first** dropdown menu and select either oldest +To change the activity sort order, select the **Oldest first** dropdown list and select either oldest or newest items to be shown first. - + ## Make an epic confidential @@ -311,6 +311,8 @@ To make an epic confidential: - **In an existing epic:** on the right sidebar, select **Edit** next to **Confidentiality**, and then select **Turn on**. +In GitLab 15.6 and later, you can also use the `/confidential` [quick action](../../../user/project/quick_actions.md). + ## Manage issues assigned to an epic This section collects instructions for all the things you can do with [issues](../../project/issues/index.md) @@ -339,9 +341,8 @@ automatically added to the epic. #### Add an existing issue to an epic -You can add existing issues to an epic, including issues in a project in an epic's group, or any of -the epic's subgroups. Newly added issues appear at the top of the list of -issues in the **Epics and Issues** tab. +You can add existing issues to an epic, including issues in a project from a [different group hierarchy](index.md#child-issues-from-different-group-hierarchies). +Newly added issues appear at the top of the list of issues in the **Epics and Issues** tab. An epic contains a list of issues and an issue can be associated with at most one epic. When you add a new issue that's already linked to an epic, the issue is automatically unlinked from its @@ -359,7 +360,8 @@ To add an existing issue to an epic: 1. Identify the issue to be added, using either of the following methods: - Paste the link of the issue. - Search for the desired issue by entering part of the issue's title, then selecting the desired - match ([introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/9126) in GitLab 12.5). + match ([introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/9126) in GitLab 12.5). Issues + from different group hierarchies do not appear in search results. To add such an issue, enter its full URL. If there are multiple issues to be added, press <kbd>Space</kbd> and repeat this step. 1. Select **Add**. @@ -486,9 +488,26 @@ New child epics appear at the top of the list of epics in the **Epics and Issues When you add an epic that's already linked to a parent epic, the link to its current parent is removed. -Epics can contain multiple nested child epics, up to a total of seven levels deep. +Epics can contain multiple nested child epics, up to a total of 7 levels deep. + +The maximum number of direct child epics is 100. + +### Child epics from other groups + +> [Introduced](https://gitlab.com/groups/gitlab-org/-/epics/8502) in GitLab 15.6 [with a flag](../../../administration/feature_flags.md) named `child_epics_from_different_hierarchies`. Disabled by default. + +FLAG: +On self-managed GitLab, by default this feature is not available. To make it available per group, ask an administrator to [enable the feature flag](../../../administration/feature_flags.md) named `child_epics_from_different_hierarchies`. +On GitLab.com, this feature is not available. The feature is not ready for production use. + +You can add a child epic that belongs to a group that is different from the parent epic's group. -Maximum number of direct child epics is 100. +Prerequisites: + +- You must have at least the Reporter role for both the child and parent epics' groups. +- Multi-level child epics must be available for both the child and parent epics' groups. + +To add a child epic from another group, paste the epic's URL when [adding an existing epic](#add-a-child-epic-to-an-epic). ### Add a child epic to an epic @@ -496,14 +515,19 @@ Prerequisites: - You must have at least the Reporter role for the parent epic's group. -To add a child epic to an epic: +To add a new epic as child epic: -1. Select **Add**. -1. Select **Add a new epic**. +1. In an epic, in the **Child issues and epics** section, select **Add > Add a new epic**. +1. Select a group from the dropdown. The epic's group is selected by default. +1. Enter a title for the new epic. +1. Select **Create epic**. + +To add an existing epic as child epic: + +1. In an epic, in the **Child issues and epics** section, select **Add > Add an existing epic**. 1. Identify the epic to be added, using either of the following methods: - Paste the link of the epic. - - Search for the desired issue by entering part of the epic's title, then selecting the desired - match ([introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/9126) in GitLab 12.5). + - Search for the desired issue by entering part of the epic's title, then selecting the desired match. This search is only available for epics within the same group hierarchy. If there are multiple epics to be added, press <kbd>Space</kbd> and repeat this step. 1. Select **Add**. diff --git a/doc/user/group/import/index.md b/doc/user/group/import/index.md index ee50cfcf182..6dcc01242c2 100644 --- a/doc/user/group/import/index.md +++ b/doc/user/group/import/index.md @@ -4,7 +4,7 @@ group: Import info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- -# Migrating groups **(FREE)** +# Migrating groups with GitLab Migration **(FREE)** > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/249160) in GitLab 13.7 for group resources [with a flag](../../feature_flags.md) named `bulk_import`. Disabled by default. > - Group items [enabled on GitLab.com and self-managed](https://gitlab.com/gitlab-org/gitlab/-/issues/338985) in GitLab 14.3. @@ -18,15 +18,15 @@ this feature, ask an administrator to [enable the feature flag](../../../adminis `bulk_import_projects`. On GitLab.com, migrating group resources is available but migrating project resources is not available. -Users with the Owner role on a top-level group can migrate it to: +Users with the Owner role on a top-level group can use GitLab Migration to migrate the group to: - Another top-level group. - The subgroup of any existing top-level group. - Another GitLab instance, including GitLab.com. -Migrating groups using the method documented here is not the same as [migrating groups using file exports](../settings/import_export.md). +Migrating groups using GitLab Migration is not the same as [migrating groups using file exports](../settings/import_export.md). Importing and exporting groups using file exports requires you to export a group to a file and then import that file in -another GitLab instance. Migrating groups using the method documented here automates this step. +another GitLab instance. Migrating groups using GitLab Migration automates this step. ## Import your groups into GitLab @@ -139,6 +139,7 @@ Migrating projects with file exports uses the same export and import mechanisms - Issue resource state events ([Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/291983) in GitLab 15.4) - Issue resource milestone events ([Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/291983) in GitLab 15.4) - Issue resource iteration events ([Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/291983) in GitLab 15.4) + - Merge request URL references ([Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/267947) in GitLab 15.6) - Labels ([Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/339419) in GitLab 14.4) - LFS Objects ([Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/339405) in GitLab 14.8) - Members ([Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/341886) in GitLab 14.8) @@ -148,6 +149,7 @@ Migrating projects with file exports uses the same export and import mechanisms - Merge request approvers ([Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/339520) in GitLab 15.3) - Merge request resource state events ([Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/291983) in GitLab 15.4) - Merge request resource milestone events ([Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/291983) in GitLab 15.4) + - Issue URL references ([Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/267947) in GitLab 15.6) - Migrate Push Rules ([Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/339403) in GitLab 14.6) - Pull Requests (including external pull requests) ([Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/339409) in GitLab 14.5) - Pipeline History ([Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/339412) in GitLab 14.6) diff --git a/doc/user/group/insights/index.md b/doc/user/group/insights/index.md index b4bca919498..9eb6d4387c1 100644 --- a/doc/user/group/insights/index.md +++ b/doc/user/group/insights/index.md @@ -79,6 +79,6 @@ important to describe those, too. Think of things that may go wrong and include This is important to minimize requests for support, and to avoid doc comments with questions that you know someone might ask. -Each scenario can be a third-level heading, e.g. `### Getting error message X`. +Each scenario can be a third-level heading, for example `### Getting error message X`. If you have none to add when creating a doc, leave this section in place but commented out to help encourage others to add to it in the future. --> diff --git a/doc/user/group/issues_analytics/index.md b/doc/user/group/issues_analytics/index.md index 4764625ff83..dbade014ec2 100644 --- a/doc/user/group/issues_analytics/index.md +++ b/doc/user/group/issues_analytics/index.md @@ -55,6 +55,6 @@ important to describe those, too. Think of things that may go wrong and include This is important to minimize requests for support, and to avoid doc comments with questions that you know someone might ask. -Each scenario can be a third-level heading, e.g. `### Getting error message X`. +Each scenario can be a third-level heading, for example `### Getting error message X`. If you have none to add when creating a doc, leave this section in place but commented out to help encourage others to add to it in the future. --> diff --git a/doc/user/group/iterations/index.md b/doc/user/group/iterations/index.md index a6435ae2aa3..518370fc7ac 100644 --- a/doc/user/group/iterations/index.md +++ b/doc/user/group/iterations/index.md @@ -278,8 +278,8 @@ To group issues by label: 1. On the top bar, select **Main menu > Groups** and find your group. 1. On the left sidebar, select **Issues > Iterations**. -1. In the **Group by** dropdown, select **Label**. -1. Select the **Filter by label** dropdown. -1. Select the labels you want to group by in the labels dropdown. +1. In the **Group by** dropdown list, select **Label**. +1. Select the **Filter by label** dropdown list. +1. Select the labels you want to group by in the labels dropdown list. You can also search for labels by typing in the search input. 1. Select any area outside the label dropdown list. The page is now grouped by the selected labels. diff --git a/doc/user/group/manage.md b/doc/user/group/manage.md index f11d9035a52..a63e6c6dd7f 100644 --- a/doc/user/group/manage.md +++ b/doc/user/group/manage.md @@ -66,6 +66,7 @@ This action removes the group. It also adds a background job to delete all proje Specifically: - In [GitLab 12.8 and later](https://gitlab.com/gitlab-org/gitlab/-/issues/33257), on [GitLab Premium](https://about.gitlab.com/pricing/premium/) or higher tiers, this action adds a background job to mark a group for deletion. By default, the job schedules the deletion 7 days in the future. You can modify this waiting period through the [instance settings](../admin_area/settings/visibility_and_access_controls.md#deletion-protection). + - In [GitLab 13.6 and later](https://gitlab.com/gitlab-org/gitlab/-/issues/39504), if the user who sets up the deletion is removed from the group before the deletion happens, the job is cancelled, and the group is no longer scheduled for deletion. @@ -201,6 +202,13 @@ To remove a member from a group: - To unassign the user from linked issues and merge requests, select the **Also unassign this user from linked issues and merge requests** checkbox. 1. Select **Remove member**. +## Ensure removed users cannot invite themselves back + +Malicious users with the Maintainer or Owner role could exploit a race condition that allows +them to invite themselves back to a group or project that a GitLab administrator has removed them from. + +To avoid this problem, GitLab administrators can [ensure removed users cannot invite themselves back](../project/members/index.md#ensure-removed-users-cannot-invite-themselves-back). + ## Add projects to a group There are two different ways to add a new project to a group: @@ -255,6 +263,12 @@ If you are changing the path so it can be claimed by another group or user, you must rename the group too. Both names and paths must be unique. +After you change the group path, the new group path is a new namespace and you must update the existing project URL in the following resources: + +- [Include statements](../../ci/yaml/includes.md#include-a-single-configuration-file). +- Docker image references in CI files. +- Variables that specify a project or namespace. + To retain ownership of the original namespace and protect the URL redirects, create a new group and transfer projects to it instead. @@ -303,9 +317,7 @@ for the group's projects to meet your group's needs. [Feature flag `invite_members_group_modal`](https://gitlab.com/gitlab-org/gitlab/-/issues/352526) removed. Similar to how you [share a project with a group](../project/members/share_project_with_groups.md), -you can share a group with another group. To invite a group, you must be a member of it. Members get direct access -to the shared group. This includes members who inherited group membership from a parent group. - +you can share a group with another group. To invite a group, you must be a member of it. To share a given group, for example, `Frontend` with another group, for example, `Engineering`: @@ -320,7 +332,7 @@ After sharing the `Frontend` group with the `Engineering` group: - The **Groups** tab lists the `Engineering` group. - The **Groups** tab lists a group regardless of whether it is a public or private group. -- All members of the `Engineering` group have access to the `Frontend` group. The same access levels of the members apply up to the maximum access level selected when sharing the group. +- All direct members of the `Engineering` group have access to the `Frontend` group. Direct members of `Engineering` that gain access to the `Frontend` group keep their same access level as in `Engineering`, but up to the maximum access level selected when sharing the group. Inherited members of the `Engineering` group do not gain access to the `Frontend` group. ## Transfer a group @@ -385,214 +397,6 @@ To enable delayed deletion of projects in a group: NOTE: In GitLab 13.11 and above the group setting for delayed project deletion is inherited by subgroups. As discussed in [Cascading settings](../../development/cascading_settings.md) inheritance can be overridden, unless enforced by an ancestor. -## Compliance frameworks **(PREMIUM)** - -> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/276221) in GitLab 13.9. -> - [Feature flag removed](https://gitlab.com/gitlab-org/gitlab/-/issues/287779) in GitLab 13.12. - -You can create a compliance framework that is a label to identify that your project has certain compliance -requirements or needs additional oversight. The label can optionally enforce -[compliance pipeline configuration](#configure-a-compliance-pipeline) to the projects on which it is -[applied](../project/settings/index.md#add-a-compliance-framework-to-a-project). - -Group owners can create, edit, and delete compliance frameworks: - -1. On the top bar, select **Main menu > Groups > View all groups** and find your group. -1. On the left sidebar, select **Settings** > **General**. -1. Expand the **Compliance frameworks** section. -1. Create, edit, or delete compliance frameworks. - -### Configure a compliance pipeline **(ULTIMATE)** - -> - [Introduced](https://gitlab.com/groups/gitlab-org/-/epics/3156) in GitLab 13.9, disabled behind `ff_evaluate_group_level_compliance_pipeline` [feature flag](../../administration/feature_flags.md). -> - [Enabled by default](https://gitlab.com/gitlab-org/gitlab/-/issues/300324) in GitLab 13.11. -> - [Feature flag removed](https://gitlab.com/gitlab-org/gitlab/-/issues/331231) in GitLab 14.2. - -Group owners can configure a compliance pipeline in a project separate to other projects. By default, the compliance -pipeline configuration (`.gitlab-ci.yml` file) is run instead of the pipeline configuration of labeled projects. - -However, the compliance pipeline configuration can reference the `.gitlab-ci.yml` file of the labeled projects so that: - -- The compliance pipeline can also run jobs of labeled project pipelines. This allows for centralized control of - pipeline configuration. -- Jobs and variables defined in the compliance pipeline can't be changed by variables in the labeled project's - `.gitlab-ci.yml` file. - -See [example configuration](#example-configuration) for help configuring a compliance pipeline that runs jobs from -labeled project pipeline configuration. - -To configure a compliance pipeline: - -1. On the top bar, select **Main menu > Groups > View all groups** and find your group. -1. On the left sidebar, select **Settings** > **General**. -1. Expand the **Compliance frameworks** section. -1. In **Compliance pipeline configuration (optional)**, add the path to the compliance framework configuration. Use the - `path/file.y[a]ml@group-name/project-name` format. For example: - - - `.compliance-ci.yml@gitlab-org/gitlab`. - - `.compliance-ci.yaml@gitlab-org/gitlab`. - -This configuration is inherited by projects where the compliance framework label is -[applied](../project/settings/index.md#add-a-compliance-framework-to-a-project). In projects with the applied compliance -framework label, the compliance pipeline configuration is run instead of the labeled project's own pipeline configuration. - -The user running the pipeline in the labeled project must at least have the Reporter role on the compliance project. - -When used to enforce scan execution, this feature has some overlap with -[scan execution policies](../application_security/policies/scan-execution-policies.md). We have not -[unified the user experience for these two features](https://gitlab.com/groups/gitlab-org/-/epics/7312). For details on -the similarities and differences between these features, see [Enforce scan execution](../application_security/index.md#enforce-scan-execution). - -#### Example configuration - -The following example `.compliance-gitlab-ci.yml` includes the `include` keyword to ensure labeled project pipeline -configuration is also executed. - -```yaml -# Allows compliance team to control the ordering and interweaving of stages/jobs. -# Stages without jobs defined will remain hidden. -stages: - - pre-compliance - - build - - test - - pre-deploy-compliance - - deploy - - post-compliance - -variables: # Can be overridden by setting a job-specific variable in project's local .gitlab-ci.yml - FOO: sast - -sast: # None of these attributes can be overridden by a project's local .gitlab-ci.yml - variables: - FOO: sast - image: ruby:2.6 - stage: pre-compliance - rules: - - if: $CI_COMMIT_BRANCH && $CI_OPEN_MERGE_REQUESTS && $CI_PIPELINE_SOURCE == "push" - when: never - - when: always # or when: on_success - allow_failure: false - before_script: - - "# No before scripts." - script: - - echo "running $FOO" - after_script: - - "# No after scripts." - -sanity check: - image: ruby:2.6 - stage: pre-deploy-compliance - rules: - - if: $CI_COMMIT_BRANCH && $CI_OPEN_MERGE_REQUESTS && $CI_PIPELINE_SOURCE == "push" - when: never - - when: always # or when: on_success - allow_failure: false - before_script: - - "# No before scripts." - script: - - echo "running $FOO" - after_script: - - "# No after scripts." - -audit trail: - image: ruby:2.7 - stage: post-compliance - rules: - - if: $CI_COMMIT_BRANCH && $CI_OPEN_MERGE_REQUESTS && $CI_PIPELINE_SOURCE == "push" - when: never - - when: always # or when: on_success - allow_failure: false - before_script: - - "# No before scripts." - script: - - echo "running $FOO" - after_script: - - "# No after scripts." - -include: # Execute individual project's configuration (if project contains .gitlab-ci.yml) - project: '$CI_PROJECT_PATH' - file: '$CI_CONFIG_PATH' - ref: '$CI_COMMIT_REF_NAME' # Must be defined or MR pipelines always use the use default branch -``` - -##### CF pipelines in Merge Requests originating in project forks - -When an MR originates in a fork, the branch to be merged usually only exists in the fork. -When creating such an MR against a project with CF pipelines, the above snippet will fail with a -`Project <project-name> reference <branch-name> does not exist!` error message. -This is because in the context of the target project, `$CI_COMMIT_REF_NAME` evaluates to a non-existing branch name. - -To get the correct context, use `$CI_MERGE_REQUEST_SOURCE_PROJECT_PATH` instead of `$CI_PROJECT_PATH`. -This variable is only availabe in -[merge request pipelines](../../ci/pipelines/merge_request_pipelines.md). - -For example, for a configuration that supports both branch pipelines, as well as merge request pipelines originating in project forks, -you need to [combine both `include` directives with `rules:if`](../../ci/yaml/includes.md#use-rules-with-include): - -```yaml -include: # Execute individual project's configuration (if project contains .gitlab-ci.yml) - - project: '$CI_MERGE_REQUEST_SOURCE_PROJECT_PATH' - file: '$CI_CONFIG_PATH' - ref: '$CI_COMMIT_REF_NAME' - rules: - - if: $CI_PIPELINE_SOURCE == 'merge_request_event' - - project: '$CI_PROJECT_PATH' - file: '$CI_CONFIG_PATH' - ref: '$CI_COMMIT_REF_NAME' - rules: - - if: $CI_PIPELINE_SOURCE != 'merge_request_event' -``` - -### Ensure compliance jobs are always run - -Compliance pipelines [use GitLab CI/CD](../../ci/index.md) to give you an incredible amount of flexibility -for defining any sort of compliance jobs you like. Depending on your goals, these jobs -can be configured to be: - -- Modified by users. -- Non-modifiable. - -Generally, if a value in a compliance job: - -- Is set, it cannot be changed or overridden by project-level configurations. -- Is not set, a project-level configuration may set. - -Either might be wanted or not depending on your use case. - -There are a few best practices for ensuring that these jobs are always run exactly -as you define them and that downstream, project-level pipeline configurations -cannot change them: - -- Add [a `rules:when:always` block](../../ci/yaml/index.md#when) to each of your compliance jobs. This ensures they are - non-modifiable and are always run. -- Explicitly set any [variables](../../ci/yaml/index.md#variables) the job references. This: - - Ensures that project-level pipeline configurations do not set them and alter their - behavior. - - Includes any jobs that drive the logic of your job. -- Explicitly set the [container image](../../ci/yaml/index.md#image) to run the job in. This ensures that your script - steps execute in the correct environment. -- Explicitly set any relevant GitLab pre-defined [job keywords](../../ci/yaml/index.md#job-keywords). - This ensures that your job uses the settings you intend and that they are not overridden by - project-level pipelines. - -### Avoid parent and child pipelines in GitLab 14.7 and earlier - -NOTE: -This advice does not apply to GitLab 14.8 and later because [a fix](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/78878) added -compatibility for combining compliance pipelines, and parent and child pipelines. - -Compliance pipelines start on the run of _every_ pipeline in a labeled project. This means that if a pipeline in the labeled project -triggers a child pipeline, the compliance pipeline runs first. This can trigger the parent pipeline, instead of the child pipeline. - -Therefore, in projects with compliance frameworks, we recommend replacing -[parent-child pipelines](../../ci/pipelines/downstream_pipelines.md#parent-child-pipelines) with the following: - -- Direct [`include`](../../ci/yaml/index.md#include) statements that provide the parent pipeline with child pipeline configuration. -- Child pipelines placed in another project that are run using the [trigger API](../../ci/triggers/index.md) rather than the parent-child - pipeline feature. - -This alternative ensures the compliance pipeline does not re-start the parent pipeline. - ## Disable email notifications > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/23585) in GitLab 12.2. @@ -722,7 +526,7 @@ This includes projects shared with the group, but it **excludes** projects in subgroups or parent groups of the group being configured. You can configure this feature for both subgroups and immediate parent groups. A project -in a subgroup has access to the templates for that subgroup, as well as +in a subgroup has access to the templates for that subgroup and any immediate parent groups. To learn how to create templates for issues and merge requests, see @@ -811,7 +615,7 @@ Group.find_by_sql("SELECT * FROM namespaces WHERE name LIKE '%oup'") If transferring a group doesn't work through the UI or API, you may want to attempt the transfer in a [Rails console session](../../administration/operations/rails_console.md#starting-a-rails-console-session): WARNING: -Any command that changes data directly could be damaging if not run correctly, or under the right conditions. We highly recommend running them in a test environment with a backup of the instance ready to be restored, just in case. +Commands that change data can cause damage if not run correctly or under the right conditions. Always run commands in a test environment first and have a backup instance ready to restore. ```ruby user = User.find_by_username('<username>') @@ -842,8 +646,27 @@ At times, a group deletion may get stuck. If needed, in a [Rails console session you can attempt to delete a group using the following command: WARNING: -Any command that changes data directly could be damaging if not run correctly, or under the right conditions. We highly recommend running them in a test environment with a backup of the instance ready to be restored, just in case. +Commands that change data can cause damage if not run correctly or under the right conditions. Always run commands in a test environment first and have a backup instance ready to restore. ```ruby GroupDestroyWorker.new.perform(group_id, user_id) ``` + +### Find a user's maximum permissions for a group or project + +Administrators can find a user's maximum permissions for a group or project. + +1. Start a [Rails console session](../../administration/operations/rails_console.md#starting-a-rails-console-session). +1. Run the following commands: + + ```ruby + user = User.find_by_username 'username' + project = Project.find_by_full_path 'group/project' + user.max_member_access_for_project project.id + ``` + + ```ruby + user = User.find_by_username 'username' + group = Group.find_by_full_path 'group' + user.max_member_access_for_group group.id + ``` diff --git a/doc/user/group/planning_hierarchy/index.md b/doc/user/group/planning_hierarchy/index.md index baa2a4d6046..f48a027ab2d 100644 --- a/doc/user/group/planning_hierarchy/index.md +++ b/doc/user/group/planning_hierarchy/index.md @@ -58,10 +58,10 @@ Epic "1"*-- "0..*" Issue In an issue, you can view the parented epic above the issue in the right sidebar under **Epic**. - + ## View ancestry of an epic In an epic, you can view the ancestors as parents in the right sidebar under **Ancestors**. - + diff --git a/doc/user/group/reporting/git_abuse_rate_limit.md b/doc/user/group/reporting/git_abuse_rate_limit.md new file mode 100644 index 00000000000..1cf3a9dbe7d --- /dev/null +++ b/doc/user/group/reporting/git_abuse_rate_limit.md @@ -0,0 +1,36 @@ +--- +stage: Anti-Abuse +group: Anti-Abuse +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments +--- + +# Git abuse rate limit **(ULTIMATE SELF)** + +> [Introduced](https://gitlab.com/groups/gitlab-org/-/epics/8066) in GitLab 15.2 [with a flag](../../../administration/feature_flags.md) named `limit_unique_project_downloads_per_namespace_user`. Disabled by default. + +FLAG: +On self-managed GitLab, by default this feature is not available. To make it available, ask an administrator to [enable the feature flag](../../../administration/feature_flags.md) named `limit_unique_project_downloads_per_namespace_user`. On GitLab.com, this feature is not available. + +Git abuse rate limiting is a feature to automatically ban users who download or clone more than a specified number of repositories in a group or any of its subgroups within a given time frame. Banned users cannot access the main group or any of its non-public subgroups via HTTP or SSH. Access to unrelated groups is unaffected. + +If the `limit_unique_project_downloads_per_namespace_user` feature flag is enabled, all users with the Owner role for the main group receive an email when a user is about to be banned. + +If automatic banning is disabled, a user is not banned automatically when they exceed the limit. However, users with the Owner role for the main group are still notified. You can use this setup to determine the correct values of the rate limit settings before enabling automatic banning. + +If automatic banning is enabled, users with the Owner role for the main group receive an email when a user is about to be banned, and the user is automatically banned from the group and its subgroups. + +## Configure Git abuse rate limiting + +1. On the left sidebar, select **Settings > Reporting**. +1. Update the Git abuse rate limit settings: + 1. Enter a number in the **Number of repositories** field, greater than or equal to `0` and less than or equal to `10,000`. This number specifies the maximum amount of unique repositories a user can download in the specified time period before they're banned. When set to `0`, Git abuse rate limiting is disabled. + 1. Enter a number in the **Reporting time period (seconds)** field, greater than or equal to `0` and less than or equal to `86,400` (10 days). This number specifies the time in seconds a user can download the maximum amount of repositories before they're banned. When set to `0`, Git abuse rate limiting is disabled. + 1. Optional. Exclude up to `100` users by adding them to the **Excluded users** field. Excluded users are not automatically banned. + 1. Optional. Turn on the **Automatically ban users from this namespace when they exceed the specified limits** toggle to enable automatic banning. +1. Select **Save changes**. + +## Unban a user + +1. On the left sidebar, select **Group information > Members**. +1. Select the **Banned** tab. +1. For the account you want to unban, select **Unban**. diff --git a/doc/user/group/repositories_analytics/index.md b/doc/user/group/repositories_analytics/index.md index 0a4d7f5eae4..9971457f2ac 100644 --- a/doc/user/group/repositories_analytics/index.md +++ b/doc/user/group/repositories_analytics/index.md @@ -58,7 +58,7 @@ To get the report: 1. Select the projects and date range you want to include in the report. 1. Select **Download test coverage data (.csv)**. -The projects dropdown shows up to 100 projects from your group. If the project you want to check is not in the dropdown list, you can select **All projects** to download the report for all projects in your group, including any projects that are not listed. There is a plan to improve this behavior in this [related issue](https://gitlab.com/gitlab-org/gitlab/-/issues/250684). +The projects dropdown list shows up to 100 projects from your group. If the project you want to check is not in the dropdown list, you can select **All projects** to download the report for all projects in your group, including any projects that are not listed. There is a plan to improve this behavior in this [related issue](https://gitlab.com/gitlab-org/gitlab/-/issues/250684). For each day that a coverage report was generated by a job in a project's pipeline, a row in the CSV includes: @@ -82,6 +82,6 @@ important to describe those, too. Think of things that may go wrong and include This is important to minimize requests for support, and to avoid doc comments with questions that you know someone might ask. -Each scenario can be a third-level heading, e.g. `### Getting error message X`. +Each scenario can be a third-level heading, for example `### Getting error message X`. If you have none to add when creating a doc, leave this section in place but commented out to help encourage others to add to it in the future. --> diff --git a/doc/user/group/roadmap/img/roadmap_blocked_icon_v15_5.png b/doc/user/group/roadmap/img/roadmap_blocked_icon_v15_5.png Binary files differindex 52130672e5a..625e4afc714 100644 --- a/doc/user/group/roadmap/img/roadmap_blocked_icon_v15_5.png +++ b/doc/user/group/roadmap/img/roadmap_blocked_icon_v15_5.png diff --git a/doc/user/group/roadmap/index.md b/doc/user/group/roadmap/index.md index 28431cb6b9a..3a9d0c833c1 100644 --- a/doc/user/group/roadmap/index.md +++ b/doc/user/group/roadmap/index.md @@ -164,6 +164,6 @@ important to describe those, too. Think of things that may go wrong and include This is important to minimize requests for support, and to avoid doc comments with questions that you know someone might ask. -Each scenario can be a third-level heading, e.g. `### Getting error message X`. +Each scenario can be a third-level heading, for example `### Getting error message X`. If you have none to add when creating a doc, leave this section in place but commented out to help encourage others to add to it in the future. --> diff --git a/doc/user/group/saml_sso/group_sync.md b/doc/user/group/saml_sso/group_sync.md index 3a50470ce50..001c73b6979 100644 --- a/doc/user/group/saml_sso/group_sync.md +++ b/doc/user/group/saml_sso/group_sync.md @@ -77,12 +77,8 @@ Users granted: ### Automatic member removal After a group sync, for GitLab subgroups, users who are not members of a mapped SAML -group are removed from the group. - -FLAG: -In [GitLab 15.1 and later](https://gitlab.com/gitlab-org/gitlab/-/issues/364144), on GitLab.com, users in the top-level -group are assigned the [default membership role](index.md#role) rather than removed. This setting is enabled with the -`saml_group_sync_retain_default_membership` feature flag and can be configured by GitLab.com administrators only. +group are removed from the group. Users in the top-level group are assigned the +[default membership role](index.md#role). For example, in the following diagram: diff --git a/doc/user/group/saml_sso/img/member_enterprise_badge_v14_0.png b/doc/user/group/saml_sso/img/member_enterprise_badge_v14_0.png Binary files differdeleted file mode 100644 index f9534b14a51..00000000000 --- a/doc/user/group/saml_sso/img/member_enterprise_badge_v14_0.png +++ /dev/null diff --git a/doc/user/group/saml_sso/index.md b/doc/user/group/saml_sso/index.md index fa6f378e811..1c5e7ff0115 100644 --- a/doc/user/group/saml_sso/index.md +++ b/doc/user/group/saml_sso/index.md @@ -121,11 +121,18 @@ It can also help to compare the XML response from your provider with our [exampl > - [Improved](https://gitlab.com/gitlab-org/gitlab/-/issues/211962) in GitLab 13.8 with allowing group owners to not go through SSO. > - [Improved](https://gitlab.com/gitlab-org/gitlab/-/issues/9152) in GitLab 13.11 with enforcing open SSO session to use Git if this setting is switched on. > - [Improved](https://gitlab.com/gitlab-org/gitlab/-/issues/339888) in GitLab 14.7 to not enforce SSO checks for Git activity originating from CI/CD jobs. -> - [Improved](https://gitlab.com/gitlab-org/gitlab/-/issues/215155) in GitLab 15.5 [with a flag](../../../administration/feature_flags.md) named `transparent_sso_enforcement` to include transparent enforcement even when SSO enforcement is not enabled. Enabled on GitLab.com. +> - [Improved](https://gitlab.com/gitlab-org/gitlab/-/issues/215155) in GitLab 15.5 [with a flag](../../../administration/feature_flags.md) named `transparent_sso_enforcement` to include transparent enforcement even when SSO enforcement is not enabled. Disabled on GitLab.com. + +FLAG: +On self-managed GitLab, transparent SSO enforcement is unavailable. On GitLab.com, transparent SSO enforcement is unavailable and can be configured by GitLab.com administrators only. SSO is enforced when users access groups and projects in the organization's group hierarchy. Users can view other groups and projects without SSO sign in. -When SAML SSO is enabled, SSO is enforced for each user with an existing SAML identity. +SSO is enforced for each user with an existing SAML identity when the following is enabled: + +- SAML SSO. +- The `:transparent_sso_enforcement` feature flag. + A user has a SAML identity if one or both of the following are true: - They have signed in to GitLab by using their GitLab group's single sign-on URL. @@ -142,6 +149,15 @@ However, users are not prompted to sign in through SSO on each visit. GitLab che has authenticated through SSO. If it's been more than 1 day since the last sign-in, GitLab prompts the user to sign in again through SSO. +When the transparent SSO enforcement feature flag is enabled, SSO is enforced as follows: + +| Project/Group visibility | Enforce SSO setting | Member with identity | Member without identity | Non-member or not signed in | +|--------------------------|---------------------|--------------------| ------ |------------------------------| +| Private | Off | Enforced | Not enforced | No access | +| Private | On | Enforced | Enforced | No access | +| Public | Off | Enforced | Not enforced | Not enforced | +| Public | On | Enforced | Enforced | Not enforced | + An [issue exists](https://gitlab.com/gitlab-org/gitlab/-/issues/297389) to add a similar SSO requirement for API activity. SSO enforcement has the following effects when enabled: @@ -306,9 +322,11 @@ To migrate users to a new email domain, users must: ## User access and management -> SAML user provisioning [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/268142) in GitLab 13.7. +> - SAML user provisioning [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/268142) in GitLab 13.7. +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/325712) in GitLab 14.0, GitLab users created by [SAML SSO](index.md#user-access-and-management) or SCIM provisioning are displayed with an **Enterprise** badge in the **Members** view. -Once Group SSO is configured and enabled, users can access the GitLab.com group through the identity provider's dashboard. If [SCIM](scim_setup.md) is configured, see the [user access and linking setup section on the SCIM page](scim_setup.md#user-access-and-linking-setup). +After group SSO is configured and enabled, users can access the GitLab.com group through the identity provider's dashboard. +If [SCIM](scim_setup.md) is configured, see [user access](scim_setup.md#user-access) on the SCIM page. When a user tries to sign in with Group SSO, GitLab attempts to find or create a user based on the following: @@ -422,7 +440,7 @@ To rescind a user's access to the group when only SAML SSO is configured, either 1. The GitLab.com group. - Use Group Sync at the top-level of your group to [automatically remove the user](group_sync.md#automatic-member-removal). -To rescind a user's access to the group when also using SCIM, refer to [Blocking access](scim_setup.md#blocking-access). +To rescind a user's access to the group when also using SCIM, refer to [Remove access](scim_setup.md#remove-access). ### Unlinking accounts diff --git a/doc/user/group/saml_sso/scim_setup.md b/doc/user/group/saml_sso/scim_setup.md index 55990336a50..18af39f4271 100644 --- a/doc/user/group/saml_sso/scim_setup.md +++ b/doc/user/group/saml_sso/scim_setup.md @@ -168,13 +168,16 @@ Prerequisites: OneLogin provides a **GitLab (SaaS)** app in their catalog, which includes a SCIM integration. Contact OneLogin if you encounter issues. -## User access and linking setup +## User access -During the synchronization process, all of your users get GitLab accounts, welcoming them -to their respective groups, with an invitation email. When implementing SCIM provisioning, -you may want to warn your security-conscious employees about this email. +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/325712) in GitLab 14.0, GitLab users created by [SAML SSO](index.md#user-access-and-management) or SCIM provisioning are displayed with an **Enterprise** badge in the **Members** view. -The following diagram is a general outline on what happens when you add users to your SCIM app: +During the synchronization process, all new users: + +- Receive GitLab accounts. +- Are welcomed to their groups with an invitation email. You may want to warn your employees to expect this email. + +The following diagram describes what happens when you add users to your SCIM app: ```mermaid graph TD @@ -186,29 +189,38 @@ graph TD During provisioning: - Both primary and secondary emails are considered when checking whether a GitLab user account exists. -- Duplicate usernames are also handled, by adding suffix `1` upon user creation. For example, - due to already existing `test_user` username, `test_user1` is used. +- Duplicate usernames are handled by adding suffix `1` when creating the user. For example, if `test_user` already + exists, `test_user1` is used. If `test_user1` already exists, GitLab increments the suffix until an unused username + is found. -If [Group SAML](index.md) has been configured and you have an existing GitLab.com account, you can link your SCIM and SAML identities: +On subsequent visits, new and existing users can access groups either: -1. Update the [primary email](../../profile/index.md#change-your-primary-email) address in your GitLab.com user account to match the - user profile email address in your identity provider. -1. [Link your SAML identity](index.md#linking-saml-to-your-existing-gitlabcom-account). +- Through the identity provider's dashboard. +- By visiting links directly. -We recommend users do this prior to turning on sync, because while synchronization is active, there may be provisioning errors for existing users. +For role information, see the [Group SAML](index.md#user-access-and-management) page. -New users and existing users on subsequent visits can access the group through the identity provider's dashboard or by visiting links directly. +### Link SCIM and SAML identities -[In GitLab 14.0 and later](https://gitlab.com/gitlab-org/gitlab/-/issues/325712), GitLab users created by [SAML SSO](index.md#user-access-and-management) or SCIM provisioning display with an **Enterprise** badge in the **Members** view. +If [group SAML](index.md) is configured and you have an existing GitLab.com account, users can link their SCIM and SAML +identities. Users should do this before synchronization is turned on because there can be provisioning errors for +existing users when synchronization is active. + +To link your SCIM and SAML identities: + +1. Update the [primary email](../../profile/index.md#change-your-primary-email) address in your GitLab.com user account + to match the user profile email address in your identity provider. +1. [Link your SAML identity](index.md#linking-saml-to-your-existing-gitlabcom-account). - +### Remove access -For role information, see the [Group SAML page](index.md#user-access-and-management) +Remove or deactivate a user on the identity provider to remove their access to: -### Blocking access +- The top-level group. +- All subgroups and projects. -To rescind access to the top-level group, all subgroups, and projects, remove or deactivate the user -on the identity provider. After the identity provider performs a sync, based on its configured schedule, the user's membership is revoked and they lose access. +After the identity provider performs a sync based on its configured schedule, the user's membership is revoked and they +lose access. NOTE: Deprovisioning does not delete the GitLab user account. diff --git a/doc/user/group/saml_sso/troubleshooting.md b/doc/user/group/saml_sso/troubleshooting.md index bde5ed1762a..7dafd2c5075 100644 --- a/doc/user/group/saml_sso/troubleshooting.md +++ b/doc/user/group/saml_sso/troubleshooting.md @@ -19,9 +19,10 @@ SAML responses are base64 encoded, so we recommend the following browser plugins - [SAML-tracer](https://addons.mozilla.org/en-US/firefox/addon/saml-tracer/) for Firefox. - [SAML Message Decoder](https://chrome.google.com/webstore/detail/saml-message-decoder/mpabchoaimgbdbbjjieoaeiibojelbhm?hl=en) for Chrome. -Specific attention should be paid to: +Pay specific attention to: -- The NameID, which we use to identify which user is signing in. If the user has previously signed in, this [must match the value we have stored](#verifying-nameid). +- The `NameID`, which we use to identify which user is signing in. If the user has previously signed in, this + [must match the value we have stored](#verify-nameid). - The presence of a `X509Certificate`, which we require to verify the response signature. - The `SubjectConfirmation` and `Conditions`, which can cause errors if misconfigured. @@ -32,7 +33,7 @@ using an identity provider. To generate a SAML Response: -1. Install one of the browser debugging tools previously mentioned. +1. Install one of the [browser debugging tools](#saml-debugging-tools). 1. Open a new browser tab. 1. Open the SAML tracer console: - Chrome: On a context menu on the page, select **Inspect**, then select the **SAML** tab in the opened developer @@ -43,16 +44,17 @@ To generate a SAML Response: [example SAML response](index.md#example-saml-response). 1. Within the SAML tracer, select the **Export** icon to save the response in JSON format. -## GitLab SAML Testing Environments +## Testing GitLab SAML -To troubleshoot, [a complete GitLab with SAML testing environment using Docker compose](https://gitlab.com/gitlab-com/support/toolbox/replication/tree/master/compose_files) -is available. +You can use one of the following to troubleshoot SAML: -If you only require a SAML provider for testing, a [quick start guide to start a Docker container](../../../administration/troubleshooting/test_environments.md#saml) with a plug and play SAML 2.0 Identity Provider (identity provider) is available. +- A [complete GitLab with SAML testing environment using Docker compose](https://gitlab.com/gitlab-com/support/toolbox/replication/tree/master/compose_files). +- A [quick start guide to start a Docker container](../../../administration/troubleshooting/test_environments.md#saml) + with a plug and play SAML 2.0 identity provider if you only require a SAML provider. +- A local environment by + [enabling SAML for groups on a self-managed instance](../../../integration/saml.md#configuring-group-saml-on-a-self-managed-gitlab-instance). -You can test the SaaS feature locally by [enabling SAML for groups on a self-managed instance](../../../integration/saml.md#configuring-group-saml-on-a-self-managed-gitlab-instance). - -## Verifying configuration +## Verify configuration For convenience, we've included some [example resources](../../../user/group/saml_sso/example_saml_config.md) used by our Support Team. While they may help you verify the SAML app configuration, they are not guaranteed to reflect the current state of third-party products. @@ -82,7 +84,7 @@ in case the customer has [configured SAML Group Sync](group_sync.md): - `json.class`: `GroupSamlGroupSyncWorker` - `json.args`: `<user ID> or <group ID>` -In the relevant log entry, the: +In the relevant log entry, the: - `json.args` are in the form `<userID>, <group ID>, [group link ID 1, group link ID 2, ..., group link ID N]`. @@ -148,13 +150,13 @@ If you do not wish to use that GitLab user with the SAML login, you can [unlink ### Message: "SAML authentication failed: User has already been taken" -The user that you're signed in with already has SAML linked to a different identity, or the NameID value has changed. +The user that you're signed in with already has SAML linked to a different identity, or the `NameID` value has changed. Here are possible causes and solutions: | Cause | Solution | | ---------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | You've tried to link multiple SAML identities to the same user, for a given identity provider. | Change the identity that you sign in with. To do so, [unlink the previous SAML identity](index.md#unlinking-accounts) from this GitLab account before attempting to sign in again. | -| The NameID changes every time the user requests SSO identification | [Check the NameID](#verifying-nameid) is not set with `Transient` format, or the NameID is not changing on subsequent requests.| +| The `NameID` changes every time the user requests SSO identification | [Check the `NameID`](#verify-nameid) is not set with `Transient` format, or the `NameID` is not changing on subsequent requests.| ### Message: "SAML authentication failed: Email has already been taken" @@ -171,9 +173,9 @@ User accounts are created in one of the following ways: ### Message: "SAML authentication failed: Extern UID has already been taken, User has already been taken" -Getting both of these errors at the same time suggests the NameID capitalization provided by the identity provider didn't exactly match the previous value for that user. +Getting both of these errors at the same time suggests the `NameID` capitalization provided by the identity provider didn't exactly match the previous value for that user. -This can be prevented by configuring the NameID to return a consistent value. Fixing this for an individual user involves changing the identifier for the user. For GitLab.com, the user needs to [unlink their SAML from the GitLab account](index.md#unlinking-accounts). +This can be prevented by configuring the `NameID` to return a consistent value. Fixing this for an individual user involves changing the identifier for the user. For GitLab.com, the user needs to [unlink their SAML from the GitLab account](index.md#unlinking-accounts). ### Message: "Request to link SAML account must be authorized" @@ -196,15 +198,15 @@ to [reset their password](https://gitlab.com/users/password/new) if both: ## Other user sign in issues -### Verifying NameID +### Verify `NameID` -In troubleshooting, any authenticated user can use the API to verify the NameID GitLab already has linked to the user by visiting [`https://gitlab.com/api/v4/user`](https://gitlab.com/api/v4/user) and checking the `extern_uid` under identities. +In troubleshooting, any authenticated user can use the API to verify the `NameID` GitLab already has linked to their user by visiting [`https://gitlab.com/api/v4/user`](https://gitlab.com/api/v4/user) and checking the `extern_uid` under identities. For self-managed, administrators can use the [users API](../../../api/users.md) to see the same information. When using SAML for groups, group members of a role with the appropriate permissions can make use of the [members API](../../../api/members.md) to view group SAML identity information for members of the group. -This can then be compared to the NameID being sent by the identity provider by decoding the message with a [SAML debugging tool](#saml-debugging-tools). We require that these match to identify users. +This can then be compared to the `NameID` being sent by the identity provider by decoding the message with a [SAML debugging tool](#saml-debugging-tools). We require that these match to identify users. ### Stuck in a login "loop" diff --git a/doc/user/group/saml_sso/troubleshooting_scim.md b/doc/user/group/saml_sso/troubleshooting_scim.md index 6f8aed4b386..22562c51e9e 100644 --- a/doc/user/group/saml_sso/troubleshooting_scim.md +++ b/doc/user/group/saml_sso/troubleshooting_scim.md @@ -8,93 +8,120 @@ info: To determine the technical writer assigned to the Stage/Group associated w This section contains possible solutions for problems you might encounter. -## How come I can't add a user after I removed them? +## User cannot be added after they are removed -As outlined in the [Blocking access section](scim_setup.md#blocking-access), when you remove a user, they are removed from the group. However, their account is not deleted. +When you remove a user, they are removed from the group but their account is not deleted +(see [remove access](scim_setup.md#remove-access)). When the user is added back to the SCIM app, GitLab cannot create a new user because the user already exists. -Solution: Have a user sign in directly to GitLab, then [manually link](scim_setup.md#user-access-and-linking-setup) their account. +To solve this problem: -## How do I diagnose why a user is unable to sign in +1. Have the user sign in directly to GitLab. +1. [Manually link](scim_setup.md#link-scim-and-saml-identities) their account. -Ensure that the user has been added to the SCIM app. +## User cannot sign in -If you receive "User is not linked to a SAML account", then most likely the user already exists in GitLab. Have the user follow the [User access and linking setup](scim_setup.md#user-access-and-linking-setup) instructions. +The following are possible solutions for problems where users cannot sign in: -The **Identity** (`extern_uid`) value stored by GitLab is updated by SCIM whenever `id` or `externalId` changes. Users cannot sign in unless the GitLab Identity (`extern_uid`) value matches the `NameId` sent by SAML. +- Ensure that the user was added to the SCIM app. +- If you receive the `User is not linked to a SAML account` error, the user probably already exists in GitLab. Have the + user follow the [Link SCIM and SAML identities](scim_setup.md#link-scim-and-saml-identities) instructions. +- The **Identity** (`extern_uid`) value stored by GitLab is updated by SCIM whenever `id` or `externalId` changes. Users + cannot sign in unless the GitLab Identity (`extern_uid`) value matches the `NameId` sent by SAML. This value is also + used by SCIM to match users on the `id`, and is updated by SCIM whenever the `id` or `externalId` values change. +- The SCIM `id` and SCIM `externalId` must be configured to the same value as the SAML `NameId`. You can trace SAML responses + using [debugging tools](troubleshooting.md#saml-debugging-tools), and check any errors against the + [SAML troubleshooting](troubleshooting.md) information. -This value is also used by SCIM to match users on the `id`, and is updated by SCIM whenever the `id` or `externalId` values change. +## Unsure if user's SAML `NameId` matches the SCIM `externalId` -It is important that this SCIM `id` and SCIM `externalId` are configured to the same value as the SAML `NameId`. SAML responses can be traced using [debugging tools](troubleshooting.md#saml-debugging-tools), and any errors can be checked against our [SAML troubleshooting docs](troubleshooting.md). +To check if a user's SAML `NameId` matches their SCIM `externalId`: -## How do I verify user's SAML NameId matches the SCIM externalId +- Administrators can use the Admin Area to [list SCIM identities for a user](../../admin_area/index.md#user-identities). +- Group owners can see the list of users and the identifier stored for each user in the group SAML SSO Settings page. +- You can use the [SCIM API](../../../api/scim.md) to manually retrieve the `external_uid` GitLab has stored for users and compare the value for each user from the [SAML API](../../../api/saml.md) . +- Have the user use a [SAML Tracer](troubleshooting.md#saml-debugging-tools) and compare the `external_uid` to + the value returned as the SAML `NameId`. -Administrators can use the Admin Area to [list SCIM identities for a user](../../admin_area/index.md#user-identities). +## Mismatched SCIM `extern_uid` and SAML `NameId` -Group owners can see the list of users and the `externalId` stored for each user in the group SAML SSO Settings page. +Whether the value was changed or you need to map to a different field, the following must map to the same field: -A possible alternative is to use the [SCIM API](../../../api/scim.md) to manually retrieve the `externalId` we have stored for users, also called the `external_uid` or `NameId`. +- `id` +- `externalId` +- `NameId` -To see how the `external_uid` compares to the value returned as the SAML NameId, you can have the user use a [SAML Tracer](troubleshooting.md#saml-debugging-tools). - -## Update or fix mismatched SCIM externalId and SAML NameId - -Whether the value was changed or you need to map to a different field, ensure `id`, `externalId`, and `NameId` all map to the same field. - -If the GitLab `externalId` doesn't match the SAML NameId, it needs to be updated in order for the user to sign in. Ideally your identity provider is configured to do such an update, but in some cases it may be unable to do so, such as when looking up a user fails due to an ID change. +If the GitLab `extern_uid` doesn't match the SAML `NameId`, it must be updated for the user to sign in. Your identity +provider should be configured to do this update. In some cases the identity provider cannot do the update, for example +when a user lookup fails because of an ID change. Be cautious if you revise the fields used by your SCIM identity provider, typically `id` and `externalId`. -We use these IDs to look up users. If the identity provider does not know the current values for these fields, +GitLab uses these IDs to look up users. If the identity provider does not know the current values for these fields, that provider may create duplicate users. -If the `externalId` for a user is not correct, and also doesn't match the SAML NameID, -you can address the problem in the following ways: +If the `extern_uid` for a user is not correct, and also doesn't match the SAML `NameID`, either: + +- Have users unlink and relink themselves, based on the + [SAML authentication failed: User has already been taken](troubleshooting.md#message-saml-authentication-failed-user-has-already-been-taken) + section. +- Unlink all users simultaneously by removing all users from the SCIM app while provisioning is turned on. +- Use the [SCIM API](../../../api/scim.md) to manually correct the `extern_uid` stored for users to match the SAML + `NameId`. To look up a user, you must know the desired value that matches the `NameId` as well as the current + `extern_uid`. -- You can have users unlink and relink themselves, based on the ["SAML authentication failed: User has already been taken"](troubleshooting.md#message-saml-authentication-failed-user-has-already-been-taken) section. -- You can unlink all users simultaneously, by removing all users from the SAML app while provisioning is turned on. -- Use the [SCIM API](../../../api/scim.md) to manually correct the `externalId` stored for users to match the SAML `NameId`. - To look up a user, you need to know the desired value that matches the `NameId` as well as the current `externalId`. +You must not: -It is important not to update these to incorrect values, since this causes users to be unable to sign in. It is also important not to assign a value to the wrong user, as this causes users to get signed into the wrong account. +- Update these to incorrect values because this causes users to be unable to sign in. +- Assign a value to the wrong user because this causes users to be signed in to the wrong account. -## I need to change my SCIM app +## Change SCIM app -Individual users can follow the instructions in the ["SAML authentication failed: User has already been taken"](index.md#change-the-saml-app) section. +When the SCIM app changes: -Alternatively, users can be removed from the SCIM app which de-links all removed users. Sync can then be turned on for the new SCIM app to [link existing users](scim_setup.md#user-access-and-linking-setup). +- Users can follow the instructions in the [Change the SAML app](index.md#change-the-saml-app) section. +- Administrators of the identity provider can: + 1. Remove users from the SCIM app, which unlinks all removed users. + 1. Turn on sync for the new SCIM app to [link existing users](scim_setup.md#link-scim-and-saml-identities). -## The SCIM app is throwing `"User has already been taken","status":409` error message +## SCIM app returns `"User has already been taken","status":409` error Changing the SAML or SCIM configuration or provider can cause the following problems: -| Problem | Solution | -| ------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| SAML and SCIM identity mismatch. | First [verify that the user's SAML NameId matches the SCIM externalId](#how-do-i-verify-users-saml-nameid-matches-the-scim-externalid) and then [update or fix the mismatched SCIM externalId and SAML NameId](#update-or-fix-mismatched-scim-externalid-and-saml-nameid). | -| SCIM identity mismatch between GitLab and the identity provider SCIM app. | You can confirm whether you're hitting the error because of your SCIM identity mismatch between your SCIM app and GitLab.com by using the [SCIM API](../../../api/scim.md) which shows up in the `id` key and compares it with the user `externalId` in the SCIM app. You can use the same [SCIM API](../../../api/scim.md) to update the SCIM `id` for the user on GitLab.com. | +- SAML and SCIM identity mismatch. To solve this problem: + 1. [Verify that the user's SAML `NameId` matches the SCIM `extern_uid`](#unsure-if-users-saml-nameid-matches-the-scim-externalid). + 1. [Update or fix the mismatched SCIM `extern_uid` and SAML `NameId`](#mismatched-scim-extern_uid-and-saml-nameid). +- SCIM identity mismatch between GitLab and the identity provider SCIM app. To solve this problem: + 1. Use the [SCIM API](../../../api/scim.md), which displays the user's `extern_uid` stored in GitLab and compares it with the user `externalId` in + the SCIM app. + 1. Use the same SCIM API to update the SCIM `extern_uid` for the user on GitLab.com. ## Search Rails logs for SCIM requests GitLab.com administrators can search for SCIM requests in the `api_json.log` using the `pubsub-rails-inf-gprd-*` index in -[Kibana](https://about.gitlab.com/handbook/support/workflows/kibana.html#using-kibana). Use the following filters based on the internal -[SCIM API](../../../development/internal_api/index.md#scim-api): +[Kibana](https://about.gitlab.com/handbook/support/workflows/kibana.html#using-kibana). Use the following filters based +on the internal [SCIM API](../../../development/internal_api/index.md#scim-api): - `json.path`: `/scim/v2/groups/<group-path>` - `json.params.value`: `<externalId>` -In a relevant log entry, the `json.params.value` shows the values of SCIM parameters GitLab receives. These values can be used to verify if SCIM parameters configured in an -identity provider's SCIM app are communicated to GitLab as intended. For example, we can use these values as a definitive source on why an account was provisioned with a certain -set of details. This information can help where an account was SCIM provisioned with details that appear to be incongruent with what might have been configured within an identity -provider's SCIM app. +In a relevant log entry, the `json.params.value` shows the values of SCIM parameters GitLab receives. Use these values +to verify if SCIM parameters configured in an identity provider's SCIM app are communicated to GitLab as intended. -## Azure +For example, use these values as a definitive source on why an account was provisioned with a certain set of +details. This information can help where an account was SCIM provisioned with details that do not match +the SCIM app configuration. -### How do I verify my SCIM configuration is correct? +## Azure Active Directory -Review the following: +The following troubleshooting information is specifically for SCIM provisioned through Azure Active Directory. -- Ensure that the SCIM value for `id` matches the SAML value for `NameId`. -- Ensure that the SCIM value for `externalId` matches the SAML value for `NameId`. +### Verify my SCIM configuration is correct + +Ensure that: + +- The matching precedence for `externalId` is 1. +- The SCIM value for `externalId` matches the SAML value for `NameId`. Review the following SCIM parameters for sensible values: @@ -102,28 +129,37 @@ Review the following SCIM parameters for sensible values: - `displayName` - `emails[type eq "work"].value` -### Testing Azure connection: invalid credentials +### `invalid credentials` error when testing connection + +When testing the connection, you may encounter an error: + +```plaintext +You appear to have entered invalid credentials. Please confirm +you are using the correct information for an administrative account +``` -When testing the connection, you may encounter an error: **You appear to have entered invalid credentials. Please confirm you are using the correct information for an administrative account**. If `Tenant URL` and `secret token` are correct, check whether your group path contains characters that may be considered invalid JSON primitives (such as `.`). Removing such characters from the group path typically resolves the error. +If `Tenant URL` and `secret token` are correct, check whether your group path contains characters that may be considered +invalid JSON primitives (such as `.`). Removing or URL encoding these characters in the group path typically resolves the error. -### (Field) can't be blank sync error +### `(Field) can't be blank` sync error -When checking the Audit Events for the Provisioning, you can sometimes see the -error `Namespace can't be blank, Name can't be blank, and User can't be blank.` +When checking the Audit Events for the provisioning, you sometimes see a `Namespace can't be blank, Name can't be blank, +and User can't be blank.` error. -This is likely caused because not all required fields (such as first name and last name) are present for all users being mapped. +This error can occur because not all required fields (such as first name and last name) are present for all users +being mapped. As a workaround, try an alternate mapping: -1. Follow the Azure mapping instructions from above. +1. Follow the [Azure mapping instructions](scim_setup.md#configure-attribute-mappings). 1. Delete the `name.formatted` target attribute entry. 1. Change the `displayName` source attribute to have `name.formatted` target attribute. -### Failed to match an entry in the source and target systems Group 'Group-Name' +### `Failed to match an entry in the source and target systems Group 'Group-Name'` error -Group provisioning in Azure can fail with the `Failed to match an entry in the source and target systems Group 'Group-Name'` error message, -and the error response can include a HTML result of the GitLab URL `https://gitlab.com/users/sign_in`. +Group provisioning in Azure can fail with the `Failed to match an entry in the source and target systems Group 'Group-Name'` +error. The error response can include a HTML result of the GitLab URL `https://gitlab.com/users/sign_in`. -This error is harmless and occurs because Group provisioning was turned on but GitLab SCIM integration does not support it nor require it. To -remove the error, follow the instructions in the Azure configuration guide to disable the option -[`Synchronize Azure Active Directory Groups to AppName`](scim_setup.md#configure-azure-active-directory). +This error is harmless and occurs because group provisioning was turned on but GitLab SCIM integration does not support +it nor require it. To remove the error, follow the instructions in the Azure configuration guide to disable the option +to [synchronize Azure Active Directory groups to AppName](scim_setup.md#configure-azure-active-directory). diff --git a/doc/user/group/subgroups/index.md b/doc/user/group/subgroups/index.md index 58f5e476f26..95c8e60af5d 100644 --- a/doc/user/group/subgroups/index.md +++ b/doc/user/group/subgroups/index.md @@ -212,6 +212,6 @@ important to describe those, too. Think of things that may go wrong and include This is important to minimize requests for support, and to avoid doc comments with questions that you know someone might ask. -Each scenario can be a third-level heading, e.g. `### Getting error message X`. +Each scenario can be a third-level heading, for example `### Getting error message X`. If you have none to add when creating a doc, leave this section in place but commented out to help encourage others to add to it in the future. --> diff --git a/doc/user/group/value_stream_analytics/index.md b/doc/user/group/value_stream_analytics/index.md index fcf4189aa32..980618ac3ae 100644 --- a/doc/user/group/value_stream_analytics/index.md +++ b/doc/user/group/value_stream_analytics/index.md @@ -104,7 +104,7 @@ The **Overview** dashboard shows the following key metrics that measure team per > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/355304) time to restore service tile in GitLab 15.0. > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/357071) change failure rate tile in GitLab 15.0. -The value stream analytics **Overview** dashboard displays the following [DORA](../../../user/analytics/index.md) metrics: +The value stream analytics **Overview** dashboard displays the following [DORA](../../../user/analytics/dora_metrics.md) metrics: - Deployment Frequency. - Lead time for changes. diff --git a/doc/user/infrastructure/clusters/connect/new_civo_cluster.md b/doc/user/infrastructure/clusters/connect/new_civo_cluster.md index 998548e3a5e..df785cce406 100644 --- a/doc/user/infrastructure/clusters/connect/new_civo_cluster.md +++ b/doc/user/infrastructure/clusters/connect/new_civo_cluster.md @@ -91,7 +91,7 @@ Refer to the [Civo Terraform provider](https://registry.terraform.io/providers/c After configuring your project, manually trigger the provisioning of your cluster. In GitLab: 1. On the left sidebar, go to **CI/CD > Pipelines**. -1. Next to **Play** (**{play}**), select the dropdown icon (**{chevron-lg-down}**). +1. Next to **Play** (**{play}**), select the dropdown list icon (**{chevron-lg-down}**). 1. Select **Deploy** to manually trigger the deployment job. When the pipeline finishes successfully, you can see your new cluster: diff --git a/doc/user/infrastructure/clusters/connect/new_eks_cluster.md b/doc/user/infrastructure/clusters/connect/new_eks_cluster.md index 164b426fd24..8a5c32150c9 100644 --- a/doc/user/infrastructure/clusters/connect/new_eks_cluster.md +++ b/doc/user/infrastructure/clusters/connect/new_eks_cluster.md @@ -92,7 +92,7 @@ View the [AWS Terraform provider](https://registry.terraform.io/providers/hashic After configuring your project, manually trigger the provisioning of your cluster. In GitLab: 1. On the left sidebar, go to **CI/CD > Pipelines**. -1. Next to **Play** (**{play}**), select the dropdown icon (**{chevron-lg-down}**). +1. Next to **Play** (**{play}**), select the dropdown list icon (**{chevron-lg-down}**). 1. Select **Deploy** to manually trigger the deployment job. When the pipeline finishes successfully, you can view the new cluster: diff --git a/doc/user/infrastructure/clusters/connect/new_gke_cluster.md b/doc/user/infrastructure/clusters/connect/new_gke_cluster.md index d04b14bf80f..6b8e2003b8d 100644 --- a/doc/user/infrastructure/clusters/connect/new_gke_cluster.md +++ b/doc/user/infrastructure/clusters/connect/new_gke_cluster.md @@ -20,7 +20,7 @@ by using the GitLab agent for Kubernetes. **Prerequisites:** -- A [Google Cloud Platform (GCP) service account](https://cloud.google.com/docs/authentication/getting-started). +- A [Google Cloud Platform (GCP) service account](https://cloud.google.com/docs/authentication#service-accounts). - [A runner](https://docs.gitlab.com/runner/install/) you can use to run the GitLab CI/CD pipeline. **Steps:** @@ -71,7 +71,7 @@ To create a GitLab agent for Kubernetes: To set up your project to communicate to GCP and the GitLab API: -1. To authenticate GCP with GitLab, create a [GCP service account](https://cloud.google.com/docs/authentication/getting-started) +1. To authenticate GCP with GitLab, create a [GCP service account](https://cloud.google.com/docs/authentication#service-accounts) with following roles: `Compute Network Viewer`, `Kubernetes Engine Admin`, `Service Account User`, and `Service Account Admin`. Both User and Admin service accounts are necessary. The User role impersonates the [default service account](https://cloud.google.com/compute/docs/access/service-accounts#default_service_account) when [creating the node pool](https://registry.terraform.io/providers/hashicorp/google/latest/docs/guides/using_gke_with_terraform#node-pool-management). @@ -117,7 +117,7 @@ Refer to the [Google Terraform provider](https://registry.terraform.io/providers After configuring your project, manually trigger the provisioning of your cluster. In GitLab: 1. On the left sidebar, go to **CI/CD > Pipelines**. -1. Next to **Play** (**{play}**), select the dropdown icon (**{chevron-lg-down}**). +1. Next to **Play** (**{play}**), select the dropdown list icon (**{chevron-lg-down}**). 1. Select **Deploy** to manually trigger the deployment job. When the pipeline finishes successfully, you can see your new cluster: diff --git a/doc/user/infrastructure/clusters/manage/management_project_applications/ingress.md b/doc/user/infrastructure/clusters/manage/management_project_applications/ingress.md index bf86fdb5a8f..14d3a7996e0 100644 --- a/doc/user/infrastructure/clusters/manage/management_project_applications/ingress.md +++ b/doc/user/infrastructure/clusters/manage/management_project_applications/ingress.md @@ -22,5 +22,5 @@ of your cluster. You can customize the installation of Ingress by updating the `applications/ingress/values.yaml` file in your cluster management project. Refer to the -[chart](https://github.com/helm/charts/tree/master/stable/nginx-ingress) +[chart](https://github.com/kubernetes/ingress-nginx/tree/master/charts/ingress-nginx) for the available configuration options. diff --git a/doc/user/infrastructure/clusters/manage/management_project_applications/vault.md b/doc/user/infrastructure/clusters/manage/management_project_applications/vault.md index 72a44ef2a21..c2190ad7cfa 100644 --- a/doc/user/infrastructure/clusters/manage/management_project_applications/vault.md +++ b/doc/user/infrastructure/clusters/manage/management_project_applications/vault.md @@ -36,17 +36,17 @@ Vault application causes downtime. To optimally use Vault in a production environment, it's ideal to have a good understanding of the internals of Vault and how to configure it. This can be done by reading the [Vault Configuration guide](../../../../../ci/secrets/index.md#configure-your-vault-server), -the [Vault documentation](https://www.vaultproject.io/docs/internals) and +the [Vault documentation](https://developer.hashicorp.com/vault/docs/internals) and the Vault Helm chart [`values.yaml` file](https://github.com/hashicorp/vault-helm/blob/v0.3.3/values.yaml). At a minimum, most users set up: -- A [seal](https://www.vaultproject.io/docs/configuration/seal) for extra encryption +- A [seal](https://developer.hashicorp.com/vault/docs/configuration/seal) for extra encryption of the main key. -- A [storage backend](https://www.vaultproject.io/docs/configuration/storage) that's +- A [storage backend](https://developer.hashicorp.com/vault/docs/configuration/storage) that's suitable for environment and storage security requirements. -- [HA Mode](https://www.vaultproject.io/docs/concepts/ha). -- The [Vault UI](https://www.vaultproject.io/docs/configuration/ui). +- [HA Mode](https://developer.hashicorp.com/vault/docs/concepts/ha). +- The [Vault UI](https://developer.hashicorp.com/vault/docs/configuration/ui). The following is an example values file (`applications/vault/values.yaml`) that configures Google Key Management Service for auto-unseal, using a Google Cloud Storage backend, enabling @@ -86,7 +86,7 @@ server: ``` After you have successfully installed Vault, you must -[initialize the Vault](https://learn.hashicorp.com/tutorials/vault/getting-started-deploy#initializing-the-vault) +[initialize the Vault](https://developer.hashicorp.com/vault/tutorials/getting-started/getting-started-deploy#initializing-the-vault) and obtain the initial root token. You need access to your Kubernetes cluster that Vault has been deployed into to do this. To initialize the Vault, get a shell to one of the Vault pods running inside Kubernetes (typically this is done diff --git a/doc/user/infrastructure/iac/index.md b/doc/user/infrastructure/iac/index.md index c2285ecc773..866b16652fa 100644 --- a/doc/user/infrastructure/iac/index.md +++ b/doc/user/infrastructure/iac/index.md @@ -99,8 +99,8 @@ in the template you fetched to customize your configuration. - To collaborate on Terraform code changes and Infrastructure-as-Code workflows, use the [Terraform integration in merge requests](mr_integration.md). - To manage GitLab resources like users, groups, and projects, use the - [GitLab Terraform provider](https://github.com/gitlabhq/terraform-provider-gitlab). It is released separately from GitLab - and its documentation is available on [the Terraform docs site](https://registry.terraform.io/providers/gitlabhq/gitlab/latest/docs). + [GitLab Terraform provider](https://gitlab.com/gitlab-org/terraform-provider-gitlab). + The GitLab Terraform provider documentation is available on [the Terraform docs site](https://registry.terraform.io/providers/gitlabhq/gitlab/latest/docs). - [Create a new cluster on Amazon Elastic Kubernetes Service (EKS)](../clusters/connect/new_eks_cluster.md). - [Create a new cluster on Google Kubernetes Engine (GKE)](../clusters/connect/new_gke_cluster.md). - [Troubleshoot](troubleshooting.md) issues with GitLab and Terraform. diff --git a/doc/user/infrastructure/iac/terraform_state.md b/doc/user/infrastructure/iac/terraform_state.md index 9cb3fdd7aaf..a690cc78121 100644 --- a/doc/user/infrastructure/iac/terraform_state.md +++ b/doc/user/infrastructure/iac/terraform_state.md @@ -59,7 +59,7 @@ encrypt plan output or modify the project visibility settings. To configure GitLab CI/CD as a backend: 1. In your Terraform project, in a `.tf` file like `backend.tf`, - define the [HTTP backend](https://www.terraform.io/docs/language/settings/backends/http.html): + define the [HTTP backend](https://developer.hashicorp.com/terraform/language/settings/backends/http): ```hcl terraform { diff --git a/doc/user/infrastructure/iac/troubleshooting.md b/doc/user/infrastructure/iac/troubleshooting.md index a1eecb909fb..3921c6a7dc8 100644 --- a/doc/user/infrastructure/iac/troubleshooting.md +++ b/doc/user/infrastructure/iac/troubleshooting.md @@ -121,3 +121,13 @@ To resolve this, ensure that: - If you have set the `TF_HTTP_PASSWORD` CI/CD variable, make sure that you either: - Set the same value as `TF_PASSWORD` - Remove `TF_HTTP_PASSWORD` variable if your CI/CD job does not explicitly use it. + +### Enable Developer role access to destructive commands + +To permit a user with the Developer role to run destructive commands, you need a workaround: + +1. [Create a project access token](../../project/settings/project_access_tokens.md#create-a-project-access-token) with `api` scope. +1. Add `TF_USERNAME` and `TF_PASSWORD` to your CI/CD variables: + 1. Set the value of `TF_USERNAME` to the username of your project access token. + 1. Set the value of `TF_PASSWORD` to the password of your project access token. + 1. Optional. Protect the variables to make them only available in pipelines that run on protected branches or protected tags. diff --git a/doc/user/markdown.md b/doc/user/markdown.md index 1a743ae99bb..b6f3ba1cfdd 100644 --- a/doc/user/markdown.md +++ b/doc/user/markdown.md @@ -331,7 +331,7 @@ However, you cannot mix the wrapping tags: ``` If your diff includes words in `` `code` `` font, make sure to escape each backtick `` ` `` with a -backslash `\`. Otherwise the diff highlight does not render correctly: +backslash <code>\</code>. Otherwise the diff highlight does not render correctly: ```markdown - {+ Just regular text +} @@ -1160,17 +1160,15 @@ These details <em>remain</em> <strong>hidden</strong> until expanded. Markdown inside these tags is also supported. -NOTE: -If your Markdown isn't rendering correctly, try adding -`{::options parse_block_html="true" /}` to the top of the page, and add -`markdown="span"` to the opening summary tag like this: `<summary markdown="span">`. - -Remember to leave a blank line after the `</summary>` tag and before the `</details>` tag, -as shown in the example: +Remember to leave a blank line before and after any Markdown sections, as shown in the example: ````html <details> -<summary>Click this to collapse/fold.</summary> +<summary> + +Click this to _collapse/fold._ + +</summary> These details _remain_ **hidden** until expanded. @@ -1187,7 +1185,7 @@ works correctly in GitLab. --> <details> -<summary>Click this to collapse/fold.</summary> +<summary>Click this to <em>collapse/fold.</em></summary> These details <em>remain</em> <b>hidden</b> until expanded. diff --git a/doc/user/packages/composer_repository/index.md b/doc/user/packages/composer_repository/index.md index 5eafb74a144..49c54ec191e 100644 --- a/doc/user/packages/composer_repository/index.md +++ b/doc/user/packages/composer_repository/index.md @@ -1,6 +1,6 @@ --- stage: Package -group: Package +group: Package Registry info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- @@ -25,58 +25,7 @@ client uses, see the [Composer API documentation](../../../api/packages/composer Composer v2.0 is recommended. Composer v1.0 is supported, but it has lower performance when working in groups with very large numbers of packages. -## Create a Composer package - -If you do not have a Composer package, create one and check it in to -a repository. This example shows a GitLab repository, but the repository -can be any public or private repository. - -WARNING: -If you are using a GitLab repository, the project must have been created from -a group's namespace, rather than a user's namespace. Composer packages -[can't be published to projects created from a user's namespace](https://gitlab.com/gitlab-org/gitlab/-/issues/235467). - -1. Create a directory called `my-composer-package` and change to that directory: - - ```shell - mkdir my-composer-package && cd my-composer-package - ``` - -1. Run [`composer init`](https://getcomposer.org/doc/03-cli.md#init) and answer the prompts. - - For namespace, enter your unique [namespace](../../../user/namespace/index.md), like your GitLab username or group name. - - A file called `composer.json` is created: - - ```json - { - "name": "<namespace>/composer-test", - "description": "Library XY", - "type": "library", - "license": "GPL-3.0-only", - "authors": [ - { - "name": "John Doe", - "email": "john@example.com" - } - ], - "require": {} - } - ``` - -1. Run Git commands to tag the changes and push them to your repository: - - ```shell - git init - git add composer.json - git commit -m 'Composer package test' - git tag v1.0.0 - git remote add origin git@gitlab.example.com:<namespace>/<project-name>.git - git push --set-upstream origin main - git push origin v1.0.0 - ``` - -The package is now in your GitLab Package Registry. +Learn how to [build a Composer package](../workflows/build_packages.md#composer). ## Publish a Composer package by using the API @@ -110,7 +59,7 @@ To publish the package with a personal access token: - `<personal-access-token>` is your personal access token. - `<project_id>` is your project ID. - `<tag>` is the Git tag name of the version you want to publish. - To publish a branch, use `branch=<branch>` instead of `tag=<tag>`. + To publish a branch, use `branch=<branch>` instead of `tag=<tag>`. To publish the package with a deploy token: @@ -125,7 +74,7 @@ To publish the package with a deploy token: - `<deploy-token>` is your deploy token - `<project_id>` is your project ID. - `<tag>` is the Git tag name of the version you want to publish. - To publish a branch, use `branch=<branch>` instead of `tag=<tag>`. + To publish a branch, use `branch=<branch>` instead of `tag=<tag>`. You can view the published package by going to **Packages and registries > Package Registry** and selecting the **Composer** tab. diff --git a/doc/user/packages/conan_repository/index.md b/doc/user/packages/conan_repository/index.md index cd5ce9a1135..3d3fe35fd65 100644 --- a/doc/user/packages/conan_repository/index.md +++ b/doc/user/packages/conan_repository/index.md @@ -1,6 +1,6 @@ --- stage: Package -group: Package +group: Package Registry info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- @@ -29,105 +29,7 @@ Package Registry. For documentation of the specific API endpoints that the Conan package manager client uses, see the [Conan API documentation](../../../api/packages/conan.md). -## Build a Conan package - -This section explains how to install Conan and build a package for your C/C++ -project. - -If you already use Conan and know how to build your own packages, go to the -[next section](#add-the-package-registry-as-a-conan-remote). - -### Install Conan - -Download the Conan package manager to your local development environment by -following the instructions at [conan.io](https://conan.io/downloads.html). - -When installation is complete, verify you can use Conan in your terminal by -running: - -```shell -conan --version -``` - -The Conan version is printed in the output: - -```plaintext -Conan version 1.20.5 -``` - -### Install CMake - -When you develop with C++ and Conan, you can select from many available -compilers. This example uses the CMake build system generator. - -To install CMake: - -- For Mac, use [Homebrew](https://brew.sh/) and run `brew install cmake`. -- For other operating systems, follow the instructions at [cmake.org](https://cmake.org/install/). - -When installation is complete, verify you can use CMake in your terminal by -running: - -```shell -cmake --version -``` - -The CMake version is printed in the output. - -### Create a project - -To test the Package Registry, you need a C++ project. If you don't already have -one, you can clone the Conan [hello world starter project](https://github.com/conan-io/hello). - -### Build a package - -To build a package: - -1. Open a terminal and navigate to your project's root folder. -1. Generate a new recipe by running `conan new` with a package name and version: - - ```shell - conan new Hello/0.1 -t - ``` - -1. Create a package for the recipe by running `conan create` with the Conan user - and channel: - - ```shell - conan create . mycompany/beta - ``` - - NOTE: - If you use an [instance remote](#add-a-remote-for-your-instance), you must - follow a specific [naming convention](#package-recipe-naming-convention-for-instance-remotes). - -A package with the recipe `Hello/0.1@mycompany/beta` is created. - -For more details about creating and managing Conan packages, see the -[Conan documentation](https://docs.conan.io/en/latest/creating_packages.html). - -#### Package without a username and a channel - -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/345055) in GitLab 14.6. - -Even though they are [recommended](https://docs.conan.io/en/latest/reference/conanfile/attributes.html#user-channel) -to distinguish your package from a similarly named existing package, -the username and channel are not mandatory fields for a Conan package. - -You can create a package without a username and channel by removing them from -the `create` command: - -```shell -conan create . -``` - -The username _and_ the channel must be blank. If only one of these fields is -blank, the request is rejected. - -NOTE: -Empty usernames and channels can only be used if you use a [project remote](#add-a-remote-for-your-project). -If you use an [instance remote](#add-a-remote-for-your-instance), the username -and the channel must be set. +Learn how to [build a Conan package](../workflows/build_packages.md#conan). ## Add the Package Registry as a Conan remote @@ -193,12 +95,12 @@ recipe `user` must be the plus sign (`+`) separated project path. Example recipe names: -| Project | Package | Supported | -| ---------------------------------- | ----------------------------------------------- | --------- | -| `foo/bar` | `my-package/1.0.0@foo+bar/stable` | Yes | -| `foo/bar-baz/buz` | `my-package/1.0.0@foo+bar-baz+buz/stable` | Yes | -| `gitlab-org/gitlab-ce` | `my-package/1.0.0@gitlab-org+gitlab-ce/stable` | Yes | -| `gitlab-org/gitlab-ce` | `my-package/1.0.0@foo/stable` | No | +| Project | Package | Supported | +| ---------------------- | ---------------------------------------------- | --------- | +| `foo/bar` | `my-package/1.0.0@foo+bar/stable` | Yes | +| `foo/bar-baz/buz` | `my-package/1.0.0@foo+bar-baz+buz/stable` | Yes | +| `gitlab-org/gitlab-ce` | `my-package/1.0.0@gitlab-org+gitlab-ce/stable` | Yes | +| `gitlab-org/gitlab-ce` | `my-package/1.0.0@foo/stable` | No | [Project remotes](#add-a-remote-for-your-project) have a more flexible naming convention. diff --git a/doc/user/packages/container_registry/index.md b/doc/user/packages/container_registry/index.md index 1ac38979950..65e7918d827 100644 --- a/doc/user/packages/container_registry/index.md +++ b/doc/user/packages/container_registry/index.md @@ -1,6 +1,6 @@ --- stage: Package -group: Package +group: Container Registry info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- @@ -70,7 +70,7 @@ To download and run a container image hosted in the GitLab Container Registry: [Authentication](#authenticate-with-the-container-registry) is needed to download images from private repository. For more information on running Docker containers, visit the -[Docker documentation](https://docs.docker.com/engine/userguide/intro/). +[Docker documentation](https://docs.docker.com/get-started/). ## Image naming convention @@ -81,9 +81,9 @@ Images follow this naming convention: ``` If your project is `gitlab.example.com/mynamespace/myproject`, for example, -then your image must be named `gitlab.example.com/mynamespace/myproject/my-app` at a minimum. +then your image must be named `gitlab.example.com/mynamespace/myproject` at a minimum. -You can append additional names to the end of an image name, up to three levels deep. +You can append additional names to the end of an image name, up to two levels deep. For example, these are all valid image names for images within the project named `myproject`: @@ -523,7 +523,7 @@ for more details about the permissions that this setting grants to users. 1. Go to your project's **Settings > General** page. 1. Expand the section **Visibility, project features, permissions**. -1. Under **Container Registry**, select an option from the dropdown: +1. Under **Container Registry**, select an option from the dropdown list: - **Everyone With Access** (Default): The Container Registry is visible to everyone with access to the project. If the project is public, the Container Registry is also public. If the project @@ -556,7 +556,7 @@ this setting. However, disabling the Container Registry disables all Container R ## Troubleshooting the GitLab Container Registry -## Migrating OCI container images to GitLab Container Registry +### Migrating OCI container images to GitLab Container Registry Migrating built container images to the GitLab registry is not a current feature. However, an [epic](https://gitlab.com/groups/gitlab-org/-/epics/5210) is open to track the work on this feature. diff --git a/doc/user/packages/container_registry/reduce_container_registry_data_transfer.md b/doc/user/packages/container_registry/reduce_container_registry_data_transfer.md index ff89e5f361f..74cbcba2ffc 100644 --- a/doc/user/packages/container_registry/reduce_container_registry_data_transfer.md +++ b/doc/user/packages/container_registry/reduce_container_registry_data_transfer.md @@ -1,6 +1,6 @@ --- stage: Package -group: Package +group: Container Registry info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- @@ -90,7 +90,7 @@ build process instead of trying to minify images afterward. ### Use multi-stage builds -With [multi-stage builds](https://docs.docker.com/develop/develop-images/multistage-build/), +With [multi-stage builds](https://docs.docker.com/build/building/multi-stage/), you use multiple `FROM` statements in your Dockerfile. Each `FROM` instruction can use a different base, and each begins a new build stage. You can selectively copy artifacts from one stage to another, leaving behind everything you don't want in the final image. This is especially useful when diff --git a/doc/user/packages/container_registry/reduce_container_registry_storage.md b/doc/user/packages/container_registry/reduce_container_registry_storage.md index 9c981aeac53..23d835ddf5f 100644 --- a/doc/user/packages/container_registry/reduce_container_registry_storage.md +++ b/doc/user/packages/container_registry/reduce_container_registry_storage.md @@ -1,6 +1,6 @@ --- stage: Package -group: Package +group: Container Registry info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- diff --git a/doc/user/packages/debian_repository/index.md b/doc/user/packages/debian_repository/index.md index 90fc0bc3bb1..7db2a341743 100644 --- a/doc/user/packages/debian_repository/index.md +++ b/doc/user/packages/debian_repository/index.md @@ -1,6 +1,6 @@ --- stage: Package -group: Package +group: Package Registry info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- diff --git a/doc/user/packages/dependency_proxy/index.md b/doc/user/packages/dependency_proxy/index.md index 3fb22437eb0..b7a9729c8ba 100644 --- a/doc/user/packages/dependency_proxy/index.md +++ b/doc/user/packages/dependency_proxy/index.md @@ -1,6 +1,6 @@ --- stage: Package -group: Package +group: Container Registry info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- @@ -299,7 +299,7 @@ hub_docker_quota_check: ## Troubleshooting -## Authentication error: "HTTP Basic: Access Denied" +### Authentication error: "HTTP Basic: Access Denied" If you receive an `HTTP Basic: Access denied` error when authenticating against the Dependency Proxy, refer to the [two-factor authentication troubleshooting guide](../../profile/account/two_factor_authentication.md#troubleshooting). diff --git a/doc/user/packages/dependency_proxy/reduce_dependency_proxy_storage.md b/doc/user/packages/dependency_proxy/reduce_dependency_proxy_storage.md index 1239d1a97ae..3aaaf0c0186 100644 --- a/doc/user/packages/dependency_proxy/reduce_dependency_proxy_storage.md +++ b/doc/user/packages/dependency_proxy/reduce_dependency_proxy_storage.md @@ -1,6 +1,6 @@ --- stage: Package -group: Package +group: Container Registry info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- diff --git a/doc/user/packages/generic_packages/index.md b/doc/user/packages/generic_packages/index.md index 930e0760eb3..563f35f2f4f 100644 --- a/doc/user/packages/generic_packages/index.md +++ b/doc/user/packages/generic_packages/index.md @@ -1,6 +1,6 @@ --- stage: Package -group: Package +group: Package Registry info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- diff --git a/doc/user/packages/go_proxy/index.md b/doc/user/packages/go_proxy/index.md index 6f6dc084720..a147c3656b7 100644 --- a/doc/user/packages/go_proxy/index.md +++ b/doc/user/packages/go_proxy/index.md @@ -1,6 +1,6 @@ --- stage: Package -group: Package +group: Package Registry info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- diff --git a/doc/user/packages/harbor_container_registry/index.md b/doc/user/packages/harbor_container_registry/index.md index 217d3d57416..f159522d77d 100644 --- a/doc/user/packages/harbor_container_registry/index.md +++ b/doc/user/packages/harbor_container_registry/index.md @@ -1,6 +1,6 @@ --- stage: Package -group: Package +group: Container Registry info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- diff --git a/doc/user/packages/helm_repository/index.md b/doc/user/packages/helm_repository/index.md index 521f04226df..bba68494c2d 100644 --- a/doc/user/packages/helm_repository/index.md +++ b/doc/user/packages/helm_repository/index.md @@ -1,6 +1,6 @@ --- stage: Package -group: Package +group: Package Registry info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- @@ -134,6 +134,8 @@ for any related errors. If you see `Validation failed: Version is invalid`, it m version in your `Chart.yaml` file does not follow [Helm Chart versioning specifications](https://helm.sh/docs/topics/charts/#charts-and-versioning). To fix the error, use the correct version syntax and upload the chart again. +Support for providing better error messages for package processing errors in the UI is proposed in issue [330515](https://gitlab.com/gitlab-org/gitlab/-/issues/330515). + ### `helm push` results in an error Helm 3.7 introduced a breaking change for the `helm-push` plugin. You can update the diff --git a/doc/user/packages/index.md b/doc/user/packages/index.md index 84a10943879..7418977e0d1 100644 --- a/doc/user/packages/index.md +++ b/doc/user/packages/index.md @@ -1,6 +1,6 @@ --- stage: Package -group: Package +group: Package Registry info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- diff --git a/doc/user/packages/infrastructure_registry/index.md b/doc/user/packages/infrastructure_registry/index.md index 8129b42905a..ac31b491a0e 100644 --- a/doc/user/packages/infrastructure_registry/index.md +++ b/doc/user/packages/infrastructure_registry/index.md @@ -1,6 +1,6 @@ --- -stage: Configure -group: Configure +stage: Package +group: Package Registry info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- diff --git a/doc/user/packages/maven_repository/index.md b/doc/user/packages/maven_repository/index.md index ec56255999a..2aa71e111fb 100644 --- a/doc/user/packages/maven_repository/index.md +++ b/doc/user/packages/maven_repository/index.md @@ -1,6 +1,6 @@ --- stage: Package -group: Package +group: Package Registry info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- @@ -14,177 +14,7 @@ Then, install the packages whenever you need to use them as a dependency. For documentation of the specific API endpoints that the Maven package manager client uses, see the [Maven API documentation](../../../api/packages/maven.md). -## Build a Maven package - -This section explains how to install Maven and build a package. - -If you already use Maven and know how to build your own packages, go to the -[next section](#authenticate-to-the-package-registry-with-maven). - -Maven repositories work well with Gradle, too. To set up a Gradle project, see [get started with Gradle](#build-a-java-project-with-gradle). - -### Install Maven - -The required minimum versions are: - -- Java 11.0.5+ -- Maven 3.6+ - -Follow the instructions at [maven.apache.org](https://maven.apache.org/install.html) -to download and install Maven for your local development environment. After -installation is complete, verify you can use Maven in your terminal by running: - -```shell -mvn --version -``` - -The output should be similar to: - -```shell -Apache Maven 3.6.1 (d66c9c0b3152b2e69ee9bac180bb8fcc8e6af555; 2019-04-04T20:00:29+01:00) -Maven home: /Users/<your_user>/apache-maven-3.6.1 -Java version: 12.0.2, vendor: Oracle Corporation, runtime: /Library/Java/JavaVirtualMachines/jdk-12.0.2.jdk/Contents/Home -Default locale: en_GB, platform encoding: UTF-8 -OS name: "mac os x", version: "10.15.2", arch: "x86_64", family: "mac" -``` - -### Create a project - -Follow these steps to create a Maven project that can be -published to the GitLab Package Registry. - -1. Open your terminal and create a directory to store the project. -1. From the new directory, run this Maven command to initialize a new package: - - ```shell - mvn archetype:generate -DgroupId=com.mycompany.mydepartment -DartifactId=my-project -DarchetypeArtifactId=maven-archetype-quickstart -DinteractiveMode=false - ``` - - The arguments are: - - - `DgroupId`: A unique string that identifies your package. Follow - the [Maven naming conventions](https://maven.apache.org/guides/mini/guide-naming-conventions.html). - - `DartifactId`: The name of the `JAR`, appended to the end of the `DgroupId`. - - `DarchetypeArtifactId`: The archetype used to create the initial structure of - the project. - - `DinteractiveMode`: Create the project using batch mode (optional). - -This message indicates that the project was set up successfully: - -```shell -... -[INFO] ------------------------------------------------------------------------ -[INFO] BUILD SUCCESS -[INFO] ------------------------------------------------------------------------ -[INFO] Total time: 3.429 s -[INFO] Finished at: 2020-01-28T11:47:04Z -[INFO] ------------------------------------------------------------------------ -``` - -In the folder where you ran the command, a new directory should be displayed. -The directory name should match the `DartifactId` parameter, which in this case, -is `my-project`. - -## Build a Java project with Gradle - -This section explains how to install Gradle and initialize a Java project. - -If you already use Gradle and know how to build your own packages, go to the -[next section](#authenticate-to-the-package-registry-with-maven). - -### Install Gradle - -If you want to create a new Gradle project, you must install Gradle. Follow -instructions at [gradle.org](https://gradle.org/install/) to download and install -Gradle for your local development environment. - -In your terminal, verify you can use Gradle by running: - -```shell -gradle -version -``` - -To use an existing Gradle project, in the project directory, -on Linux execute `gradlew`, or on Windows execute `gradlew.bat`. - -The output should be similar to: - -```plaintext ------------------------------------------------------------- -Gradle 6.0.1 ------------------------------------------------------------- - -Build time: 2019-11-18 20:25:01 UTC -Revision: fad121066a68c4701acd362daf4287a7c309a0f5 - -Kotlin: 1.3.50 -Groovy: 2.5.8 -Ant: Apache Ant(TM) version 1.10.7 compiled on September 1 2019 -JVM: 11.0.5 (Oracle Corporation 11.0.5+10) -OS: Windows 10 10.0 amd64 -``` - -### Create a Java project - -Follow these steps to create a Maven project that can be -published to the GitLab Package Registry. - -1. Open your terminal and create a directory to store the project. -1. From this new directory, run this Maven command to initialize a new package: - - ```shell - gradle init - ``` - - The output should be: - - ```plaintext - Select type of project to generate: - 1: basic - 2: application - 3: library - 4: Gradle plugin - Enter selection (default: basic) [1..4] - ``` - -1. Enter `3` to create a new Library project. The output should be: - - ```plaintext - Select implementation language: - 1: C++ - 2: Groovy - 3: Java - 4: Kotlin - 5: Scala - 6: Swift - ``` - -1. Enter `3` to create a new Java Library project. The output should be: - - ```plaintext - Select build script DSL: - 1: Groovy - 2: Kotlin - Enter selection (default: Groovy) [1..2] - ``` - -1. Enter `1` to create a new Java Library project that is described in Groovy DSL, or `2` to create one that is described in Kotlin DSL. The output should be: - - ```plaintext - Select test framework: - 1: JUnit 4 - 2: TestNG - 3: Spock - 4: JUnit Jupiter - ``` - -1. Enter `1` to initialize the project with JUnit 4 testing libraries. The output should be: - - ```plaintext - Project name (default: test): - ``` - -1. Enter a project name or press <kbd>Enter</kbd> to use the directory name as project name. +Learn how to build a [Maven](../workflows/build_packages.md#maven) or [Gradle](../workflows/build_packages.md#gradle) package. ## Authenticate to the Package Registry with Maven @@ -561,11 +391,11 @@ for download. **Only packages that have the same path as the project** are exposed by the instance-level endpoint. -| Project | Package | Instance-level endpoint available | -| ------- | ------- | --------------------------------- | -| `foo/bar` | `foo/bar/1.0-SNAPSHOT` | Yes | -| `gitlab-org/gitlab` | `foo/bar/1.0-SNAPSHOT` | No | -| `gitlab-org/gitlab` | `gitlab-org/gitlab/1.0-SNAPSHOT` | Yes | +| Project | Package | Instance-level endpoint available | +| ------------------- | -------------------------------- | --------------------------------- | +| `foo/bar` | `foo/bar/1.0-SNAPSHOT` | Yes | +| `gitlab-org/gitlab` | `foo/bar/1.0-SNAPSHOT` | No | +| `gitlab-org/gitlab` | `gitlab-org/gitlab/1.0-SNAPSHOT` | Yes | This example shows how relevant `repository` section of your `pom.xml`. You still need a project-specific URL in the `distributionManagement` section. @@ -838,7 +668,7 @@ dependencies { ### Request forwarding to Maven Central -> [Introduced](<https://gitlab.com/gitlab-org/gitlab/-/issues/362657>) behind a [feature flag](../../feature_flags.md), disabled by default in GitLab 15.4 +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/362657) behind a [feature flag](../../feature_flags.md), disabled by default in GitLab 15.4 FLAG: On self-managed GitLab, by default this feature is not available. To make it available, ask an administrator to [enable the feature flag](../../../administration/feature_flags.md) named `maven_central_request_forwarding`. @@ -1053,20 +883,20 @@ that you can use when performing tasks with GitLab CI/CD. - Specify where to find the `pom.xml` file (`-f,--file`): - ```yaml - package: - script: - - 'mvn --no-transfer-progress -f helloworld/pom.xml package' - ``` + ```yaml + package: + script: + - 'mvn --no-transfer-progress -f helloworld/pom.xml package' + ``` - Specify where to find the user settings (`-s,--settings`) instead of [the default location](https://maven.apache.org/settings.html). There's also a `-gs,--global-settings` option: - ```yaml - package: - script: - - 'mvn -s settings/ci.xml package' - ``` + ```yaml + package: + script: + - 'mvn -s settings/ci.xml package' + ``` ### Verify your Maven settings diff --git a/doc/user/packages/npm_registry/index.md b/doc/user/packages/npm_registry/index.md index 44266559999..5d2efc52ba9 100644 --- a/doc/user/packages/npm_registry/index.md +++ b/doc/user/packages/npm_registry/index.md @@ -1,6 +1,6 @@ --- stage: Package -group: Package +group: Package Registry info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- @@ -20,72 +20,7 @@ WARNING: Never hardcode GitLab tokens (or any tokens) directly in `.npmrc` files or any other files that can be committed to a repository. -## Build an npm package - -This section covers how to install npm or Yarn and build a package for your -JavaScript project. - -If you already use npm and know how to build your own packages, go to -the [next section](#authenticate-to-the-package-registry). - -### Install npm - -Install Node.js and npm in your local development environment by following -the instructions at [npmjs.com](https://docs.npmjs.com/downloading-and-installing-node-js-and-npm/). - -When installation is complete, verify you can use npm in your terminal by -running: - -```shell -npm --version -``` - -The npm version is shown in the output: - -```plaintext -6.10.3 -``` - -### Install Yarn - -As an alternative to npm, you can install Yarn in your local environment by following the -instructions at [classic.yarnpkg.com](https://classic.yarnpkg.com/en/docs/install). - -When installation is complete, verify you can use Yarn in your terminal by -running: - -```shell -yarn --version -``` - -The Yarn version is shown in the output: - -```plaintext -1.19.1 -``` - -### Create a project - -To create a project: - -1. Create an empty directory. -1. Go to the directory and initialize an empty package by running: - - ```shell - npm init - ``` - - Or if you're using Yarn: - - ```shell - yarn init - ``` - -1. Enter responses to the questions. Ensure the **package name** follows - the [naming convention](#package-naming-convention) and is scoped to the - project or group where the registry exists. - -A `package.json` file is created. +Learn how to build an [npm](../workflows/build_packages.md#npm) or [yarn](../workflows/build_packages.md#yarn) package. ## Use the GitLab endpoint for npm packages @@ -244,15 +179,15 @@ In this case, the `@scope` needs to be `@registries-group` and not `@source-code For example, if your project is `https://gitlab.example.com/my-org/engineering-group/team-amazing/analytics`, the root namespace is `my-org`. When you publish a package, it must have `my-org` as the scope. -| Project | Package | Supported | -| ---------------------- | ----------------------- | --------- | -| `my-org/bar` | `@my-org/bar` | Yes | -| `my-org/bar/baz` | `@my-org/baz` | Yes | -| `My-Org/Bar/baz` | `@my-org/Baz` | Yes | -| `My-Org/Bar/baz` | `@My-Org/Baz` | Yes | -| `my-org/bar/buz` | `@my-org/anything` | Yes | -| `gitlab-org/gitlab` | `@gitlab-org/gitlab` | Yes | -| `gitlab-org/gitlab` | `@foo/bar` | No | +| Project | Package | Supported | +| ------------------- | -------------------- | --------- | +| `my-org/bar` | `@my-org/bar` | Yes | +| `my-org/bar/baz` | `@my-org/baz` | Yes | +| `My-Org/Bar/baz` | `@my-org/Baz` | Yes | +| `My-Org/Bar/baz` | `@My-Org/Baz` | Yes | +| `my-org/bar/buz` | `@my-org/anything` | Yes | +| `gitlab-org/gitlab` | `@gitlab-org/gitlab` | Yes | +| `gitlab-org/gitlab` | `@foo/bar` | No | In GitLab, this regex validates all package names from all package managers: @@ -290,7 +225,9 @@ You can also define `"publishConfig"` for your project in `package.json`. For ex ```json { -"publishConfig": { "@foo:registry":" https://gitlab.example.com/api/v4/projects/<your_project_id>/packages/npm/" } + "publishConfig": { + "@foo:registry": " https://gitlab.example.com/api/v4/projects/<your_project_id>/packages/npm/" + } } ``` @@ -342,13 +279,13 @@ To publish and install with the project-level npm endpoint, set the following co ```yaml npmScopes: foo: - npmRegistryServer: "https://gitlab.example.com/api/v4/projects/<your_project_id>/packages/npm/" - npmPublishRegistry: "https://gitlab.example.com/api/v4/projects/<your_project_id>/packages/npm/" + npmRegistryServer: 'https://gitlab.example.com/api/v4/projects/<your_project_id>/packages/npm/' + npmPublishRegistry: 'https://gitlab.example.com/api/v4/projects/<your_project_id>/packages/npm/' npmRegistries: //gitlab.example.com/api/v4/projects/<your_project_id>/packages/npm/: npmAlwaysAuth: true - npmAuthToken: "<your_token>" + npmAuthToken: '<your_token>' ``` For the instance-level npm endpoint, use this Yarn 2 configuration in `.yarnrc.yml`: @@ -356,12 +293,12 @@ For the instance-level npm endpoint, use this Yarn 2 configuration in `.yarnrc.y ```yaml npmScopes: foo: - npmRegistryServer: "https://gitlab.example.com/api/v4/packages/npm/" + npmRegistryServer: 'https://gitlab.example.com/api/v4/packages/npm/' npmRegistries: //gitlab.example.com/api/v4/packages/npm/: npmAlwaysAuth: true - npmAuthToken: "<your_token>" + npmAuthToken: '<your_token>' ``` In this configuration: @@ -561,7 +498,7 @@ should look like: { "name": "@foo/my-package", "version": "1.0.0", - "description": "Example package for GitLab npm registry", + "description": "Example package for GitLab npm registry" } ``` diff --git a/doc/user/packages/nuget_repository/index.md b/doc/user/packages/nuget_repository/index.md index d4b87d70447..956202bb990 100644 --- a/doc/user/packages/nuget_repository/index.md +++ b/doc/user/packages/nuget_repository/index.md @@ -1,6 +1,6 @@ --- stage: Package -group: Package +group: Package Registry info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- @@ -22,47 +22,7 @@ The Package Registry works with: For documentation of the specific API endpoints that these clients use, see the [NuGet API documentation](../../../api/packages/nuget.md). -## Install NuGet - -The required minimum versions are: - -- [NuGet CLI 5.1 or later](https://www.nuget.org/downloads). If you have - [Visual Studio](https://visualstudio.microsoft.com/vs/), the NuGet CLI is - probably already installed. -- Alternatively, you can use [.NET SDK 3.0 or later](https://dotnet.microsoft.com/download/dotnet/3.0), - which installs the NuGet CLI. -- NuGet protocol version 3 or later. - -Verify that the [NuGet CLI](https://www.nuget.org/) is installed by running: - -```shell -nuget help -``` - -The output should be similar to: - -```plaintext -NuGet Version: 5.1.0.6013 -usage: NuGet <command> [args] [options] -Type 'NuGet help <command>' for help on a specific command. - -Available commands: - -[output truncated] -``` - -### Install NuGet on macOS - -For macOS, you can use [Mono](https://www.mono-project.com/) to run the -NuGet CLI. - -1. If you use Homebrew, to install Mono, run `brew install mono`. -1. Download the Windows C# binary `nuget.exe` from the [NuGet CLI page](https://www.nuget.org/downloads). -1. Run this command: - - ```shell - mono nuget.exe - ``` +Learn how to [install NuGet](../workflows/build_packages.md#nuget). ## Use the GitLab endpoint for NuGet Packages @@ -162,6 +122,7 @@ To use the [project-level](#use-the-gitlab-endpoint-for-nuget-packages) NuGet en 1. In the **NuGet** section, select **Sources** to view a list of all your NuGet sources. 1. Select **Add**. 1. Complete the following fields: + - **Name**: Name for the source. - **Location**: `https://gitlab.example.com/api/v4/projects/<your_project_id>/packages/nuget/index.json`, where `<your_project_id>` is your project ID, and `gitlab.example.com` is @@ -191,6 +152,7 @@ To use the [group-level](#use-the-gitlab-endpoint-for-nuget-packages) NuGet endp 1. In the **NuGet** section, select **Sources** to view a list of all your NuGet sources. 1. Select **Add**. 1. Complete the following fields: + - **Name**: Name for the source. - **Location**: `https://gitlab.example.com/api/v4/groups/<your_group_id>/-/packages/nuget/index.json`, where `<your_group_id>` is your group ID, and `gitlab.example.com` is diff --git a/doc/user/packages/package_registry/index.md b/doc/user/packages/package_registry/index.md index 2d8cb46f933..8e160cbb195 100644 --- a/doc/user/packages/package_registry/index.md +++ b/doc/user/packages/package_registry/index.md @@ -1,6 +1,6 @@ --- stage: Package -group: Package +group: Package Registry info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- diff --git a/doc/user/packages/package_registry/reduce_package_registry_storage.md b/doc/user/packages/package_registry/reduce_package_registry_storage.md index e6996d9dc3e..1085cf5c239 100644 --- a/doc/user/packages/package_registry/reduce_package_registry_storage.md +++ b/doc/user/packages/package_registry/reduce_package_registry_storage.md @@ -1,6 +1,6 @@ --- stage: Package -group: Package +group: Package Registry info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- diff --git a/doc/user/packages/pypi_repository/index.md b/doc/user/packages/pypi_repository/index.md index fb1b9ce78ab..f6ed9654882 100644 --- a/doc/user/packages/pypi_repository/index.md +++ b/doc/user/packages/pypi_repository/index.md @@ -1,6 +1,6 @@ --- stage: Package -group: Package +group: Package Registry info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- @@ -20,148 +20,7 @@ The Package Registry works with: For documentation of the specific API endpoints that the `pip` and `twine` clients use, see the [PyPI API documentation](../../../api/packages/pypi.md). -## Build a PyPI package - -This section explains how to create a PyPI package. - -If you already use PyPI and know how to build your own packages, go to the -[next section](#authenticate-with-the-package-registry). - -### Install pip and twine - -Install a recent version of [pip](https://pypi.org/project/pip/) and -[twine](https://pypi.org/project/twine/). - -### Create a project - -Create a test project. - -1. Open your terminal. -1. Create a directory called `MyPyPiPackage`, and then go to that directory: - - ```shell - mkdir MyPyPiPackage && cd MyPyPiPackage - ``` - -1. Create another directory and go to it: - - ```shell - mkdir mypypipackage && cd mypypipackage - ``` - -1. Create the required files in this directory: - - ```shell - touch __init__.py - touch greet.py - ``` - -1. Open the `greet.py` file, and then add: - - ```python - def SayHello(): - print("Hello from MyPyPiPackage") - return - ``` - -1. Open the `__init__.py` file, and then add: - - ```python - from .greet import SayHello - ``` - -1. To test the code, in your `MyPyPiPackage` directory, start the Python prompt. - - ```shell - python - ``` - -1. Run this command: - - ```python - >>> from mypypipackage import SayHello - >>> SayHello() - ``` - -A message indicates that the project was set up successfully: - -```plaintext -Python 3.8.2 (v3.8.2:7b3ab5921f, Feb 24 2020, 17:52:18) -[Clang 6.0 (clang-600.0.57)] on darwin -Type "help", "copyright", "credits" or "license" for more information. ->>> from mypypipackage import SayHello ->>> SayHello() -Hello from MyPyPiPackage -``` - -### Create a package - -After you create a project, you can create a package. - -1. In your terminal, go to the `MyPyPiPackage` directory. -1. Create a `pyproject.toml` file: - - ```shell - touch pyproject.toml - ``` - - This file contains all the information about the package. For more information - about this file, see [creating `pyproject.toml`](https://packaging.python.org/en/latest/tutorials/packaging-projects/#creating-pyproject-toml). - Because GitLab identifies packages based on - [Python normalized names (PEP-503)](https://www.python.org/dev/peps/pep-0503/#normalized-names), - ensure your package name meets these requirements. See the [installation section](#authenticate-with-a-ci-job-token) - for details. - -1. Open the `pyproject.toml` file, and then add basic information: - - ```toml - [build-system] - requires = ["setuptools>=61.0"] - build-backend = "setuptools.build_meta" - - [project] - name = "mypypipackage" - version = "0.0.1" - authors = [ - { name="Example Author", email="author@example.com" }, - ] - description = "A small example package" - requires-python = ">=3.7" - classifiers = [ - "Programming Language :: Python :: 3", - "Operating System :: OS Independent", - ] - - [tool.setuptools.packages] - find = {} - ``` - -1. Save the file. -1. Install the package build library: - - ```shell - pip install build - ``` - -1. Build the package: - - ```shell - python -m build - ``` - -The output should be visible in a newly-created `dist` folder: - -```shell -ls dist -``` - -The output should appear similar to the following: - -```plaintext -mypypipackage-0.0.1-py3-none-any.whl mypypipackage-0.0.1.tar.gz -``` - -The package is now ready to be published to the Package Registry. +Learn how to [build a PyPI package](../workflows/build_packages.md#pypi). ## Authenticate with the Package Registry diff --git a/doc/user/packages/rubygems_registry/index.md b/doc/user/packages/rubygems_registry/index.md index f21b210f4f5..7710ad3db01 100644 --- a/doc/user/packages/rubygems_registry/index.md +++ b/doc/user/packages/rubygems_registry/index.md @@ -1,6 +1,6 @@ --- stage: Package -group: Package +group: Package Registry info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- diff --git a/doc/user/packages/terraform_module_registry/index.md b/doc/user/packages/terraform_module_registry/index.md index 6dd118c39b4..2b99ff807ec 100644 --- a/doc/user/packages/terraform_module_registry/index.md +++ b/doc/user/packages/terraform_module_registry/index.md @@ -1,6 +1,6 @@ --- -stage: Configure -group: Configure +stage: Package +group: Package Registry info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- @@ -24,9 +24,11 @@ When you publish a Terraform Module, if it does not exist, it is created. Prerequisites: -- A package with the same name and version must not already exist in the top-level namespace. +- The package name and version [must be unique in the top-level namespace](../infrastructure_registry/index.md#how-module-resolution-works). - Your project and group names must not include a dot (`.`). For example, `source = "gitlab.example.com/my.group/project.name"`. - You must [authenticate with the API](../../../api/index.md#authentication). If authenticating with a deploy token, it must be configured with the `write_package_registry` scope. +- The name of a module [must be unique within the scope of its group](../infrastructure_registry/index.md#how-module-resolution-works), otherwise an + [error occurs](#troubleshooting). ```plaintext PUT /projects/:id/packages/terraform/modules/:module-name/:module-system/:module-version/file @@ -141,3 +143,7 @@ For examples of the Terraform module registry, check the projects below: - The [_GitLab local file_ project](https://gitlab.com/mattkasa/gitlab-local-file) creates a minimal Terraform module and uploads it into the Terraform module registry using GitLab CI/CD. - The [_Terraform module test_ project](https://gitlab.com/mattkasa/terraform-module-test) uses the module from the previous example. + +## Troubleshooting + +- Publishing a module with a duplicate name results in a `{"message":"Access Denied"}` error. There's an ongoing discussion about allowing duplicate module names [in this issue](https://gitlab.com/gitlab-org/gitlab/-/issues/368040). diff --git a/doc/user/packages/workflows/build_packages.md b/doc/user/packages/workflows/build_packages.md new file mode 100644 index 00000000000..ec971195ea9 --- /dev/null +++ b/doc/user/packages/workflows/build_packages.md @@ -0,0 +1,504 @@ +--- +stage: Package +group: Package Registry +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments +--- + +# Build packages + +Learn how to install and build packages different package formats. + +- [Composer](#composer) +- [Conan](#conan) +- [Maven](#maven) +- [Gradle](#gradle) +- [npm](#npm) +- [Yarn](#yarn) +- [NuGet](#nuget) +- [PyPI](#pypi) + +## Composer + +1. Create a directory called `my-composer-package` and change to that directory: + + ```shell + mkdir my-composer-package && cd my-composer-package + ``` + +1. Run [`composer init`](https://getcomposer.org/doc/03-cli.md#init) and answer the prompts. + + For namespace, enter your unique [namespace](../../../user/namespace/index.md), like your GitLab username or group name. + + A file called `composer.json` is created: + + ```json + { + "name": "<namespace>/composer-test", + "description": "Library XY", + "type": "library", + "license": "GPL-3.0-only", + "authors": [ + { + "name": "John Doe", + "email": "john@example.com" + } + ], + "require": {} + } + ``` + +## Conan + +### Install Conan + +Download the Conan package manager to your local development environment by +following the instructions at [conan.io](https://conan.io/downloads.html). + +When installation is complete, verify you can use Conan in your terminal by +running: + +```shell +conan --version +``` + +The Conan version is printed in the output: + +```plaintext +Conan version 1.20.5 +``` + +### Install CMake + +When you develop with C++ and Conan, you can select from many available +compilers. This example uses the CMake build system generator. + +To install CMake: + +- For Mac, use [Homebrew](https://brew.sh/) and run `brew install cmake`. +- For other operating systems, follow the instructions at [cmake.org](https://cmake.org/install/). + +When installation is complete, verify you can use CMake in your terminal by +running: + +```shell +cmake --version +``` + +The CMake version is printed in the output. + +### Create a project + +To test the Package Registry, you need a C++ project. If you don't already have +one, you can clone the Conan [hello world starter project](https://github.com/conan-io/hello). + +### Build a Conan package + +To build a package: + +1. Open a terminal and navigate to your project's root folder. +1. Generate a new recipe by running `conan new` with a package name and version: + + ```shell + conan new Hello/0.1 -t + ``` + +1. Create a package for the recipe by running `conan create` with the Conan user + and channel: + + ```shell + conan create . mycompany/beta + ``` + + NOTE: + If you use an [instance remote](../conan_repository/index.md#add-a-remote-for-your-instance), you must + follow a specific [naming convention](../conan_repository/index.md#package-recipe-naming-convention-for-instance-remotes). + +A package with the recipe `Hello/0.1@mycompany/beta` is created. + +For more details about creating and managing Conan packages, see the +[Conan documentation](https://docs.conan.io/en/latest/creating_packages.html). + +## Maven + +### Install Maven + +The required minimum versions are: + +- Java 11.0.5+ +- Maven 3.6+ + +Follow the instructions at [maven.apache.org](https://maven.apache.org/install.html) +to download and install Maven for your local development environment. After +installation is complete, verify you can use Maven in your terminal by running: + +```shell +mvn --version +``` + +The output should be similar to: + +```shell +Apache Maven 3.6.1 (d66c9c0b3152b2e69ee9bac180bb8fcc8e6af555; 2019-04-04T20:00:29+01:00) +Maven home: /Users/<your_user>/apache-maven-3.6.1 +Java version: 12.0.2, vendor: Oracle Corporation, runtime: /Library/Java/JavaVirtualMachines/jdk-12.0.2.jdk/Contents/Home +Default locale: en_GB, platform encoding: UTF-8 +OS name: "mac os x", version: "10.15.2", arch: "x86_64", family: "mac" +``` + +### Build a Maven package + +1. Open your terminal and create a directory to store the project. +1. From the new directory, run this Maven command to initialize a new package: + + ```shell + mvn archetype:generate -DgroupId=com.mycompany.mydepartment -DartifactId=my-project -DarchetypeArtifactId=maven-archetype-quickstart -DinteractiveMode=false + ``` + + The arguments are: + + - `DgroupId`: A unique string that identifies your package. Follow + the [Maven naming conventions](https://maven.apache.org/guides/mini/guide-naming-conventions.html). + - `DartifactId`: The name of the `JAR`, appended to the end of the `DgroupId`. + - `DarchetypeArtifactId`: The archetype used to create the initial structure of + the project. + - `DinteractiveMode`: Create the project using batch mode (optional). + +This message indicates that the project was set up successfully: + +```shell +... +[INFO] ------------------------------------------------------------------------ +[INFO] BUILD SUCCESS +[INFO] ------------------------------------------------------------------------ +[INFO] Total time: 3.429 s +[INFO] Finished at: 2020-01-28T11:47:04Z +[INFO] ------------------------------------------------------------------------ +``` + +In the folder where you ran the command, a new directory should be displayed. +The directory name should match the `DartifactId` parameter, which in this case, +is `my-project`. + +## Gradle + +### Install Gradle + +If you want to create a new Gradle project, you must install Gradle. Follow +instructions at [gradle.org](https://gradle.org/install/) to download and install +Gradle for your local development environment. + +In your terminal, verify you can use Gradle by running: + +```shell +gradle -version +``` + +To use an existing Gradle project, in the project directory, +on Linux execute `gradlew`, or on Windows execute `gradlew.bat`. + +The output should be similar to: + +```plaintext +------------------------------------------------------------ +Gradle 6.0.1 +------------------------------------------------------------ + +Build time: 2019-11-18 20:25:01 UTC +Revision: fad121066a68c4701acd362daf4287a7c309a0f5 + +Kotlin: 1.3.50 +Groovy: 2.5.8 +Ant: Apache Ant(TM) version 1.10.7 compiled on September 1 2019 +JVM: 11.0.5 (Oracle Corporation 11.0.5+10) +OS: Windows 10 10.0 amd64 +``` + +### Create a package + +1. Open your terminal and create a directory to store the project. +1. From this new directory, run this command to initialize a new package: + + ```shell + gradle init + ``` + + The output should be: + + ```plaintext + Select type of project to generate: + 1: basic + 2: application + 3: library + 4: Gradle plugin + Enter selection (default: basic) [1..4] + ``` + +1. Enter `3` to create a new Library project. The output should be: + + ```plaintext + Select implementation language: + 1: C++ + 2: Groovy + 3: Java + 4: Kotlin + 5: Scala + 6: Swift + ``` + +1. Enter `3` to create a new Java Library project. The output should be: + + ```plaintext + Select build script DSL: + 1: Groovy + 2: Kotlin + Enter selection (default: Groovy) [1..2] + ``` + +1. Enter `1` to create a new Java Library project that is described in Groovy DSL, or `2` to create one that is described in Kotlin DSL. The output should be: + + ```plaintext + Select test framework: + 1: JUnit 4 + 2: TestNG + 3: Spock + 4: JUnit Jupiter + ``` + +1. Enter `1` to initialize the project with JUnit 4 testing libraries. The output should be: + + ```plaintext + Project name (default: test): + ``` + +1. Enter a project name or press <kbd>Enter</kbd> to use the directory name as project name. + +## npm + +### Install npm + +Install Node.js and npm in your local development environment by following +the instructions at [npmjs.com](https://docs.npmjs.com/downloading-and-installing-node-js-and-npm/). + +When installation is complete, verify you can use npm in your terminal by +running: + +```shell +npm --version +``` + +The npm version is shown in the output: + +```plaintext +6.10.3 +``` + +### Create an npm package + +1. Create an empty directory. +1. Go to the directory and initialize an empty package by running: + + ```shell + npm init + ``` + +1. Enter responses to the questions. Ensure the **package name** follows + the [naming convention](../npm_registry/index.md#package-naming-convention) and is scoped to the project or group where the registry exists. + +## Yarn + +### Install Yarn + +As an alternative to npm, you can install Yarn in your local environment by following the +instructions at [classic.yarnpkg.com](https://classic.yarnpkg.com/en/docs/install). + +When installation is complete, verify you can use Yarn in your terminal by +running: + +```shell +yarn --version +``` + +The Yarn version is shown in the output: + +```plaintext +1.19.1 +``` + +### Create a package + +1. Create an empty directory. +1. Go to the directory and initialize an empty package by running: + + ```shell + yarn init + ``` + +1. Enter responses to the questions. Ensure the **package name** follows + the [naming convention](../npm_registry/index.md#package-naming-convention) and is scoped to the + project or group where the registry exists. + +A `package.json` file is created. + +## NuGet + +### Install NuGet + +Follow the instructions from [Microsoft](https://learn.microsoft.com/en-us/nuget/install-nuget-client-tools) to install NuGet. If you have +[Visual Studio](https://visualstudio.microsoft.com/vs/), NuGet is +probably already installed. + +Verify that the [NuGet CLI](https://www.nuget.org/) is installed by running: + +```shell +nuget help +``` + +The output should be similar to: + +```plaintext +NuGet Version: 5.1.0.6013 +usage: NuGet <command> [args] [options] +Type 'NuGet help <command>' for help on a specific command. + +Available commands: + +[output truncated] +``` + +## PyPI + +### Install pip and twine + +Install a recent version of [pip](https://pypi.org/project/pip/) and +[twine](https://pypi.org/project/twine/). + +### Create a project + +Create a test project. + +1. Open your terminal. +1. Create a directory called `MyPyPiPackage`, and then go to that directory: + + ```shell + mkdir MyPyPiPackage && cd MyPyPiPackage + ``` + +1. Create another directory and go to it: + + ```shell + mkdir mypypipackage && cd mypypipackage + ``` + +1. Create the required files in this directory: + + ```shell + touch __init__.py + touch greet.py + ``` + +1. Open the `greet.py` file, and then add: + + ```python + def SayHello(): + print("Hello from MyPyPiPackage") + return + ``` + +1. Open the `__init__.py` file, and then add: + + ```python + from .greet import SayHello + ``` + +1. To test the code, in your `MyPyPiPackage` directory, start the Python prompt. + + ```shell + python + ``` + +1. Run this command: + + ```python + >>> from mypypipackage import SayHello + >>> SayHello() + ``` + +A message indicates that the project was set up successfully: + +```plaintext +Python 3.8.2 (v3.8.2:7b3ab5921f, Feb 24 2020, 17:52:18) +[Clang 6.0 (clang-600.0.57)] on darwin +Type "help", "copyright", "credits" or "license" for more information. +>>> from mypypipackage import SayHello +>>> SayHello() +Hello from MyPyPiPackage +``` + +### Create a PyPI package + +After you create a project, you can create a package. + +1. In your terminal, go to the `MyPyPiPackage` directory. +1. Create a `pyproject.toml` file: + + ```shell + touch pyproject.toml + ``` + + This file contains all the information about the package. For more information + about this file, see [creating `pyproject.toml`](https://packaging.python.org/en/latest/tutorials/packaging-projects/#creating-pyproject-toml). + Because GitLab identifies packages based on + [Python normalized names (PEP-503)](https://www.python.org/dev/peps/pep-0503/#normalized-names), + ensure your package name meets these requirements. See the [installation section](../pypi_repository/index.md#authenticate-with-a-ci-job-token) + for details. + +1. Open the `pyproject.toml` file, and then add basic information: + + ```toml + [build-system] + requires = ["setuptools>=61.0"] + build-backend = "setuptools.build_meta" + + [project] + name = "mypypipackage" + version = "0.0.1" + authors = [ + { name="Example Author", email="author@example.com" }, + ] + description = "A small example package" + requires-python = ">=3.7" + classifiers = [ + "Programming Language :: Python :: 3", + "Operating System :: OS Independent", + ] + + [tool.setuptools.packages] + find = {} + ``` + +1. Save the file. +1. Install the package build library: + + ```shell + pip install build + ``` + +1. Build the package: + + ```shell + python -m build + ``` + +The output should be visible in a newly-created `dist` folder: + +```shell +ls dist +``` + +The output should appear similar to the following: + +```plaintext +mypypipackage-0.0.1-py3-none-any.whl mypypipackage-0.0.1.tar.gz +``` + +The package is now ready to be published to the Package Registry. diff --git a/doc/user/packages/workflows/project_registry.md b/doc/user/packages/workflows/project_registry.md index 25c64c93b73..df4b087f6d5 100644 --- a/doc/user/packages/workflows/project_registry.md +++ b/doc/user/packages/workflows/project_registry.md @@ -1,6 +1,6 @@ --- stage: Package -group: Package +group: Package Registry info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- diff --git a/doc/user/packages/workflows/working_with_monorepos.md b/doc/user/packages/workflows/working_with_monorepos.md index 5606e257ea8..572cd309e67 100644 --- a/doc/user/packages/workflows/working_with_monorepos.md +++ b/doc/user/packages/workflows/working_with_monorepos.md @@ -1,6 +1,6 @@ --- stage: Package -group: Package +group: Package Registry info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- diff --git a/doc/user/permissions.md b/doc/user/permissions.md index 102abf2b427..8e152a8c190 100644 --- a/doc/user/permissions.md +++ b/doc/user/permissions.md @@ -64,7 +64,7 @@ The following table lists project permissions available for each role: | [Analytics](analytics/index.md):<br>View [code review analytics](analytics/code_review_analytics.md) | | ✓ | ✓ | ✓ | ✓ | | [Analytics](analytics/index.md):<br>View [repository analytics](analytics/repository_analytics.md) | | ✓ | ✓ | ✓ | ✓ | | [Application security](application_security/index.md):<br>View licenses in [dependency list](application_security/dependency_list/index.md) | | | ✓ | ✓ | ✓ | -| [Application security](application_security/index.md):<br>Create and run [on-demand DAST scans](application_security/dast/index.md#on-demand-scans) | | | ✓ | ✓ | ✓ | +| [Application security](application_security/index.md):<br>Create and run [on-demand DAST scans](application_security/dast/proxy-based.md#on-demand-scans) | | | ✓ | ✓ | ✓ | | [Application security](application_security/index.md):<br>Manage [security policy](application_security/policies/index.md) | | | ✓ | ✓ | ✓ | | [Application security](application_security/index.md):<br>View [dependency list](application_security/dependency_list/index.md) | | | ✓ | ✓ | ✓ | | [Application security](application_security/index.md):<br>Create a [CVE ID Request](application_security/cve_id_request.md) | | | | ✓ | ✓ | @@ -73,16 +73,17 @@ The following table lists project permissions available for each role: | [Clusters](infrastructure/clusters/index.md):<br>Manage clusters | | | | ✓ | ✓ | | [Container Registry](packages/container_registry/index.md):<br>Create, edit, delete [cleanup policies](packages/container_registry/index.md#delete-images-by-using-a-cleanup-policy) | | | | ✓ | ✓ | | [Container Registry](packages/container_registry/index.md):<br>Push an image to the Container Registry | | | ✓ | ✓ | ✓ | -| [Container Registry](packages/container_registry/index.md):<br>Pull an image from the Container Registry | ✓ (*20*) | ✓ (*20*) | ✓ | ✓ | ✓ | +| [Container Registry](packages/container_registry/index.md):<br>Pull an image from the Container Registry | ✓ (*19*) | ✓ (*19*) | ✓ | ✓ | ✓ | | [Container Registry](packages/container_registry/index.md):<br>Remove a Container Registry image | | | ✓ | ✓ | ✓ | -| [GitLab Pages](project/pages/index.md):<br>View Pages protected by [access control](project/pages/introduction.md#gitlab-pages-access-control) | ✓ | ✓ | ✓ | ✓ | ✓ | +| [GitLab Pages](project/pages/index.md):<br>View Pages protected by [access control](project/pages/pages_access_control.md) | ✓ | ✓ | ✓ | ✓ | ✓ | | [GitLab Pages](project/pages/index.md):<br>Manage | | | | ✓ | ✓ | | [GitLab Pages](project/pages/index.md):<br>Manage GitLab Pages domains and certificates | | | | ✓ | ✓ | | [GitLab Pages](project/pages/index.md):<br>Remove GitLab Pages | | | | ✓ | ✓ | | [Incident Management](../operations/incident_management/index.md):<br>View [alerts](../operations/incident_management/alerts.md) | | ✓ | ✓ | ✓ | ✓ | | [Incident Management](../operations/incident_management/index.md):<br>Assign an alert | ✓ | ✓ | ✓ | ✓ | ✓ | +| [Incident Management](../operations/incident_management/index.md):<br>[Change an alert status](../operations/incident_management/alerts.md#change-an-alerts-status) | | ✓ | ✓ | ✓ | ✓ | | [Incident Management](../operations/incident_management/index.md):<br>View [incident](../operations/incident_management/incidents.md) | ✓ | ✓ | ✓ | ✓ | ✓ | -| [Incident Management](../operations/incident_management/index.md):<br>Create [incident](../operations/incident_management/incidents.md) | (*16*) | ✓ | ✓ | ✓ | ✓ | +| [Incident Management](../operations/incident_management/index.md):<br>Create [incident](../operations/incident_management/incidents.md) | | ✓ | ✓ | ✓ | ✓ | | [Incident Management](../operations/incident_management/index.md):<br>View [on-call schedules](../operations/incident_management/oncall_schedules.md) | | ✓ | ✓ | ✓ | ✓ | | [Incident Management](../operations/incident_management/index.md):<br>Participate in on-call rotation | ✓ | ✓ | ✓ | ✓ | ✓ | | [Incident Management](../operations/incident_management/index.md):<br>View [escalation policies](../operations/incident_management/escalation_policies.md) | | ✓ | ✓ | ✓ | ✓ | @@ -91,16 +92,16 @@ The following table lists project permissions available for each role: | [Issue boards](project/issue_board.md):<br>Create or delete lists | | ✓ | ✓ | ✓ | ✓ | | [Issue boards](project/issue_board.md):<br>Move issues between lists | | ✓ | ✓ | ✓ | ✓ | | [Issues](project/issues/index.md):<br>Add Labels | ✓ (*15*) | ✓ | ✓ | ✓ | ✓ | -| [Issues](project/issues/index.md):<br>Add to epic | | ✓ (*23*) | ✓ (*23*) | ✓ (*23*) | ✓ (*23*) | +| [Issues](project/issues/index.md):<br>Add to epic | | ✓ (*22*) | ✓ (*22*) | ✓ (*22*) | ✓ (*22*) | | [Issues](project/issues/index.md):<br>Assign | ✓ (*15*) | ✓ | ✓ | ✓ | ✓ | -| [Issues](project/issues/index.md):<br>Create (*18*) | ✓ | ✓ | ✓ | ✓ | ✓ | +| [Issues](project/issues/index.md):<br>Create (*17*) | ✓ | ✓ | ✓ | ✓ | ✓ | | [Issues](project/issues/index.md):<br>Create [confidential issues](project/issues/confidential_issues.md) | ✓ | ✓ | ✓ | ✓ | ✓ | | [Issues](project/issues/index.md):<br>View [Design Management](project/issues/design_management.md) pages | ✓ | ✓ | ✓ | ✓ | ✓ | | [Issues](project/issues/index.md):<br>View [related issues](project/issues/related_issues.md) | ✓ | ✓ | ✓ | ✓ | ✓ | | [Issues](project/issues/index.md):<br>Set [weight](project/issues/issue_weight.md) | ✓ (*15*) | ✓ | ✓ | ✓ | ✓ | | [Issues](project/issues/index.md):<br>Set [parent epic](group/epics/manage_epics.md#add-an-existing-issue-to-an-epic) | | ✓ | ✓ | ✓ | ✓ | | [Issues](project/issues/index.md):<br>View [confidential issues](project/issues/confidential_issues.md) | (*2*) | ✓ | ✓ | ✓ | ✓ | -| [Issues](project/issues/index.md):<br>Close / reopen (*19*) | | ✓ | ✓ | ✓ | ✓ | +| [Issues](project/issues/index.md):<br>Close / reopen (*18*) | | ✓ | ✓ | ✓ | ✓ | | [Issues](project/issues/index.md):<br>Lock threads | | ✓ | ✓ | ✓ | ✓ | | [Issues](project/issues/index.md):<br>Manage [related issues](project/issues/related_issues.md) | | ✓ | ✓ | ✓ | ✓ | | [Issues](project/issues/index.md):<br>Manage tracker | | ✓ | ✓ | ✓ | ✓ | @@ -118,7 +119,7 @@ The following table lists project permissions available for each role: | [Merge requests](project/merge_requests/index.md):<br>Apply code change suggestions | | | ✓ | ✓ | ✓ | | [Merge requests](project/merge_requests/index.md):<br>Approve (*8*) | | | ✓ | ✓ | ✓ | | [Merge requests](project/merge_requests/index.md):<br>Assign | | | ✓ | ✓ | ✓ | -| [Merge requests](project/merge_requests/index.md):<br>Create (*17*) | | | ✓ | ✓ | ✓ | +| [Merge requests](project/merge_requests/index.md):<br>Create (*16*) | | | ✓ | ✓ | ✓ | | [Merge requests](project/merge_requests/index.md):<br>Add labels | | | ✓ | ✓ | ✓ | | [Merge requests](project/merge_requests/index.md):<br>Lock threads | | | ✓ | ✓ | ✓ | | [Merge requests](project/merge_requests/index.md):<br>Manage or accept | | | ✓ | ✓ | ✓ | @@ -153,7 +154,7 @@ The following table lists project permissions available for each role: | [Projects](project/index.md):<br>View project [Audit Events](../administration/audit_events.md) | | | ✓ (*10*) | ✓ | ✓ | | [Projects](project/index.md):<br>Add [deploy keys](project/deploy_keys/index.md) | | | | ✓ | ✓ | | [Projects](project/index.md):<br>Add new [team members](project/members/index.md) | | | | ✓ | ✓ | -| [Projects](project/index.md):<br>Manage [team members](project/members/index.md) | | | | ✓ (*21*) | ✓ | +| [Projects](project/index.md):<br>Manage [team members](project/members/index.md) | | | | ✓ (*20*) | ✓ | | [Projects](project/index.md):<br>Change [project features visibility](public_access.md) level | | | | ✓ (*13*) | ✓ | | [Projects](project/index.md):<br>Configure [webhooks](project/integrations/webhooks.md) | | | | ✓ | ✓ | | [Projects](project/index.md):<br>Delete [wiki](project/wiki/index.md) pages | | | ✓ | ✓ | ✓ | @@ -161,7 +162,7 @@ The following table lists project permissions available for each role: | [Projects](project/index.md):<br>Edit project badges | | | | ✓ | ✓ | | [Projects](project/index.md):<br>Edit project settings | | | | ✓ | ✓ | | [Projects](project/index.md):<br>Export project | | | | ✓ | ✓ | -| [Projects](project/index.md):<br>Manage [project access tokens](project/settings/project_access_tokens.md) (*11*) | | | | ✓ (*21*) | ✓ | +| [Projects](project/index.md):<br>Manage [project access tokens](project/settings/project_access_tokens.md) (*11*) | | | | ✓ (*20*) | ✓ | | [Projects](project/index.md):<br>Manage [Project Operations](../operations/index.md) | | | | ✓ | ✓ | | [Projects](project/index.md):<br>Rename project | | | | ✓ | ✓ | | [Projects](project/index.md):<br>Share (invite) projects with groups | | | | ✓ (*7*) | ✓ (*7*) | @@ -203,10 +204,10 @@ The following table lists project permissions available for each role: | [Security dashboard](application_security/security_dashboard/index.md):<br>Use security dashboard | | | ✓ | ✓ | ✓ | | [Security dashboard](application_security/security_dashboard/index.md):<br>View vulnerability | | | ✓ | ✓ | ✓ | | [Security dashboard](application_security/security_dashboard/index.md):<br>View vulnerability findings in [dependency list](application_security/dependency_list/index.md) | | | ✓ | ✓ | ✓ | -| [Tasks](tasks.md):<br>Create (*18*) | ✓ | ✓ | ✓ | ✓ | ✓ | +| [Tasks](tasks.md):<br>Create (*17*) | ✓ | ✓ | ✓ | ✓ | ✓ | | [Tasks](tasks.md):<br>Edit | | ✓ | ✓ | ✓ | ✓ | | [Tasks](tasks.md):<br>Remove from issue | | ✓ | ✓ | ✓ | ✓ | -| [Tasks](tasks.md):<br>Delete (*22*) | | | | | ✓ | +| [Tasks](tasks.md):<br>Delete (*21*) | | | | | ✓ | | [Terraform](infrastructure/index.md):<br>Read Terraform state | | | ✓ | ✓ | ✓ | | [Terraform](infrastructure/index.md):<br>Manage Terraform state | | | | ✓ | ✓ | | [Test cases](../ci/test_cases/index.md):<br>Archive | | ✓ | ✓ | ✓ | ✓ | @@ -239,15 +240,13 @@ The following table lists project permissions available for each role: Developer role. 15. Guest users can only set metadata (for example, labels, assignees, or milestones) when creating an issue. They cannot change the metadata on existing issues. -16. In GitLab 14.5 or later, Guests are not allowed to [create incidents](../operations/incident_management/incidents.md#incident-creation). - In GitLab 15.1 and later, a Guest who created an issue that was promoted to an incident cannot edit, close, or reopen their incident. -17. In projects that accept contributions from external members, users can create, edit, and close their own merge requests. -18. Authors and assignees can modify the title and description even if they don't have the Reporter role. -19. Authors and assignees can close and reopen issues even if they don't have the Reporter role. -20. The ability to view the Container Registry and pull images is controlled by the [Container Registry's visibility permissions](packages/container_registry/index.md#container-registry-visibility-permissions). -21. Maintainers cannot create, demote, or remove Owners, and they cannot promote users to the Owner role. They also cannot approve Owner role access requests. -22. Authors of tasks can delete them even if they don't have the Owner role, but they have to have at least the Guest role for the project. -23. You must have permission to [view the epic](group/epics/manage_epics.md#who-can-view-an-epic). +16. In projects that accept contributions from external members, users can create, edit, and close their own merge requests. +17. Authors and assignees can modify the title and description even if they don't have the Reporter role. +18. Authors and assignees can close and reopen issues even if they don't have the Reporter role. +19. The ability to view the Container Registry and pull images is controlled by the [Container Registry's visibility permissions](packages/container_registry/index.md#container-registry-visibility-permissions). +20. Maintainers cannot create, demote, or remove Owners, and they cannot promote users to the Owner role. They also cannot approve Owner role access requests. +21. Authors of tasks can delete them even if they don't have the Owner role, but they have to have at least the Guest role for the project. +22. You must have permission to [view the epic](group/epics/manage_epics.md#who-can-view-an-epic). <!-- markdownlint-enable MD029 --> @@ -470,7 +469,7 @@ project and should only have access to that project. External users: -- Cannot create project, groups, and snippets within their personal namespaces. +- Cannot create project, groups, and snippets in their personal namespaces. - Can only create projects (including forks), subgroups, and snippets within top-level groups to which they are explicitly granted access. - Can only access public projects and projects to which they are explicitly granted access, thus hiding all other internal or private ones from them (like being diff --git a/doc/user/product_analytics/index.md b/doc/user/product_analytics/index.md new file mode 100644 index 00000000000..8e340fff32a --- /dev/null +++ b/doc/user/product_analytics/index.md @@ -0,0 +1,48 @@ +--- +stage: Analyze +group: Product Analytics +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments +--- + +# Product analytics **(ULTIMATE)** **Alpha** + +> Introduced in GitLab 15.4 [with a flag](../../administration/feature_flags.md) named `cube_api_proxy`. Disabled by default. + +FLAG: +On self-managed GitLab, by default this feature is not available. To make it available per project or for your entire instance, ask an administrator to [enable the feature flag](../../administration/feature_flags.md) named `cube_api_proxy`. +On GitLab.com, this feature is not available. +This feature is not ready for production use. + +## Overview + +You can view the [product category](https://about.gitlab.com/direction/analytics/product-analytics/) page for more information about our direction. This page is a work in progress and will be updated as we add more features. + +## Product analytics dashboards + +Each project can define an unlimited number of dashboards. These dashboards are defined using our YAML schema and stored +in the `.gitlab/product_analytics/dashboards/` directory. The name of the file is the name of the dashboard, and visualizations are shared across dashboards.. + +Project maintainers can enforce approval rules on dashboard changes, and dashboards can be versioned in source control. + +### Define a dashboard + +To define a dashboard: + +1. In `.gitlab/product_analytics/dashboards/`, create a directory named like the dashboard. Each dashboard should have its own directory. +1. In the new directory, create a `.yaml` file with the same name as the directory. This file contains the dashboard definition, and must conform to the JSON schema defined in `ee/app/validators/json_schemas/product_analytics_dashboard.json`. +1. In the `.gitlab/product_analytics/dashboards/visualizations/` directory, create a `yaml` file. This file defines the visualization type for the dashboard, and must conform to the schema in +`ee/app/validators/json_schemas/product_analytics_visualization.json`. + +The example below includes three dashboards and one visualization that applies to all dashboards. + +```plaintext +.gitlab/product_analytics/dashboards +├── conversion_funnels +│ └── conversion_funnels.yaml +├── demographic_breakdown +│ └── demographic_breakdown.yaml +├── north_star_metrics +| └── north_star_metrics.yaml +├── visualizations +│ └── example_line_chart.yaml +``` diff --git a/doc/user/profile/account/create_accounts.md b/doc/user/profile/account/create_accounts.md index f2636fdaa68..60b8fe8ab88 100644 --- a/doc/user/profile/account/create_accounts.md +++ b/doc/user/profile/account/create_accounts.md @@ -49,3 +49,19 @@ Users are: - Created when first signing with [Group SAML](../../group/saml_sso/index.md). - Automatically created by [SCIM](../../group/saml_sso/scim_setup.md) when the user is created in the identity provider. + +## Create users through the Rails console + +WARNING: +Commands that change data can cause damage if not run correctly or under the right conditions. Always run commands in a test environment first and have a backup instance ready to restore. + +To create a user through the Rails console: + +1. [Start a Rails console session](../../../administration/operations/rails_console.md#starting-a-rails-console-session). +1. Run the following commands: + + ```ruby + u = User.new(username: 'test_user', email: 'test@example.com', name: 'Test User', password: 'password', password_confirmation: 'password') + u.skip_confirmation! # Use it only if you wish user to be automatically confirmed. If skipped, user receives confirmation e-mail + u.save! + ``` diff --git a/doc/user/profile/account/two_factor_authentication.md b/doc/user/profile/account/two_factor_authentication.md index 3dc768f6606..39826bf59c4 100644 --- a/doc/user/profile/account/two_factor_authentication.md +++ b/doc/user/profile/account/two_factor_authentication.md @@ -61,7 +61,6 @@ To enable 2FA with a one-time password: 1. Install a compatible application. For example: - Cloud-based (recommended because you can restore access if you lose the hardware device): - [Authy](https://authy.com/) - - [Duo Mobile](https://duo.com/product/multi-factor-authentication-mfa/duo-mobile-app) - Other: - [Google Authenticator](https://support.google.com/accounts/answer/1066447?hl=en) - [Microsoft Authenticator](https://www.microsoft.com/en-us/security/mobile-authenticator-app) @@ -73,6 +72,9 @@ To enable 2FA with a one-time password: 1. Enter your current password. 1. Select **Submit**. +NOTE: +DUO [cannot be used for 2FA](https://gitlab.com/gitlab-org/gitlab/-/issues/15760). + If you entered the correct pin, GitLab displays a list of [recovery codes](#recovery-codes). Download them and keep them in a safe place. diff --git a/doc/user/profile/active_sessions.md b/doc/user/profile/active_sessions.md index d0600e5e80d..430d1c3dc9f 100644 --- a/doc/user/profile/active_sessions.md +++ b/doc/user/profile/active_sessions.md @@ -51,6 +51,6 @@ important to describe those, too. Think of things that may go wrong and include This is important to minimize requests for support, and to avoid doc comments with questions that you know someone might ask. -Each scenario can be a third-level heading, e.g. `### Getting error message X`. +Each scenario can be a third-level heading, for example `### Getting error message X`. If you have none to add when creating a doc, leave this section in place but commented out to help encourage others to add to it in the future. --> diff --git a/doc/user/profile/contributions_calendar.md b/doc/user/profile/contributions_calendar.md new file mode 100644 index 00000000000..6df7ad56c5e --- /dev/null +++ b/doc/user/profile/contributions_calendar.md @@ -0,0 +1,136 @@ +--- +stage: Manage +group: Workspace +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments +type: concepts, howto +--- + +# Contributions calendar **(FREE)** + +The contributions calendar displays a [user's events](#user-contribution-events) from the past 12 months. +This includes contributions made in forked and [private](#show-private-contributions-on-your-user-profile-page) repositories. + + + +The gradient color of the tiles represents the number of contributions made per day. The gradient ranges from blank (0 contributions) to dark blue (more than 30 contributions). + +NOTE: +The contribution calendar only displays contributions from the last 12 months, but issue [24264](https://gitlab.com/gitlab-org/gitlab/-/issues/24264) proposes to change this to more than 12 months. General improvements to the user profile are proposed in issue [8488](https://gitlab.com/groups/gitlab-org/-/epics/8488). + +## User contribution events + +GitLab tracks the following contribution events: + +- `approved` + - Merge request +- `closed` + - [Epic](../group/epics/index.md) + - Issue + - Merge request + - Milestone +- `commented` on any `Noteable` record. + - Alert + - Commit + - Design + - Issue + - Merge request + - Snippet +- `created` + - Design + - [Epic](../group/epics/index.md) + - Issue + - Merge request + - Milestone + - Project + - Wiki page +- `destroyed` + - Design + - Milestone + - Wiki page +- `expired` + - Project membership +- `joined` + - Project membership +- `left` + - Project membership +- `merged` + - Merge request +- `pushed` commits to (or deleted commits from) a repository, individually or in bulk. + - Project +- `reopened` + - [Epic](../group/epics/index.md) + - Issue + - Merge request + - Milestone +- `updated` + - Design + - Wiki page + +### View daily contributions + +To view your daily contributions: + +1. On the top bar, in the top-right corner, select your avatar. +1. Select your name from the dropdown list. +1. In the contributions calendar: + - To view the number of contributions for a specific day, hover over a tile. + - To view all contributions for a specific day, select a tile. A list displays all contributions and the time they were made. + +### Show private contributions on your user profile page + +The contributions calendar graph and recent activity list displays your +[contribution actions](#user-contribution-events) to private projects. + +To view private contributions: + +1. On the top bar, in the top-right corner, select your avatar. +1. Select **Edit profile**. +1. In the **Main settings** section, select the **Include private contributions on my profile** checkbox. +1. Select **Update profile settings**. + +## User activity + +### Follow a user's activity + +You can follow users whose activity you're interested in. +In [GitLab 15.5 and later](https://gitlab.com/gitlab-org/gitlab/-/issues/360755), +the maximum number of users you can follow is 300. + +To follow a user, either: + +- From a user's profile, select **Follow**. +- Hover over a user's name, and select **Follow** ([introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/76050) + in GitLab 15.0). + +To view the activity of users you follow: + +1. In the GitLab menu, select **Activity**. +1. Select the **Followed users** tab. + +### Retrieve user activity as a feed + +GitLab provides RSS feeds of user activity. To subscribe to the +RSS feed of a user's activity: + +1. Go to the [user's profile](index.md#access-your-user-profile). +1. In the top right, select the feed symbol **{rss}** to display the results as an RSS feed in Atom format. + +The URL of the result contains both a feed token, and +the user's activity that you're authorized to view. +You can add this URL to your feed reader. + +### Reset the user activity feed token + +Feed tokens are sensitive and can reveal information from confidential issues. +If you think your feed token has been exposed, you should reset it. + +To reset your feed token: + +1. On the top bar, in the top right corner, select your avatar. +1. Select **Edit profile**. +1. On the left sidebar, select **Access Tokens**. +1. Scroll down. In the **Feed token** section, select the + **reset this token** link. +1. On the confirmation box, select **OK**. + +A new token is generated. diff --git a/doc/user/profile/img/contributions_calendar_v15_6.png b/doc/user/profile/img/contributions_calendar_v15_6.png Binary files differnew file mode 100644 index 00000000000..363636ec0cd --- /dev/null +++ b/doc/user/profile/img/contributions_calendar_v15_6.png diff --git a/doc/user/profile/index.md b/doc/user/profile/index.md index 66f6b4b52de..4adf6c351df 100644 --- a/doc/user/profile/index.md +++ b/doc/user/profile/index.md @@ -137,7 +137,7 @@ To add links to other accounts: ## Show private contributions on your user profile page -In the user contribution calendar graph and recent activity list, you can see your [contribution actions](#user-contribution-events) to private projects. +In the user contribution calendar graph and recent activity list, you can see your [contribution actions](contributions_calendar.md#user-contribution-events) to private projects. To show private contributions: @@ -311,7 +311,7 @@ git config --global user.email <your email address> ## User activity -GitLab tracks user contribution activity. You can follow or unfollow other users from either: +GitLab tracks [user contribution activity](contributions_calendar.md). You can follow or unfollow other users from either: - Their [user profiles](#access-your-user-profile). - The small popover that appears when you hover over a user's name ([introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/76050) @@ -326,83 +326,6 @@ To view a user's activity in a top-level Activity view: 1. In the GitLab menu, select **Activity**. 1. Select the **Followed users** tab. -### User contribution events - -Each of these contribution events is tracked: - -- `approved` - - Merge request -- `closed` - - [Epic](../group/epics/index.md) - - Issue - - Merge request - - Milestone -- `commented` on any `Noteable` record. - - Alert - - Commit - - Design - - Issue - - Merge request - - Snippet -- `created` - - Design - - [Epic](../group/epics/index.md) - - Issue - - Merge request - - Milestone - - Project - - Wiki page -- `destroyed` - - Design - - Milestone - - Wiki page -- `expired` - - Project membership -- `joined` - - Project membership -- `left` - - Project membership -- `merged` - - Merge request -- `pushed` commits to (or deleted commits from) a repository, individually or in bulk. - - Project -- `reopened` - - [Epic](../group/epics/index.md) - - Issue - - Merge request - - Milestone -- `updated` - - Design - - Wiki page - -### Retrieve user activity as a feed - -GitLab provides RSS feeds of user activity. To subscribe to the -RSS feed of a user's activity: - -1. Go to the [user's profile](#access-your-user-profile). -1. In the top right, select the feed symbol **{rss}** to display the results as an RSS feed in Atom format. - -The URL of the result contains both a feed token, and -the user's activity that you're authorized to view. -You can add this URL to your feed reader. - -### Reset the user activity feed token - -Feed tokens are sensitive and can reveal information from confidential issues. -If you think your feed token has been exposed, you should reset it. - -To reset your feed token: - -1. On the top bar, in the top right corner, select your avatar. -1. Select **Edit profile**. -1. On the left sidebar, select **Access Tokens**. -1. Scroll down. In the **Feed token** section, select the - **reset this token** link. -1. On the confirmation box, select **OK**. - -A new token is generated. - ## Troubleshooting ### Why do I keep getting signed out? diff --git a/doc/user/profile/notifications.md b/doc/user/profile/notifications.md index 7942ee38be9..1deb4842107 100644 --- a/doc/user/profile/notifications.md +++ b/doc/user/profile/notifications.md @@ -109,7 +109,7 @@ To select a notification level for a group, use either of these methods: Or: 1. On the top bar, select **Main menu > Groups** and find your group. -1. Select the notification dropdown, next to the bell icon (**{notifications}**). +1. Select the notification dropdown list, next to the bell icon (**{notifications}**). 1. Select the desired [notification level](#notification-levels). #### Change email address used for group notifications @@ -140,7 +140,7 @@ To select a notification level for a project, use either of these methods: Or: 1. On the top bar, select **Main menu > Projects** and find your project. -1. Select the notification dropdown, next to the bell icon (**{notifications}**). +1. Select the notification dropdown list, next to the bell icon (**{notifications}**). 1. Select the desired [notification level](#notification-levels). <i class="fa fa-youtube-play youtube" aria-hidden="true"></i> @@ -194,6 +194,7 @@ Users are notified of the following events: | Personal access tokens expiring soon | User | Security email, always sent. | | Personal access tokens have been created | User | Security email, always sent. _[Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/337591) in GitLab 14.9._ | | Personal access tokens have expired | User | Security email, always sent. | +| Personal access token has been revoked | User | Security email, always sent. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/98911) in GitLab 15.5. | | Project access level changed | User | Sent when user project access level is changed. | | SSH key has expired | User | Security email, always sent. _[Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/322637) in GitLab 13.12._ | | Two-factor authentication disabled | User | Security email, always sent. | @@ -227,20 +228,20 @@ to change their user notification settings to **Watch** instead. ### Edit notification settings for issues, merge requests, and epics -To enable notifications on a specific issue, merge request, or epic, you must turn on the -**Notifications** toggle in the right sidebar. +To toggle notifications on an issue, merge request, or epic: on the right sidebar, turn on or off the **Notifications** toggle. -- To subscribe, **turn on** if you are not a participant in the discussion, but want to receive - notifications on each update. +When you **turn on** notifications, you start receiving notifications on each update, even if you +haven't participated in the discussion. +When you turn notifications on in an epic, you aren't automatically subscribed to the issues linked +to the epic. - When you turn notifications on in an epic, you aren't automatically subscribed to the issues linked - to the epic. - -- To unsubscribe, **turn off** if you are receiving notifications for updates but no longer want to - receive them. +When you **turn off** notifications, you stop receiving notifications for updates. +Turning this toggle off only unsubscribes you from updates related to this issue, merge request, or epic. +Learn how to [opt out of all emails from GitLab](#opt-out-of-all-gitlab-emails). - Turning this toggle off only unsubscribes you from updates related to this issue, merge request, or epic. - Learn how to [opt out of all emails from GitLab](#opt-out-of-all-gitlab-emails). +<!-- Delete when the `moved_mr_sidebar` feature flag is removed --> +If you don't see this action on the right sidebar, your project or instance may have +enabled a feature flag for [moved sidebar actions](../project/merge_requests/index.md#move-sidebar-actions). ### Notification events on issues, merge requests, and epics @@ -356,13 +357,13 @@ a merge request or an issue. The following table lists all GitLab-specific email headers: | Header | Description | -|---------------------------------------------------------------|------------------------------------------------------------------------------------------------------------------------------------------------| +| ------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------- | | `List-Id` | The path of the project in an RFC 2919 mailing list identifier. You can use it for email organization with filters. | | `X-GitLab-(Resource)-ID` | The ID of the resource the notification is for. The resource, for example, can be `Issue`, `MergeRequest`, `Commit`, or another such resource. | | `X-GitLab-Discussion-ID` | The ID of the thread the comment belongs to, in notification emails for comments. | | `X-GitLab-Group-Id` | The group's ID. Only present on notification emails for [epics](../group/epics/index.md). | | `X-GitLab-Group-Path` | The group's path. Only present on notification emails for [epics](../group/epics/index.md) | -| [`X-GitLab-NotificationReason`](#x-gitlab-notificationreason) | The reason for the notification. This can be `mentioned`, `assigned`, or `own_activity`. | +| `X-GitLab-NotificationReason` | The reason for the notification. [See possible values.](#x-gitlab-notificationreason). | | `X-GitLab-Pipeline-Id` | The ID of the pipeline the notification is for, in notification emails for pipelines. | | `X-GitLab-Project-Id` | The project's ID. | | `X-GitLab-Project-Path` | The project's path. | @@ -376,21 +377,35 @@ The value is one of the following, in order of priority: - `own_activity` - `assigned` +- `review_requested` - `mentioned` +- `subscribed` The reason for the notification is also included in the footer of the notification email. For example, an email with the reason `assigned` has this sentence in the footer: > You are receiving this email because you have been assigned an item on \<configured GitLab hostname>. -For example, an alert notification email can have one of -[the alert's](../../operations/incident_management/alerts.md) statuses: +#### On-call alerts notifications **(PREMIUM)** + +An [on-call alert](../../operations/incident_management/oncall_schedules.md) +notification email can have one of [the alert's](../../operations/incident_management/alerts.md) statuses: - `alert_triggered` - `alert_acknowledged` - `alert_resolved` - `alert_ignored` +#### Incident escalation notifications **(PREMIUM)** + +An [incident escalation](../../operations/incident_management/escalation_policies.md) +notification email can have one of [the incident's](../../operations/incident_management/incidents.md) status: + +- `incident_triggered` +- `incident_acknowledged` +- `incident_resolved` +- `incident_ignored` + Expanding the list of events included in the `X-GitLab-NotificationReason` header is tracked in [issue 20689](https://gitlab.com/gitlab-org/gitlab/-/issues/20689). diff --git a/doc/user/profile/personal_access_tokens.md b/doc/user/profile/personal_access_tokens.md index c7fe68c0609..507ad6378bc 100644 --- a/doc/user/profile/personal_access_tokens.md +++ b/doc/user/profile/personal_access_tokens.md @@ -89,8 +89,10 @@ At any time, you can revoke a personal access token. ## View the last time a token was used -Token usage is updated once every 24 hours. It is updated each time the token is used to request -[API resources](../../api/api_resources.md) and the [GraphQL API](../../api/graphql/index.md). +Token usage information is updated every 24 hours. GitLab considers a token used when the token is used to: + +- Authenticate with the [REST](../../api/index.md) or [GraphQL](../../api/graphql/index.md) APIs. +- Perform a Git operation. To view the last time a token was used: @@ -195,17 +197,29 @@ This code can be shortened into a single-line shell command using the sudo gitlab-rails runner "PersonalAccessToken.find_by_token('token-string-here123').revoke!" ``` -<!-- ## Troubleshooting +## Troubleshooting + +### Unrevoke a personal access token **(FREE SELF)** + +If a personal access token is revoked accidentally by any method, administrators can unrevoke that token. + +WARNING: +Running the following commands changes data directly. This could be damaging if not done correctly, or under the right conditions. You should first run these commands in a test environment with a backup of the instance ready to be restored, just in case. + +1. Open a [Rails console](../../administration/operations/rails_console.md#starting-a-rails-console-session). +1. Unrevoke the token: -Include any troubleshooting steps that you can foresee. If you know beforehand what issues -one might have when setting this up, or when something is changed, or on upgrading, it's -important to describe those, too. Think of things that may go wrong and include them here. -This is important to minimize requests for support, and to avoid doc comments with -questions that you know someone might ask. + ```ruby + token = PersonalAccessToken.find_by_token('<token_string>') + token.update!(revoked:false) + ``` + + For example, to unrevoke a token of `token-string-here123`: -Each scenario can be a third-level heading, e.g. `### Getting error message X`. -If you have none to add when creating a doc, leave this section in place -but commented out to help encourage others to add to it in the future. --> + ```ruby + token = PersonalAccessToken.find_by_token('token-string-here123') + token.update!(revoked:false) + ``` ## Alternatives to personal access tokens diff --git a/doc/user/profile/preferences.md b/doc/user/profile/preferences.md index 6849918886a..dce8684d993 100644 --- a/doc/user/profile/preferences.md +++ b/doc/user/profile/preferences.md @@ -137,7 +137,7 @@ You can include the following options for your default dashboard view: ### Group overview content -The **Group overview content** dropdown allows you to choose what information is +The **Group overview content** dropdown list allows you to choose what information is displayed on a group's home page. You can choose between 2 options: @@ -230,6 +230,6 @@ important to describe those, too. Think of things that may go wrong and include This is important to minimize requests for support, and to avoid doc comments with questions that you know someone might ask. -Each scenario can be a third-level heading, e.g. `### Getting error message X`. +Each scenario can be a third-level heading, for example `### Getting error message X`. If you have none to add when creating a doc, leave this section in place but commented out to help encourage others to add to it in the future. --> diff --git a/doc/user/profile/user_passwords.md b/doc/user/profile/user_passwords.md index 04d149c9709..b8dbdcdd956 100644 --- a/doc/user/profile/user_passwords.md +++ b/doc/user/profile/user_passwords.md @@ -60,7 +60,8 @@ Self-managed installations can configure the following additional password requi ## Block weak passwords -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/23610) in GitLab 15.4 [with a flag](../../administration/feature_flags.md) named `block_weak_passwords`, weak passwords aren't accepted. Disabled by default. +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/23610) in GitLab 15.4 [with a flag](../../administration/feature_flags.md) named `block_weak_passwords`, weak passwords aren't accepted. Disabled by default on self-managed. +> - [Enabled](https://gitlab.com/gitlab-org/gitlab/-/issues/363445) on GitLab.com. FLAG: On self-managed GitLab, by default blocking weak passwords is not available. To make it available, ask an administrator diff --git a/doc/user/project/badges.md b/doc/user/project/badges.md index d43af92e9c6..5d1d10fc37d 100644 --- a/doc/user/project/badges.md +++ b/doc/user/project/badges.md @@ -128,7 +128,7 @@ To add a new badge to a group or project with a custom image: 1. Select **Add badge**. To learn how to use custom images generated via a pipeline, see our documentation on -[accessing the latest job artifacts by URL](../../ci/pipelines/job_artifacts.md#access-the-latest-job-artifacts-by-url). +[accessing the latest job artifacts by URL](../../ci/pipelines/job_artifacts.md#access-the-latest-job-artifacts). ## API diff --git a/doc/user/project/clusters/add_existing_cluster.md b/doc/user/project/clusters/add_existing_cluster.md index 363197107f9..0b0555806ce 100644 --- a/doc/user/project/clusters/add_existing_cluster.md +++ b/doc/user/project/clusters/add_existing_cluster.md @@ -69,7 +69,7 @@ To add a Kubernetes cluster to your project, group, or instance: 1. Project's **{cloud-gear}** **Infrastructure > Kubernetes clusters** page, for a project-level cluster. 1. Group's **{cloud-gear}** **Kubernetes** page, for a group-level cluster. 1. **Main menu > Admin > Kubernetes** page, for an instance-level cluster. -1. On the **Kubernetes clusters** page, select the **Connect with a certificate** option from the **Actions** dropdown menu. +1. On the **Kubernetes clusters** page, select the **Connect with a certificate** option from the **Actions** dropdown list. 1. On the **Connect a cluster** page, fill in the details: 1. **Kubernetes cluster name** (required) - The name you wish to give the cluster. 1. **Environment scope** (required) - The diff --git a/doc/user/project/clusters/deploy_to_cluster.md b/doc/user/project/clusters/deploy_to_cluster.md index e8acf5a2727..65e4a5d9fde 100644 --- a/doc/user/project/clusters/deploy_to_cluster.md +++ b/doc/user/project/clusters/deploy_to_cluster.md @@ -131,7 +131,7 @@ However, sometimes GitLab cannot create them. In such instances, your job can fa This job failed because the necessary resources were not successfully created. ``` -To find the cause of this error when creating a namespace and service account, check the [logs](../../../administration/logs/index.md#kuberneteslog). +To find the cause of this error when creating a namespace and service account, check the [logs](../../../administration/logs/index.md#kuberneteslog-deprecated). Reasons for failure include: diff --git a/doc/user/project/clusters/kubernetes_pod_logs.md b/doc/user/project/clusters/kubernetes_pod_logs.md deleted file mode 100644 index 51e4f1c2db2..00000000000 --- a/doc/user/project/clusters/kubernetes_pod_logs.md +++ /dev/null @@ -1,12 +0,0 @@ ---- -stage: Monitor -group: Respond -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments -remove_date: '2022-10-18' -redirect_to: '../../clusters/agent/index.md' ---- - -# Kubernetes Logs (removed) **(FREE SELF)** - -This feature was [deprecated](https://gitlab.com/groups/gitlab-org/configure/-/epics/8) in GitLab 14.5 -and [removed](https://gitlab.com/gitlab-org/gitlab/-/issues/360193) in GitLab 15.2. diff --git a/doc/user/project/clusters/runbooks/index.md b/doc/user/project/clusters/runbooks/index.md index c4ca82f7db4..df0ffff8561 100644 --- a/doc/user/project/clusters/runbooks/index.md +++ b/doc/user/project/clusters/runbooks/index.md @@ -166,11 +166,11 @@ the components outlined above and the pre-loaded demo runbook. [GitLab Access Token](../../../profile/personal_access_tokens.md) and your Project ID in the **Setup** section of the demo runbook: - 1. Double-click the **DevOps-Runbook-Demo** folder located on the left panel. + 1. Select the **DevOps-Runbook-Demo** folder located on the left panel.  - 1. Double-click the `Nurtch-DevOps-Demo.ipynb` runbook. + 1. Select the `Nurtch-DevOps-Demo.ipynb` runbook.  diff --git a/doc/user/project/import/clearcase.md b/doc/user/project/import/clearcase.md index 1dc62cbbe35..0878476d59f 100644 --- a/doc/user/project/import/clearcase.md +++ b/doc/user/project/import/clearcase.md @@ -27,7 +27,7 @@ The following table illustrates the main differences between ClearCase and Git: | Server | UNIX, Windows legacy systems | UNIX, macOS | | License | Proprietary | GPL | -_Taken from the slides [ClearCase and the journey to Git](https://docplayer.net/42708453-Clearcase-the-journey-to-git-migrating-your-skills-and-vobs-to-git.html) provided by [collab.net](https://www.collab.net/)_ +_Taken from the slides [ClearCase and the journey to Git](https://docplayer.net/42708453-Clearcase-the-journey-to-git-migrating-your-skills-and-vobs-to-git.html) provided by `collab.net`_ ## Why migrate diff --git a/doc/user/project/import/github.md b/doc/user/project/import/github.md index 03f6fd20b0a..c0b13c76322 100644 --- a/doc/user/project/import/github.md +++ b/doc/user/project/import/github.md @@ -112,8 +112,8 @@ If you are not using the GitHub integration, you can still perform an authorizat 1. Select **Generate token**. 1. Copy the token hash. 1. Go back to GitLab and provide the token to the GitHub importer. -1. Hit the **List Your GitHub Repositories** button and wait while GitLab reads your repositories' information. - Once done, you are taken to the importer page to select the repositories to import. +1. Select **List Your GitHub Repositories** and wait while GitLab reads your repositories' information. + When done, you are taken to the importer page to select the repositories to import. To use a newer personal access token in imports after previously performing these steps, sign out of your GitLab account and sign in again, or revoke the older personal access token in GitHub. @@ -202,18 +202,21 @@ The following items of a project are imported: - Merge Request description. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/18052) in GitLab 15.5. All attachment imports are disabled by default behind - `github_importer_attachments_import` [feature flag](../../../administration/feature_flags.md). From GitLab 15.5, can be imported - [as an additional item](#select-additional-items-to-import). The feature flag was removed. + `github_importer_attachments_import` [feature flag](../../../administration/feature_flags.md). From GitLab 15.5, can + be imported [as an additional item](#select-additional-items-to-import). The feature flag was removed. - Pull request review comments. - Regular issue and pull request comments. - [Git Large File Storage (LFS) Objects](../../../topics/git/lfs/index.md). -- Pull request reviews (GitLab.com and GitLab 13.7 and later). -- Pull request "merged by" information (GitLab.com and GitLab 13.7 and later). -- Pull request comments replies in discussions ([GitLab.com and GitLab 14.5 and later](https://gitlab.com/gitlab-org/gitlab/-/issues/336596)). -- Diff Notes suggestions ([GitLab.com and GitLab 14.7 and later](https://gitlab.com/gitlab-org/gitlab/-/issues/340624)). -- Issue events and pull requests events. [Introduced](https://gitlab.com/groups/gitlab-org/-/epics/7673) in GitLab 15.4 with `github_importer_issue_events_import` - [feature flag](../../../administration/feature_flags.md) disabled by default. - From GitLab 15.5, can be imported [as an additional item](#select-additional-items-to-import). The feature flag was removed. +- Pull request reviews. +- Pull request assigned reviewers. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/355137) in GitLab 15.6. +- Pull request "merged by" information. +- Pull request comments replies in discussions. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/336596) in + GitLab 14.5. +- Diff Notes suggestions. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/340624) in GitLab 14.7. +- Issue events and pull requests events. [Introduced](https://gitlab.com/groups/gitlab-org/-/epics/7673) in GitLab 15.4 + with `github_importer_issue_events_import` [feature flag](../../../administration/feature_flags.md) disabled by default. + From GitLab 15.5, can be imported [as an additional item](#select-additional-items-to-import). The feature flag was + removed. References to pull requests and issues are preserved. Each imported repository maintains visibility level unless that [visibility level is restricted](../../public_access.md#restrict-use-of-public-or-internal-projects), in which case it @@ -225,7 +228,13 @@ Supported GitHub branch protection rules are mapped to GitLab branch protection - GitHub rule **Require conversation resolution before merging** for the project's default branch is mapped to the [**All threads must be resolved** GitLab setting](../../discussions/index.md#prevent-merge-unless-all-threads-are-resolved). [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/371110) in GitLab 15.5. - GitHub rule **Require a pull request before merging** is mapped to the **No one** option in the **Allowed to push** list of the branch protection rule. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/370951) in GitLab 15.5. -- GitHub rule **Require signed commits** for the project's default branch is mapped to the **Reject unsigned commits** GitLab setting. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/370949) in GitLab 15.5. +- GitHub rule **Require a pull request before merging - Require review from Code Owners** is mapped to the + [**Code owner approval** branch protection rule](../protected_branches.md#require-code-owner-approval-on-a-protected-branch). Requires GitLab Premium or higher. + [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/376683) in GitLab 15.6. +- GitHub rule **Require signed commits** for the project's default branch is mapped to the **Reject unsigned commits** GitLab push rule. Requires GitLab Premium or higher. + [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/370949) in GitLab 15.5. +- GitHub rule **Allow force pushes - Everyone** is mapped to the [**Allowed to force push** branch protection rule](../protected_branches.md#allow-force-push-on-a-protected-branch). [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/370943) in GitLab 15.6. +- GitHub rule **Allow force pushes - Specify who can force push** is proposed in issue [370945](https://gitlab.com/gitlab-org/gitlab/-/issues/370945). - Support for GitHub rule **Require status checks to pass before merging** was proposed in issue [370948](https://gitlab.com/gitlab-org/gitlab/-/issues/370948). However, this rule cannot be translated during project import into GitLab due to technical difficulties. You can still create [status checks](../merge_requests/status_checks.md) in GitLab yourself. @@ -263,6 +272,112 @@ To disable the feature, run this command: Feature.disable(:github_importer_lower_per_page_limit, group) ``` +## Import from GitHub Enterprise on an internal network + +If your GitHub Enterprise instance is on a internal network that is unaccessible to the internet, you can use a reverse proxy +to allow GitLab.com to access the instance. + +The proxy needs to: + +- Forward requests to the GitHub Enterprise instance. +- Convert to the public proxy hostname all occurrences of the internal hostname in: + - The API response body. + - The API response `Link` header. + +GitHub API uses the `Link` header for pagination. + +After configuring the proxy, test it by making API requests. Below there are some examples of commands to test the API: + +```shell +curl --header "Authorization: Bearer <YOUR-TOKEN>" "https://{PROXY_HOSTNAME}/user" + +### URLs in the response body should use the proxy hostname + +{ + "login": "example_username", + "id": 1, + "url": "https://{PROXY_HOSTNAME}/users/example_username", + "html_url": "https://{PROXY_HOSTNAME}/example_username", + "followers_url": "https://{PROXY_HOSTNAME}/api/v3/users/example_username/followers", + ... + "created_at": "2014-02-11T17:03:25Z", + "updated_at": "2022-10-18T14:36:27Z" +} +``` + +```shell +curl --head --header "Authorization: Bearer <YOUR-TOKEN>" "https://{PROXY_DOMAIN}/api/v3/repos/{repository_path}/pulls?states=all&sort=created&direction=asc" + +### Link header should use the proxy hostname + +HTTP/1.1 200 OK +Date: Tue, 18 Oct 2022 21:42:55 GMT +Server: GitHub.com +Content-Type: application/json; charset=utf-8 +Cache-Control: private, max-age=60, s-maxage=60 +... +X-OAuth-Scopes: repo +X-Accepted-OAuth-Scopes: +github-authentication-token-expiration: 2022-11-22 18:13:46 UTC +X-GitHub-Media-Type: github.v3; format=json +X-RateLimit-Limit: 5000 +X-RateLimit-Remaining: 4997 +X-RateLimit-Reset: 1666132381 +X-RateLimit-Used: 3 +X-RateLimit-Resource: core +Link: <https://{PROXY_DOMAIN}/api/v3/repositories/1/pulls?page=2>; rel="next", <https://{PROXY_DOMAIN}/api/v3/repositories/1/pulls?page=11>; rel="last" +``` + +Also test that cloning the repository using the proxy does not fail: + +```shell +git clone -c http.extraHeader="Authorization: basic <base64 encode YOUR-TOKEN>" --mirror https://{PROXY_DOMAIN}/{REPOSITORY_PATH}.git +``` + +### Sample reverse proxy configuration + +The following configuration is an example on how to configure Apache HTTP Server as a reverse proxy + +WARNING: +For simplicity, the snippet does not have configuration to encrypt the connection between the client and the proxy. However, for security reasons you should include that +configuration. See [sample Apache TLS/SSL configuration](https://ssl-config.mozilla.org/#server=apache&version=2.4.41&config=intermediate&openssl=1.1.1k&guideline=5.6). + +```plaintext +# Required modules +LoadModule filter_module lib/httpd/modules/mod_filter.so +LoadModule reflector_module lib/httpd/modules/mod_reflector.so +LoadModule substitute_module lib/httpd/modules/mod_substitute.so +LoadModule deflate_module lib/httpd/modules/mod_deflate.so +LoadModule headers_module lib/httpd/modules/mod_headers.so +LoadModule proxy_module lib/httpd/modules/mod_proxy.so +LoadModule proxy_connect_module lib/httpd/modules/mod_proxy_connect.so +LoadModule proxy_http_module lib/httpd/modules/mod_proxy_http.so +LoadModule ssl_module lib/httpd/modules/mod_ssl.so + +<VirtualHost GITHUB_ENTERPRISE_HOSTNAME:80> + ServerName GITHUB_ENTERPRISE_HOSTNAME + + # Enables reverse-proxy configuration with SSL support + SSLProxyEngine On + ProxyPass "/" "https://GITHUB_ENTERPRISE_HOSTNAME/" + ProxyPassReverse "/" "https://GITHUB_ENTERPRISE_HOSTNAME/" + + # Replaces occurrences of the local GitHub Enterprise URL with the Proxy URL + # GitHub Enterprise compresses the responses, the filters INFLATE and DEFLATE needs to be used to + # decompress and compress the response back + AddOutputFilterByType INFLATE;SUBSTITUTE;DEFLATE application/json + Substitute "s|https://GITHUB_ENTERPRISE_HOSTNAME|https://PROXY_HOSTNAME|ni" + SubstituteMaxLineLength 50M + + # GitHub API uses the response header "Link" for the API pagination + # For example: + # <https://example.com/api/v3/repositories/1/issues?page=2>; rel="next", <https://example.com/api/v3/repositories/1/issues?page=3>; rel="last" + # The directive below replaces all occurrences of the GitHub Enterprise URL with the Proxy URL if the + # response header Link is present + Header edit* Link "https://GITHUB_ENTERPRISE_HOSTNAME" "https://PROXY_HOSTNAME" +</VirtualHost> +``` + ## Automate group and project import **(PREMIUM)** For information on automating user, group, and project import API calls, see @@ -279,8 +394,8 @@ repository to be imported manually. Administrators can manually import the repos 1. Run the following series of commands in the console: ```ruby - project_id = <PROJECT_ID> - github_access_token = <GITHUB_ACCESS_TOKEN> + project_id = <PROJECT_ID> + github_access_token = <GITHUB_ACCESS_TOKEN> github_repository_path = '<GROUP>/<REPOSITORY>' github_repository_url = "https://#{github_access_token}@github.com/#{github_repository_path}.git" diff --git a/doc/user/project/import/tfvc.md b/doc/user/project/import/tfvc.md index 3625355340b..0a03513467e 100644 --- a/doc/user/project/import/tfvc.md +++ b/doc/user/project/import/tfvc.md @@ -7,7 +7,7 @@ type: concepts # Migrate from TFVC to Git **(FREE)** -Team Foundation Server (TFS), renamed [Azure DevOps Server](https://azure.microsoft.com/en-us/services/devops/server/) +Team Foundation Server (TFS), renamed [Azure DevOps Server](https://azure.microsoft.com/en-us/products/devops/server/) in 2019, is a set of tools developed by Microsoft which also includes [Team Foundation Version Control](https://learn.microsoft.com/en-us/azure/devops/repos/tfvc/what-is-tfvc?view=azure-devops) (TFVC), a centralized version control system similar to Git. diff --git a/doc/user/project/integrations/gitlab_slack_application.md b/doc/user/project/integrations/gitlab_slack_application.md index 07c99653a0e..8d6fdf882b7 100644 --- a/doc/user/project/integrations/gitlab_slack_application.md +++ b/doc/user/project/integrations/gitlab_slack_application.md @@ -4,35 +4,35 @@ group: Integrations info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- -# GitLab Slack application **(FREE SAAS)** +# GitLab for Slack app **(FREE SAAS)** NOTE: -The GitLab Slack application is only configurable for GitLab.com. It will **not** -work for on-premises installations where you can configure the -[Slack slash commands](slack_slash_commands.md) integration instead. We're planning -to make this configurable for all GitLab installations, but there's -no ETA - see [#28164](https://gitlab.com/gitlab-org/gitlab/-/issues/28164). +The GitLab for Slack app is only configurable for GitLab.com. It does **not** +work for on-premises installations where you can configure +[Slack slash commands](slack_slash_commands.md) instead. See +[Slack application integration for self-managed instances](https://gitlab.com/gitlab-org/gitlab/-/issues/28164) +for our plans to make the app configurable for all GitLab installations. -Slack provides a native application which you can enable via your project's +Slack provides a native application that you can enable with your project's integrations on GitLab.com. ## Slack App Directory -The simplest way to enable the GitLab Slack application for your workspace is to -install the [GitLab application](https://slack-platform.slack.com/apps/A676ADMV5-gitlab) from -the [Slack App Directory](https://slack.com/apps). +To enable the GitLab for Slack app for your workspace, +install the [GitLab application](https://slack-platform.slack.com/apps/A676ADMV5-gitlab) +from the [Slack App Directory](https://slack.com/apps). -Selecting install takes you to the [GitLab Slack application landing page](https://gitlab.com/-/profile/slack/edit) -where you can select a project to enable the GitLab Slack application for. +On the [GitLab for Slack app landing page](https://gitlab.com/-/profile/slack/edit), +you can select a GitLab project to link with your Slack workspace. ## Configuration -Alternatively, you can configure the Slack application with a project's +Alternatively, you can configure the GitLab for Slack app with your project's integration settings. -Keep in mind that you must have the appropriate permissions for your Slack -workspace to be able to install a new application. Read more in Slack's -documentation on [Adding an app to your workspace](https://slack.com/help/articles/202035138-Add-apps-to-your-Slack-workspace). +You must have the appropriate permissions for your Slack +workspace to be able to install a new application. See +[Add apps to your Slack workspace](https://slack.com/help/articles/202035138-Add-apps-to-your-Slack-workspace). To enable the GitLab integration for your Slack workspace: @@ -41,23 +41,21 @@ To enable the GitLab integration for your Slack workspace: 1. Select **Install Slack app**. 1. Select **Allow** on Slack's confirmation screen. -That's all! You can now start using the Slack slash commands. - You can also select **Reinstall Slack app** to update the app in your Slack workspace -to the latest version. See the [Version history](#version-history) for details. +to the latest version. See [Version history](#version-history) for details. ## Create a project alias for Slack To create a project alias on GitLab.com for Slack integration: 1. Go to your project's home page. -1. Go to **Settings > Integrations** (only visible on GitLab.com) +1. Go to **Settings > Integrations** (only visible on GitLab.com). 1. On the **Integrations** page, select **Slack application**. 1. The current **Project Alias**, if any, is displayed. To edit this value, select **Edit**. 1. Enter your desired alias, and select **Save changes**. -Some Slack commands require a project alias, and fail with the following error +Some Slack commands require a project alias and fail with the following error if the project alias is incorrect or missing from the command: ```plaintext @@ -66,17 +64,15 @@ GitLab error: project or alias not found ## Usage -After confirming the installation, you, and everyone else in your Slack workspace, -can use all the [slash commands](../../../integration/slash_commands.md). - -When you perform your first slash command, you are asked to authorize your -Slack user on GitLab.com. +After installing the app, everyone in your Slack workspace can +use the [slash commands](../../../integration/slash_commands.md). +When you perform your first slash command, you are asked to +authorize your Slack user on GitLab.com. The only difference with the [manually configurable Slack slash commands](slack_slash_commands.md) -is that all the commands should be prefixed with the `/gitlab` keyword. - -For example, to show the issue number `1001` under the `gitlab-org/gitlab` -project, you would do: +is that you must prefix all commands with the `/gitlab` keyword. For example, +to show the issue number `1001` under the `gitlab-org/gitlab` +project, you must run the following command: ```plaintext /gitlab gitlab-org/gitlab issue show 1001 @@ -84,15 +80,11 @@ project, you would do: ## Version history -### 15.0+ - -In GitLab 15.0 the Slack app is updated to [Slack's new granular permissions app model](https://medium.com/slack-developer-blog/more-precision-less-restrictions-a3550006f9c3). - -There is no change in functionality. A reinstall is not required but recommended. +In GitLab 15.0 and later, the GitLab for Slack app is updated to [Slack's new granular permissions model](https://medium.com/slack-developer-blog/more-precision-less-restrictions-a3550006f9c3). While there is no change in functionality, you should reinstall the app. ## Troubleshooting -When you work with the Slack app, the +When you work with the GitLab for Slack app, the [App Home](https://api.slack.com/start/overview#app_home) might not display properly. As a workaround, ensure your app is up to date. diff --git a/doc/user/project/integrations/hangouts_chat.md b/doc/user/project/integrations/hangouts_chat.md index 1be0db223ac..c6e102b1d74 100644 --- a/doc/user/project/integrations/hangouts_chat.md +++ b/doc/user/project/integrations/hangouts_chat.md @@ -28,7 +28,7 @@ notifications to Google Chat: To enable the integration in Google Chat: 1. Enter the room where you want to receive notifications from GitLab. -1. Open the room dropdown menu on the top-left and select **Manage webhooks**. +1. Open the room dropdown list on the top-left and select **Manage webhooks**. 1. Enter the name for your webhook, for example "GitLab integration". 1. Optional. Add an avatar for your bot. 1. Select **Save**. diff --git a/doc/user/project/integrations/mlflow_client.md b/doc/user/project/integrations/mlflow_client.md new file mode 100644 index 00000000000..82bfd08e926 --- /dev/null +++ b/doc/user/project/integrations/mlflow_client.md @@ -0,0 +1,81 @@ +--- +stage: Create +group: Incubation +info: Machine Learning Experiment Tracking is a GitLab Incubation Engineering program. No technical writer assigned to this group. +--- + +# MLFlow Client Integration **(FREE)** + +> [Introduced](https://gitlab.com/groups/gitlab-org/-/epics/8560) in GitLab 15.6 as an [Alpha](../../../policy/alpha-beta-support.md#alpha-features) release [with a flag](../../../administration/feature_flags.md) named `ml_experiment_tracking`. Disabled by default. + +DISCLAIMER: +MLFlow Client Integration is an experimental feature being developed by the Incubation Engineering Department, +and will receive significant changes over time. + +[MLFlow](https://mlflow.org/) is one of the most popular open source tools for Machine Learning Experiment Tracking. +GitLabs works as a backend to the MLFlow Client, [logging experiments](../ml/experiment_tracking/index.md). +Setting up your integrations requires minimal changes to existing code. + +GitLab plays the role of proxy server, both for artifact storage and tracking data. It reflects the +MLFlow [Scenario 5](https://www.mlflow.org/docs/latest/tracking.html#scenario-5-mlflow-tracking-server-enabled-with-proxied-artifact-storage-access). + +## Enable MFlow Client Integration + +Complete this task to enable MFlow Client Integration. + +Prerequisites: + +- A [personal access token](../../../user/profile/personal_access_tokens.md) for the project, with minimum access level of `api`. +- The project ID. To find the project ID, on the top bar, select **Main menu > Projects** and find your project. On the left sidebar, select **Settings > General**. + +1. Set the tracking URI and token environment variables on the host that runs the code (your local environment, CI pipeline, or remote host). + + For example: + + ```shell + export MLFLOW_TRACKING_URI="http://<your gitlab endpoint>/api/v4/projects/<your project id>/ml/mlflow" + export MLFLOW_TRACKING_TOKEN="<your_access_token>" + ``` + +1. If your training code contains the call to `mlflow.set_tracking_uri()`, remove it. + +When running the training code, MLFlow will create experiments, runs, log parameters, metrics, +and artifacts on GitLab. + +After experiments are logged, they are listed under `/<your project>/-/ml/experiments`. Runs are registered as Model Candidates, +that can be explored by selecting an experiment. + +## Limitations + +- The API GitLab supports is the one defined at MLFlow version 1.28.0. +- API endpoints not listed above are not supported. +- During creation of experiments and runs, tags are ExperimentTags and RunTags are ignored. +- MLFLow Model Registry is not supported. + +## Supported methods and caveats + +This is a list of methods we support from the MLFlow client. Other methods might be supported but were not +tested. More information can be found in the [MLFlow Documentation](https://www.mlflow.org/docs/1.28.0/python_api/mlflow.html). + +### `set_experiment` + +Accepts both experiment_name and experiment_id + +### `start_run()` + +- Nested runs have not been tested. +- `run_name` is not supported + +### `log_param()`, `log_params()`, `log_metric()`, `log_metrics()` + +Work as defined by the documentation + +### `log_artifact()`, `log_artifacts()` + +`artifact_path` must be empty string. + +### `log_model()` + +This is an experimental method in MLFlow, and partial support is offered. It stores the model artifacts, but does +not log the model information. The `artifact_path` parameter must be set to `''`, because Generic Packages do not support folder +structure. diff --git a/doc/user/project/integrations/prometheus_library/nginx_ingress.md b/doc/user/project/integrations/prometheus_library/nginx_ingress.md index 99466a67417..947210541f4 100644 --- a/doc/user/project/integrations/prometheus_library/nginx_ingress.md +++ b/doc/user/project/integrations/prometheus_library/nginx_ingress.md @@ -38,7 +38,7 @@ Next, the Ingress needs to be annotated for Prometheus monitoring. Two new annot - `prometheus.io/scrape: "true"` - `prometheus.io/port: "10254"` -Managing these settings depends on how NGINX Ingress has been deployed. If you have deployed via the [official Helm chart](https://github.com/helm/charts/tree/master/stable/nginx-ingress), metrics can be enabled with `controller.stats.enabled` along with the required annotations. Alternatively it is possible to edit the NGINX Ingress YAML directly in the [Kubernetes dashboard](https://github.com/kubernetes/dashboard). +Managing these settings depends on how NGINX Ingress has been deployed. If you have deployed via the [official Helm chart](https://github.com/kubernetes/ingress-nginx/tree/master/charts/ingress-nginx), metrics can be enabled with `controller.stats.enabled` along with the required annotations. Alternatively it is possible to edit the NGINX Ingress YAML directly in the [Kubernetes dashboard](https://github.com/kubernetes/dashboard). ## Specifying the Environment label diff --git a/doc/user/project/integrations/prometheus_library/nginx_ingress_vts.md b/doc/user/project/integrations/prometheus_library/nginx_ingress_vts.md index e26f93351a1..e6f2ac2753a 100644 --- a/doc/user/project/integrations/prometheus_library/nginx_ingress_vts.md +++ b/doc/user/project/integrations/prometheus_library/nginx_ingress_vts.md @@ -38,7 +38,7 @@ Next, the Ingress needs to be annotated for Prometheus monitoring. Two new annot - `prometheus.io/scrape: "true"` - `prometheus.io/port: "10254"` -Managing these settings depends on how NGINX Ingress has been deployed. If you have deployed via the [official Helm chart](https://github.com/helm/charts/tree/master/stable/nginx-ingress), metrics can be enabled with `controller.stats.enabled` along with the required annotations. Alternatively it is possible edit the NGINX Ingress YAML directly in the [Kubernetes dashboard](https://github.com/kubernetes/dashboard). +Managing these settings depends on how NGINX Ingress has been deployed. If you have deployed via the [official Helm chart](https://github.com/kubernetes/ingress-nginx/tree/master/charts/ingress-nginx), metrics can be enabled with `controller.stats.enabled` along with the required annotations. Alternatively it is possible edit the NGINX Ingress YAML directly in the [Kubernetes dashboard](https://github.com/kubernetes/dashboard). ## Specifying the Environment label diff --git a/doc/user/project/integrations/servicenow.md b/doc/user/project/integrations/servicenow.md index d528d1a5547..fcc8db7cb87 100644 --- a/doc/user/project/integrations/servicenow.md +++ b/doc/user/project/integrations/servicenow.md @@ -35,7 +35,5 @@ You can: For more information, refer to the following ServiceNow resources: - [ServiceNow DevOps home page](https://www.servicenow.com/products/devops.html) -- [Install DevOps](https://docs.servicenow.com/bundle/paris-devops/page/product/enterprise-dev-ops/task/activate-dev-ops.html) -- [Install DevOps Integrations](https://docs.servicenow.com/bundle/paris-devops/page/product/enterprise-dev-ops/task/activate-dev-ops-integrations.html) +- [ServiceNow DevOps documentation](https://docs.servicenow.com/bundle/tokyo-devops/page/product/enterprise-dev-ops/concept/dev-ops-bundle-landing-page.html) - [GitLab SCM and Continuous Integration for DevOps](https://store.servicenow.com/sn_appstore_store.do#!/store/application/54dc4eacdbc2dcd02805320b7c96191e/) -- [Model a GitLab CI pipeline in DevOps](https://docs.servicenow.com/bundle/paris-devops/page/product/enterprise-dev-ops/task/model-gitlab-pipeline-dev-ops.html). diff --git a/doc/user/project/integrations/slack.md b/doc/user/project/integrations/slack.md index 9fe0c76ec4f..d34c558ebbc 100644 --- a/doc/user/project/integrations/slack.md +++ b/doc/user/project/integrations/slack.md @@ -4,9 +4,9 @@ group: Integrations info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- -# Slack notifications service **(FREE)** +# Slack notifications integration **(FREE)** -The Slack notifications service enables your GitLab project to send events +The Slack notifications integration enables your GitLab project to send events (such as issue creation) to your existing Slack team as notifications. Setting up Slack notifications requires configuration changes for both Slack and GitLab. @@ -45,7 +45,7 @@ to control GitLab from Slack. Slash commands are configured separately. 1. Optional. In **Username**, enter the username of the Slack bot that sends the notifications. 1. Select the **Notify only broken pipelines** checkbox to notify only on failures. -1. In the **Branches for which notifications are to be sent** dropdown, select which types of branches +1. In the **Branches for which notifications are to be sent** dropdown list, select which types of branches to send notifications for. 1. Leave the **Labels to be notified** field blank to get all notifications, or add labels that the issue or merge request must have to trigger a @@ -91,7 +91,7 @@ the error message and keep troubleshooting from there. You might see an entry like the following in your Sidekiq log: ```plaintext -2019-01-10_13:22:08.42572 2019-01-10T13:22:08.425Z 6877 TID-abcdefg Integrations::ExecuteWorker JID-3bade5fb3dd47a85db6d78c5 ERROR: {:class=>"Integrations::ExecuteWorker :service_class=>"SlackService", :message=>"SSL_connect returned=1 errno=0 state=error: certificate verify failed"} +2019-01-10_13:22:08.42572 2019-01-10T13:22:08.425Z 6877 TID-abcdefg Integrations::ExecuteWorker JID-3bade5fb3dd47a85db6d78c5 ERROR: {:class=>"Integrations::ExecuteWorker :integration_class=>"SlackService", :message=>"SSL_connect returned=1 errno=0 state=error: certificate verify failed"} ``` This issue occurs when there is a problem with GitLab communicating with Slack, @@ -128,20 +128,20 @@ the GitLab OpenSSL trust store is incorrect. Typical causes are: - Overriding the trust store with `gitlab_rails['env'] = {"SSL_CERT_FILE" => "/path/to/file.pem"}`. - Accidentally modifying the default CA bundle `/opt/gitlab/embedded/ssl/certs/cacert.pem`. -### Bulk update to disable the Slack Notification service +### Bulk update to disable the Slack Notification integration To disable notifications for all projects that have Slack integration enabled, [start a rails console session](../../../administration/operations/rails_console.md#starting-a-rails-console-session) and use a script similar to the following: WARNING: -Any command that changes data directly could be damaging if not run correctly, or under the right conditions. We highly recommend running them in a test environment with a backup of the instance ready to be restored, just in case. +Commands that change data can cause damage if not run correctly or under the right conditions. Always run commands in a test environment first and have a backup instance ready to restore. ```ruby # Grab all projects that have the Slack notifications enabled p = Project.find_by_sql("SELECT p.id FROM projects p LEFT JOIN integrations s ON p.id = s.project_id WHERE s.type_new = 'Slack' AND s.active = true") -# Disable the service on each of the projects that were found. +# Disable the integration on each of the projects that were found. p.each do |project| - project.slack_service.update!(:active, false) + project.slack_integration.update!(:active, false) end ``` diff --git a/doc/user/project/integrations/slack_slash_commands.md b/doc/user/project/integrations/slack_slash_commands.md index cb698ac0ee0..f4c789239f3 100644 --- a/doc/user/project/integrations/slack_slash_commands.md +++ b/doc/user/project/integrations/slack_slash_commands.md @@ -14,7 +14,7 @@ GitLab can also send events (for example, `issue created`) to Slack as notificat The [Slack notifications service](slack.md) is configured separately. NOTE: -For GitLab.com, use the [GitLab Slack app](gitlab_slack_application.md) instead. +For GitLab.com, use the [GitLab for Slack app](gitlab_slack_application.md) instead. ## Configure GitLab and Slack diff --git a/doc/user/project/integrations/unify_circuit.md b/doc/user/project/integrations/unify_circuit.md index c13f642d9e9..77530b4b417 100644 --- a/doc/user/project/integrations/unify_circuit.md +++ b/doc/user/project/integrations/unify_circuit.md @@ -22,7 +22,7 @@ In GitLab: 1. Select the checkboxes corresponding to the GitLab events you want to receive in Unify Circuit. 1. Paste the **Webhook URL** that you copied from the Unify Circuit configuration step. 1. Select the **Notify only broken pipelines** checkbox to notify only on failures. -1. In the **Branches for which notifications are to be sent** dropdown, select which types of branches to send notifications for. +1. In the **Branches for which notifications are to be sent** dropdown list, select which types of branches to send notifications for. 1. Select `Save changes` or optionally select **Test settings**. Your Unify Circuit conversation now starts receiving GitLab event notifications. diff --git a/doc/user/project/integrations/webex_teams.md b/doc/user/project/integrations/webex_teams.md index 930ca8e99b8..be4528ba70d 100644 --- a/doc/user/project/integrations/webex_teams.md +++ b/doc/user/project/integrations/webex_teams.md @@ -13,7 +13,7 @@ You can configure GitLab to send notifications to a Webex Teams space: ## Create a webhook for the space -1. Go to the [Incoming Webhooks app page](https://apphub.webex.com/applications/incoming-webhooks-cisco-systems-38054-23307). +1. Go to the [Incoming Webhooks app page](https://apphub.webex.com/applications/incoming-webhooks-cisco-systems-38054-23307-75252). 1. Select **Connect**, and sign in to Webex Teams if required. 1. Enter a name for the webhook and select the space to receive the notifications. 1. Select **ADD**. diff --git a/doc/user/project/integrations/webhook_events.md b/doc/user/project/integrations/webhook_events.md index c0f0f5a0cd4..60187b9a682 100644 --- a/doc/user/project/integrations/webhook_events.md +++ b/doc/user/project/integrations/webhook_events.md @@ -611,7 +611,8 @@ Payload example: "name": "User1", "username": "user1", "avatar_url": "http://www.gravatar.com/avatar/e64c7d89f26bd1972efa854d13d7dd61?s=40\u0026d=identicon" - } + }, + "detailed_merge_status": "checking" } } ``` @@ -947,7 +948,8 @@ Payload example: "type": "ProjectLabel", "group_id": 41 }], - "action": "open" + "action": "open", + "detailed_merge_status": "mergeable" }, "labels": [{ "id": 206, @@ -1017,7 +1019,7 @@ Payload example: ``` NOTE: -The fields `assignee_id`, and `state` are deprecated. +The fields `assignee_id`, `state`, `merge_status` are deprecated. ## Wiki page events @@ -1132,6 +1134,7 @@ Payload example: "target_project_id": 1, "state": "opened", "merge_status": "can_be_merged", + "detailed_merge_status": "mergeable", "url": "http://192.168.64.1:3005/gitlab-org/gitlab-test/merge_requests/1" }, "user":{ @@ -1359,6 +1362,15 @@ Payload example: ## Job events +- Number of retries [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/382046) in GitLab 15.6 [with a flag](../../../administration/feature_flags.md) + named `job_webhook_retries_count`. Disabled by default. + +FLAG: +On self-managed GitLab, by default this feature is not available. To make it available, +ask an administrator to [enable the feature flag](../../../administration/feature_flags.md) named +`job_webhook_retries_count`. +On GitLab.com, this feature is not available. + Job events are triggered when the status of a job changes. The `commit.id` in the payload is the ID of the pipeline, not the ID of the commit. @@ -1391,6 +1403,7 @@ Payload example: "build_duration": null, "build_allow_failure": false, "build_failure_reason": "script_failure", + "retries_count": 2, // 2 indicates this is the 2nd retry of this job "pipeline_id": 2366, "project_id": 380, "project_name": "gitlab-org/gitlab-test", diff --git a/doc/user/project/integrations/webhooks.md b/doc/user/project/integrations/webhooks.md index 9fc9d6e2eda..ef6957ac2d8 100644 --- a/doc/user/project/integrations/webhooks.md +++ b/doc/user/project/integrations/webhooks.md @@ -115,6 +115,15 @@ a test request to re-enable a [disabled webhook](#re-enable-disabled-webhooks). For example, to test `push events`, your project should have at least one commit. The webhook uses this commit in the webhook. +NOTE: +Testing is not supported for some types of events for project and groups webhooks. +Read more in [issue 379201](https://gitlab.com/gitlab-org/gitlab/-/issues/379201). + +Prerequisites: + +- To test project webhooks, you must have at least the Maintainer role for the project. +- To test group webhooks, you must have the Owner role for the group. + To test a webhook: 1. In your project or group, on the left sidebar, select **Settings > Webhooks**. @@ -177,9 +186,13 @@ that the request is legitimate. ## Filter push events by branch -Push events can be filtered by branch using a branch name or wildcard pattern -to limit which push events are sent to your webhook endpoint. By default, -all push events are sent to your webhook endpoint. You can configure branch filtering +You can filter push events by branch. Use one of the following options to filter which push events are sent to your webhook endpoint: + +- **All branches**: push events from all branches. +- **Wildcard pattern**: push events from a branch that matches a wildcard pattern (for example, `*-stable` or `production/*`). +- **Regular expression**: push events from a branch that matches a regular expression (for example, `(feature|hotfix)/*`). + +You can configure branch filtering in the [webhook settings](#configure-a-webhook-in-gitlab) in your project. ## How image URLs are displayed in the webhook body @@ -233,6 +246,11 @@ Webhook requests to your endpoint include the following headers: GitLab records the history of each webhook request. You can view requests made in the last 2 days in the **Recent events** table. +Prerequisites: + +- To troubleshoot project webhooks, you must have at least the Maintainer role for the project. +- To troubleshoot group webhooks, you must have the Owner role for the group. + To view the table: 1. In your project or group, on the left sidebar, select **Settings > Webhooks**. diff --git a/doc/user/project/issue_board.md b/doc/user/project/issue_board.md index 1cf902d2290..c2952b23615 100644 --- a/doc/user/project/issue_board.md +++ b/doc/user/project/issue_board.md @@ -707,6 +707,14 @@ A few things to remember: ## Troubleshooting issue boards +### `There was a problem fetching users` on group issue board when filtering by Author or Assignee + +If you get a banner with `There was a problem fetching users` error when filtering by author or assignee on +group issue board, make sure that you are added as a member to the current group. +Non-members do not have permission to list group members when filtering by author or assignee on issue boards. + +To fix this error, you should add all of your users to the top-level group with at least the Guest role. + ### Use Rails console to fix issue boards not loading and timing out If you see issue board not loading and timing out in UI, use Rails console to call the Issue Rebalancing service to fix it: diff --git a/doc/user/project/issues/confidential_issues.md b/doc/user/project/issues/confidential_issues.md index b1bb3f0dbf8..2b302a60d63 100644 --- a/doc/user/project/issues/confidential_issues.md +++ b/doc/user/project/issues/confidential_issues.md @@ -16,9 +16,9 @@ keep security vulnerabilities private or prevent surprises from leaking out. You can make an issue confidential when you create or edit an issue. When you create a new issue, a checkbox right below the text area is available -to mark the issue as confidential. Check that box and hit the **Create issue** -button to create the issue. For existing issues, edit them, check the -confidential checkbox and hit **Save changes**. +to mark the issue as confidential. Check that box and select **Create issue** +to create the issue. For existing issues, edit them, check the +confidential checkbox and select **Save changes**. When you create a confidential issue in a project, the project becomes listed in the **Contributed projects** section in your [profile](../../profile/index.md). **Contributed projects** does not show information about the confidential issue; it only shows the project name. diff --git a/doc/user/project/issues/design_management.md b/doc/user/project/issues/design_management.md index 593557967ed..27d935d0ed1 100644 --- a/doc/user/project/issues/design_management.md +++ b/doc/user/project/issues/design_management.md @@ -4,7 +4,7 @@ group: Product Planning info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- -# Design Management **(FREE)** +# Design management **(FREE)** > - [Introduced](https://gitlab.com/groups/gitlab-org/-/epics/660) in GitLab 12.2. > - Support for SVGs [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/12771) in GitLab 12.4. diff --git a/doc/user/project/issues/due_dates.md b/doc/user/project/issues/due_dates.md index 6293fe981de..f41c90a74a5 100644 --- a/doc/user/project/issues/due_dates.md +++ b/doc/user/project/issues/due_dates.md @@ -35,7 +35,7 @@ The last way to set a due date is by using [quick actions](../quick_actions.md), You can see issues with their due dates in the issues list. Overdue issues have their icon and date colored red. -To sort issues by their due dates, select **Due date** from the dropdown menu on the right. +To sort issues by their due dates, select **Due date** from the dropdown list on the right. Issues are then sorted from the earliest due date to the latest. To display issues with the latest due dates at the top, select **Sort direction** (**{sort-lowest}**). diff --git a/doc/user/project/issues/managing_issues.md b/doc/user/project/issues/managing_issues.md index 9ec6d5b300c..8cd211a51c7 100644 --- a/doc/user/project/issues/managing_issues.md +++ b/doc/user/project/issues/managing_issues.md @@ -207,7 +207,8 @@ the appropriate project and followed up from there. ### Fields in the new issue form -> Adding the new issue to an epic [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/13847) in GitLab 13.1. +> - Adding the new issue to an epic [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/13847) in GitLab 13.1. +> - Iteration field [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/233517) in GitLab 15.6. When you're creating a new issue, you can complete the following fields: @@ -222,6 +223,7 @@ When you're creating a new issue, you can complete the following fields: - [Due date](due_dates.md) - [Milestone](../milestones/index.md) - [Labels](../labels.md) +- [Iteration](../../group/iterations/index.md) ## Edit an issue @@ -340,6 +342,29 @@ To move an issue: ### Bulk move issues **(FREE SELF)** +#### From the issues list + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/15991) in GitLab 15.6. + +You can move multiple issues at the same time when you’re in a project. +You can't move tasks or test cases. + +Prerequisite: + +- You must have at least the Reporter role for the project. + +To move multiple issues at the same time: + +1. On the top bar, select **Main menu > Projects** and find your project. +1. On the left sidebar, select **Issues**. +1. Select **Edit issues**. A sidebar on the right of your screen appears. +1. Select the checkboxes next to each issue you want to move. +1. From the right sidebar, select **Move selected**. +1. From the dropdown list, select the destination project. +1. Select **Move**. + +#### From the Rails console + You can move all open issues from one project to another. Prerequisites: @@ -586,7 +611,7 @@ To filter the list of issues: 1. Select or type the operator to use for filtering the attribute. The following operators are available: - `=`: Is - - `!=`: Is not ([Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/18059) in GitLab 12.7) + - `!=`: Is not one of 1. Enter the text to filter the attribute by. You can filter some attributes by **None** or **Any**. 1. Repeat this process to filter by multiple attributes. Multiple attributes are joined by a logical @@ -595,6 +620,21 @@ To filter the list of issues: GitLab displays the results on-screen, but you can also [retrieve them as an RSS feed](../../search/index.md#retrieve-search-results-as-feed). +### Filter with the OR operator + +> OR filtering for assignees was [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/23532) in GitLab 15.6 [with a flag](../../../administration/feature_flags.md) named `or_issuable_queries`. Disabled by default. + +FLAG: +On self-managed GitLab, by default this feature is not available. +To make it available, ask an administrator to [enable the feature flag](../../../administration/feature_flags.md) named `or_issuable_queries`. +The feature is not ready for production use. + +When this feature is enabled, you can use the OR operator (**is one of: `||`**) +when you [filter the list of issues](#filter-the-list-of-issues). + +`is one of` represents an inclusive OR. For example, if you filter by `Assignee is one of Sidney Jones` and +`Assignee is one of Zhang Wei`, GitLab shows issues where either Sidney, Zhang, or both of them are assignees. + ### Filter issues by ID > [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/39908) in GitLab 12.1. @@ -648,7 +688,7 @@ you can see the change without having to refresh the page. The following sections are updated in real time: - [Assignee](#assignee) -- Labels, [if enabled](../labels.md#real-time-changes-to-labels) +- [Labels](../labels.md#assign-and-unassign-labels) ## Assignee diff --git a/doc/user/project/issues/related_issues.md b/doc/user/project/issues/related_issues.md index 53ad7196920..1a8f19b80ba 100644 --- a/doc/user/project/issues/related_issues.md +++ b/doc/user/project/issues/related_issues.md @@ -1,6 +1,6 @@ --- stage: Plan -group: Project Management +group: Product Planning info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- diff --git a/doc/user/project/labels.md b/doc/user/project/labels.md index 826e0b21ea7..bb72ab0052d 100644 --- a/doc/user/project/labels.md +++ b/doc/user/project/labels.md @@ -28,10 +28,21 @@ You can use two types of labels in GitLab: ## Assign and unassign labels -> Unassigning labels with the **X** button [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/216881) in GitLab 13.5. +> - Unassigning labels with the **X** button [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/216881) in GitLab 13.5. +> - Real-time updates in the sidebar [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/241538) in GitLab 14.10 with a [feature flag](../../administration/feature_flags.md) named `realtime_labels`, disabled by default. +> - Real-time updates in the sidebar [enabled on GitLab.com](https://gitlab.com/gitlab-org/gitlab/-/issues/357370#note_991987201) in GitLab 15.1. +> - Real-time updates in the sidebar [enabled by default](https://gitlab.com/gitlab-org/gitlab/-/issues/357370) in GitLab 15.5. +> - Real-time updates in the sidebar [generally available](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/103199) in GitLab 15.6. Feature flag `realtime_labels` removed. You can assign labels to any issue, merge request, or epic. +Changed labels are immediately visible to other users, without refreshing the page, on the following: + +- Epics +- Incidents +- Issues +- Merge requests + To assign or unassign a label: 1. In the **Labels** section of the sidebar, select **Edit**. @@ -410,7 +421,7 @@ To subscribe to a label: 1. To the right of any label, select **Subscribe**. 1. Optional. If you are subscribing to a group label from a project, select either: - **Subscribe at project level** to be notified about events in this project. - - **Subscribe at group level**: to be notified about events in the whole group. + - **Subscribe at group level** to be notified about events in the whole group. ## Set label priority @@ -444,23 +455,6 @@ The labels higher in the list get higher priority. To learn what happens when you sort by priority or label priority, see [Sorting and ordering issue lists](issues/sorting_issue_lists.md). -## Real-time changes to labels - -> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/241538) in GitLab 14.10 with a [feature flag](../../administration/feature_flags.md) named `realtime_labels`, disabled by default. -> - [Enabled on GitLab.com](https://gitlab.com/gitlab-org/gitlab/-/issues/357370#note_991987201) in GitLab 15.1. -> - [Enabled by default](https://gitlab.com/gitlab-org/gitlab/-/issues/357370) in GitLab 15.5. - -FLAG: -On self-managed GitLab, to prevent updating labels in real-time, you can ask an administrator to [disable the feature flag](../../administration/feature_flags.md) named `realtime_labels`. -On GitLab.com, this feature is available. - -Changed labels are immediately visible to other users, without refreshing the page, on the following: - -- Epics -- Incidents -- Issues -- Merge requests - ## Troubleshooting ### Some label titles end with `_duplicate<number>` diff --git a/doc/user/project/members/index.md b/doc/user/project/members/index.md index a8f1b634127..e8ec954df8f 100644 --- a/doc/user/project/members/index.md +++ b/doc/user/project/members/index.md @@ -45,26 +45,14 @@ flowchart RL > - [Changed](https://gitlab.com/gitlab-org/gitlab/-/issues/247208) in GitLab 13.11 from a form to a modal window [with a flag](../../feature_flags.md). Disabled by default. > - Modal window [enabled on GitLab.com and self-managed](https://gitlab.com/gitlab-org/gitlab/-/issues/247208) in GitLab 14.8. -> - [Generally available](https://gitlab.com/gitlab-org/gitlab/-/issues/352526) in GitLab 14.9. - [Feature flag `invite_members_group_modal`](https://gitlab.com/gitlab-org/gitlab/-/issues/352526) removed. +> - [Generally available](https://gitlab.com/gitlab-org/gitlab/-/issues/352526) in GitLab 14.9. [Feature flag `invite_members_group_modal`](https://gitlab.com/gitlab-org/gitlab/-/issues/352526) removed. Add users to a project so they become members and have permission to perform actions. -The maximum role (access level) you set depends on if you have the Owner or Maintainer role for the group. For example, the maximum -role that can be set is: - -- Owner (`50`), if you have the Owner role for the project. -- Maintainer (`40`), if you have the Maintainer role on the project. - -In GitLab 14.8 and earlier, direct members of a project have a maximum role of Maintainer. -The Owner [role](../../permissions.md#project-members-permissions) can only be added at the group level. - Prerequisite: -- You must have the Maintainer or Owner role: - - To remove direct members with the Maintainer role and below, you must have the Maintainer role. - - To remove members with the Owner role, you must have the Owner role. +- You must have the Owner or Maintainer role. To add a user to a project: @@ -73,7 +61,7 @@ To add a user to a project: 1. Select **Invite members**. 1. Enter an email address and select a [role](../../permissions.md). 1. Optional. Select an **Access expiration date**. - On that date, the user can no longer access the project. + From that date onwards, the user can no longer access the project. 1. Select **Invite**. If the user has a GitLab account, they are added to the members list. @@ -86,12 +74,22 @@ deleted after 90 days. If the user does not have a GitLab account, they are prompted to create an account using the email address the invitation was sent to. +### Which roles you can assign + +The maximum role you can assign depends on whether you have the Owner or Maintainer +role for the group. For example, the maximum role you can set is: + +- Owner (`50`), if you have the Owner role for the project. +- Maintainer (`40`), if you have the Maintainer role on the project. + +In GitLab 14.8 and earlier, direct members of a project have a maximum role of Maintainer. +The Owner [role](../../permissions.md#project-members-permissions) can be added for the group only. + ## Add groups to a project > - [Changed](https://gitlab.com/gitlab-org/gitlab/-/issues/247208) in GitLab 13.11 from a form to a modal window [with a flag](../../feature_flags.md). Disabled by default. > - Modal window [enabled on GitLab.com and self-managed](https://gitlab.com/gitlab-org/gitlab/-/issues/247208) in GitLab 14.8. -> - [Generally available](https://gitlab.com/gitlab-org/gitlab/-/issues/352526) in GitLab 14.9. - [Feature flag `invite_members_group_modal`](https://gitlab.com/gitlab-org/gitlab/-/issues/352526) removed. +> - [Generally available](https://gitlab.com/gitlab-org/gitlab/-/issues/352526) in GitLab 14.9. [Feature flag `invite_members_group_modal`](https://gitlab.com/gitlab-org/gitlab/-/issues/352526) removed. When you add a group to a project, each user in the group gets access to the project. Each user's access is based on: @@ -99,19 +97,20 @@ Each user's access is based on: - The role they're assigned in the group. - The maximum role you choose when you invite the group. -Prerequisite: +Prerequisites: - You must have the Maintainer or Owner role. - Sharing the project with other groups must not be [prevented](../../group/access_and_permissions.md#prevent-a-project-from-being-shared-with-groups). -To add groups to a project: +To add a group to a project: 1. On the top bar, select **Main menu > Projects** and find your project. 1. On the left sidebar, select **Project information > Members**. 1. Select **Invite a group**. 1. Select a group. 1. Select the highest [role](../../permissions.md) for users in the group. -1. Optional. Select an **Access expiration date**. On that date, the group can no longer access the project. +1. Optional. Select an **Access expiration date**. + From that date onwards, the group can no longer access the project. 1. Select **Invite**. The members of the group are not displayed on the **Members** tab. @@ -169,7 +168,9 @@ group itself. Prerequisites: -- You must have the Maintainer or Owner role. +- To remove direct members with the: + - Maintainer, Developer, Reporter, or Guest role, you must have the Maintainer role. + - Owner role, you must have the Owner role. - Optional. Unassign the member from all issues and merge requests that are assigned to them. @@ -187,6 +188,21 @@ To remove a member from a project: [from being forked outside their group](../../group/access_and_permissions.md#prevent-project-forking-outside-group). 1. Select **Remove member**. +## Ensure removed users cannot invite themselves back + +Malicious users with the Maintainer or Owner role could exploit a race condition that allows +them to invite themselves back to a group or project that a GitLab administrator has removed them from. + +To avoid this problem, GitLab administrators can: + +- Remove the malicious user session from the [GitLab Rails console](../../../administration/operations/rails_console.md). +- Impersonate the malicious user to: + - Remove the user from the project. + - Log the user out of GitLab. +- Block the malicious user account. +- Remove the malicious user account. +- Change the password for the malicious user account. + ## Filter and sort members > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/21727) in GitLab 12.6. diff --git a/doc/user/project/merge_requests/approvals/index.md b/doc/user/project/merge_requests/approvals/index.md index ea03427161e..eb460225858 100644 --- a/doc/user/project/merge_requests/approvals/index.md +++ b/doc/user/project/merge_requests/approvals/index.md @@ -68,7 +68,7 @@ if a user approves a merge request and is shown in the reviewer list, a green ch After a merge request receives the [number and type of approvals](rules.md) you configure, it can merge unless it's blocked for another reason. Merge requests can be blocked by other problems, -such as merge conflicts, [pending discussions](../../../discussions/index.md#prevent-merge-unless-all-threads-are-resolved), +such as merge conflicts, [unresolved threads](../../../discussions/index.md#prevent-merge-unless-all-threads-are-resolved), or a [failed CI/CD pipeline](../merge_when_pipeline_succeeds.md). To prevent merge request authors from approving their own merge requests, diff --git a/doc/user/project/merge_requests/authorization_for_merge_requests.md b/doc/user/project/merge_requests/authorization_for_merge_requests.md index ba28432e90a..52944ee3143 100644 --- a/doc/user/project/merge_requests/authorization_for_merge_requests.md +++ b/doc/user/project/merge_requests/authorization_for_merge_requests.md @@ -68,6 +68,6 @@ important to describe those, too. Think of things that may go wrong and include This is important to minimize requests for support, and to avoid doc comments with questions that you know someone might ask. -Each scenario can be a third-level heading, e.g. `### Getting error message X`. +Each scenario can be a third-level heading, for example `### Getting error message X`. If you have none to add when creating a doc, leave this section in place but commented out to help encourage others to add to it in the future. --> diff --git a/doc/user/project/merge_requests/changes.md b/doc/user/project/merge_requests/changes.md index 6703cbf8b03..6e8b0cb1a75 100644 --- a/doc/user/project/merge_requests/changes.md +++ b/doc/user/project/merge_requests/changes.md @@ -149,7 +149,7 @@ The feature is not ready for production use. To avoid displaying the changes that are already on target branch in the diff, we compare the merge request's source branch with HEAD of the target branch. -When there are conflicts between the source and target branch, we show the -conflicts on the merge request diff: +When there are conflicts between the source and target branch, we show an alert +per conflicted file on the merge request diff: - + diff --git a/doc/user/project/merge_requests/conflicts.md b/doc/user/project/merge_requests/conflicts.md index 902095bcbce..24f22924a08 100644 --- a/doc/user/project/merge_requests/conflicts.md +++ b/doc/user/project/merge_requests/conflicts.md @@ -172,6 +172,6 @@ important to describe those, too. Think of things that may go wrong and include This is important to minimize requests for support, and to avoid doc comments with questions that you know someone might ask. -Each scenario can be a third-level heading, e.g. `### Getting error message X`. +Each scenario can be a third-level heading, for example `### Getting error message X`. If you have none to add when creating a doc, leave this section in place but commented out to help encourage others to add to it in the future. --> diff --git a/doc/user/project/merge_requests/drafts.md b/doc/user/project/merge_requests/drafts.md index 2beb7406518..0bc9b337e3b 100644 --- a/doc/user/project/merge_requests/drafts.md +++ b/doc/user/project/merge_requests/drafts.md @@ -88,6 +88,6 @@ important to describe those, too. Think of things that may go wrong and include This is important to minimize requests for support, and to avoid doc comments with questions that you know someone might ask. -Each scenario can be a third-level heading, e.g. `### Getting error message X`. +Each scenario can be a third-level heading, for example `### Getting error message X`. If you have none to add when creating a doc, leave this section in place but commented out to help encourage others to add to it in the future. --> diff --git a/doc/user/project/merge_requests/img/conflict_ui_v14_0.png b/doc/user/project/merge_requests/img/conflict_ui_v14_0.png Binary files differdeleted file mode 100644 index 92c532351cb..00000000000 --- a/doc/user/project/merge_requests/img/conflict_ui_v14_0.png +++ /dev/null diff --git a/doc/user/project/merge_requests/img/conflict_ui_v15_6.png b/doc/user/project/merge_requests/img/conflict_ui_v15_6.png Binary files differnew file mode 100644 index 00000000000..d5d5ad14edb --- /dev/null +++ b/doc/user/project/merge_requests/img/conflict_ui_v15_6.png diff --git a/doc/user/project/merge_requests/index.md b/doc/user/project/merge_requests/index.md index e73c339e000..8309b04758a 100644 --- a/doc/user/project/merge_requests/index.md +++ b/doc/user/project/merge_requests/index.md @@ -250,6 +250,28 @@ This feature works only when a merge request is merged. Selecting **Remove sourc after merging does not retarget open merge requests. This improvement is [proposed as a follow-up](https://gitlab.com/gitlab-org/gitlab/-/issues/321559). +## Move sidebar actions + +<!-- When the `moved_mr_sidebar` feature flag is removed, delete this topic and update the steps for these actions +like in https://gitlab.com/gitlab-org/gitlab/-/merge_requests/87727/diffs?diff_id=522279685#5d9afba799c4af9920dab533571d7abb8b9e9163 --> + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/85584) in GitLab 14.10 [with a flag](../../../administration/feature_flags.md) named `moved_mr_sidebar`. Disabled by default. + +FLAG: +On self-managed GitLab, by default this feature is not available. To make it available per project or for your entire instance, ask an administrator to [enable the feature flag](../../../administration/feature_flags.md) named `moved_mr_sidebar`. +On GitLab.com, this feature is not available. + +When this feature flag is enabled, you can find the following actions in +**Merge request actions** (**{ellipsis_v}**) on the top right: + +- The [notifications](../../profile/notifications.md#edit-notification-settings-for-issues-merge-requests-and-epics) toggle +- Mark merge request as ready or [draft](../merge_requests/drafts.md) +- Close merge request +- [Lock discussion](../../discussions/index.md#prevent-comments-by-locking-the-discussion) +- Copy reference + +When this feature flag is disabled, these actions are in the right sidebar. + ## Merge request workflows For a software developer working in a team: @@ -289,3 +311,76 @@ For a web developer writing a webpage for your company's website: - [Commits](commits.md) - [CI/CD pipelines](../../../ci/index.md) - [Push options](../push_options.md) for merge requests + +## Troubleshooting + +### Rebase a merge request from the Rails console **(FREE SELF)** + +In addition to the `/rebase` [quick action](../quick_actions.md#issues-merge-requests-and-epics), +users with access to the [Rails console](../../../administration/operations/rails_console.md) +can rebase a merge request from the Rails console. Replace `<username>`, +`<namespace/project>`, and `<iid>` with appropriate values: + +WARNING: +Any command that changes data directly could be damaging if not run correctly, +or under the right conditions. We highly recommend running them in a test environment +with a backup of the instance ready to be restored, just in case. + +```ruby +u = User.find_by_username('<username>') +p = Project.find_by_full_path('<namespace/project>') +m = p.merge_requests.find_by(iid: <iid>) +MergeRequests::RebaseService.new(project: m.target_project, current_user: u).execute(m) +``` + +### Fix incorrect merge request status **(FREE SELF)** + +If a merge request remains **Open** after its changes are merged, +users with access to the [Rails console](../../../administration/operations/rails_console.md) +can correct the merge request's status. Replace `<username>`, `<namespace/project>`, +and `<iid>` with appropriate values: + +WARNING: +Any command that changes data directly could be damaging if not run correctly, +or under the right conditions. We highly recommend running them in a test environment +with a backup of the instance ready to be restored, just in case. + +```ruby +u = User.find_by_username('<username>') +p = Project.find_by_full_path('<namespace/project>') +m = p.merge_requests.find_by(iid: <iid>) +MergeRequests::PostMergeService.new(project: p, current_user: u).execute(m) +``` + +Running this command against a merge request with unmerged changes causes the +merge request to display an incorrect message: `merged into <branch-name>`. + +### Close a merge request from the Rails console **(FREE SELF)** + +If closing a merge request doesn't work through the UI or API, you may want to attempt to close it in a [Rails console session](../../../administration/operations/rails_console.md#starting-a-rails-console-session): + +WARNING: +Commands that change data can cause damage if not run correctly or under the right conditions. Always run commands in a test environment first and have a backup instance ready to restore. + +```ruby +u = User.find_by_username('<username>') +p = Project.find_by_full_path('<namespace/project>') +m = p.merge_requests.find_by(iid: <iid>) +MergeRequests::CloseService.new(project: p, current_user: u).execute(m) +``` + +### Delete a merge request from the Rails console **(FREE SELF)** + +If deleting a merge request doesn't work through the UI or API, you may want to attempt to delete it in a [Rails console session](../../../administration/operations/rails_console.md#starting-a-rails-console-session): + +WARNING: +Any command that changes data directly could be damaging if not run correctly, +or under the right conditions. We highly recommend running them in a test environment +with a backup of the instance ready to be restored, just in case. + +```ruby +u = User.find_by_username('<username>') +p = Project.find_by_full_path('<namespace/project>') +m = p.merge_requests.find_by(iid: <iid>) +Issuable::DestroyService.new(project: m.project, current_user: u).execute(m) +``` diff --git a/doc/user/project/merge_requests/revert_changes.md b/doc/user/project/merge_requests/revert_changes.md index 3b07f75a3a7..76f351f1346 100644 --- a/doc/user/project/merge_requests/revert_changes.md +++ b/doc/user/project/merge_requests/revert_changes.md @@ -95,6 +95,6 @@ important to describe those, too. Think of things that may go wrong and include This is important to minimize requests for support, and to avoid doc comments with questions that you know someone might ask. -Each scenario can be a third-level heading, e.g. `### Getting error message X`. +Each scenario can be a third-level heading, for example `### Getting error message X`. If you have none to add when creating a doc, leave this section in place but commented out to help encourage others to add to it in the future. --> diff --git a/doc/user/project/merge_requests/reviews/data_usage.md b/doc/user/project/merge_requests/reviews/data_usage.md index 56fb84c8085..3a3af7a24bc 100644 --- a/doc/user/project/merge_requests/reviews/data_usage.md +++ b/doc/user/project/merge_requests/reviews/data_usage.md @@ -19,7 +19,7 @@ This data extraction job can take a few hours to complete (possibly up to a day) ### Generating suggestions -Once Suggested Reviewers is enabled and the data extraction is complete, new merge requests or new commits to existing merge requests will automatically trigger a Suggested Reviewers ML model inference and generate up to 5 suggested reviewers. These suggestions are contextual to the changes in the merge request. Additional commits to merge requests may change the reviewer suggestions which will automatically update in the reviewer dropdown. +Once Suggested Reviewers is enabled and the data extraction is complete, new merge requests or new commits to existing merge requests will automatically trigger a Suggested Reviewers ML model inference and generate up to 5 suggested reviewers. These suggestions are contextual to the changes in the merge request. Additional commits to merge requests may change the reviewer suggestions which will automatically update in the reviewer dropdown list. ## Progressive enhancement diff --git a/doc/user/project/merge_requests/reviews/index.md b/doc/user/project/merge_requests/reviews/index.md index ce1fc94395c..4c503211513 100644 --- a/doc/user/project/merge_requests/reviews/index.md +++ b/doc/user/project/merge_requests/reviews/index.md @@ -39,7 +39,7 @@ Project Maintainers or Owners can enable suggested reviewers by visiting the [pr Enabling suggested reviewers will trigger GitLab to create an ML model for your project that will be used to generate reviewers. The larger your project, the longer this can take, but usually, the model will be ready to generate suggestions within a few hours. -No action is required once the feature is enabled. Once the model is ready, recommendations will populate the Reviewer dropdown in the right-hand sidebar of a merge request with new commits. +No action is required once the feature is enabled. Once the model is ready, recommendations will populate the Reviewer dropdown list in the right-hand sidebar of a merge request with new commits. ## Review a merge request diff --git a/doc/user/project/merge_requests/reviews/suggestions.md b/doc/user/project/merge_requests/reviews/suggestions.md index 2d3682c62d4..832f78d18a1 100644 --- a/doc/user/project/merge_requests/reviews/suggestions.md +++ b/doc/user/project/merge_requests/reviews/suggestions.md @@ -12,7 +12,7 @@ type: index, reference As a reviewer, you're able to suggest code changes with a Markdown syntax in merge request diff threads. Then, the merge request author (or other users with appropriate -[permission](../../../permissions.md)) can apply these suggestions with a click. +[permission](../../../permissions.md)) can apply these suggestions. This action generates a commit in the merge request, authored by the user that suggested the changes. 1. Choose a line of code to be changed, add a new comment, then select @@ -47,8 +47,11 @@ After the author applies a suggestion: ## Multi-line suggestions +> [Changed](https://gitlab.com/gitlab-org/gitlab/-/issues/232339) in GitLab 13.11: suggestions in multi-line comments also become multi-line. + Reviewers can also suggest changes to multiple lines with a single suggestion -within merge request diff threads by adjusting the range offsets. The +within merge request diff threads by selecting and dragging selection to all +relevant line numbers or by adjusting the range offsets. The offsets are relative to the position of the diff thread, and specify the range to be replaced by the suggestion when it is applied. diff --git a/doc/user/project/merge_requests/squash_and_merge.md b/doc/user/project/merge_requests/squash_and_merge.md index e83e17072d6..9f87f1e2e0d 100644 --- a/doc/user/project/merge_requests/squash_and_merge.md +++ b/doc/user/project/merge_requests/squash_and_merge.md @@ -83,6 +83,6 @@ important to describe those, too. Think of things that may go wrong and include This is important to minimize requests for support, and to avoid doc comments with questions that you know someone might ask. -Each scenario can be a third-level heading, e.g. `### Getting error message X`. +Each scenario can be a third-level heading, for example `### Getting error message X`. If you have none to add when creating a doc, leave this section in place but commented out to help encourage others to add to it in the future. --> diff --git a/doc/user/project/merge_requests/versions.md b/doc/user/project/merge_requests/versions.md index 6f29272f003..a864a9849b0 100644 --- a/doc/user/project/merge_requests/versions.md +++ b/doc/user/project/merge_requests/versions.md @@ -16,12 +16,12 @@ request diffs. ## Selecting a version By default, the latest version of changes is shown. However, you -can select an older one from version dropdown. +can select an older one from version dropdown list. - + Merge request versions are based on push not on commit. So, if you pushed 5 -commits in a single push, it displays as a single option in the dropdown. If you +commits in a single push, it displays as a single option in the dropdown list. If you pushed 5 times, that counts for 5 options. You can also compare the merge request version with an older one to see what has @@ -76,6 +76,6 @@ important to describe those, too. Think of things that may go wrong and include This is important to minimize requests for support, and to avoid doc comments with questions that you know someone might ask. -Each scenario can be a third-level heading, e.g. `### Getting error message X`. +Each scenario can be a third-level heading, for example `### Getting error message X`. If you have none to add when creating a doc, leave this section in place but commented out to help encourage others to add to it in the future. --> diff --git a/doc/user/project/milestones/burndown_and_burnup_charts.md b/doc/user/project/milestones/burndown_and_burnup_charts.md index 01eeee9d3b9..81b334c0a02 100644 --- a/doc/user/project/milestones/burndown_and_burnup_charts.md +++ b/doc/user/project/milestones/burndown_and_burnup_charts.md @@ -145,6 +145,6 @@ important to describe those, too. Think of things that may go wrong and include This is important to minimize requests for support, and to avoid doc comments with questions that you know someone might ask. -Each scenario can be a third-level heading, e.g. `### Getting error message X`. +Each scenario can be a third-level heading, for example `### Getting error message X`. If you have none to add when creating a doc, leave this section in place but commented out to help encourage others to add to it in the future. --> diff --git a/doc/user/project/milestones/index.md b/doc/user/project/milestones/index.md index 76c5e32eb2b..bbe4aadc50d 100644 --- a/doc/user/project/milestones/index.md +++ b/doc/user/project/milestones/index.md @@ -238,6 +238,6 @@ important to describe those, too. Think of things that may go wrong and include This is important to minimize requests for support, and to avoid doc comments with questions that you know someone might ask. -Each scenario can be a third-level heading, e.g. `### Getting error message X`. +Each scenario can be a third-level heading, for example `### Getting error message X`. If you have none to add when creating a doc, leave this section in place but commented out to help encourage others to add to it in the future. --> diff --git a/doc/user/project/ml/experiment_tracking/img/candidates.png b/doc/user/project/ml/experiment_tracking/img/candidates.png Binary files differnew file mode 100644 index 00000000000..df70a01a2bd --- /dev/null +++ b/doc/user/project/ml/experiment_tracking/img/candidates.png diff --git a/doc/user/project/ml/experiment_tracking/img/experiments.png b/doc/user/project/ml/experiment_tracking/img/experiments.png Binary files differnew file mode 100644 index 00000000000..a6472406b90 --- /dev/null +++ b/doc/user/project/ml/experiment_tracking/img/experiments.png diff --git a/doc/user/project/ml/experiment_tracking/index.md b/doc/user/project/ml/experiment_tracking/index.md new file mode 100644 index 00000000000..e274bd7f38e --- /dev/null +++ b/doc/user/project/ml/experiment_tracking/index.md @@ -0,0 +1,76 @@ +--- +stage: Create +group: Incubation +info: Machine Learning Experiment Tracking is a GitLab Incubation Engineering program. No technical writer assigned to this group. +--- + +# Machine Learning Experiment Tracking **(FREE)** + +DISCLAIMER: +Machine Learning Experiment Tracking is an experimental feature being developed by the Incubation Engineering Department, +and will receive significant changes over time. This feature is being release with the aim of getting user feedback, but +is not stable and can lead to performance degradation. See below on how to disable this feature. + +When creating machine learning models, data scientists often experiment with different parameters, configurations, feature +engineering, and so on, to improve the performance of the model. Keeping track of all this metadata and the associated +artifacts so that the data scientist can later replicate the experiment is not trivial. Machine learning experiment +tracking enables them to log parameters, metrics, and artifacts directly into GitLab, giving easy access later on. + + + + + +## What is an experiment? + +An experiment is a collection of comparable model candidates. Experiments can be long lived (for example, when they represent +a use case), or short lived (results from hyperparameter tuning triggered by a merge request), but usually hold model candidates +that have a similar set of parameters and metrics. + +## Model candidate + +A model candidate is a variation of the training of a machine learning model, that can be eventually promoted to a version +of the model. The goal of a data scientist is to find the model candidate whose parameter values lead to the best model +performance, as indicated by the given metrics. + +Example parameters: + +- Algorithm (linear regression, decision tree, and so on). +- Hyperparameters for the algorithm (learning rate, tree depth, number of epochs). +- Features included. + +## Usage + +### User access management + +An experiment is always associated to a project. Only users with access to the project an experiment is associated with +can view that experiment data. + +### Tracking new experiments and trials + +Experiment and trials can only be tracked through the [MLFlow](https://www.mlflow.org/docs/latest/tracking.html) client +integration. More information on how to use GitLab as a backend for MLFlow Client can be found [at the documentation page](../../integrations/mlflow_client.md). + +### Exploring model candidates + +To list the current active experiments, navigate to `https/-/ml/experiments`. To display all trials +that have been logged, along with their metrics and parameters, selecting an experiment. + +### Logging artifacts + +Trial artifacts are saved as [generic packages](../../../packages/generic_packages/index.md), and follow all their +conventions. After an artifact is logged for a candidate, all artifacts logged for the candidate are listed in the +package registry. The package name for a candidate is `ml_candidate_<candidate_id>`, with version `-`. + +### Limitations and future + +- Searching experiments, searching trials, visual comparison of trials, and creating, deleting and updating experiments and trials through GitLab UI is under development. +- No support for experiment and trial metadata that do not classify as parameters or metrics. + +## Disabling or enabling the Feature + +On self-managed GitLab, ML Experiment Tracking is disabled by default. To enable the feature, ask an administrator to [disable the feature flag](../../../../administration/feature_flags.md) named `ml_experiment_tracking`. +On GitLab.com, this feature is currently on private testing. + +## Feedback, roadmap and reports + +For updates on the development, feedback and bug reports, refer to the [development epic](https://gitlab.com/groups/gitlab-org/-/epics/8560). diff --git a/doc/user/project/pages/custom_domains_ssl_tls_certification/index.md b/doc/user/project/pages/custom_domains_ssl_tls_certification/index.md index 5ee1fa103a4..6378d962ffe 100644 --- a/doc/user/project/pages/custom_domains_ssl_tls_certification/index.md +++ b/doc/user/project/pages/custom_domains_ssl_tls_certification/index.md @@ -250,7 +250,7 @@ can use the following setup: on the top nav. - Select **Create Page Rule**. - Enter the domain `www.domain.com` and select **+ Add a Setting**. - - From the dropdown menu, choose **Forwarding URL**, then select the + - From the dropdown list, choose **Forwarding URL**, then select the status code **301 - Permanent Redirect**. - Enter the destination URL `https://domain.com`. diff --git a/doc/user/project/pages/getting_started/pages_ci_cd_template.md b/doc/user/project/pages/getting_started/pages_ci_cd_template.md index 01583dbe45d..caf98e8a8a4 100644 --- a/doc/user/project/pages/getting_started/pages_ci_cd_template.md +++ b/doc/user/project/pages/getting_started/pages_ci_cd_template.md @@ -18,9 +18,9 @@ these steps, you may have to do additional configuration for the Pages site to g 1. On the top bar, select **Main menu > Projects** and find your project. 1. On the left sidebar, select the project's name. -1. From the **Add** (**{plus}**) dropdown, select **New file**. -1. From the **Select a template type** dropdown, select `.gitlab-ci.yml`. -1. From the **Apply a template** dropdown, in the **Pages** section, select the name of your SSG. +1. From the **Add** (**{plus}**) dropdown list, select **New file**. +1. From the **Select a template type** dropdown list, select `.gitlab-ci.yml`. +1. From the **Apply a template** dropdown list, in the **Pages** section, select the name of your SSG. 1. In the **Commit message** box, type the commit message. 1. Select **Commit changes**. @@ -30,5 +30,9 @@ You can watch the pipeline run by navigating to **CI/CD > Pipelines**. When the pipeline is finished, go to **Settings > Pages** to find the link to your Pages website. +To view the HTML and other assets that were created for the site, +go to the **Pipelines** tab, view the job, and on the right side, +select **Download artifacts**. + For every change pushed to your repository, GitLab CI/CD runs a new pipeline that immediately publishes your changes to the Pages site. diff --git a/doc/user/project/pages/getting_started/pages_forked_sample_project.md b/doc/user/project/pages/getting_started/pages_forked_sample_project.md index 33cf677e1be..69c60cab4b3 100644 --- a/doc/user/project/pages/getting_started/pages_forked_sample_project.md +++ b/doc/user/project/pages/getting_started/pages_forked_sample_project.md @@ -29,6 +29,10 @@ When the pipeline is finished, go to **Settings > Pages** to find the link to yo For every change pushed to your repository, GitLab CI/CD runs a new pipeline that immediately publishes your changes to the Pages site. +To view the HTMl and other assets that were created for the site, +go to the **Pipelines** tab, view the job, and on the right side, +select **Download artifacts**. + You can take some **optional** further steps: - Remove the fork relationship. If you want to contribute to the project you forked from, diff --git a/doc/user/project/pages/getting_started/pages_from_scratch.md b/doc/user/project/pages/getting_started/pages_from_scratch.md index e34544c96f8..c0a1e8f16e0 100644 --- a/doc/user/project/pages/getting_started/pages_from_scratch.md +++ b/doc/user/project/pages/getting_started/pages_from_scratch.md @@ -420,6 +420,10 @@ Now GitLab CI/CD not only builds the website, but also: - **Caches** dependencies installed with Bundler. - **Continuously deploys** every push to the `main` branch. +To view the HTMl and other assets that were created for the site, +go to the **Pipelines** tab, view the job, and on the right side, +select **Download artifacts**. + ## Related topics For more information, see the following blog posts. @@ -427,7 +431,7 @@ For more information, see the following blog posts. - Use GitLab CI/CD `environments` to [deploy your web app to staging and production](https://about.gitlab.com/blog/2021/02/05/ci-deployment-and-environments/). - Learn how to run jobs - [sequentially, in parallel, or build a custom pipeline](https://about.gitlab.com/blog/2016/07/29/the-basics-of-gitlab-ci/). + [sequentially, in parallel, or build a custom pipeline](https://about.gitlab.com/blog/2020/12/10/basics-of-gitlab-ci-updated/). - Learn [how to pull specific directories from different projects](https://about.gitlab.com/blog/2016/12/07/building-a-new-gitlab-docs-site-with-nanoc-gitlab-ci-and-gitlab-pages/) to deploy this website, <https://docs.gitlab.com>. - Learn [how to use GitLab Pages to produce a code coverage report](https://about.gitlab.com/blog/2016/11/03/publish-code-coverage-report-with-gitlab-pages/). diff --git a/doc/user/project/pages/getting_started/pages_new_project_template.md b/doc/user/project/pages/getting_started/pages_new_project_template.md index d7e304dc6f8..e4890954d13 100644 --- a/doc/user/project/pages/getting_started/pages_new_project_template.md +++ b/doc/user/project/pages/getting_started/pages_new_project_template.md @@ -28,3 +28,7 @@ your Pages website. For every change pushed to your repository, GitLab CI/CD runs a new pipeline that immediately publishes your changes to the Pages site. + +To view the HTMl and other assets that were created for the site, +go to the **Pipelines** tab, view the job, and on the right side, +select **Download artifacts**. diff --git a/doc/user/project/pages/getting_started/pages_ui.md b/doc/user/project/pages/getting_started/pages_ui.md index 064255c094f..ba97fcb8749 100644 --- a/doc/user/project/pages/getting_started/pages_ui.md +++ b/doc/user/project/pages/getting_started/pages_ui.md @@ -41,6 +41,10 @@ To build your YAML file from the GitLab UI: 1. Commit your `.gitlab-ci.yml` to your repository. This commit triggers your first GitLab Pages deployment. +To view the HTMl and other assets that were created for the site, +go to **CI/CD > Pipelines**, view the job, and on the right side, +select **Download artifacts**. + ## Troubleshooting ### If you can't see the "Get Started with Pages" interface diff --git a/doc/user/project/pages/img/remove_pages_v15_3.png b/doc/user/project/pages/img/remove_pages_v15_3.png Binary files differdeleted file mode 100644 index f740daf5c0b..00000000000 --- a/doc/user/project/pages/img/remove_pages_v15_3.png +++ /dev/null diff --git a/doc/user/project/pages/index.md b/doc/user/project/pages/index.md index 47c0885c26b..9305b92bf7d 100644 --- a/doc/user/project/pages/index.md +++ b/doc/user/project/pages/index.md @@ -107,7 +107,7 @@ These GitLab Pages website examples can teach you advanced techniques to use and adapt for your own needs: - [Posting to your GitLab Pages blog from iOS](https://about.gitlab.com/blog/2016/08/19/posting-to-your-gitlab-pages-blog-from-ios/). -- [GitLab CI: Run jobs sequentially, in parallel, or build a custom pipeline](https://about.gitlab.com/blog/2016/07/29/the-basics-of-gitlab-ci/). +- [GitLab CI: Run jobs sequentially, in parallel, or build a custom pipeline](https://about.gitlab.com/blog/2020/12/10/basics-of-gitlab-ci-updated/). - [GitLab CI: Deployment & environments](https://about.gitlab.com/blog/2021/02/05/ci-deployment-and-environments/). - [Building a new GitLab docs site with Nanoc, GitLab CI, and GitLab Pages](https://about.gitlab.com/blog/2016/12/07/building-a-new-gitlab-docs-site-with-nanoc-gitlab-ci-and-gitlab-pages/). - [Publish code coverage reports with GitLab Pages](https://about.gitlab.com/blog/2016/11/03/publish-code-coverage-report-with-gitlab-pages/). diff --git a/doc/user/project/pages/introduction.md b/doc/user/project/pages/introduction.md index ed154c0dfca..a9b8960d629 100644 --- a/doc/user/project/pages/introduction.md +++ b/doc/user/project/pages/introduction.md @@ -40,20 +40,19 @@ If you are using [GitLab Pages on GitLab.com](#gitlab-pages-on-gitlabcom) to hos Visit the [GitLab Pages group](https://gitlab.com/groups/pages) for a complete list of example projects. Contributions are very welcome. -## Custom error codes Pages +## Custom error codes pages -You can provide your own 403 and 404 error pages by creating the `403.html` and -`404.html` files respectively in the root directory of the `public/` directory -that are included in the artifacts. Usually this is the root directory of -your project, but that may differ depending on your static generator -configuration. +You can provide your own `403` and `404` error pages by creating `403.html` and +`404.html` files in the root of the `public/` directory. Usually this is +the root directory of your project, but that may differ +depending on your static generator configuration. If the case of `404.html`, there are different scenarios. For example: - If you use project Pages (served under `/projectname/`) and try to access `/projectname/non/existing_file`, GitLab Pages tries to serve first `/projectname/404.html`, and then `/404.html`. -- If you use user/group Pages (served under `/`) and try to access +- If you use user or group Pages (served under `/`) and try to access `/non/existing_file` GitLab Pages tries to serve `/404.html`. - If you use a custom domain and try to access `/non/existing_file`, GitLab Pages tries to serve only `/404.html`. @@ -63,34 +62,34 @@ If the case of `404.html`, there are different scenarios. For example: You can configure redirects for your site using a `_redirects` file. To learn more, read the [redirects documentation](redirects.md). -## GitLab Pages Access Control +## Remove your pages -To restrict access to your website, enable [GitLab Pages Access Control](pages_access_control.md). +To remove your pages: -## Unpublishing your Pages - -If you ever feel the need to purge your Pages content, you can do so by going -to your project's settings through the gear icon in the top right, and then -navigating to **Pages**. Select the **Remove pages** button to delete your Pages -website. - - +1. On the top bar, select **Main menu > Projects** and find your project. +1. On the left sidebar, select **Settings > Pages**. +1. Select **Remove pages**. ## Subdomains of subdomains When using Pages under the top-level domain of a GitLab instance (`*.example.io`), you can't use HTTPS with subdomains of subdomains. If your namespace or group name contains a dot (for example, `foo.bar`) the domain -`https://foo.bar.example.io` does _not_ work. +`https://foo.bar.example.io` does **not** work. This limitation is because of the [HTTP Over TLS protocol](https://www.rfc-editor.org/rfc/rfc2818#section-3.1). HTTP pages work as long as you don't redirect HTTP to HTTPS. -## GitLab Pages and subgroups +## GitLab Pages in projects and groups + +You must host your GitLab Pages website in a project. This project can be +[private, internal, or public](../../../user/public_access.md) and belong +to a [group](../../group/index.md) or [subgroup](../../group/subgroups/index.md). + +For [group websites](../../project/pages/getting_started_part_one.md#user-and-group-website-examples), +the group must be at the top level and not a subgroup. -You must host your GitLab Pages website in a project. This project can belong to a [group](../../group/index.md) or -[subgroup](../../group/subgroups/index.md). For -[group websites](../../project/pages/getting_started_part_one.md#gitlab-pages-default-domain-names), the group must be -at the top level and not a subgroup. +For [project websites](../../project/pages/getting_started_part_one.md#project-website-examples), +you can create your project first and access it under `http(s)://namespace.example.io/projectname`. ## Specific configuration options for Pages @@ -129,7 +128,7 @@ pages: See this document for a [step-by-step guide](getting_started/pages_from_scratch.md). -### `.gitlab-ci.yml` for a repository where there's also actual code +### `.gitlab-ci.yml` for a repository with code Remember that GitLab Pages are by default branch/tag agnostic and their deployment relies solely on what you specify in `.gitlab-ci.yml`. You can limit @@ -257,26 +256,6 @@ instead. Here are some examples of what happens given the above Pages site: Note that when `public/data/index.html` exists, it takes priority over the `public/data.html` file for both the `/data` and `/data/` URL paths. -## Frequently Asked Questions - -### Can you download your generated pages? - -Sure. All you need to do is download the artifacts archive from the job page. - -### Can you use GitLab Pages if your project is private? - -Yes. GitLab Pages doesn't care whether you set your project's visibility level -to private, internal or public. - -### Can you create a personal or a group website? - -Yes. See the documentation about [GitLab Pages domain names, URLs, and base URLs](getting_started_part_one.md). - -### Do you need to create a user/group website before creating a project website? - -No, you don't. You can create your project first and access it under -`http(s)://namespace.example.io/projectname`. - ## Known issues For a list of known issues, visit the GitLab [public issue tracker](https://gitlab.com/gitlab-org/gitlab/-/issues?label_name[]=Category%3APages). diff --git a/doc/user/project/pages/pages_access_control.md b/doc/user/project/pages/pages_access_control.md index be1ea236c87..b98772a6702 100644 --- a/doc/user/project/pages/pages_access_control.md +++ b/doc/user/project/pages/pages_access_control.md @@ -20,7 +20,7 @@ For a demonstration, see [Pages access controls](https://www.youtube.com/watch?v 1. Toggle the **Pages** button to enable the access control. If you don't see the toggle button, that means it isn't enabled. Ask your administrator to [enable it](../../../administration/pages/index.md#access-control). -1. The Pages access control dropdown allows you to set who can view pages hosted +1. The Pages access control dropdown list allows you to set who can view pages hosted with GitLab Pages, depending on your project's visibility: - If your project is private: diff --git a/doc/user/project/pages/public_folder.md b/doc/user/project/pages/public_folder.md index b5d498402d5..a19e296b954 100644 --- a/doc/user/project/pages/public_folder.md +++ b/doc/user/project/pages/public_folder.md @@ -116,7 +116,7 @@ GitLab Pages supports only static sites. ``` 1. Configure your Nuxt.js application for - [Static Site Generation](https://nuxtjs.org/docs/features/deployment-targets#static-hosting). + [Static Site Generation](https://nuxtjs.org/docs/features/deployment-targets/#static-hosting). ### Vite diff --git a/doc/user/project/protected_branches.md b/doc/user/project/protected_branches.md index f9dcf838c33..ab97ff08123 100644 --- a/doc/user/project/protected_branches.md +++ b/doc/user/project/protected_branches.md @@ -260,6 +260,6 @@ important to describe those, too. Think of things that may go wrong and include This is important to minimize requests for support, and to avoid doc comments with questions that you know someone might ask. -Each scenario can be a third-level heading, e.g. `### Getting error message X`. +Each scenario can be a third-level heading, for example `### Getting error message X`. If you have none to add when creating a doc, leave this section in place but commented out to help encourage others to add to it in the future. --> diff --git a/doc/user/project/protected_tags.md b/doc/user/project/protected_tags.md index 22ce81409a3..152a55d24b7 100644 --- a/doc/user/project/protected_tags.md +++ b/doc/user/project/protected_tags.md @@ -114,6 +114,6 @@ important to describe those, too. Think of things that may go wrong and include This is important to minimize requests for support, and to avoid doc comments with questions that you know someone might ask. -Each scenario can be a third-level heading, e.g. `### Getting error message X`. +Each scenario can be a third-level heading, for example `### Getting error message X`. If you have none to add when creating a doc, leave this section in place but commented out to help encourage others to add to it in the future. --> diff --git a/doc/user/project/quick_actions.md b/doc/user/project/quick_actions.md index 3471123f8b5..7b7619cfeb5 100644 --- a/doc/user/project/quick_actions.md +++ b/doc/user/project/quick_actions.md @@ -52,7 +52,7 @@ threads. Some quick actions might not be available to all subscription tiers. | Command | Issue | Merge request | Epic | Action | |:-------------------------------------------------------------------------------------------------|:-----------------------|:-----------------------|:-----------------------|:----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| -| `/add_contacts [contact:email1@example.com] [contact:email2@example.com]` | **{check-circle}** Yes | **{dotted-circle}** No | **{dotted-circle}** No | Add one or more [CRM contacts](../crm/index.md) ([introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/73413) in GitLab 14.6). | +| `/add_contacts [contact:email1@example.com] [contact:email2@example.com]` | **{check-circle}** Yes | **{dotted-circle}** No | **{dotted-circle}** No | Add one or more active [CRM contacts](../crm/index.md) ([introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/73413) in GitLab 14.6). | | `/approve` | **{dotted-circle}** No | **{check-circle}** Yes | **{dotted-circle}** No | Approve the merge request. | | `/assign @user1 @user2` | **{check-circle}** Yes | **{check-circle}** Yes | **{dotted-circle}** No | Assign one or more users. | | `/assign me` | **{check-circle}** Yes | **{check-circle}** Yes | **{dotted-circle}** No | Assign yourself. | @@ -65,7 +65,7 @@ threads. Some quick actions might not be available to all subscription tiers. | `/clear_weight` | **{check-circle}** Yes | **{dotted-circle}** No | **{dotted-circle}** No | Clear weight. | | `/clone <path/to/project> [--with_notes]` | **{check-circle}** Yes | **{dotted-circle}** No | **{dotted-circle}** No | Clone the issue to given project, or the current one if no arguments are given ([introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/9421) in GitLab 13.7). Copies as much data as possible as long as the target project contains equivalent labels, milestones, and so on. Does not copy comments or system notes unless `--with_notes` is provided as an argument. | | `/close` | **{check-circle}** Yes | **{check-circle}** Yes | **{check-circle}** Yes | Close. | -| `/confidential` | **{check-circle}** Yes | **{dotted-circle}** No | **{dotted-circle}** No | Make confidential. | +| `/confidential` | **{check-circle}** Yes | **{dotted-circle}** No | **{check-circle}** Yes | Mark issue or epic as confidential. Support for epics [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/213741) in GitLab 15.6. | | `/copy_metadata <!merge_request>` | **{check-circle}** Yes | **{check-circle}** Yes | **{dotted-circle}** No | Copy labels and milestone from another merge request in the project. | | `/copy_metadata <#issue>` | **{check-circle}** Yes | **{check-circle}** Yes | **{dotted-circle}** No | Copy labels and milestone from another issue in the project. | | `/create_merge_request <branch name>` | **{check-circle}** Yes | **{dotted-circle}** No | **{dotted-circle}** No | Create a new merge request starting from the current issue. | @@ -74,13 +74,13 @@ threads. Some quick actions might not be available to all subscription tiers. | `/due <date>` | **{check-circle}** Yes | **{dotted-circle}** No | **{dotted-circle}** No | Set due date. Examples of valid `<date>` include `in 2 days`, `this Friday` and `December 31st`. | | `/duplicate <#issue>` | **{check-circle}** Yes | **{dotted-circle}** No | **{dotted-circle}** No | Close this issue and mark as a duplicate of another issue. Also, mark both as related. | | `/epic <epic>` | **{check-circle}** Yes | **{dotted-circle}** No | **{dotted-circle}** No | Add to epic `<epic>`. The `<epic>` value should be in the format of `&epic`, `group&epic`, or a URL to an epic. | -| `/estimate <time>` | **{check-circle}** Yes | **{check-circle}** Yes | **{dotted-circle}** No | Set time estimate. For example, `/estimate 1mo 2w 3d 4h 5m`. Learn more about [time tracking](time_tracking.md). | +| `/estimate <time>` or `/estimate_time <time>` | **{check-circle}** Yes | **{check-circle}** Yes | **{dotted-circle}** No | Set time estimate. For example, `/estimate 1mo 2w 3d 4h 5m`. Learn more about [time tracking](time_tracking.md). Alias `/estimate_time` [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/16501) in GitLab 15.6. | | `/health_status <value>` | **{check-circle}** Yes | **{dotted-circle}** No | **{dotted-circle}** No | Set [health status](issues/managing_issues.md#health-status). Valid options for `<value>` are `on_track`, `needs_attention`, and `at_risk` ([introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/213814) in GitLab 14.7). | | `/invite_email email1 email2` | **{check-circle}** Yes | **{dotted-circle}** No | **{dotted-circle}** No | Add up to six email participants. This action is behind feature flag `issue_email_participants` and is not yet supported in issue templates. | | `/iteration *iteration:"iteration name"` | **{check-circle}** Yes | **{dotted-circle}** No | **{dotted-circle}** No | Set iteration. For example, to set the `Late in July` iteration: `/iteration *iteration:"Late in July"` ([introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/196795) in GitLab 13.1). | | `/label ~label1 ~label2` | **{check-circle}** Yes | **{check-circle}** Yes | **{check-circle}** Yes | Add one or more labels. Label names can also start without a tilde (`~`), but mixed syntax is not supported. | | `/lock` | **{check-circle}** Yes | **{check-circle}** Yes | **{dotted-circle}** No | Lock the discussions. | -| `/link` | **{check-circle}** Yes | ****{dotted-circle}** No | **{dotted-circle}** No | Add a link and description to [linked resources](../../operations/incident_management/linked_resources.md) in an incident ([introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/374964) in GitLab 15.5). | +| `/link` | **{check-circle}** Yes | **{dotted-circle}** No | **{dotted-circle}** No | Add a link and description to [linked resources](../../operations/incident_management/linked_resources.md) in an incident ([introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/374964) in GitLab 15.5). | | `/merge` | **{dotted-circle}** No | **{check-circle}** Yes | **{dotted-circle}** No | Merge changes. Depending on the project setting, this may be [when the pipeline succeeds](merge_requests/merge_when_pipeline_succeeds.md), or adding to a [Merge Train](../../ci/pipelines/merge_trains.md). | | `/milestone %milestone` | **{check-circle}** Yes | **{check-circle}** Yes | **{dotted-circle}** No | Set milestone. | | `/move <path/to/project>` | **{check-circle}** Yes | **{dotted-circle}** No | **{dotted-circle}** No | Move this issue to another project. Be careful when moving an issue to a project with different access rules. Before moving the issue, make sure it does not contain sensitive data. | @@ -99,7 +99,7 @@ threads. Some quick actions might not be available to all subscription tiers. | `/remove_contacts [contact:email1@example.com] [contact:email2@example.com]` | **{check-circle}** Yes | **{dotted-circle}** No | **{dotted-circle}** No | Remove one or more [CRM contacts](../crm/index.md) ([introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/73413) in GitLab 14.6). | | `/remove_due_date` | **{check-circle}** Yes | **{dotted-circle}** No | **{dotted-circle}** No | Remove due date. | | `/remove_epic` | **{check-circle}** Yes | **{dotted-circle}** No | **{dotted-circle}** No | Remove from epic. | -| `/remove_estimate` | **{check-circle}** Yes | **{check-circle}** Yes | **{dotted-circle}** No | Remove time estimate. | +| `/remove_estimate` or `/remove_time_estimate` | **{check-circle}** Yes | **{check-circle}** Yes | **{dotted-circle}** No | Remove time estimate. Alias `/remove_time_estimate` [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/16501) in GitLab 15.6. | | `/remove_iteration` | **{check-circle}** Yes | **{dotted-circle}** No | **{dotted-circle}** No | Remove iteration ([introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/196795) in GitLab 13.1). | | `/remove_milestone` | **{check-circle}** Yes | **{check-circle}** Yes | **{dotted-circle}** No | Remove milestone. | | `/remove_parent_epic` | **{dotted-circle}** No | **{dotted-circle}** No | **{check-circle}** Yes | Remove parent epic from epic ([introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/10556) in GitLab 12.1). | @@ -108,7 +108,7 @@ threads. Some quick actions might not be available to all subscription tiers. | `/reopen` | **{check-circle}** Yes | **{check-circle}** Yes | **{check-circle}** Yes | Reopen. | | `/severity <severity>` | **{check-circle}** Yes | **{check-circle}** No | **{check-circle}** No | Set the severity. Issue type must be `Incident`. Options for `<severity>` are `S1` ... `S4`, `critical`, `high`, `medium`, `low`, `unknown`. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/334045) in GitLab 14.2. | | `/shrug <comment>` | **{check-circle}** Yes | **{check-circle}** Yes | **{check-circle}** Yes | Append the comment with `¯\＿(ツ)＿/¯`. | -| `/spend <time> [<date>]` | **{check-circle}** Yes | **{check-circle}** Yes | **{dotted-circle}** No | Add or subtract spent time. Optionally, specify the date that time was spent on. For example, `/spend 1mo 2w 3d 4h 5m 2018-08-26` or `/spend -1h 30m`. Learn more about [time tracking](time_tracking.md). | +| `/spend <time> [<date>]` or `/spend_time <time> [<date>]` | **{check-circle}** Yes | **{check-circle}** Yes | **{dotted-circle}** No | Add or subtract spent time. Optionally, specify the date that time was spent on. For example, `/spend 1mo 2w 3d 4h 5m 2018-08-26` or `/spend -1h 30m`. Learn more about [time tracking](time_tracking.md). Alias `/spend_time` [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/16501) in GitLab 15.6. | | `/submit_review` | **{dotted-circle}** No | **{check-circle}** Yes | **{dotted-circle}** No | Submit a pending review ([introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/8041) in GitLab 12.7). | | `/subscribe` | **{check-circle}** Yes | **{check-circle}** Yes | **{check-circle}** Yes | Subscribe to notifications. | | `/tableflip <comment>` | **{check-circle}** Yes | **{check-circle}** Yes | **{check-circle}** Yes | Append the comment with `(╯°□°)╯︵ ┻━┻`. | @@ -145,6 +145,6 @@ important to describe those, too. Think of things that may go wrong and include This is important to minimize requests for support, and to avoid doc comments with questions that you know someone might ask. -Each scenario can be a third-level heading, e.g. `### Getting error message X`. +Each scenario can be a third-level heading, for example `### Getting error message X`. If you have none to add when creating a doc, leave this section in place but commented out to help encourage others to add to it in the future. --> diff --git a/doc/user/project/releases/index.md b/doc/user/project/releases/index.md index fd01dd451c2..75a25678125 100644 --- a/doc/user/project/releases/index.md +++ b/doc/user/project/releases/index.md @@ -462,6 +462,20 @@ In the API: This includes associated Git-tag-names, release description, author information of the releases. However, other repository-related information, such as [source code](release_fields.md#source-code), [release evidence](#release-evidence) are redacted. +### Publish releases without giving access to source code + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/216485) in GitLab 15.6. + +Releases can be made accessible to non-project members while keeping repository-related information such as +[source code](release_fields.md#source-code) and [release evidence](#release-evidence) private. This is useful for +projects that use releases as a way to give access to new versions of software but do not want the source code to +be public. + +To make releases available publicly, set the following [project settings](../settings/index.md#project-feature-settings): + +- Repository is enabled and set to **Only Project Members** +- Releases is enabled and set to **Everyone With Access** + ### Create, update, and delete a release and its assets - Users with at least the Developer role diff --git a/doc/user/project/releases/release_fields.md b/doc/user/project/releases/release_fields.md index 5ae1c69847d..c06e746268f 100644 --- a/doc/user/project/releases/release_fields.md +++ b/doc/user/project/releases/release_fields.md @@ -93,7 +93,7 @@ By default, GitLab fetches the release using `released_at` time. The use of the The assets associated with a release are accessible through a permanent URL. GitLab always redirects this URL to the actual asset location, so even if the assets move to a different location, you can continue -to use the same URL. This is defined during [link creation](../../../api/releases/links.md#create-a-link) or [updating](../../../api/releases/links.md#update-a-link) using the `filepath` API attribute. +to use the same URL. This is defined during [link creation](../../../api/releases/links.md#create-a-release-link) or [updating](../../../api/releases/links.md#update-a-release-link) using the `filepath` API attribute. The format of the URL is: @@ -133,7 +133,7 @@ The format of the URL is: https://host/namespace/project/-/releases/permalink/latest/downloads/:filepath ``` -If you have an asset with [`filepath`](../../../api/releases/links.md#create-a-link) for the `v11.9.0-rc2` latest release in the `gitlab-org` +If you have an asset with [`filepath`](../../../api/releases/links.md#create-a-release-link) for the `v11.9.0-rc2` latest release in the `gitlab-org` namespace and `gitlab-runner` project on `gitlab.com`, for example: ```json diff --git a/doc/user/project/remote_development/index.md b/doc/user/project/remote_development/index.md index 879978f550a..62220dd2fde 100644 --- a/doc/user/project/remote_development/index.md +++ b/doc/user/project/remote_development/index.md @@ -6,6 +6,14 @@ info: To determine the technical writer assigned to the Stage/Group associated w # Remote Development **(FREE)** +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/95169) in GitLab 15.6 [with a flag](../../../administration/feature_flags.md) named `vscode_web_ide`. Disabled by default. + +FLAG: +On self-managed GitLab, by default this feature is not available. To make it available, ask an administrator to [enable the feature flag](../../../administration/feature_flags.md) named `vscode_web_ide`. On GitLab.com, this feature is available. The feature is not ready for production use. + +WARNING: +This feature is in [Alpha](../../../policy/alpha-beta-support.md#alpha-features) and subject to change without notice. + DISCLAIMER: This page contains information related to upcoming products, features, and functionality. It is important to note that the information presented is for informational purposes only. @@ -43,7 +51,7 @@ To point a domain to your remote machine, create an `A` record from `example.rem #### Install Certbot -[Certbot](https://certbot.eff.org/) is a free and open-source software tool that automatically uses Let's Encrypt certificates on manually administrated websites to enable HTTPS. +[Certbot](https://certbot.eff.org/) is a free and open-source software tool that automatically uses Let's Encrypt certificates on manually administered websites to enable HTTPS. To install Certbot, run the following command: @@ -89,7 +97,7 @@ docker run -d \ -v "${CERTS_DIR}/fullchain.pem:/gitlab-rd-web-ide/certs/fullchain.pem" \ -v "${CERTS_DIR}/privkey.pem:/gitlab-rd-web-ide/certs/privkey.pem" \ -v "${PROJECTS_DIR}:/projects" \ - registry.gitlab.com/gitlab-com/create-stage/editor-poc/remote-development/gitlab-rd-web-ide-docker:0.1 \ + registry.gitlab.com/gitlab-com/create-stage/editor-poc/remote-development/gitlab-rd-web-ide-docker:0.1-alpha \ --log-level warn --domain "${DOMAIN}" --ignore-version-mismatch ``` diff --git a/doc/user/project/repository/branches/index.md b/doc/user/project/repository/branches/index.md index 9755b5cb944..6cc7394e7b3 100644 --- a/doc/user/project/repository/branches/index.md +++ b/doc/user/project/repository/branches/index.md @@ -124,6 +124,6 @@ important to describe those, too. Think of things that may go wrong and include This is important to minimize requests for support, and to avoid doc comments with questions that you know someone might ask. -Each scenario can be a third-level heading, e.g. `### Getting error message X`. +Each scenario can be a third-level heading, for example `### Getting error message X`. If you have none to add when creating a doc, leave this section in place but commented out to help encourage others to add to it in the future. --> diff --git a/doc/user/project/repository/gpg_signed_commits/index.md b/doc/user/project/repository/gpg_signed_commits/index.md index 910b09449d8..a1f57f51f26 100644 --- a/doc/user/project/repository/gpg_signed_commits/index.md +++ b/doc/user/project/repository/gpg_signed_commits/index.md @@ -253,3 +253,19 @@ If you must unverify both future and past commits, - [OpenPGP Best Practices](https://riseup.net/en/security/message-security/openpgp/best-practices) - [Creating a new GPG key with subkeys](https://www.void.gr/kargig/blog/2013/12/02/creating-a-new-gpg-key-with-subkeys/) (advanced) - [Review existing GPG keys in your instance](../../../admin_area/credentials_inventory.md#review-existing-gpg-keys) + +## Troubleshooting + +### Fix verification problems with signed commits + +Commits can be signed with [X.509 certificates](../x509_signed_commits/index.md) +or a GPG key. The verification process for both methods can fail for multiple reasons: + +| Value | Description | Possible Fixes | +|-----------------------------|-------------|----------------| +| `UNVERIFIED` | The commit signature is not valid. | Sign the commit with a valid signature. | +| `SAME_USER_DIFFERENT_EMAIL` | The GPG key used to sign the commit does not contain the committer email, but does contain a different valid email for the committer. | Amend the commit to use an email address that matches the GPG key, or update the GPG key [to include the email address](https://security.stackexchange.com/a/261468). | +| `OTHER_USER` | The signature and GPG key are valid, but the key belongs to a different user than the committer. | Amend the commit to use the correct email address, or amend the commit to use a GPG key associated with your user. | +| `UNVERIFIED_KEY` | The key associated with the GPG signature has no verified email address associated with the committer. | Add and verify the email to your GitLab profile, [update the GPG key to include the email address](https://security.stackexchange.com/a/261468), or amend the commit to use a different committer email address. | +| `UNKNOWN_KEY` | The GPG key associated with the GPG signature for this commit is unknown to GitLab. | [Add the GPG key](#add-a-gpg-key-to-your-account) to your GitLab profile. | +| `MULTIPLE_SIGNATURES` | Multiple GPG or X.509 signatures have been found for the commit. | Amend the commit to use only one GPG or X.509 signature. | diff --git a/doc/user/project/repository/index.md b/doc/user/project/repository/index.md index bcdb626d0f2..83389c15eba 100644 --- a/doc/user/project/repository/index.md +++ b/doc/user/project/repository/index.md @@ -153,7 +153,7 @@ contents of the file's [markup language](https://en.wikipedia.org/wiki/Lightweig | [reStructuredText](https://docutils.sourceforge.io/rst.html) | `rst` | | [AsciiDoc](../../asciidoc.md) | `adoc`, `ad`, `asciidoc` | | [Textile](https://textile-lang.com/) | `textile` | -| [Rdoc](http://rdoc.sourceforge.net/doc/index.html) | `rdoc` | +| [Rdoc](https://rdoc.sourceforge.net/doc/index.html) | `rdoc` | | [Org mode](https://orgmode.org/) | `org` | | [creole](http://www.wikicreole.org/) | `creole` | | [MediaWiki](https://www.mediawiki.org/wiki/MediaWiki) | `wiki`, `mediawiki` | diff --git a/doc/user/project/repository/jupyter_notebooks/index.md b/doc/user/project/repository/jupyter_notebooks/index.md index 4f6cff576ea..28f0ffb6fbd 100644 --- a/doc/user/project/repository/jupyter_notebooks/index.md +++ b/doc/user/project/repository/jupyter_notebooks/index.md @@ -28,24 +28,20 @@ GitLab. > - [Introduced](https://gitlab.com/groups/gitlab-org/-/epics/6589) in GitLab 14.5 as an [Alpha](../../../../policy/alpha-beta-support.md#alpha-features) release [with a flag](../../../../administration/feature_flags.md) named `jupyter_clean_diffs`. Enabled by default. > - [Generally available](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/75500) in GitLab 14.9. Feature flag `jupyter_clean_diffs` removed. > - [Reintroduced toggle](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/85079) in GitLab 15.0 [with a flag](../../../../administration/feature_flags.md) named `ipynb_semantic_diff`. Enabled by default. - -FLAG: -On self-managed GitLab, by default semantic diffs are available. To hide the feature, ask an administrator to [disable the feature flag](../../../../administration/feature_flags.md) named `ipynb_semantic_diff`. -On GitLab.com, this feature is available. +> - [Generally available](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/95373) in GitLab 15.6. Feature flag `ipynb_semantic_diff` removed. When commits include changes to Jupyter Notebook files, GitLab: - Transforms the machine-readable `.ipynb` file into a human-readable Markdown file. - Displays a cleaner version of the diff that includes syntax highlighting. - Enables switching between raw and rendered diffs on the Commit and Compare pages. (Not available on merge request pages.) +- Renders images on the diffs. Code suggestions are not available on diffs and merge requests for `.ipynb` files. - +Cleaner notebook diffs are not generated when the notebook is too large. -This feature is an [Alpha](../../../../policy/alpha-beta-support.md#alpha-features) release, -and might lead to performance degradation. On self-managed GitLab, if unexpected issues -arise, disable the feature. + ## Jupyter Git integration diff --git a/doc/user/project/repository/mirror/index.md b/doc/user/project/repository/mirror/index.md index fa6d8d82559..2b578977bd2 100644 --- a/doc/user/project/repository/mirror/index.md +++ b/doc/user/project/repository/mirror/index.md @@ -325,6 +325,9 @@ Use case: If you have multiple users using their own GitHub credentials to set u repository mirroring, mirroring breaks when people leave the company. Use this script to migrate disparate mirroring users and tokens into a single service account: +WARNING: +Commands that change data can cause damage if not run correctly or under the right conditions. Always run commands in a test environment first and have a backup instance ready to restore. + ```ruby svc_user = User.find_by(username: 'ourServiceUser') token = 'githubAccessToken' diff --git a/doc/user/project/repository/push_rules.md b/doc/user/project/repository/push_rules.md index 917fca03967..7ae085812ae 100644 --- a/doc/user/project/repository/push_rules.md +++ b/doc/user/project/repository/push_rules.md @@ -267,7 +267,7 @@ to use them as normal characters in a match condition. ## Related topics -- [Server hooks](../../../administration/server_hooks.md), to create complex custom push rules +- [Git server hooks](../../../administration/server_hooks.md) (previously called server hooks), to create complex custom push rules - [Signing commits with GPG](gpg_signed_commits/index.md) - [Protected branches](../protected_branches.md) @@ -304,7 +304,7 @@ For example, to enable **Check whether the commit author is a GitLab user** and and create a filter for allowing commits from a specific email domain only through rails console: WARNING: -Any command that changes data directly could be damaging if not run correctly, or under the right conditions. We highly recommend running them in a test environment with a backup of the instance ready to be restored, just in case. +Commands that change data can cause damage if not run correctly or under the right conditions. Always run commands in a test environment first and have a backup instance ready to restore. ``` ruby Project.find_each do |p| diff --git a/doc/user/project/repository/reducing_the_repo_size_using_git.md b/doc/user/project/repository/reducing_the_repo_size_using_git.md index c85dd4555ca..9c977e4da40 100644 --- a/doc/user/project/repository/reducing_the_repo_size_using_git.md +++ b/doc/user/project/repository/reducing_the_repo_size_using_git.md @@ -72,6 +72,12 @@ To purge files from a GitLab repository: cd project.git ``` +1. Because cloning from a bundle file sets the `origin` remote to the local bundle file, change it to the URL of your repository: + + ```shell + git remote set-url origin https://gitlab.example.com/<namespace>/<project_name>.git + ``` + 1. Using either `git filter-repo` or `git-sizer`, analyze your repository and review the results to determine which items you want to purge: @@ -84,38 +90,39 @@ To purge files from a GitLab repository: git-sizer ``` -1. Proceed to purging any files from the history of your repository. Because we are - trying to remove internal refs, we rely on the `commit-map` produced by each run to tell us - which internal refs to remove. - - NOTE: - `git filter-repo` creates a new `commit-map` file every run, and overwrites the `commit-map` from - the previous run. You need this file from **every** run. Do the next step every time you run - `git filter-repo`. +1. Purge the history of your repository using relevant `git filter-repo` options. + Two common options are: - To purge specific files, the `--path` and `--invert-paths` options can be combined: + - `--path` and `--invert-paths` to purge specific files: - ```shell - git filter-repo --path path/to/file.ext --invert-paths - ``` + ```shell + git filter-repo --path path/to/file.ext --invert-paths + ``` - To generally purge all files larger than 10M, the `--strip-blobs-bigger-than` option can be used: + - `--strip-blobs-bigger-than` to purge all files larger than for example 10M: - ```shell - git filter-repo --strip-blobs-bigger-than 10M - ``` + ```shell + git filter-repo --strip-blobs-bigger-than 10M + ``` See the [`git filter-repo` documentation](https://htmlpreview.github.io/?https://github.com/newren/git-filter-repo/blob/docs/html/git-filter-repo.html#EXAMPLES) for more examples and the complete documentation. -1. Because cloning from a bundle file sets the `origin` remote to the local bundle file, delete this `origin` remote, and set it to the URL to your repository: +1. Because you are trying to remove internal refs, + you'll later rely on `commit-map` files produced by each run + to tell you which internal refs to remove. + Every `git filter-repo` run creates a new `commit-map`, + and overwrites the `commit-map` from the previous run. + You can use the following command to back up each `commit-map` file: ```shell - git remote remove origin - git remote add origin https://gitlab.example.com/<namespace>/<project_name>.git + cp .git/filter-repo/commit-map ./_filter_repo_commit_map_$(date +%s) ``` + Repeat this step and all following steps (including the [repository cleanup](#repository-cleanup) step) + every time you run any `git filter-repo` command. + 1. Force push your changes to overwrite all branches on GitLab: ```shell diff --git a/doc/user/project/repository/web_editor.md b/doc/user/project/repository/web_editor.md index e1f05cd5501..773662edb17 100644 --- a/doc/user/project/repository/web_editor.md +++ b/doc/user/project/repository/web_editor.md @@ -10,13 +10,13 @@ Sometimes it's easier to make quick changes directly from the GitLab interface than to clone the project and use the Git command-line tool. In this feature highlight, we look at how you can create a new file, directory, branch, or tag from the file browser. All of these actions are available from a single -dropdown menu. +dropdown list. ## Create a file From a project's files page, select the '+' button to the right of the branch selector. -Choose **New file** from the dropdown. - +Choose **New file** from the dropdown list. + Enter a filename in the **Filename** box. Then, add file content in the editor area. Add a descriptive commit message and choose a branch. The branch field @@ -55,6 +55,18 @@ NOTE: The **Set up CI/CD** button does not appear on an empty repository. For the button to display, add a file to your repository. +## Preview Markdown + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/378966) in GitLab 15.6. + +To preview Markdown content in the Web Editor, select the **Preview** tab. +In this tab, you can see a live Markdown preview that updates as you type alongside your content. + +To close the preview panel, do one of the following: + +- Select the **Write** tab. +- From the context menu, select **Hide Live Preview**. + ## Highlight lines > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/56159) in GitLab 13.10 for GitLab SaaS instances. @@ -85,7 +97,7 @@ this case, you need to upload a file. From a project's files page, select the '+' button to the right of the branch selector. Choose **Upload file** from the dropdown: - + After the upload dialog pops up, there are two ways to upload your file. Either drag and drop a file on the popup or use the **click to upload** link. After you @@ -104,7 +116,7 @@ directory. From a project's files page, select the plus button (`+`) to the right of the branch selector. Choose **New directory** from the dropdown. - + In the new directory dialog, enter a directory name, a commit message, and choose the target branch. Select **Create directory** to finish. @@ -138,6 +150,7 @@ The **Create merge request** button doesn't display if: - A branch with the same name already exists. - A merge request already exists for this branch. - Your project has an active fork relationship. +- Your project is private and the issue is confidential. To make this button appear, one possible workaround is to [remove your project's fork relationship](../settings/index.md#remove-a-fork-relationship). @@ -177,9 +190,9 @@ merge request is merged. If you want to make changes to several files before creating a new merge request, you can create a new branch upfront. -1. From a project's files page, choose **New branch** from the dropdown. +1. From a project's files page, choose **New branch** from the dropdown list. -  +  1. Enter a new **Branch name**. 1. Optional. Change the **Create from** field to choose which branch, tag, or @@ -202,9 +215,9 @@ Tags help you mark major milestones such as production releases and release candidates. You can create a tag from a branch or a commit SHA: -1. From a project's files page, choose **New tag** from the dropdown. +1. From a project's files page, choose **New tag** from the dropdown list. -  +  1. Give the tag a name such as `v1.0.0`. 1. Choose the branch or SHA from which you want to create this new tag. @@ -238,6 +251,6 @@ important to describe those, too. Think of things that may go wrong and include This is important to minimize requests for support, and to avoid doc comments with questions that you know someone might ask. -Each scenario can be a third-level heading, e.g. `### Getting error message X`. +Each scenario can be a third-level heading, for example `### Getting error message X`. If you have none to add when creating a doc, leave this section in place but commented out to help encourage others to add to it in the future. --> diff --git a/doc/user/project/repository/x509_signed_commits/index.md b/doc/user/project/repository/x509_signed_commits/index.md index 4a17b2ab84c..e16f5e4defe 100644 --- a/doc/user/project/repository/x509_signed_commits/index.md +++ b/doc/user/project/repository/x509_signed_commits/index.md @@ -163,6 +163,11 @@ can start signing your tags: ## Troubleshooting +For committers without administrator access, review the list of +[verification problems with signed commits](../gpg_signed_commits/index.md#fix-verification-problems-with-signed-commits) +for possible fixes. The other troubleshooting suggestions on this page require +administrator access. + ### Re-verify commits GitLab stores the status of any checked commits in the database. You can use a diff --git a/doc/user/project/requirements/index.md b/doc/user/project/requirements/index.md index 3b1af1f688c..922accf9d28 100644 --- a/doc/user/project/requirements/index.md +++ b/doc/user/project/requirements/index.md @@ -5,7 +5,7 @@ group: Certify info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- -# Requirements Management **(ULTIMATE)** +# Requirements management **(ULTIMATE)** NOTE: In 14.4, Requirements was moved under **Issues**. @@ -120,8 +120,8 @@ You can search for a requirement from the requirements list page based on the fo To search for a requirement: 1. In a project, go to **Issues > Requirements > List**. -1. Select the **Search or filter results** field. A dropdown menu appears. -1. Select the requirement author or status from the dropdown or enter plain text to search by requirement title. +1. Select the **Search or filter results** field. A dropdown list appears. +1. Select the requirement author or status from the dropdown list or enter plain text to search by requirement title. 1. Press <kbd>Enter</kbd> on your keyboard to filter the list. You can also sort the requirements list by: diff --git a/doc/user/project/settings/import_export.md b/doc/user/project/settings/import_export.md index a47873f5179..3c12bb9b80f 100644 --- a/doc/user/project/settings/import_export.md +++ b/doc/user/project/settings/import_export.md @@ -254,168 +254,3 @@ and the exports between them are compatible. - [Project import/export administration Rake tasks](../../../administration/raketasks/project_import_export.md) - [Group import/export](../../group/settings/import_export.md) - [Group import/export API](../../../api/group_import_export.md) - -## Troubleshooting - -### Project fails to import due to mismatch - -If the [shared runners enablement](../../../ci/runners/runners_scope.md#enable-shared-runners-for-a-project) -does not match between the exported project, and the project import, the project fails to import. -Review [issue 276930](https://gitlab.com/gitlab-org/gitlab/-/issues/276930), and either: - -- Ensure shared runners are enabled in both the source and destination projects. -- Disable shared runners on the parent group when you import the project. - -### Import workarounds for large repositories - -[Maximum import size limitations](#import-a-project-and-its-data) -can prevent an import from being successful. If changing the import limits is not possible, you can -try one of the workarounds listed here. - -#### Workaround option 1 - -The following local workflow can be used to temporarily -reduce the repository size for another import attempt: - -1. Create a temporary working directory from the export: - - ```shell - EXPORT=<filename-without-extension> - - mkdir "$EXPORT" - tar -xf "$EXPORT".tar.gz --directory="$EXPORT"/ - cd "$EXPORT"/ - git clone project.bundle - - # Prevent interference with recreating an importable file later - mv project.bundle ../"$EXPORT"-original.bundle - mv ../"$EXPORT".tar.gz ../"$EXPORT"-original.tar.gz - - git switch --create smaller-tmp-main - ``` - -1. To reduce the repository size, work on this `smaller-tmp-main` branch: - [identify and remove large files](../repository/reducing_the_repo_size_using_git.md) - or [interactively rebase and fixup](../../../topics/git/git_rebase.md#interactive-rebase) - to reduce the number of commits. - - ```shell - # Reduce the .git/objects/pack/ file size - cd project - git reflog expire --expire=now --all - git gc --prune=now --aggressive - - # Prepare recreating an importable file - git bundle create ../project.bundle <default-branch-name> - cd .. - mv project/ ../"$EXPORT"-project - cd .. - - # Recreate an importable file - tar -czf "$EXPORT"-smaller.tar.gz --directory="$EXPORT"/ . - ``` - -1. Import this new, smaller file into GitLab. -1. In a full clone of the original repository, - use `git remote set-url origin <new-url> && git push --force --all` - to complete the import. -1. Update the imported repository's - [branch protection rules](../protected_branches.md) and - its [default branch](../repository/branches/default.md), and - delete the temporary, `smaller-tmp-main` branch, and - the local, temporary data. - -#### Workaround option 2 - -NOTE: -This workaround does not account for LFS objects. - -Rather than attempting to push all changes at once, this workaround: - -- Separates the project import from the Git Repository import -- Incrementally pushes the repository to GitLab - -1. Make a local clone of the repository to migrate. In a later step, you push this clone outside of - the project export. -1. Download the export and remove the `project.bundle` (which contains the Git repository): - - ```shell - tar -czvf new_export.tar.gz --exclude='project.bundle' @old_export.tar.gz - ``` - -1. Import the export without a Git repository. It asks you to confirm to import without a - repository. -1. Save this bash script as a file and run it after adding the appropriate origin. - - ```shell - #!/bin/sh - - # ASSUMPTIONS: - # - The GitLab location is "origin" - # - The default branch is "main" - # - This will attempt to push in chunks of 500MB (dividing the total size by 500MB). - # Decrease this size to push in smaller chunks if you still receive timeouts. - - git gc - SIZE=$(git count-objects -v 2> /dev/null | grep size-pack | awk '{print $2}') - - # Be conservative... and try to push 2GB at a time - # (given this assumes each commit is the same size - which is wrong) - BATCHES=$(($SIZE / 500000)) - TOTAL_COMMITS=$(git rev-list --count HEAD) - if (( BATCHES > TOTAL_COMMITS )); then - BATCHES=$TOTAL_COMMITS - fi - - INCREMENTS=$(( ($TOTAL_COMMITS / $BATCHES) - 1 )) - - for (( BATCH=BATCHES; BATCH>=1; BATCH-- )) - do - COMMIT_NUM=$(( $BATCH - $INCREMENTS )) - COMMIT_SHA=$(git log -n $COMMIT_NUM --format=format:%H | tail -1) - git push -u origin ${COMMIT_SHA}:refs/heads/main - done - git push -u origin main - git push -u origin --all - git push -u origin --tags - ``` - -### Manually execute export steps - -Exports sometimes fail without giving enough information to troubleshoot. In these cases, it can be -helpful to [open a rails console session](../../../administration/operations/rails_console.md#starting-a-rails-console-session) -and loop through [all the defined exporters](https://gitlab.com/gitlab-org/gitlab/-/blob/b67a5b5a12498d57cd877023b7385f7251e57de8/app/services/projects/import_export/export_service.rb#L65). -Execute each line individually, rather than pasting the entire block at once, so you can see any -errors each command returns. - -```shell -# User needs to have permission to export -u = User.find_by_username('someuser') -p = Project.find_by_full_path('some/project') -e = Projects::ImportExport::ExportService.new(p,u) - -e.send(:version_saver).send(:save) -e.send(:repo_saver).send(:save) -## continue using `e.send(:exporter_name).send(:save)` going through the list of exporters - -# The following line should show you the export_path similar to /var/opt/gitlab/gitlab-rails/shared/tmp/gitlab_exports/@hashed/49/94/4994.... -s = Gitlab::ImportExport::Saver.new(exportable: p, shared:p.import_export_shared) - -# To try and upload use: -s.send(:compress_and_save) -s.send(:save_upload) -``` - -### Import using the REST API fails when using a group access token - -[Group access tokens](../../group/settings/group_access_tokens.md) -don't work for project or group import operations. When a group access token initiates an import, -the import fails with this message: - -```plaintext -Error adding importer user to Project members. -Validation failed: User project bots cannot be added to other groups / projects -``` - -To use [Import REST API](../../../api/project_import_export.md), -pass regular user account credentials such as [personal access tokens](../../profile/personal_access_tokens.md). diff --git a/doc/user/project/settings/import_export_troubleshooting.md b/doc/user/project/settings/import_export_troubleshooting.md new file mode 100644 index 00000000000..81ceba7139b --- /dev/null +++ b/doc/user/project/settings/import_export_troubleshooting.md @@ -0,0 +1,280 @@ +--- +stage: Manage +group: Import +info: "To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments" +--- + +# Troubleshooting file export project migrations + +If you have problems with [migrating projects using file exports](import_export.md), see the possible solutions below. + +## Troubleshooting commands + +Finds information about the status of the import and further logs using the JID: + +```ruby +# Rails console +Project.find_by_full_path('group/project').import_state.slice(:jid, :status, :last_error) +> {"jid"=>"414dec93f941a593ea1a6894", "status"=>"finished", "last_error"=>nil} +``` + +```shell +# Logs +grep JID /var/log/gitlab/sidekiq/current +grep "Import/Export error" /var/log/gitlab/sidekiq/current +grep "Import/Export backtrace" /var/log/gitlab/sidekiq/current +tail /var/log/gitlab/gitlab-rails/importer.log +``` + +## Project fails to import due to mismatch + +If the [shared runners enablement](../../../ci/runners/runners_scope.md#enable-shared-runners-for-a-project) +does not match between the exported project, and the project import, the project fails to import. +Review [issue 276930](https://gitlab.com/gitlab-org/gitlab/-/issues/276930), and either: + +- Ensure shared runners are enabled in both the source and destination projects. +- Disable shared runners on the parent group when you import the project. + +## Import workarounds for large repositories + +[Maximum import size limitations](import_export.md#import-a-project-and-its-data) +can prevent an import from being successful. If changing the import limits is not possible, you can +try one of the workarounds listed here. + +### Workaround option 1 + +The following local workflow can be used to temporarily +reduce the repository size for another import attempt: + +1. Create a temporary working directory from the export: + + ```shell + EXPORT=<filename-without-extension> + + mkdir "$EXPORT" + tar -xf "$EXPORT".tar.gz --directory="$EXPORT"/ + cd "$EXPORT"/ + git clone project.bundle + + # Prevent interference with recreating an importable file later + mv project.bundle ../"$EXPORT"-original.bundle + mv ../"$EXPORT".tar.gz ../"$EXPORT"-original.tar.gz + + git switch --create smaller-tmp-main + ``` + +1. To reduce the repository size, work on this `smaller-tmp-main` branch: + [identify and remove large files](../repository/reducing_the_repo_size_using_git.md) + or [interactively rebase and fixup](../../../topics/git/git_rebase.md#interactive-rebase) + to reduce the number of commits. + + ```shell + # Reduce the .git/objects/pack/ file size + cd project + git reflog expire --expire=now --all + git gc --prune=now --aggressive + + # Prepare recreating an importable file + git bundle create ../project.bundle <default-branch-name> + cd .. + mv project/ ../"$EXPORT"-project + cd .. + + # Recreate an importable file + tar -czf "$EXPORT"-smaller.tar.gz --directory="$EXPORT"/ . + ``` + +1. Import this new, smaller file into GitLab. +1. In a full clone of the original repository, + use `git remote set-url origin <new-url> && git push --force --all` + to complete the import. +1. Update the imported repository's + [branch protection rules](../protected_branches.md) and + its [default branch](../repository/branches/default.md), and + delete the temporary, `smaller-tmp-main` branch, and + the local, temporary data. + +### Workaround option 2 + +NOTE: +This workaround does not account for LFS objects. + +Rather than attempting to push all changes at once, this workaround: + +- Separates the project import from the Git Repository import +- Incrementally pushes the repository to GitLab + +1. Make a local clone of the repository to migrate. In a later step, you push this clone outside of + the project export. +1. Download the export and remove the `project.bundle` (which contains the Git repository): + + ```shell + tar -czvf new_export.tar.gz --exclude='project.bundle' @old_export.tar.gz + ``` + +1. Import the export without a Git repository. It asks you to confirm to import without a + repository. +1. Save this bash script as a file and run it after adding the appropriate origin. + + ```shell + #!/bin/sh + + # ASSUMPTIONS: + # - The GitLab location is "origin" + # - The default branch is "main" + # - This will attempt to push in chunks of 500MB (dividing the total size by 500MB). + # Decrease this size to push in smaller chunks if you still receive timeouts. + + git gc + SIZE=$(git count-objects -v 2> /dev/null | grep size-pack | awk '{print $2}') + + # Be conservative... and try to push 2GB at a time + # (given this assumes each commit is the same size - which is wrong) + BATCHES=$(($SIZE / 500000)) + TOTAL_COMMITS=$(git rev-list --count HEAD) + if (( BATCHES > TOTAL_COMMITS )); then + BATCHES=$TOTAL_COMMITS + fi + + INCREMENTS=$(( ($TOTAL_COMMITS / $BATCHES) - 1 )) + + for (( BATCH=BATCHES; BATCH>=1; BATCH-- )) + do + COMMIT_NUM=$(( $BATCH - $INCREMENTS )) + COMMIT_SHA=$(git log -n $COMMIT_NUM --format=format:%H | tail -1) + git push -u origin ${COMMIT_SHA}:refs/heads/main + done + git push -u origin main + git push -u origin --all + git push -u origin --tags + ``` + +## Manually execute export steps + +You usually export a project through [the web interface](import_export.md#export-a-project-and-its-data) or through [the API](../../../api/project_import_export.md). Exporting using these +methods can sometimes fail without giving enough information to troubleshoot. In these cases, +[open a rails console session](../../../administration/operations/rails_console.md#starting-a-rails-console-session) +Execute each line individually, rather than pasting the entire block at once, so you can see any +errors each command returns. + +```shell +# User needs to have permission to export +u = User.find_by_username('someuser') +p = Project.find_by_full_path('some/project') +e = Projects::ImportExport::ExportService.new(p,u) + +e.send(:version_saver).send(:save) +e.send(:repo_saver).send(:save) +## continue using `e.send(:exporter_name).send(:save)` going through the list of exporters + +# The following line should show you the export_path similar to /var/opt/gitlab/gitlab-rails/shared/tmp/gitlab_exports/@hashed/49/94/4994.... +s = Gitlab::ImportExport::Saver.new(exportable: p, shared:p.import_export_shared) + +# To try and upload use: +s.send(:compress_and_save) +s.send(:save_upload) +``` + +After the project is successfully uploaded, the exported project is located in a `.tar.gz` file in `/var/opt/gitlab/gitlab-rails/uploads/-/system/import_export_upload/export_file/`. + +## Import using the REST API fails when using a group access token + +[Group access tokens](../../group/settings/group_access_tokens.md) +don't work for project or group import operations. When a group access token initiates an import, +the import fails with this message: + +```plaintext +Error adding importer user to Project members. +Validation failed: User project bots cannot be added to other groups / projects +``` + +To use [Import REST API](../../../api/project_import_export.md), +pass regular user account credentials such as [personal access tokens](../../profile/personal_access_tokens.md). + +## Troubleshooting performance issues + +Read through the current performance problems using the Import/Export below. + +### OOM errors + +Out of memory (OOM) errors are normally caused by the [Sidekiq Memory Killer](../../../administration/sidekiq/sidekiq_memory_killer.md): + +```shell +SIDEKIQ_MEMORY_KILLER_MAX_RSS = 2000000 +SIDEKIQ_MEMORY_KILLER_HARD_LIMIT_RSS = 3000000 +SIDEKIQ_MEMORY_KILLER_GRACE_TIME = 900 +``` + +An import status `started`, and the following Sidekiq logs signal a memory issue: + +```shell +WARN: Work still in progress <struct with JID> +``` + +### Timeouts + +Timeout errors occur due to the `Gitlab::Import::StuckProjectImportJobsWorker` marking the process as failed: + +```ruby +module Gitlab + module Import + class StuckProjectImportJobsWorker + include Gitlab::Import::StuckImportJob + # ... + end + end +end + +module Gitlab + module Import + module StuckImportJob + # ... + IMPORT_JOBS_EXPIRATION = 15.hours.to_i + # ... + def perform + stuck_imports_without_jid_count = mark_imports_without_jid_as_failed! + stuck_imports_with_jid_count = mark_imports_with_jid_as_failed! + + track_metrics(stuck_imports_with_jid_count, stuck_imports_without_jid_count) + end + # ... + end + end +end +``` + +```shell +Marked stuck import jobs as failed. JIDs: xyz +``` + +```plaintext + +-----------+ +-----------------------------------+ + |Export Job |--->| Calls ActiveRecord `as_json` and | + +-----------+ | `to_json` on all project models | + +-----------------------------------+ + + +-----------+ +-----------------------------------+ + |Import Job |--->| Loads all JSON in memory, then | + +-----------+ | inserts into the DB in batches | + +-----------------------------------+ +``` + +### Problems and solutions + +| Problem | Possible solutions | +| -------- | -------- | +| [Slow JSON](https://gitlab.com/gitlab-org/gitlab/-/issues/25251) loading/dumping models from the database | [split the worker](https://gitlab.com/gitlab-org/gitlab/-/issues/25252) | +| | Batch export +| | Optimize SQL +| | Move away from `ActiveRecord` callbacks (difficult) +| High memory usage (see also some [analysis](https://gitlab.com/gitlab-org/gitlab/-/issues/18857) | DB Commit sweet spot that uses less memory | +| | [Netflix Fast JSON API](https://github.com/Netflix/fast_jsonapi) may help | +| | Batch reading/writing to disk and any SQL + +### Temporary solutions + +While the performance problems are not tackled, there is a process to workaround +importing big projects, using a foreground import: + +[Foreground import](https://gitlab.com/gitlab-com/gl-infra/reliability/-/issues/5384) of big projects for customers. +(Using the import template in the [infrastructure tracker](https://gitlab.com/gitlab-com/gl-infra/reliability/-/issues)) diff --git a/doc/user/project/settings/index.md b/doc/user/project/settings/index.md index 4407986f354..a872a339433 100644 --- a/doc/user/project/settings/index.md +++ b/doc/user/project/settings/index.md @@ -45,7 +45,7 @@ If you're an instance administrator, you can administer all project topics from ## Add a compliance framework to a project **(PREMIUM)** -[Compliance frameworks](../../group/manage.md#compliance-frameworks) can be assigned to projects within group that has a +[Compliance frameworks](../../group/compliance_frameworks.md) can be assigned to projects within group that has a compliance framework using either: - The GitLab UI: @@ -91,9 +91,12 @@ Use the toggles to enable or disable features in the project. | **Wiki** | ✓ | Enables a separate system for [documentation](../wiki/index.md). | | **Snippets** | ✓ | Enables [sharing of code and text](../../snippets.md). | | **Pages** | ✓ | Allows you to [publish static websites](../pages/index.md). | -| **Operations** | ✓ | Control access to Operations-related features, including [Operations Dashboard](../../../operations/index.md), [Environments and Deployments](../../../ci/environments/index.md), [Feature Flags](../../../operations/feature_flags.md). | | **Metrics Dashboard** | ✓ | Control access to [metrics dashboard](../integrations/prometheus.md). | | **Releases** | ✓ | Control access to [Releases](../releases/index.md). | +| **Environments** | ✓ | Control access to [Environments and Deployments](../../../ci/environments/index.md). | +| **Feature flags** | ✓ | Control access to [Feature Flags](../../../operations/feature_flags.md). | +| **Monitor** | ✓ | Control access to [Monitor](../../../operations/index.md) features. | +| **Infrastructure** | ✓ | Control access to [Infrastructure](../../infrastructure/index.md) features. | When you disable a feature, the following additional features are also disabled: @@ -277,7 +280,7 @@ To delete a project: 1. In the "Delete project" section, select **Delete project**. 1. Confirm the action when asked to. -This action deletes a project including all associated resources (issues, merge requests, and so on). +This action deletes a project including all associated resources (such as issues and merge requests). WARNING: The default deletion behavior for projects was changed to [delayed project deletion](https://gitlab.com/gitlab-org/gitlab/-/issues/32935) diff --git a/doc/user/project/web_ide/index.md b/doc/user/project/web_ide/index.md index a0eac9376f2..0200e9c4e7d 100644 --- a/doc/user/project/web_ide/index.md +++ b/doc/user/project/web_ide/index.md @@ -202,12 +202,12 @@ left. ## Switching merge requests To switch between your authored and assigned merge requests, select the -dropdown in the top of the sidebar to open a list of merge requests. You must commit or discard all your changes before switching to a different merge +dropdown list in the top of the sidebar to open a list of merge requests. You must commit or discard all your changes before switching to a different merge request. ## Switching branches -To switch between branches of the current project repository, select the dropdown +To switch between branches of the current project repository, select the dropdown list in the top of the sidebar to open a list of branches. You must commit or discard all your changes before switching to a different branch. @@ -459,3 +459,12 @@ The Web IDE has a few limitations: - If the terminal displays **Connection Failure**, then the terminal could not connect to the runner. Try to stop and restart the terminal. If the problem persists, double check your runner configuration. + +## VSCode Reimplementation + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/95169) in GitLab 15.4 [with a flag](../../../administration/feature_flags.md) named `vscode_web_ide`. Disabled by default. + +As announced in [this blog post](https://about.gitlab.com/blog/2022/05/23/the-future-of-the-gitlab-web-ide/), +the current implementation of the Web IDE will be replaced with a [VSCode inspired implementation](https://gitlab.com/groups/gitlab-org/-/epics/7683). + +This effort is currently under development. Follow [this epic](https://gitlab.com/groups/gitlab-org/-/epics/7683) for updates and more information. diff --git a/doc/user/project/working_with_projects.md b/doc/user/project/working_with_projects.md index 705e49df039..067a0303277 100644 --- a/doc/user/project/working_with_projects.md +++ b/doc/user/project/working_with_projects.md @@ -516,7 +516,7 @@ If a project or repository has been updated but the state is not reflected in th You can do so through [a Rails console session](../../administration/operations/rails_console.md#starting-a-rails-console-session) and one of the following: WARNING: -Any command that changes data directly could be damaging if not run correctly, or under the right conditions. We highly recommend running them in a test environment with a backup of the instance ready to be restored, just in case. +Commands that change data can cause damage if not run correctly or under the right conditions. Always run commands in a test environment first and have a backup instance ready to restore. ```ruby ## Clear project cache @@ -545,7 +545,7 @@ end If a project cannot be deleted, you can attempt to delete it through [Rails console](../../administration/operations/rails_console.md#starting-a-rails-console-session). WARNING: -Any command that changes data directly could be damaging if not run correctly, or under the right conditions. We highly recommend running them in a test environment with a backup of the instance ready to be restored, just in case. +Commands that change data can cause damage if not run correctly or under the right conditions. Always run commands in a test environment first and have a backup instance ready to restore. ```ruby project = Project.find_by_full_path('<project_path>') @@ -569,7 +569,7 @@ To toggle a specific feature, you can [start a Rails console session](../../admi and run the following function: WARNING: -Any command that changes data directly could be damaging if not run correctly, or under the right conditions. We highly recommend running them in a test environment with a backup of the instance ready to be restored, just in case. +Commands that change data can cause damage if not run correctly or under the right conditions. Always run commands in a test environment first and have a backup instance ready to restore. ```ruby projects = Group.find_by_name('_group_name').projects diff --git a/doc/user/public_access.md b/doc/user/public_access.md index 703932e50f6..bdc711f2098 100644 --- a/doc/user/public_access.md +++ b/doc/user/public_access.md @@ -18,6 +18,11 @@ for your GitLab instance). For example, <https://gitlab.com/public>. You can control the visibility of individual features with [project feature settings](permissions.md#project-features). +The visibility setting of a project must be at least as restrictive +as the visibility of its parent group. +For example, a private group can include only private projects, +while a public group can include private, internal, and public projects. + ## Public projects and groups Public projects can be cloned **without any** authentication over HTTPS. @@ -73,11 +78,12 @@ Prerequisite: ## Change group visibility -Prerequisite: +Prerequisites: - You must have the Owner role for a group. - Subgroups and projects must already have visibility settings that are at least as - restrictive as the new setting of the parent group. + restrictive as the new setting of the parent group. For example, you cannot set a group + to private if a subgroup or project in that group is public. 1. On the top bar, select **Main menu > Groups** and find your project. 1. On the left sidebar, select **Settings > General**. @@ -101,6 +107,6 @@ important to describe those, too. Think of things that may go wrong and include This is important to minimize requests for support, and to avoid doc comments with questions that you know someone might ask. -Each scenario can be a third-level heading, e.g. `### Getting error message X`. +Each scenario can be a third-level heading, for example `### Getting error message X`. If you have none to add when creating a doc, leave this section in place but commented out to help encourage others to add to it in the future. --> diff --git a/doc/user/reserved_names.md b/doc/user/reserved_names.md index f8c735eaea8..1ab22fb846d 100644 --- a/doc/user/reserved_names.md +++ b/doc/user/reserved_names.md @@ -26,7 +26,7 @@ under the `TOP_LEVEL_ROUTES`, `PROJECT_WILDCARD_ROUTES` and `GROUP_ROUTES` lists ## Reserved project names -It is currently not possible to create a project with the following names: +It is not possible to create a project with the following names: - `\-` - `badges` @@ -52,7 +52,7 @@ It is currently not possible to create a project with the following names: ## Reserved group names -Currently, the following names are reserved as top level groups: +The following names are reserved as top level groups: - `\-` - `.well-known` diff --git a/doc/user/search/advanced_search.md b/doc/user/search/advanced_search.md index 86b59572e77..925fc7e6036 100644 --- a/doc/user/search/advanced_search.md +++ b/doc/user/search/advanced_search.md @@ -37,9 +37,39 @@ You can use Advanced Search in: ## Syntax -See [Advanced Search syntax](global_search/advanced_search_syntax.md) for more information. +Advanced Search uses [Elasticsearch syntax](https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-simple-query-string-query.html#simple-query-string-syntax). The syntax supports both exact and fuzzy search queries. -## Search by ID +<!-- markdownlint-disable --> -- To search by issue ID, use the `#` prefix followed by the issue ID (for example, [`#23456`](https://gitlab.com/search?snippets=&scope=issues&repository_ref=&search=%2323456&group_id=9970&project_id=278964)). -- To search by merge request ID, use the `!` prefix followed by the merge request ID (for example, [`!23456`](https://gitlab.com/search?snippets=&scope=merge_requests&repository_ref=&search=%2123456&group_id=9970&project_id=278964)). +| Syntax | Description | Example | +|--------------|---------------------------------|------------------------------------------------------------------------------------------------------------------------------------------------------| +| `"` | Exact search | [`"gem sidekiq"`](https://gitlab.com/search?group_id=9970&project_id=278964&scope=blobs&search=%22gem+sidekiq%22) | +| <code>|</code> | Or | [<code>display | banner</code>](https://gitlab.com/search?group_id=9970&project_id=278964&scope=blobs&search=display+%7C+banner) | +| `+` | And | [`display +banner`](https://gitlab.com/search?group_id=9970&project_id=278964&repository_ref=&scope=blobs&search=display+%2Bbanner&snippets=) | +| `-` | Exclude | [`display -banner`](https://gitlab.com/search?group_id=9970&project_id=278964&scope=blobs&search=display+-banner) | +| `*` | Partial | [`bug error 50*`](https://gitlab.com/search?group_id=9970&project_id=278964&repository_ref=&scope=blobs&search=bug+error+50%2A&snippets=) | +| `\` | Escape | [`\*md`](https://gitlab.com/search?snippets=&scope=blobs&repository_ref=&search=%5C*md&group_id=9970&project_id=278964) | +| `#` | Issue ID | [`#23456`](https://gitlab.com/search?snippets=&scope=issues&repository_ref=&search=%2323456&group_id=9970&project_id=278964) | +| `!` | Merge request ID | [`!23456`](https://gitlab.com/search?snippets=&scope=merge_requests&repository_ref=&search=%2123456&group_id=9970&project_id=278964) | + +### Code search + +| Syntax | Description | Example | +|--------------|---------------------------------|------------------------------------------------------------------------------------------------------------------------------------------------------| +| `filename:` | Filename | [`filename:*spec.rb`](https://gitlab.com/search?snippets=&scope=blobs&repository_ref=&search=filename%3A*spec.rb&group_id=9970&project_id=278964) | +| `path:` | Repository location | [`path:spec/workers/`](https://gitlab.com/search?group_id=9970&project_id=278964&repository_ref=&scope=blobs&search=path%3Aspec%2Fworkers&snippets=) | +| `extension:` | File extension without `.` | [`extension:js`](https://gitlab.com/search?group_id=9970&project_id=278964&repository_ref=&scope=blobs&search=extension%3Ajs&snippets=) | +| `blob:` | Git object ID | [`blob:998707*`](https://gitlab.com/search?snippets=false&scope=blobs&repository_ref=&search=blob%3A998707*&group_id=9970) | + +`extension:` and `blob:` return exact matches only. + +### Examples + +| Query | Description | +|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|----------------------------------------------------------------------| +| [`rails -filename:gemfile.lock`](https://gitlab.com/search?group_id=9970&project_id=278964&repository_ref=&scope=blobs&search=rails+-filename%3Agemfile.lock&snippets=) | Returns `rails` in all files except the `gemfile.lock` file. | +| [`RSpec.describe Resolvers -*builder`](https://gitlab.com/search?group_id=9970&project_id=278964&scope=blobs&search=RSpec.describe+Resolvers+-*builder) | Returns `RSpec.describe Resolvers` that does not start with `builder`. | +| [<code>bug | (display +banner)</code>](https://gitlab.com/search?snippets=&scope=issues&repository_ref=&search=bug+%7C+%28display+%2Bbanner%29&group_id=9970&project_id=278964) | Returns `bug` or both `display` and `banner`. | +| [<code>helper -extension:yml -extension:js</code>](https://gitlab.com/search?group_id=9970&project_id=278964&repository_ref=&scope=blobs&search=helper+-extension%3Ayml+-extension%3Ajs&snippets=) | Returns `helper` in all files except files with a `.yml` or `.js` extension. | + +<!-- markdownlint-enable --> diff --git a/doc/user/search/global_search/advanced_search_syntax.md b/doc/user/search/global_search/advanced_search_syntax.md index cd4bff0a20e..c34c5c0c8fc 100644 --- a/doc/user/search/global_search/advanced_search_syntax.md +++ b/doc/user/search/global_search/advanced_search_syntax.md @@ -1,52 +1,11 @@ --- -stage: Data Stores -group: Global Search -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments +redirect_to: '../advanced_search.md' +remove_date: '2023-02-02' --- -# Advanced Search syntax **(PREMIUM)** +This document was moved to [another location](../advanced_search.md). -With [Advanced Search](../advanced_search.md), you can perform a thorough -search of your entire GitLab instance. - -The Advanced Search syntax supports fuzzy or exact search queries with prefixes, -boolean operators, and more. Advanced Search uses -[Elasticsearch's syntax](https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-simple-query-string-query.html#simple-query-string-syntax). - -WARNING: -Advanced Search searches default project branches only. - -## General search - -<!-- markdownlint-disable --> - -| Use | Description | Example | -|-----|--------------|-----------------------------------------------------------------------------------------------------------------------------------------------| -| `"` | Exact search | [`"gem sidekiq"`](https://gitlab.com/search?group_id=9970&project_id=278964&scope=blobs&search=%22gem+sidekiq%22) | -| <code>|</code> | Or | [<code>display | banner</code>](https://gitlab.com/search?group_id=9970&project_id=278964&scope=blobs&search=display+%7C+banner) | -| `+` | And | [`display +banner`](https://gitlab.com/search?group_id=9970&project_id=278964&repository_ref=&scope=blobs&search=display+%2Bbanner&snippets=) | -| `-` | Exclude | [`display -banner`](https://gitlab.com/search?group_id=9970&project_id=278964&scope=blobs&search=display+-banner) | -| `*` | Partial | [`bug error 50*`](https://gitlab.com/search?group_id=9970&project_id=278964&repository_ref=&scope=blobs&search=bug+error+50%2A&snippets=) | -| `\` | Escape | [`\*md`](https://gitlab.com/search?snippets=&scope=blobs&repository_ref=&search=%5C*md&group_id=9970&project_id=278964) | - -## Code search - -| Use | Description | Example | -|--------------|---------------------------------|------------------------------------------------------------------------------------------------------------------------------------------------------| -| `filename:` | Filename | [`filename:*spec.rb`](https://gitlab.com/search?snippets=&scope=blobs&repository_ref=&search=filename%3A*spec.rb&group_id=9970&project_id=278964) | -| `path:` | Repository location | [`path:spec/workers/`](https://gitlab.com/search?group_id=9970&project_id=278964&repository_ref=&scope=blobs&search=path%3Aspec%2Fworkers&snippets=) | -| `extension:` | File extension, without the `.` | [`extension:js`](https://gitlab.com/search?group_id=9970&project_id=278964&repository_ref=&scope=blobs&search=extension%3Ajs&snippets=) | -| `blob:` | Git object ID | [`blob:998707*`](https://gitlab.com/search?snippets=false&scope=blobs&repository_ref=&search=blob%3A998707*&group_id=9970) | - -`extension` and `blob` return exact matches only. - -## Examples - -| Example | Description | -|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|----------------------------------------------------------------------| -| [`rails -filename:gemfile.lock`](https://gitlab.com/search?group_id=9970&project_id=278964&repository_ref=&scope=blobs&search=rails+-filename%3Agemfile.lock&snippets=) | Show _rails_ in all files except the _`gemfile.lock`_ file. | -| [`RSpec.describe Resolvers -*builder`](https://gitlab.com/search?group_id=9970&project_id=278964&scope=blobs&search=RSpec.describe+Resolvers+-*builder) | Show all _RSpec.describe Resolvers_ that don't start with _builder_. | -| [<code>bug | (display +banner)</code>](https://gitlab.com/search?snippets=&scope=issues&repository_ref=&search=bug+%7C+%28display+%2Bbanner%29&group_id=9970&project_id=278964) | Show _bug_ **or** _display_ **and** _banner_. | -| [<code>helper -extension:yml -extension:js</code>](https://gitlab.com/search?group_id=9970&project_id=278964&repository_ref=&scope=blobs&search=helper+-extension%3Ayml+-extension%3Ajs&snippets=) | Show _helper_ in all files, except for files with _`.yml`_ **or** _`.js`_ extensions. | - -<!-- markdownlint-enable --> +<!-- This redirect file can be deleted after <2023-02-02>. --> +<!-- Redirects that point to other docs in the same project expire in three months. --> +<!-- Redirects that point to docs in a different project or site (for example, link is not relative and starts with `https:`) expire in one year. --> +<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/redirects.html --> diff --git a/doc/user/shortcuts.md b/doc/user/shortcuts.md index bf233bef6a2..f9e61ad78ad 100644 --- a/doc/user/shortcuts.md +++ b/doc/user/shortcuts.md @@ -272,7 +272,8 @@ These shortcuts are available when editing a file with the | <kbd>Command</kbd> + <kbd>Shift</kbd> + <kbd>8</kbd> | <kbd>Control</kbd> + <kbd>Shift</kbd> + <kbd>8</kbd> | Unordered list | | <kbd>Command</kbd> + <kbd>Shift</kbd> + <kbd>9</kbd> | <kbd>Control</kbd> + <kbd>Shift</kbd> + <kbd>9</kbd> | Task list | | <kbd>Command</kbd> + <kbd>Shift</kbd> + <kbd>b</kbd> | <kbd>Control</kbd> + <kbd>Shift</kbd> + <kbd>b</kbd> | Blockquote | -| <kbd>Command</kbd> + <kbd>Alt</kbd> + <kbd>c</kbd> | <kbd>Control</kbd> + <kbd>Shift</kbd> + <kbd>c</kbd> | Code block | +| <kbd>Command</kbd> + <kbd>Alt</kbd> + <kbd>c</kbd> | <kbd>Control</kbd> + <kbd>Alt</kbd> + <kbd>c</kbd> | Code block | +| <kbd>Command</kbd> + <kbd>Shift</kbd> + <kbd>h</kbd> | <kbd>Control</kbd> + <kbd>Shift</kbd> + <kbd>h</kbd> | Highlight | | <kbd>Command</kbd> + <kbd>,</kbd> | <kbd>Control</kbd> + <kbd>,</kbd> | Subscript | | <kbd>Command</kbd> + <kbd>.</kbd> | <kbd>Control</kbd> + <kbd>.</kbd> | Superscript | | <kbd>Tab</kbd> | <kbd>Tab</kbd> | Indent list | diff --git a/doc/user/ssh.md b/doc/user/ssh.md index be76ce487b6..85332446334 100644 --- a/doc/user/ssh.md +++ b/doc/user/ssh.md @@ -252,6 +252,29 @@ To generate ED25519_SK or ECDSA_SK SSH keys, you must use OpenSSH 8.2 or later: A public and private key are generated. [Add the public SSH key to your GitLab account](#add-an-ssh-key-to-your-gitlab-account). +## Generate an SSH key pair with a password manager + +### Generate an SSH key pair with 1Password + +You can use [1Password](https://1password.com/) and the [1Password browser extension](https://support.1password.com/getting-started-browser/) to either: + +- Automatically generate a new SSH key. +- Use an existing SSH in your 1Password vault to authenticate with GitLab. + +1. Sign in to GitLab. +1. On the top bar, in the top right corner, select your avatar. +1. Select **Preferences**. +1. On the left sidebar, select **SSH Keys**. +1. Select **Key**, and you should see the 1Password helper appear. +1. Select the 1Password icon and unlock 1Password. +1. You can then select **Create SSH Key** or select an existing SSH key to fill in the public key. +1. In the **Title** box, type a description, like `Work Laptop` or + `Home Workstation`. +1. Optional. Update **Expiration date** to modify the default expiration date. +1. Select **Add key**. + +To learn more about using 1Password with SSH keys, see [1Password's documentation](https://developer.1password.com/docs/ssh/get-started). + ## Add an SSH key to your GitLab account > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/271239) in GitLab 15.4, default expiration date suggested in UI. @@ -499,4 +522,4 @@ You can troubleshoot this by trying the following: - Verify your IDO/U2F hardware security key supports the key type provided. - Verify the version of OpenSSH is 8.2 or greater by - running `ssh -v`. + running `ssh -V`. diff --git a/doc/user/tasks.md b/doc/user/tasks.md index 10a86724bf1..5c9290e57cb 100644 --- a/doc/user/tasks.md +++ b/doc/user/tasks.md @@ -18,7 +18,7 @@ For the latest updates, check the [Tasks Roadmap](https://gitlab.com/groups/gitl FLAG: On self-managed GitLab, by default this feature is available. To hide the feature, -ask an administrator to [disable the feature flags](../administration/feature_flags.md) named `work_items` and `work_items_hierarchy`. +ask an administrator to [disable the feature flags](../administration/feature_flags.md) named `work_items`. On GitLab.com, this feature is available. Use tasks to track steps needed for the [issue](project/issues/index.md) to be closed. @@ -50,9 +50,26 @@ Prerequisites: To create a task: 1. In the issue description, in the **Tasks** section, select **Add**. +1. Select **New task**. 1. Enter the task title. 1. Select **Create task**. +## Add existing tasks to an issue + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/381868) in GitLab 15.6. + +Prerequisites: + +- You must have at least the Guest role for the project, or the project must be public. + +To add a task: + +1. In the issue description, in the **Tasks** section, select **Add**. +1. Select **Existing task**. +1. Search tasks by title. +1. Select one or multiple tasks to add to the issue. +1. Select **Add task**. + ## Edit a task Prerequisites: @@ -159,6 +176,30 @@ To set a start date: The due date must be the same or later than the start date. If you select a start date to be later than the due date, the due date is then changed to the same day. +## Add a task to a milestone + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/367463) in GitLab 15.5 [with a flag](../administration/feature_flags.md) named `work_items_mvc_2`. Disabled by default. + +FLAG: +On self-managed GitLab, by default this feature is not available. To make it available per group, ask an administrator to [enable the feature flag](../administration/feature_flags.md) named `work_items_mvc_2`. On GitLab.com, this feature is not available. The feature is not ready for production use. + +You can add a task to a [milestone](project/milestones/index.md). +You can see the milestone title when you view a task. +If you create a task for an issue that already belongs to a milestone, +the new task inherits the milestone. + +Prerequisites: + +- You must have at least the Reporter role for the project. + +To add a task to a milestone: + +1. In the issue description, in the **Tasks** section, select the title of the task you want to edit. + The task window opens. +1. Next to **Milestone**, select **Add to milestone**. +If a task already belongs to a milestone, the dropdown list shows the current milestone. +1. From the dropdown list, select the milestone to be associated with the task. + ## Set task weight **(PREMIUM)** > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/362550) in GitLab 15.3. diff --git a/doc/user/usage_quotas.md b/doc/user/usage_quotas.md index 1b265e86dd4..7df5ade215d 100644 --- a/doc/user/usage_quotas.md +++ b/doc/user/usage_quotas.md @@ -7,52 +7,6 @@ info: To determine the technical writer assigned to the Stage/Group associated w # Storage usage quota **(FREE)** -> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/13294) in GitLab 12.0. -> - Moved to GitLab Free. - -## Namespace storage limit - -Namespaces on GitLab SaaS have a storage limit. For more information, see our [pricing page](https://about.gitlab.com/pricing/). -This limit is not visible on the Usage quotas page, but will be prior to the limit being [applied](#namespace-storage-limit-application-schedule). Self-managed deployments are not affected. - -Storage types that add to the total namespace storage are: - -- Git repository -- Git LFS -- Artifacts -- Container registry -- Package registry -- Dependency proxy -- Wiki -- Snippets - -If your total namespace storage exceeds the available namespace storage quota, all projects under the namespace are locked. -A locked project cannot push to the repository, run pipelines and jobs, or build and push packages. - -To prevent exceeding the namespace storage quota, you can: - -- Reduce storage consumption by following the suggestions in the [Manage Your Storage Usage](#manage-your-storage-usage) section of this page. -- Apply for [GitLab for Education](https://about.gitlab.com/solutions/education/join/), [GitLab for Open Source](https://about.gitlab.com/solutions/open-source/join/), or [GitLab for Startups](https://about.gitlab.com/solutions/startups/) if you meet the eligibility requirements. -- Consider using a [self-managed instance](../subscriptions/self_managed/index.md) of GitLab which does not have these limits on the free tier. -- [Purchase additional storage](../subscriptions/gitlab_com/index.md#purchase-more-storage-and-transfer) units at $60/year for 10GB of storage. -- [Start a trial](https://about.gitlab.com/free-trial/) or [upgrade to GitLab Premium or Ultimate](https://about.gitlab.com/pricing) which include higher limits and features that enable growing teams to ship faster without sacrificing on quality. -- [Talk to an expert](https://page.gitlab.com/usage_limits_help.html) to learn more about your options and ask questions. - -### Namespace storage limit application schedule - -Information on when namespace-level storage limits will be applied is available on these FAQ pages for the [Free](https://about.gitlab.com/pricing/faq-efficient-free-tier/#storage-limits-on-gitlab-saas-free-tier) and [Paid](https://about.gitlab.com/pricing/faq-paid-storage-transfer/) tier. - -## Project storage limit - -Projects on GitLab SaaS have a 10GB storage limit on their Git repository and LFS storage. -After namespace-level storage limits are applied, the project limit will be removed. A namespace has either a namespace-level storage limit or a project-level storage limit, but not both. - -When a project's repository and LFS reaches the quota, the project is locked. -You cannot push changes to a locked project. To monitor the size of each -repository in a namespace, including a breakdown for each project, -[view storage usage](#view-storage-usage). To allow a project's repository and LFS to exceed the free quota -you must purchase additional storage. For more details, see [Excess storage usage](#excess-storage-usage). - ## View storage usage Prerequisites: @@ -72,7 +26,7 @@ is updated every 90 minutes. If your namespace shows `'Not applicable.'`, push a commit to any project in the namespace to recalculate the storage. -## Storage usage statistics +### Storage usage statistics > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/68898) project-level graph in GitLab 14.4 [with a flag](../administration/feature_flags.md) named `project_storage_ui`. Disabled by default. > - Enabled on GitLab.com in GitLab 14.4. @@ -97,7 +51,22 @@ Depending on your role, you can also use the following methods to manage or redu - [Reduce container registry storage](packages/container_registry/reduce_container_registry_storage.md). - [Reduce wiki repository size](../administration/wikis/index.md#reduce-wiki-repository-size). -## Excess storage usage +## Manage your transfer usage + +Depending on your role, to manage your transfer usage you can [reduce Container Registry data transfers](packages/container_registry/reduce_container_registry_data_transfer.md). + +## Project storage limit + +Projects on GitLab SaaS have a 10GB storage limit on their Git repository and LFS storage. +After namespace-level storage limits are applied, the project limit will be removed. A namespace has either a namespace-level storage limit or a project-level storage limit, but not both. + +When a project's repository and LFS reaches the quota, the project is locked. +You cannot push changes to a locked project. To monitor the size of each +repository in a namespace, including a breakdown for each project, +[view storage usage](#view-storage-usage). To allow a project's repository and LFS to exceed the free quota +you must purchase additional storage. For more details, see [Excess storage usage](#excess-storage-usage). + +### Excess storage usage Excess storage usage is the amount that a project's repository and LFS exceeds the [project storage limit](#project-storage-limit). If no purchased storage is available the project is locked. You cannot push changes to a locked project. @@ -112,7 +81,7 @@ The **Storage** tab of the **Usage Quotas** page warns you of the following: - Projects that are locked because purchased storage available is zero. Locked projects are marked with an information icon (**{information-o}**) beside their name. -### Excess storage example +#### Excess storage example The following example describes an excess storage scenario for a namespace: @@ -141,8 +110,34 @@ available decreases. All projects remain unlocked because 40 GB purchased storag | Yellow | 5 GB | 0 GB | 10 GB | Not locked | | **Totals** | **45 GB** | **10 GB** | - | - | -## Manage your transfer usage +## Namespace storage limit -Depending on your role, you can use the following methods to manage or reduce your transfer: +Namespaces on GitLab SaaS have a storage limit. For more information, see our [pricing page](https://about.gitlab.com/pricing/). +This limit is not visible on the **Usage quotas** page, but will be prior to the limit being [applied](#namespace-storage-limit-application-schedule). Self-managed deployments are not affected. + +Storage types that add to the total namespace storage are: + +- Git repository +- Git LFS +- Artifacts +- Container registry +- Package registry +- Dependency proxy +- Wiki +- Snippets + +If your total namespace storage exceeds the available namespace storage quota, all projects under the namespace are locked. +A locked project cannot push to the repository, run pipelines and jobs, or build and push packages. + +To prevent exceeding the namespace storage quota, you can: + +- Reduce storage consumption by following the suggestions in the [Manage Your Storage Usage](#manage-your-storage-usage) section of this page. +- Apply for [GitLab for Education](https://about.gitlab.com/solutions/education/join/), [GitLab for Open Source](https://about.gitlab.com/solutions/open-source/join/), or [GitLab for Startups](https://about.gitlab.com/solutions/startups/) if you meet the eligibility requirements. +- Consider using a [self-managed instance](../subscriptions/self_managed/index.md) of GitLab which does not have these limits on the free tier. +- [Purchase additional storage](../subscriptions/gitlab_com/index.md#purchase-more-storage-and-transfer) units at $60/year for 10GB of storage. +- [Start a trial](https://about.gitlab.com/free-trial/) or [upgrade to GitLab Premium or Ultimate](https://about.gitlab.com/pricing/) which include higher limits and features that enable growing teams to ship faster without sacrificing on quality. +- [Talk to an expert](https://page.gitlab.com/usage_limits_help.html) to learn more about your options and ask questions. + +### Namespace storage limit application schedule -- [Reduce Container Registry data transfers](packages/container_registry/reduce_container_registry_data_transfer.md). +Information on when namespace-level storage limits will be applied is available on these FAQ pages for the [Free](https://about.gitlab.com/pricing/faq-efficient-free-tier/#storage-limits-on-gitlab-saas-free-tier) and [Paid](https://about.gitlab.com/pricing/faq-paid-storage-transfer/) tier. diff --git a/doc/user/workspace/index.md b/doc/user/workspace/index.md index d7e014672aa..5bcd96cd4a5 100644 --- a/doc/user/workspace/index.md +++ b/doc/user/workspace/index.md @@ -15,7 +15,7 @@ The development, release, and timing of any products, features, or functionality sole discretion of GitLab Inc. NOTE: -Workspace is currently in development. +Workspace is in development. Workspace will be above the [top-level namespaces](../namespace/index.md) for you to manage everything you do as a GitLab administrator, including: |
