diff options
Diffstat (limited to 'doc/user')
219 files changed, 5753 insertions, 4122 deletions
diff --git a/doc/user/admin_area/external_users.md b/doc/user/admin_area/external_users.md new file mode 100644 index 00000000000..8b968a3da01 --- /dev/null +++ b/doc/user/admin_area/external_users.md @@ -0,0 +1,77 @@ +--- +stage: Manage +group: Authentication and Authorization +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments +--- + +# External users **(FREE SELF)** + +In cases where it is desired that a user has access only to some internal or +private projects, there is the option of creating **External Users**. This +feature may be useful when for example a contractor is working on a given +project and should only have access to that project. + +External users: + +- Cannot create project, groups, and snippets in their personal namespaces. +- Can only create projects (including forks), subgroups, and snippets within top-level groups to which they are explicitly granted access. +- Can only access public projects and projects to which they are explicitly granted access, + thus hiding all other internal or private ones from them (like being + logged out). +- Can only access public groups and groups to which they are explicitly granted access, + thus hiding all other internal or private ones from them (like being + logged out). +- Can only access public snippets. + +Access can be granted by adding the user as member to the project or group. +Like usual users, they receive a role in the project or group with all +the abilities that are mentioned in the [permissions table](../permissions.md#project-members-permissions). +For example, if an external user is added as Guest, and your project is internal or +private, they do not have access to the code; you need to grant the external +user access at the Reporter level or above if you want them to have access to the code. You should +always take into account the +[project's visibility and permissions settings](../project/settings/index.md#configure-project-visibility-features-and-permissions) +as well as the permission level of the user. + +NOTE: +External users still count towards a license seat. + +An administrator can flag a user as external by either of the following methods: + +- [Through the API](../../api/users.md#user-modification). +- Using the GitLab UI: + 1. On the top bar, select **Main menu > Admin**. + 1. On the left sidebar, select **Overview > Users** to create a new user or edit an existing one. + There, you can find the option to flag the user as external. + +Additionally, users can be set as external users using: + +- [SAML groups](../../integration/saml.md#external-groups). +- [LDAP groups](../../administration/auth/ldap/ldap_synchronization.md#external-groups). + +## Set a new user to external + +By default, new users are not set as external users. This behavior can be changed +by an administrator: + +1. On the top bar, select **Main menu > Admin**. +1. On the left sidebar, select **Settings > General**. +1. Expand the **Account and limit** section. + +If you change the default behavior of creating new users as external, you +have the option to narrow it down by defining a set of internal users. +The **Internal users** field allows specifying an email address regex pattern to +identify default internal users. New users whose email address matches the regex +pattern are set to internal by default rather than an external collaborator. + +The regex pattern format is in Ruby, but it needs to be convertible to JavaScript, +and the ignore case flag is set (`/regex pattern/i`). Here are some examples: + +- Use `\.internal@domain\.com$` to mark email addresses ending with + `.internal@domain.com` as internal. +- Use `^(?:(?!\.ext@domain\.com).)*$\r?` to mark users with email addresses + not including `.ext@domain.com` as internal. + +WARNING: +Be aware that this regex could lead to a +[regular expression denial of service (ReDoS) attack](https://en.wikipedia.org/wiki/ReDoS). diff --git a/doc/user/admin_area/index.md b/doc/user/admin_area/index.md index 4c34d82dc02..c9b6a077c73 100644 --- a/doc/user/admin_area/index.md +++ b/doc/user/admin_area/index.md @@ -163,7 +163,7 @@ You can impersonate a user in the following ways: 1. Select **Impersonate**. - With the API, using [impersonation tokens](../../api/index.md#impersonation-tokens). -All impersonation activities are [captured with audit events](../../administration/audit_events.md#impersonation-data). +All impersonation activities are [captured with audit events](../../administration/audit_events.md#user-impersonation). By default, impersonation is enabled. GitLab can be configured to [disable impersonation](../../api/index.md#disable-impersonation). diff --git a/doc/user/admin_area/license_file.md b/doc/user/admin_area/license_file.md index d821d9bab23..9c35a8c1269 100644 --- a/doc/user/admin_area/license_file.md +++ b/doc/user/admin_area/license_file.md @@ -25,7 +25,8 @@ Otherwise, to add your license: 1. Select **Add license**. NOTE: -In GitLab 14.1.x through 14.10.x, you can access the **Add License** page directly from the URL, `<YourGitLabURL>/admin/license/new`. In GitLab 15.0 and later, the path is `<YourGitLabURL>/admin/subscription`. +In GitLab 14.7.x to 14.9.x, you can add the license file with the UI. +In GitLab 14.1.x to 14.7, if you have already activated your subscription with an activation code, you cannot access **Add License** from the Admin Area. You must access **Add License** directly from the URL, `<YourGitLabURL>/admin/license/new`. ## Add your license file during installation diff --git a/doc/user/admin_area/monitoring/background_migrations.md b/doc/user/admin_area/monitoring/background_migrations.md index f16cb219f2b..b4a6f7f66fb 100644 --- a/doc/user/admin_area/monitoring/background_migrations.md +++ b/doc/user/admin_area/monitoring/background_migrations.md @@ -1,244 +1,11 @@ --- -stage: Data Stores -group: Database -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments +redirect_to: '../../../update/background_migrations.md' +remove_date: '2023-03-11' --- -# Batched background migrations **(FREE SELF)** +This document was moved to [another location](../../../update/background_migrations.md). -> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/51332) in GitLab 13.11. -> - [Deployed behind a feature flag](../../../user/feature_flags.md), disabled by default. -> - [Enabled by default](https://gitlab.com/gitlab-org/gitlab/-/issues/329511) in GitLab 13.12. -> - Enabled on GitLab.com. -> - Recommended for production use. -> - For GitLab self-managed instances, GitLab administrators can opt to [disable it](#enable-or-disable-batched-background-migrations). - -There can be [risks when disabling released features](../../../administration/feature_flags.md#risks-when-disabling-released-features). -Refer to this feature's version history for more details. - -To update database tables in batches, GitLab can use batched background migrations. These migrations -are created by GitLab developers and run automatically on upgrade. However, such migrations are -limited in scope to help with migrating some `integer` database columns to `bigint`. This is needed to -prevent integer overflow for some tables. - -## Check the status of background migrations - -All migrations must have a `Finished` status before you [upgrade GitLab](../../../update/index.md). -You can [check the status of existing migrations](../../../update/index.md#batched-background-migrations). - -## Enable or disable batched background migrations - -WARNING: -If you disable this feature flag, GitLab upgrades may fail. - -Batched background migrations are under development but ready for production use. -It is deployed behind a feature flag that is **enabled by default**. -[GitLab administrators with access to the GitLab Rails console](../../../administration/feature_flags.md) -can opt to disable it. - -To enable it: - -```ruby -Feature.enable(:execute_batched_migrations_on_schedule) -``` - -To disable it: - -```ruby -Feature.disable(:execute_batched_migrations_on_schedule) -``` - -### Pause batched background migrations in GitLab 14.x - -To pause an ongoing batched background migration, use the `disable` command above. -This command causes the migration to complete the current batch, and then wait to start the next batch. - -Use the following database queries to see the state of the current batched background migration: - -1. Obtain the ID of the running migration: - - ```sql - SELECT - id, - job_class_name, - table_name, - column_name, - job_arguments - FROM batched_background_migrations - WHERE status <> 3; - ``` - -1. Run this query, replacing `XX` with the ID you obtained in the previous step, - to see the status of the migration: - - ```sql - SELECT - started_at, - finished_at, - finished_at - started_at AS duration, - min_value, - max_value, - batch_size, - sub_batch_size - FROM batched_background_migration_jobs - WHERE batched_background_migration_id = XX - ORDER BY id DESC - limit 10; - ``` - -1. Run the query multiple times within a few minutes to ensure no new row has been added. - If no new row has been added, the migration has been paused. - -1. After confirming the migration has paused, restart the migration (using the `enable` - command above) to proceed with the batch when ready. On larger instances, - background migrations can take as long as 48 hours to complete each batch. - -## Automatic batch size optimization - -> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/60133) in GitLab 13.12. -> - [Deployed behind a feature flag](../../../user/feature_flags.md), disabled by default. -> - [Enabled by default](https://gitlab.com/gitlab-org/gitlab/-/issues/329511) in GitLab 13.12. -> - Enabled on GitLab.com. -> - Recommended for production use. -> - For GitLab self-managed instances, GitLab administrators can opt to [disable it](#enable-or-disable-automatic-batch-size-optimization). - -There can be [risks when disabling released features](../../../administration/feature_flags.md#risks-when-disabling-released-features). -Refer to this feature's version history for more details. - -To maximize throughput of batched background migrations (in terms of the number of tuples updated per time unit), batch sizes are automatically adjusted based on how long the previous batches took to complete. - -## Enable or disable automatic batch size optimization - -Automatic batch size optimization for batched background migrations is under development but ready for production use. -It is deployed behind a feature flag that is **enabled by default**. -[GitLab administrators with access to the GitLab Rails console](../../../administration/feature_flags.md) -can opt to disable it. - -To enable it: - -```ruby -Feature.enable(:optimize_batched_migrations) -``` - -To disable it: - -```ruby -Feature.disable(:optimize_batched_migrations) -``` - -## Troubleshooting - -### Database migrations failing because of batched background migration not finished - -When updating to GitLab 14.2 or later there might be a database migration failing with a message like: - -```plaintext -StandardError: An error has occurred, all later migrations canceled: - -Expected batched background migration for the given configuration to be marked as 'finished', but it is 'active': - {:job_class_name=>"CopyColumnUsingBackgroundMigrationJob", :table_name=>"push_event_payloads", :column_name=>"event_id", :job_arguments=>[["event_id"], ["event_id_convert_to_bigint"]]} -``` - -First, check if you have followed the [version-specific upgrade instructions for 14.2](../../../update/index.md#1420). -If you have, you can [manually finish the batched background migration](#manually-finishing-a-batched-background-migration). -If you haven't, choose one of the following methods: - -1. [Rollback and upgrade](#roll-back-and-follow-the-required-upgrade-path) through one of the required -versions before updating to 14.2+. -1. [Roll forward](#roll-forward-and-finish-the-migrations-on-the-upgraded-version), staying on the current -version and manually ensuring that the batched migrations complete successfully. - -#### Roll back and follow the required upgrade path - -1. [Rollback and restore the previously installed version](../../../update/restore_after_failure.md#roll-back-to-an-earlier-version-and-restore-a-backup) -1. Update to either 14.0.5 or 14.1 **before** updating to 14.2+ -1. [Check the status](#check-the-status-of-background-migrations) of the batched background migrations and -make sure they are all marked as finished before attempting to upgrade again. If any remain marked as active, -you can [manually finish them](#manually-finishing-a-batched-background-migration). - -#### Roll forward and finish the migrations on the upgraded version - -##### For a deployment with downtime - -To run all the batched background migrations, it can take a significant amount of time -depending on the size of your GitLab installation. - -1. [Check the status](#check-the-status-of-background-migrations) of the batched background migrations in the -database, and [manually run them](#manually-finishing-a-batched-background-migration) with the appropriate -arguments until the status query returns no rows. -1. When the status of all of all them is marked as complete, re-run migrations for your installation. -1. [Complete the database migrations](../../../administration/raketasks/maintenance.md#run-incomplete-database-migrations) from your GitLab upgrade: - - ```plaintext - sudo gitlab-rake db:migrate - ``` - -1. Run a reconfigure: - - ```plaintext - sudo gitlab-ctl reconfigure - ``` - -1. Finish the upgrade for your installation. - -##### For a no-downtime deployment - -As the failing migrations are post-deployment migrations, you can remain on a running instance of the upgraded -version and wait for the batched background migrations to finish normally. - -1. [Check the status](#check-the-status-of-background-migrations) of the batched background migration from -the error message, and make sure it is listed as finished. If it is still active, either wait until it is done, -or [manually finish it](#manually-finishing-a-batched-background-migration). -1. Re-run migrations for your installation, so the remaining post-deployment migrations finish. - -### Manually finishing a batched background migration - -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/62634) in GitLab 14.1 - -If you need to manually finish a batched background migration due to an -error, you can run: - -```shell -sudo gitlab-rake gitlab:background_migrations:finalize[<job_class_name>,<table_name>,<column_name>,'<job_arguments>'] -``` - -Replace the values in angle brackets with the correct -arguments. For example, if you receive an error similar to this: - -```plaintext -StandardError: An error has occurred, all later migrations canceled: - -Expected batched background migration for the given configuration to be marked as 'finished', but it is 'active': - {:job_class_name=>"CopyColumnUsingBackgroundMigrationJob", :table_name=>"push_event_payloads", :column_name=>"event_id", :job_arguments=>[["event_id"], ["event_id_convert_to_bigint"]]} -``` - -Plug the arguments from the error message into the command: - -```shell -sudo gitlab-rake gitlab:background_migrations:finalize[CopyColumnUsingBackgroundMigrationJob,push_event_payloads,event_id,'[["event_id"]\, ["event_id_convert_to_bigint"]]'] -``` - -If you need to manually run a batched background migration to continue an upgrade, you can -[check the status](#check-the-status-of-background-migrations) in the database and get the -arguments from the query results. For example, if the query returns this: - -```plaintext - job_class_name | table_name | column_name | job_arguments ----------------------------------------+------------+-------------+------------------------------------ - CopyColumnUsingBackgroundMigrationJob | events | id | [["id"], ["id_convert_to_bigint"]] - ``` - -The results from the query can be plugged into the command: - -```shell -sudo gitlab-rake gitlab:background_migrations:finalize[CopyColumnUsingBackgroundMigrationJob,events,id,'[["id"]\, ["id_convert_to_bigint"]]'] -``` - -### The `BackfillNamespaceIdForNamespaceRoute` batched migration job fails - -In GitLab 14.8, the `BackfillNamespaceIdForNamespaceRoute` batched background migration job -may fail to complete. When retried, a `500 Server Error` is returned. This issue was -[resolved](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/82387) in GitLab 14.9. - -To resolve this issue, [upgrade GitLab](../../../update/index.md) from 14.8 to 14.9. -You can ignore the failed batch migration until after you update to GitLab 14.9. +<!-- This redirect file can be deleted after <2023-03-11>. --> +<!-- Redirects that point to other docs in the same project expire in three months. --> +<!-- Redirects that point to docs in a different project or site (link is not relative and starts with `https:`) expire in one year. --> +<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/redirects.html --> diff --git a/doc/user/admin_area/monitoring/health_check.md b/doc/user/admin_area/monitoring/health_check.md index 2939a8b0418..668d34af024 100644 --- a/doc/user/admin_area/monitoring/health_check.md +++ b/doc/user/admin_area/monitoring/health_check.md @@ -133,7 +133,7 @@ This check is being exempt from Rack Attack. ## Sidekiq -Learn how to configure the [Sidekiq health checks](../../../administration/sidekiq_health_check.md). +Learn how to configure the [Sidekiq health checks](../../../administration/sidekiq/sidekiq_health_check.md). <!-- ## Troubleshooting diff --git a/doc/user/admin_area/settings/account_and_limit_settings.md b/doc/user/admin_area/settings/account_and_limit_settings.md index 44a6bbb9d8e..b235b812416 100644 --- a/doc/user/admin_area/settings/account_and_limit_settings.md +++ b/doc/user/admin_area/settings/account_and_limit_settings.md @@ -57,7 +57,7 @@ You can change the maximum push size for your instance: 1. On the left sidebar, select **Settings > General**, then expand **Account and limit**. 1. Increase or decrease by changing the value in **Maximum push size (MB)**. -For GitLab.com application limits, read [GitLab application limits](../../../administration/instance_limits.md#max-push-size). +For GitLab.com push size limits, read [accounts and limit settings](../../gitlab_com/index.md#account-and-limit-settings). NOTE: When you [add files to a repository](../../project/repository/web_editor.md#create-a-file) diff --git a/doc/user/admin_area/settings/continuous_integration.md b/doc/user/admin_area/settings/continuous_integration.md index adca9c85af1..81e51aaef37 100644 --- a/doc/user/admin_area/settings/continuous_integration.md +++ b/doc/user/admin_area/settings/continuous_integration.md @@ -46,7 +46,10 @@ limit on the number of [CI/CD minutes](../../../ci/pipelines/cicd_minutes.md) yo ## Enable a specific runner for multiple projects -To enable a specific runner for one or more projects: +If you have already registered a [specific runner](../../../ci/runners/runners_scope.md#specific-runners) +you can assign that runner to other projects. + +To enable a specific runner for more than one project: 1. On the top bar, select **Main menu > Admin**. 1. From the left sidebar, select **Overview > Runners**. @@ -326,7 +329,7 @@ To set the maximum file size: 1. Enter the maximum file size, in bytes. 1. Select **Save size limits**. -## Prevent users from registering runners +## Restrict runner registration by all users in an instance > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/22225) in GitLab 14.1. > - [Enabled on GitLab.com and self-managed](https://gitlab.com/gitlab-org/gitlab/-/issues/368008) in GitLab 15.5. @@ -335,7 +338,7 @@ GitLab administrators can adjust who is allowed to register runners, by showing By default, all members of a project and group are able to register runners. -To change this: +To restrict all users in an instance from registering runners: 1. On the top bar, select **Main menu > Admin**. 1. On the left sidebar, select **Settings > CI/CD**. @@ -347,6 +350,22 @@ To change this: WARNING: When the registration sections are hidden in the UI, members of the project or group that need to register runners must contact the administrators. If you plan to prevent registration, ensure users have access to the runners they need to run jobs. +## Restrict runner registration by all members in a group + +Prerequisites: + +- Runner registration must be enabled for [all users in the instance](#restrict-runner-registration-by-all-users-in-an-instance). + +GitLab administrators can adjust group permissions to restrict runner registration by group members. + +To restrict runner registration by members in a specific group: + +1. On the top bar, select **Main menu > Admin**. +1. On the left sidebar, select **Overview > Groups** and find your group. +1. Select **Edit**. +1. Clear the **New group runners can be registered** checkbox if you want to disable runner registration by all members in the group. If the setting is read-only, you must enable runner registration for the [instance](#restrict-runner-registration-by-all-users-in-an-instance). +1. Select **Save changes**. + ## Troubleshooting ### 413 Request Entity Too Large diff --git a/doc/user/admin_area/settings/img/mirror_settings.png b/doc/user/admin_area/settings/img/mirror_settings.png Binary files differdeleted file mode 100644 index 090db6808a7..00000000000 --- a/doc/user/admin_area/settings/img/mirror_settings.png +++ /dev/null diff --git a/doc/user/admin_area/settings/img/mirror_settings_v15_7.png b/doc/user/admin_area/settings/img/mirror_settings_v15_7.png Binary files differnew file mode 100644 index 00000000000..5da41dbeceb --- /dev/null +++ b/doc/user/admin_area/settings/img/mirror_settings_v15_7.png diff --git a/doc/user/admin_area/settings/index.md b/doc/user/admin_area/settings/index.md index 08c4a0d0167..b2b702bde7c 100644 --- a/doc/user/admin_area/settings/index.md +++ b/doc/user/admin_area/settings/index.md @@ -200,3 +200,12 @@ for the entire GitLab instance: 1. On the top bar, select **Main menu > Admin**. 1. On the left sidebar, select **Settings > Preferences**. 1. Scroll to the **Localization** section, and select your desired first day of the week. + +## Default language + +You can change the [Default language](../../profile/preferences.md) +for the entire GitLab instance: + +1. On the top bar, select **Main menu > Admin**. +1. On the left sidebar, select **Settings > Preferences**. +1. Scroll to the **Localization** section, and select your desired default language. diff --git a/doc/user/admin_area/settings/sidekiq_job_limits.md b/doc/user/admin_area/settings/sidekiq_job_limits.md index 7a437d877ca..a25bc0e85e4 100644 --- a/doc/user/admin_area/settings/sidekiq_job_limits.md +++ b/doc/user/admin_area/settings/sidekiq_job_limits.md @@ -9,7 +9,7 @@ type: reference > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/68982) in GitLab 14.3. -[Sidekiq](../../../administration/sidekiq.md) jobs get stored in +[Sidekiq](../../../administration/sidekiq/index.md) jobs get stored in Redis. To avoid excessive memory for Redis, we: - Compress job arguments before storing them in Redis. diff --git a/doc/user/admin_area/settings/sign_in_restrictions.md b/doc/user/admin_area/settings/sign_in_restrictions.md index 4ea420d7ca6..6ec3d082114 100644 --- a/doc/user/admin_area/settings/sign_in_restrictions.md +++ b/doc/user/admin_area/settings/sign_in_restrictions.md @@ -68,7 +68,7 @@ For more information, see the [list of settings that can be accessed through API Open the [Rails console](../../../administration/operations/rails_console.md) and run the following: ```ruby -::Gitlab::CurrentSettings.update_attributes!(admin_mode: true) +::Gitlab::CurrentSettings.update!(admin_mode true) ``` #### Use the UI to enable Admin Mode @@ -84,7 +84,7 @@ To enable Admin Mode through the UI: To turn on Admin Mode for your current session and access potentially dangerous resources: -1. On the top bar, select **Enable Admin Mode**. +1. On the top bar, select **Main menu > Enter Admin Mode**. 1. Try to access any part of the UI with `/admin` in the URL (which requires administrator access). When Admin Mode status is disabled or turned off, administrators cannot access resources unless @@ -95,7 +95,11 @@ if they try to open a private group or project, unless they are members of that authentication are supported by Admin Mode. Admin Mode status is stored in the current user session and remains active until either: - It is explicitly disabled. -- It is disabled automatically after a timeout. +- It is disabled automatically after six hours. + +### Turn off Admin Mode for your session + +To turn off Admin Mode for your current session, on the top bar, select **Main menu > Leave Admin mode**. ### Limitations of Admin Mode diff --git a/doc/user/admin_area/settings/sign_up_restrictions.md b/doc/user/admin_area/settings/sign_up_restrictions.md index 76415596dce..8aabe503065 100644 --- a/doc/user/admin_area/settings/sign_up_restrictions.md +++ b/doc/user/admin_area/settings/sign_up_restrictions.md @@ -60,7 +60,7 @@ To enforce confirmation of the email address used for new sign ups: 1. On the top bar, select **Main menu > Admin**. 1. On the left sidebar, select **Settings > General**, and expand **Sign-up restrictions**. -1. Select the **Send confirmation email on sign-up** checkbox, then select **Save changes**. +1. Under **Email confirmation settings**, select **Hard**. ## User cap @@ -190,6 +190,12 @@ To disable it: Feature.disable(:soft_email_confirmation) ``` +## Set up LDAP user filter + +You can limit GitLab access to a subset of the LDAP users on your LDAP server. + +See the [documentation on setting up an LDAP user filter](../../../administration/auth/ldap/index.md#set-up-ldap-user-filter) for more information. + <!-- ## Troubleshooting Include any troubleshooting steps that you can foresee. If you know beforehand what issues diff --git a/doc/user/admin_area/settings/terraform_limits.md b/doc/user/admin_area/settings/terraform_limits.md new file mode 100644 index 00000000000..4e54c7a3459 --- /dev/null +++ b/doc/user/admin_area/settings/terraform_limits.md @@ -0,0 +1,27 @@ +--- +stage: Configure +group: Configure +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments +type: reference +--- + +# Terraform limits **(FREE SELF)** + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/352951) in GitLab 15.7. + +You can limit the total storage of [Terraform state files](../../../administration/terraform_state.md). +The limit applies to each individual +state file version, and is checked whenever a new version is created. + +To add a storage limit: + +1. On the top bar, select **Main menu > Admin**. +1. On the left sidebar, select **Settings > Preferences**. +1. Expand **Terraform limits**. +1. Adjust the size limit. + +## Available settings + +| Setting | Default | Description | +|------------------------------------|---------|---------------------------------------------------------------------------------------------------------------------------------------------------------| +| Terraform state size limit (bytes) | 0 | Terraform state files that exceed this size are not saved, and associated Terraform operations are rejected. Set to 0 to allow files of unlimited size. | diff --git a/doc/user/admin_area/settings/visibility_and_access_controls.md b/doc/user/admin_area/settings/visibility_and_access_controls.md index 6d878bcb01c..4da0f5da3f4 100644 --- a/doc/user/admin_area/settings/visibility_and_access_controls.md +++ b/doc/user/admin_area/settings/visibility_and_access_controls.md @@ -219,7 +219,7 @@ If only one protocol is enabled: - The project page shows only the allowed protocol's URL, with no option to change it. -- GitLab shows a tooltip when you hover over the URL's protocol, if user action +- GitLab shows a tooltip when you hover over the protocol for the URL, if user action (such as adding a SSH key or setting a password) is required:  @@ -273,7 +273,7 @@ This option is enabled by default. By disabling it, both [pull mirroring](../../project/repository/mirror/pull.md) and [push mirroring](../../project/repository/mirror/push.md) no longer work in every repository. They can only be re-enabled by an administrator user on a per-project basis. - + ## Configure globally-allowed IP address ranges diff --git a/doc/user/analytics/dora_metrics.md b/doc/user/analytics/dora_metrics.md index b5f37203817..fba0d0e98ff 100644 --- a/doc/user/analytics/dora_metrics.md +++ b/doc/user/analytics/dora_metrics.md @@ -104,7 +104,7 @@ To retrieve metrics for change failure rate, use the [GraphQL](../../api/graphql ### Insights: Custom DORA reporting -Custom charts to visualize DORA data with Insights YAML-based reports. +Custom charts to visualize DORA data with [Insights YAML-based reports](../../user/project/insights/index.md#dora-query-parameters). With this new visualization, software leaders can track metrics improvements, understand patterns in their metrics trends, and compare performance between groups and projects. @@ -113,7 +113,7 @@ With this new visualization, software leaders can track metrics improvements, un Deployment frequency is calculated based on the deployments record, which is created for typical push-based deployments. These deployment records are not created for pull-based deployments, for example when Container Images are connected to GitLab with an agent. -To track DORA metrics in these cases, you can [create a deployment record](../../api/deployments.md#create-a-deployment) using the Deployments API. +To track DORA metrics in these cases, you can [create a deployment record](../../api/deployments.md#create-a-deployment) using the Deployments API. See also the documentation page for [Track deployments of an external deployment tool](../../ci/environments/external_deployment_tools.md). ### Supported DORA metrics in GitLab diff --git a/doc/user/analytics/index.md b/doc/user/analytics/index.md index 01fe466755c..38d92180c7d 100644 --- a/doc/user/analytics/index.md +++ b/doc/user/analytics/index.md @@ -12,7 +12,7 @@ Instance-level analytics make it possible to aggregate analytics across GitLab, so that users can view information across multiple projects and groups in one place. -[Learn more about instance-level analytics](../admin_area/analytics/index.md). +For more information, see [instance-level analytics](../admin_area/analytics/index.md). ## Group-level analytics diff --git a/doc/user/analytics/value_stream_analytics.md b/doc/user/analytics/value_stream_analytics.md index ab68c897da8..0906f7d17a7 100644 --- a/doc/user/analytics/value_stream_analytics.md +++ b/doc/user/analytics/value_stream_analytics.md @@ -204,7 +204,7 @@ Value stream analytics records the following times for each stage: - **Review**: 14:00 to 19:00: 5 hrs - **Staging**: 19:00 to 19:30: 30 minutes -There are some additional considerations for this example: +Keep in mind the following observations related to this example: - Although this example specifies the issue number in a later commit, the process still collects analytics data for the issue. diff --git a/doc/user/application_security/api_fuzzing/create_har_files.md b/doc/user/application_security/api_fuzzing/create_har_files.md index e7eaf47121b..5e033a75902 100644 --- a/doc/user/application_security/api_fuzzing/create_har_files.md +++ b/doc/user/application_security/api_fuzzing/create_har_files.md @@ -109,7 +109,7 @@ responses in HAR format. have an account, first create an account. 1. Browse pages that call an API. Fiddler automatically captures the requests. 1. Select one or more requests, then from the context menu, select **Export > Selected Sessions**. -1. In the **Choose Format** dropdown list select **HTTPArchive v1.2**. +1. In the **Choose Format** dropdown list select **HTTP Archive v1.2**. 1. Enter a filename and select **Save**. Fiddler shows a popup message confirming the export has succeeded. diff --git a/doc/user/application_security/api_fuzzing/index.md b/doc/user/application_security/api_fuzzing/index.md index 03eed6fdbf8..e4ca512bdc6 100644 --- a/doc/user/application_security/api_fuzzing/index.md +++ b/doc/user/application_security/api_fuzzing/index.md @@ -1551,13 +1551,13 @@ When testing an API it can be useful to exclude certain paths. For example, you To verify the paths are excluded, review the `Tested Operations` and `Excluded Operations` portion of the job output. You should not see any excluded paths listed under `Tested Operations`. ```plaintext -2021-05-27 21:51:08 [INF] API Security: --[ Tested Operations ]------------------------- -2021-05-27 21:51:08 [INF] API Security: 201 POST http://target:7777/api/users CREATED -2021-05-27 21:51:08 [INF] API Security: ------------------------------------------------ -2021-05-27 21:51:08 [INF] API Security: --[ Excluded Operations ]----------------------- -2021-05-27 21:51:08 [INF] API Security: GET http://target:7777/api/messages -2021-05-27 21:51:08 [INF] API Security: POST http://target:7777/api/messages -2021-05-27 21:51:08 [INF] API Security: ------------------------------------------------ +2021-05-27 21:51:08 [INF] API Fuzzing: --[ Tested Operations ]------------------------- +2021-05-27 21:51:08 [INF] API Fuzzing: 201 POST http://target:7777/api/users CREATED +2021-05-27 21:51:08 [INF] API Fuzzing: ------------------------------------------------ +2021-05-27 21:51:08 [INF] API Fuzzing: --[ Excluded Operations ]----------------------- +2021-05-27 21:51:08 [INF] API Fuzzing: GET http://target:7777/api/messages +2021-05-27 21:51:08 [INF] API Fuzzing: POST http://target:7777/api/messages +2021-05-27 21:51:08 [INF] API Fuzzing: ------------------------------------------------ ``` #### Examples of excluding paths @@ -1589,7 +1589,7 @@ variables: While testing an API you may might want to exclude a parameter (query string, header, or body element) from testing. This may be needed because a parameter always causes a failure, slows down testing, or for other reasons. To exclude parameters you can use one of the following variables: `FUZZAPI_EXCLUDE_PARAMETER_ENV` or `FUZZAPI_EXCLUDE_PARAMETER_FILE`. -The `FUZZAPI_EXCLUDE_PARAMETER_ENV` allows providing a JSON string containing excluded parameters. This is a good option if the JSON is short and will not often change. Another option is the variable `FUZZAPI_EXCLUDE_PARAMETER_FILE`. This variable is set to a file path that can be checked into the repository, created by another job as an artifact, or generated at runtime from a pre script using `FUZZAPI_PRE_SCRIPT`. +The `FUZZAPI_EXCLUDE_PARAMETER_ENV` allows providing a JSON string containing excluded parameters. This is a good option if the JSON is short and will not often change. Another option is the variable `FUZZAPI_EXCLUDE_PARAMETER_FILE`. This variable is set to a file path that can be checked into the repository, created by another job as an artifact, or generated at runtime from a pre-script using `FUZZAPI_PRE_SCRIPT`. #### Exclude parameters using a JSON document @@ -1821,13 +1821,13 @@ As an alternative to excluding by paths, you can filter by any other component i In your job output you can check if any URLs matched any provided regular expression from `FUZZAPI_EXCLUDE_URLS`. Matching operations are listed in the **Excluded Operations** section. Operations listed in the **Excluded Operations** should not be listed in the **Tested Operations** section. For example the following portion of a job output: ```plaintext -2021-05-27 21:51:08 [INF] API Security: --[ Tested Operations ]------------------------- -2021-05-27 21:51:08 [INF] API Security: 201 POST http://target:7777/api/users CREATED -2021-05-27 21:51:08 [INF] API Security: ------------------------------------------------ -2021-05-27 21:51:08 [INF] API Security: --[ Excluded Operations ]----------------------- -2021-05-27 21:51:08 [INF] API Security: GET http://target:7777/api/messages -2021-05-27 21:51:08 [INF] API Security: POST http://target:7777/api/messages -2021-05-27 21:51:08 [INF] API Security: ------------------------------------------------ +2021-05-27 21:51:08 [INF] API Fuzzing: --[ Tested Operations ]------------------------- +2021-05-27 21:51:08 [INF] API Fuzzing: 201 POST http://target:7777/api/users CREATED +2021-05-27 21:51:08 [INF] API Fuzzing: ------------------------------------------------ +2021-05-27 21:51:08 [INF] API Fuzzing: --[ Excluded Operations ]----------------------- +2021-05-27 21:51:08 [INF] API Fuzzing: GET http://target:7777/api/messages +2021-05-27 21:51:08 [INF] API Fuzzing: POST http://target:7777/api/messages +2021-05-27 21:51:08 [INF] API Fuzzing: ------------------------------------------------ ``` NOTE: @@ -2242,18 +2242,18 @@ The first step to resolving performance issues is to understand what is contribu The API Fuzzing job output contains helpful information about how fast we are testing, how fast each operation being tested responds, and summary information. Let's take a look at some sample output to see how it can be used in tracking down performance issues: ```shell -API Security: Loaded 10 operations from: assets/har-large-response/large_responses.har -API Security: -API Security: Testing operation [1/10]: 'GET http://target:7777/api/large_response_json'. -API Security: - Parameters: (Headers: 4, Query: 0, Body: 0) -API Security: - Request body size: 0 Bytes (0 bytes) -API Security: -API Security: Finished testing operation 'GET http://target:7777/api/large_response_json'. -API Security: - Excluded Parameters: (Headers: 0, Query: 0, Body: 0) -API Security: - Performed 767 requests -API Security: - Average response body size: 130 MB -API Security: - Average call time: 2 seconds and 82.69 milliseconds (2.082693 seconds) -API Security: - Time to complete: 14 minutes, 8 seconds and 788.36 milliseconds (848.788358 seconds) +API Fuzzing: Loaded 10 operations from: assets/har-large-response/large_responses.har +API Fuzzing: +API Fuzzing: Testing operation [1/10]: 'GET http://target:7777/api/large_response_json'. +API Fuzzing: - Parameters: (Headers: 4, Query: 0, Body: 0) +API Fuzzing: - Request body size: 0 Bytes (0 bytes) +API Fuzzing: +API Fuzzing: Finished testing operation 'GET http://target:7777/api/large_response_json'. +API Fuzzing: - Excluded Parameters: (Headers: 0, Query: 0, Body: 0) +API Fuzzing: - Performed 767 requests +API Fuzzing: - Average response body size: 130 MB +API Fuzzing: - Average call time: 2 seconds and 82.69 milliseconds (2.082693 seconds) +API Fuzzing: - Time to complete: 14 minutes, 8 seconds and 788.36 milliseconds (848.788358 seconds) ``` This job console output snippet starts by telling us how many operations were found (10), followed by notifications that testing has started on a specific operation and a summary of the operation has been completed. The summary is the most interesting part of this log output. In the summary, we can see that it took API Fuzzing 767 requests to fully test this operation and its related fields. We can also see that the average response time was 2 seconds and the time to complete was 14 minutes for this one operation. @@ -2443,7 +2443,7 @@ See the following documentation sections for assistance: See [Performance Tuning and Testing Speed](#performance-tuning-and-testing-speed) -### Error waiting for API Security 'http://127.0.0.1:5000' to become available +### Error waiting for API Fuzzing 'http://127.0.0.1:5000' to become available A bug exists in versions of the API Fuzzing analyzer prior to v1.6.196 that can cause a background process to fail under certain conditions. The solution is to update to a newer version of the API Fuzzing analyzer. @@ -2456,6 +2456,11 @@ If the issue is occurring with versions v1.6.196 or greater, contact Support and 1. The `gl-api-security-scanner.log` file available as a job artifact. In the right-hand panel of the job details page, select the **Browse** button. 1. The `apifuzzer_fuzz` job definition from your `.gitlab-ci.yml` file. +**Error message** + +- In [GitLab 15.6 and later](https://gitlab.com/gitlab-org/gitlab/-/issues/376078), `Error waiting for API Fuzzing 'http://127.0.0.1:5000' to become available` +- In GitLab 15.5 and earlier, `Error waiting for API Security 'http://127.0.0.1:5000' to become available`. + ### `Failed to start session with scanner. Please retry, and if the problem persists reach out to support.` The API Fuzzing engine outputs an error message when it cannot establish a connection with the scanner application component. The error message is shown in the job output window of the `apifuzzer_fuzz` job. A common cause for this issue is that the background component cannot use the selected port as it's already in use. This error can occur intermittently if timing plays a part (race condition). This issue occurs most often with Kubernetes environments when other services are mapped into the container causing port conflicts. @@ -2648,6 +2653,50 @@ API Fuzzing uses the specified media types in the OpenAPI document to generate r 1. Review the supported media types in the [OpenAPI Specification](#openapi-specification) section. 1. Edit your OpenAPI document, allowing at least a given operation to accept any of the supported media types. Alternatively, a supported media type could be set in the OpenAPI document level and get applied to all operations. This step may require changes in your application to ensure the supported media type is accepted by the application. +### ``Error, error occurred trying to download `<URL>`: There was an error when retrieving content from Uri:' <URL>'. Error:The SSL connection could not be established, see inner exception.`` + +API fuzzing is compatible with a broad range of TLS configurations, including outdated protocols and ciphers. +Despite broad support, you might encounter connection errors. This error occurs because API fuzzing could not establish a secure connection with the server at the given URL. + +To resolve the issue: + +If the host in the error message supports non-TLS connections, change `https://` to `http://` in your configuration. +For example, if an error occurs with the following configuration: + +```yaml +stages: + - fuzz + +include: + - template: API-Fuzzing.gitlab-ci.yml + +variables: + FUZZAPI_TARGET_URL: https://test-deployment/ + FUZZAPI_OPENAPI: https://specs/openapi.json +``` + +Change the prefix of `FUZZAPI_OPENAPI` from `https://` to `http://`: + +```yaml +stages: + - fuzz + +include: + - template: API-Fuzzing.gitlab-ci.yml + +variables: + FUZZAPI_TARGET_URL: https://test-deployment/ + FUZZAPI_OPENAPI: http://specs/openapi.json +``` + +If you cannot use a non-TLS connection to access the URL, contact the Support team for help. + +You can expedite the investigation with the [testssl.sh tool](https://testssl.sh/). From a machine with a bash shell and connectivity to the affected server: + +1. Download the latest release `zip` or `tar.gz` file and extract from <https://github.com/drwetter/testssl.sh/releases>. +1. Run `./testssl.sh --log https://specs`. +1. Attach the log file to your support ticket. + ## Get support or request an improvement To get support for your particular problem use the [getting help channels](https://about.gitlab.com/get-help/). diff --git a/doc/user/application_security/configuration/index.md b/doc/user/application_security/configuration/index.md index d9ba4640855..ea422f0b33c 100644 --- a/doc/user/application_security/configuration/index.md +++ b/doc/user/application_security/configuration/index.md @@ -67,7 +67,7 @@ You can configure the following security controls: - Can be configured by adding a configuration block to your agent configuration. For more details, read [Operational Container Scanning](../../clusters/agent/vulnerabilities.md#enable-operational-container-scanning). - [Secret Detection](../secret_detection/index.md) - Select **Configure with a merge request** to create a merge request with the changes required to - enable Secret Detection. For more details, read [Enable Secret Detection using a merge request](../secret_detection/index.md#enable-secret-detection-using-a-merge-request). + enable Secret Detection. For more details, read [Use an automatically configured merge request](../secret_detection/index.md#use-an-automatically-configured-merge-request). - [API Fuzzing](../api_fuzzing/index.md) - Select **Enable API Fuzzing** to use API Fuzzing for the current project. For more details, read [API Fuzzing](../../../user/application_security/api_fuzzing/index.md#enable-web-api-fuzzing). - [Coverage Fuzzing](../coverage_fuzzing/index.md) diff --git a/doc/user/application_security/container_scanning/index.md b/doc/user/application_security/container_scanning/index.md index 6fc01a716b2..e198f967eea 100644 --- a/doc/user/application_security/container_scanning/index.md +++ b/doc/user/application_security/container_scanning/index.md @@ -90,12 +90,12 @@ To enable container scanning in your pipeline, you need the following: ## Configuration To enable container scanning, add the -[`Container-Scanning.gitlab-ci.yml` template](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/ci/templates/Security/Container-Scanning.gitlab-ci.yml) +[`Container-Scanning.gitlab-ci.yml` template](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/ci/templates/Jobs/Container-Scanning.gitlab-ci.yml) to your `.gitlab-ci.yml` file: ```yaml include: - - template: Security/Container-Scanning.gitlab-ci.yml + - template: Jobs/Container-Scanning.gitlab-ci.yml ``` The included template: @@ -117,7 +117,7 @@ registry, and scans the image: ```yaml include: - template: Jobs/Build.gitlab-ci.yml - - template: Security/Container-Scanning.gitlab-ci.yml + - template: Jobs/Container-Scanning.gitlab-ci.yml container_scanning: variables: @@ -142,7 +142,7 @@ enables verbose output for the analyzer: ```yaml include: - - template: Security/Container-Scanning.gitlab-ci.yml + - template: Jobs/Container-Scanning.gitlab-ci.yml variables: SECURE_LOG_LEVEL: 'debug' @@ -154,7 +154,7 @@ To scan images located in a registry other than the project's, use the following ```yaml include: - - template: Security/Container-Scanning.gitlab-ci.yml + - template: Jobs/Container-Scanning.gitlab-ci.yml container_scanning: variables: @@ -178,7 +178,7 @@ container_scanning: - export AWS_ECR_PASSWORD=$(aws ecr get-login-password --region region) include: - - template: Security/Container-Scanning.gitlab-ci.yml + - template: Jobs/Container-Scanning.gitlab-ci.yml CS_IMAGE: <aws_account_id>.dkr.ecr.<region>.amazonaws.com/<image>:<tag> CS_REGISTRY_USER: AWS CS_REGISTRY_PASSWORD: "$AWS_ECR_PASSWORD" @@ -199,7 +199,7 @@ For example: ```yaml include: - - template: Security/Container-Scanning.gitlab-ci.yml + - template: Jobs/Container-Scanning.gitlab-ci.yml container_scanning: variables: @@ -223,7 +223,7 @@ By default, the report only includes packages managed by the Operating System (O ```yaml include: - - template: Security/Container-Scanning.gitlab-ci.yml + - template: Jobs/Container-Scanning.gitlab-ci.yml container_scanning: variables: @@ -345,7 +345,7 @@ This example sets `GIT_STRATEGY` to `fetch`: ```yaml include: - - template: Security/Container-Scanning.gitlab-ci.yml + - template: Jobs/Container-Scanning.gitlab-ci.yml container_scanning: variables: @@ -391,7 +391,7 @@ duplicated: ```yaml include: - - template: Security/Container-Scanning.gitlab-ci.yml + - template: Jobs/Container-Scanning.gitlab-ci.yml container_scanning: variables: @@ -567,7 +567,7 @@ and you may be able to make occasional updates on your own. For more information, see [the specific steps on how to update an image with a pipeline](#automating-container-scanning-vulnerability-database-updates-with-a-pipeline). -For details on saving and transporting Docker images as a file, see Docker's documentation on +For details on saving and transporting Docker images as a file, see the Docker documentation on [`docker save`](https://docs.docker.com/engine/reference/commandline/save/), [`docker load`](https://docs.docker.com/engine/reference/commandline/load/), [`docker export`](https://docs.docker.com/engine/reference/commandline/export/), and [`docker import`](https://docs.docker.com/engine/reference/commandline/import/). @@ -577,7 +577,7 @@ For details on saving and transporting Docker images as a file, see Docker's doc ```yaml include: - - template: Security/Container-Scanning.gitlab-ci.yml + - template: Jobs/Container-Scanning.gitlab-ci.yml container_scanning: image: $CI_REGISTRY/namespace/gitlab-container-scanning @@ -628,7 +628,7 @@ This example shows the configuration needed to scan images in a private [Google ```yaml include: - - template: Security/Container-Scanning.gitlab-ci.yml + - template: Jobs/Container-Scanning.gitlab-ci.yml container_scanning: variables: diff --git a/doc/user/application_security/dast/authentication.md b/doc/user/application_security/dast/authentication.md new file mode 100644 index 00000000000..d4f91639dbc --- /dev/null +++ b/doc/user/application_security/dast/authentication.md @@ -0,0 +1,527 @@ +--- +stage: Secure +group: Dynamic Analysis +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments +type: reference, howto +--- + +# Authentication (DAST) **(ULTIMATE)** + +WARNING: +**Never** run an authenticated scan against a production server. +Authenticated scans may perform *any* function that the authenticated user can, +including modifying or deleting data, submitting forms, and following links. +Only run an authenticated scan against a test server. + +Authentication logs a user in before a DAST scan so that the analyzer can test +as much of the application as possible when searching for vulnerabilities. + +DAST uses a browser to authenticate the user so that the login form has the necessary JavaScript +and styling required to submit the form. DAST finds the username and password fields and fills them with their respective values. +The login form is submitted, and when the response returns, a series of checks verify if authentication was successful. +DAST saves the credentials for reuse when crawling the target application. + +If DAST fails to authenticate, the scan halts and the CI job fails. + +Authentication supports single-step login forms, multi-step login forms, single sign-on, and authenticating to URLs outside of the configured target URL. + +## Getting started + +NOTE: +We recommend periodically confirming that the analyzer's authentication is still working, as this tends to break over +time due to changes to the application. + +To run a DAST authenticated scan: + +- Read the [prerequisite](#prerequisites) conditions for authentication. +- [Update your target website](#update-the-target-website) to a landing page of an authenticated user. +- If your login form has the username, password and submit button on a single page, use the [CI/CD variables](#available-cicd-variables) to configure [single-step](#configuration-for-a-single-step-login-form) login form authentication. +- If your login form has the username and password fields on different pages, use the [CI/CD variables](#available-cicd-variables) to configure [multi-step](#configuration-for-a-multi-step-login-form) login form authentication. +- Make sure the user isn't [logged out](#excluding-logout-urls) during the scan. + +### Prerequisites + +- You are using either the [DAST proxy-based analyzer](proxy-based.md) or the [DAST browser-based analyzer](browser_based.md). +- You know the URL of the login form of your application. Alternatively, you know how to navigate to the login form from the authentication URL (see [clicking to navigate to the login form](#clicking-to-navigate-to-the-login-form)). +- You have the username and password of the user you would like to authenticate as during the scan. +- You know the [selectors](#finding-an-elements-selector) of the username and password HTML fields that DAST will use to input the respective values. +- You know the element's [selector](#finding-an-elements-selector) that will submit the login form when selected. +- You have thought about how you can [verify](#verifying-authentication-is-successful) whether or not authentication was successful. +- You have checked the [known limitations](#known-limitations) to ensure DAST can authenticate to your application. + +### Available CI/CD variables + +| CI/CD variable | Type | Description | +|:-----------------------------------------------|:------------------------------------------|:-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| +| `DAST_AUTH_COOKIES` | string | Set to a comma-separated list of cookie names to specify which cookies are used for authentication. | +| `DAST_AUTH_REPORT` | boolean | Used in combination with exporting the `gl-dast-debug-auth-report.html` artifact to aid in debugging authentication issues. | +| `DAST_AUTH_URL` <sup>1</sup> | URL | The URL of the page containing the sign-in HTML form on the target website. `DAST_USERNAME` and `DAST_PASSWORD` are submitted with the login form to create an authenticated scan. Example: `https://login.example.com`. | +| `DAST_AUTH_VERIFICATION_LOGIN_FORM` | boolean | Verifies successful authentication by checking for the absence of a login form once the login form has been submitted. | +| `DAST_AUTH_VERIFICATION_SELECTOR` | [selector](#finding-an-elements-selector) | Verifies successful authentication by checking for presence of a selector once the login form has been submitted. Example: `css:.user-photo`. | +| `DAST_AUTH_VERIFICATION_URL` <sup>1</sup> | URL | Verifies successful authentication by checking the URL in the browser once the login form has been submitted. Example: `"https://example.com/loggedin_page"`. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/207335) in GitLab 13.8. | +| `DAST_BROWSER_PATH_TO_LOGIN_FORM` <sup>1</sup> | [selector](#finding-an-elements-selector) | Comma-separated list of selectors that are selected prior to attempting to enter `DAST_USERNAME` and `DAST_PASSWORD` into the login form. Example: `"css:.navigation-menu,css:.login-menu-item"`. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/326633) in GitLab 14.1. | +| `DAST_EXCLUDE_URLS` <sup>1</sup> | URLs | The URLs to skip during the authenticated scan; comma-separated. Regular expression syntax can be used to match multiple URLs. For example, `.*` matches an arbitrary character sequence. | +| `DAST_FIRST_SUBMIT_FIELD` | string | The `id` or `name` of the element that when selected submits the username form of a multi-page login process. For example, `css:button[type='user-submit']`. [Introduced](https://gitlab.com/gitlab-org/gitlab-ee/issues/9894) in GitLab 12.4. | +| `DAST_PASSWORD` <sup>1</sup> | string | The password to authenticate to in the website. Example: `P@55w0rd!` | +| `DAST_PASSWORD_FIELD` | string | The selector of password field at the sign-in HTML form. Example: `id:password` | +| `DAST_SUBMIT_FIELD` | string | The `id` or `name` of the element that when selected submits the login form or the password form of a multi-page login process. For example, `css:button[type='submit']`. [Introduced](https://gitlab.com/gitlab-org/gitlab-ee/issues/9894) in GitLab 12.4. | +| `DAST_USERNAME` <sup>1</sup> | string | The username to authenticate to in the website. Example: `admin` | +| `DAST_USERNAME_FIELD` <sup>1</sup> | string | The selector of username field at the sign-in HTML form. Example: `name:username` | + +1. Available to an on-demand proxy-based DAST scan. + +### Update the target website + +The target website, defined using the CI/CD variable `DAST_WEBSITE`, is the URL DAST uses to begin crawling your application. + +For best crawl results on an authenticated scan, the target website should be a URL accessible only after the user is authenticated. +Often, this is the URL of the page the user lands on after they're logged in. + +For example: + +```yaml +include: + - template: DAST.gitlab-ci.yml + +dast: + variables: + DAST_WEBSITE: "https://example.com/dashboard/welcome" + DAST_AUTH_URL: "https://example.com/login" +``` + +### Configuration for a single-step login form + +A single-step login form has all login form elements on a single page. +Configuration requires the CI/CD variables `DAST_AUTH_URL`, `DAST_USERNAME`, `DAST_USERNAME_FIELD`, `DAST_PASSWORD`, `DAST_PASSWORD_FIELD`, and `DAST_SUBMIT_FIELD` to be defined for the DAST job. + +It is recommended to set up the URL and selectors of fields in the job definition YAML, for example: + +```yaml +include: + - template: DAST.gitlab-ci.yml + +dast: + variables: + DAST_WEBSITE: "https://example.com" + DAST_AUTH_URL: "https://example.com/login" + DAST_USERNAME_FIELD: "css:[name=username]" + DAST_PASSWORD_FIELD: "css:[name=password]" + DAST_SUBMIT_FIELD: "css:button[type=submit]" +``` + +Do **not** define `DAST_USERNAME` and `DAST_PASSWORD` in the YAML job definition file as this could present a security risk. Instead, create them as masked CI/CD variables using the GitLab UI. +See [Custom CI/CI variables](../../../ci/variables/index.md#custom-cicd-variables) for more information. + +### Configuration for a multi-step login form + +A multi-step login form has two pages. The first page has a form with the username and a next submit button. +If the username is valid, a second form on the subsequent page has the password and the form submit button. + +Configuration requires the CI/CD variables `DAST_AUTH_URL`, `DAST_USERNAME`, `DAST_USERNAME_FIELD`, `DAST_FIRST_SUBMIT_FIELD`, `DAST_PASSWORD`, `DAST_PASSWORD_FIELD`, and `DAST_SUBMIT_FIELD` to be defined for the DAST job. + +It is recommended to set up the URL and selectors of fields in the job definition YAML, for example: + +```yaml +include: + - template: DAST.gitlab-ci.yml + +dast: + variables: + DAST_WEBSITE: "https://example.com" + DAST_AUTH_URL: "https://example.com/login" + DAST_USERNAME_FIELD: "css:[name=username]" + DAST_FIRST_SUBMIT_FIELD: "css:button[name=next]" + DAST_PASSWORD_FIELD: "css:[name=password]" + DAST_SUBMIT_FIELD: "css:button[type=submit]" +``` + +Do **not** define `DAST_USERNAME` and `DAST_PASSWORD` in the YAML job definition file as this could present a security risk. Instead, create them as masked CI/CD variables using the GitLab UI. +See [Custom CI/CI variables](../../../ci/variables/index.md#custom-cicd-variables) for more information. + +### Configuration for Single Sign-On (SSO) + +If a user can log into an application, then in most cases, DAST will also be able to log in. +This is the case even when an application uses Single Sign-on. Applications using SSO solutions should configure DAST +authentication using the [single-step](#configuration-for-a-single-step-login-form) or [multi-step](#configuration-for-a-multi-step-login-form) login form configuration guides. + +DAST supports authentication processes where a user is redirected to an external Identity Provider's site to log in. +Check the [known limitations](#known-limitations) of DAST authentication to determine if your SSO authentication process is supported. + +### Clicking to navigate to the login form + +Define `DAST_BROWSER_PATH_TO_LOGIN_FORM` to provide a path of elements to click on from the `DAST_AUTH_URL` so that DAST can access the +login form. This is useful for applications that show the login form in a pop-up (modal) window or when the login form does not +have a unique URL. + +For example: + +```yaml +include: + - template: DAST.gitlab-ci.yml + +dast: + variables: + DAST_WEBSITE: "https://example.com" + DAST_AUTH_URL: "https://example.com/login" + DAST_BROWSER_PATH_TO_LOGIN_FORM: "css:.navigation-menu,css:.login-menu-item" +``` + +### Excluding logout URLs + +If DAST crawls the logout URL while running an authenticated scan, the user will be logged out, resulting in the remainder of the scan being unauthenticated. +It is therefore recommended to exclude logout URLs using the CI/CD variable `DAST_EXCLUDE_URLS`. DAST will not access any excluded URLs, ensuring the user remains logged in. + +Provided URLs can be either absolute URLs, or regular expressions of URL paths relative to the base path of the `DAST_WEBSITE`. For example: + +```yaml +include: + - template: DAST.gitlab-ci.yml + +dast: + variables: + DAST_WEBSITE: "https://example.com/welcome/home" + DAST_EXCLUDE_URLS: "https://example.com/logout,/user/.*/logout" +``` + +### Finding an element's selector + +Selectors are used by CI/CD variables to specify the location of an element displayed on a page in a browser. +Selectors have the format `type`:`search string`. DAST searches for the selector using the search string based on the type. + +| Selector type | Example | Description | +|---------------|------------------------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| +| `css` | `css:.password-field` | Searches for a HTML element having the supplied CSS selector. Selectors should be as specific as possible for performance reasons. | +| `id` | `id:element` | Searches for an HTML element with the provided element ID. | +| `name` | `name:element` | Searches for an HTML element with the provided element name. | +| `xpath` | `xpath://input[@id="my-button"]/a` | Searches for a HTML element with the provided XPath. Note that XPath searches are expected to be less performant than other searches. | +| None provided | `a.click-me` | Defaults to searching using a CSS selector. **{warning}** **[Deprecated](https://gitlab.com/gitlab-org/gitlab/-/issues/383348)** in GitLab 15.8. Replaced by explicitly declaring the selector type. | + +#### Find selectors with Google Chrome + +Chrome DevTools element selector tool is an effective way to find a selector. + +1. Open Chrome and navigate to the page where you would like to find a selector, for example, the login page for your site. +1. Open the `Elements` tab in Chrome DevTools with the keyboard shortcut `Command + Shift + c` in macOS or `Ctrl + Shift + c` in Windows. +1. Select the `Select an element in the page to select it` tool. +  +1. Select the field on your page that you would like to know the selector for. +1. Once the tool is active, highlight a field you wish to view the details of. +  +1. Once highlighted, you can see the element's details, including attributes that would make a good candidate for a selector. + +In this example, the `id="user_login"` appears to be a good candidate. You can use this as a selector as the DAST username field by setting +`DAST_USERNAME_FIELD: "id:user_login"`. + +#### Choose the right selector + +Judicious choice of selector leads to a scan that is resilient to the application changing. + +In order of preference, it is recommended to choose as selectors: + +- `id` fields. These are generally unique on a page, and rarely change. +- `name` fields. These are generally unique on a page, and rarely change. +- `class` values specific to the field, such as the selector `"css:.username"` for the `username` class on the username field. +- Presence of field specific data attributes, such as the selector, `"css:[data-username]"` when the `data-username` field has any value on the username field. +- Multiple `class` hierarchy values, such as the selector `"css:.login-form .username"` when there are multiple elements with class `username` but only one nested inside the element with the class `login-form`. + +When using selectors to locate specific fields we recommend you avoid searching on: + +- Any `id`, `name`, `attribute`, `class` or `value` that is dynamically generated. +- Generic class names, such as `column-10` and `dark-grey`. +- XPath searches as they are less performant than other selector searches. +- Unscoped searches, such as those beginning with `css:*` and `xpath://*`. + +## Verifying authentication is successful + +Once DAST has submitted the login form, a verification process takes place +to determine if authentication succeeded. The scan will halt with an error if authentication is unsuccessful. + +Following the submission of the login form, authentication is determined to be unsuccessful when: + +- The login submit HTTP response has a `400` or `500` series status code. +- Any [verification check](#verification-checks) fails. +- An [authentication token](#authentication-tokens) with a sufficiently random value is not set during the authentication process. + +### Verification checks + +Verification checks run checks on the state of the browser once authentication is complete +to determine further if authentication succeeded. + +DAST will test for the absence of a login form if no verification checks are configured. + +#### Verify based on the URL + +Define `DAST_AUTH_VERIFICATION_URL` as the URL displayed in the browser tab once the login form is successfully submitted. + +DAST will compare the verification URL to the URL in the browser after authentication. +If they are not the same, authentication is unsuccessful. + +For example: + +```yaml +include: + - template: DAST.gitlab-ci.yml + +dast: + variables: + DAST_WEBSITE: "https://example.com" + DAST_AUTH_VERIFICATION_URL: "https://example.com/user/welcome" +``` + +#### Verify based on presence of an element + +Define `DAST_AUTH_VERIFICATION_SELECTOR` as a [selector](#finding-an-elements-selector) that will find one or many elements on the page +displayed once the login form is successfully submitted. If no element is found, authentication is unsuccessful. +Searching for the selector on the page displayed when login fails should return no elements. + +For example: + +```yaml +include: + - template: DAST.gitlab-ci.yml + +dast: + variables: + DAST_WEBSITE: "https://example.com" + DAST_AUTH_VERIFICATION_SELECTOR: "css:.welcome-user" +``` + +#### Verify based on absence of a login form + +Define `DAST_AUTH_VERIFICATION_LOGIN_FORM` as `"true"` to indicate that DAST should search for the login form on the +page displayed once the login form is successfully submitted. If a login form is still present after logging in, authentication is unsuccessful. + +For example: + +```yaml +include: + - template: DAST.gitlab-ci.yml + +dast: + variables: + DAST_WEBSITE: "https://example.com" + DAST_AUTH_VERIFICATION_LOGIN_FORM: "true" +``` + +### Authentication tokens + +DAST records authentication tokens set during the authentication process. +Authentication tokens are loaded into new browsers when DAST opens them so the user can remain logged in throughout the scan. + +To record tokens, DAST takes a snapshot of cookies, local storage, and session storage values set by the application before +the authentication process. DAST does the same after authentication and uses the difference to determine which were created +by the authentication process. + +DAST considers cookies, local storage and session storage values set with sufficiently "random" values to be authentication tokens. +For example, `sessionID=HVxzpS8GzMlPAc2e39uyIVzwACIuGe0H` would be viewed as an authentication token, while `ab_testing_group=A1` would not. + +The CI/CD variable `DAST_AUTH_COOKIES` can be used to specify the names of authentication cookies and bypass the randomness check used by DAST. +Not only can this make the authentication process more robust, but it can also increase vulnerability check accuracy for checks that +inspect authentication tokens. + +For example: + +```yaml +include: + - template: DAST.gitlab-ci.yml + +dast: + variables: + DAST_WEBSITE: "https://example.com" + DAST_AUTH_COOKIES: "sessionID,refreshToken" +``` + +## Known limitations + +- DAST cannot bypass a CAPTCHA if the authentication flow includes one. Please turn these off in the testing environment for the application being scanned. +- DAST cannot handle multi-factor authentication like one-time passwords (OTP) by using SMS, biometrics, or authenticator apps. Please turn these off in the testing environment for the application being scanned. +- DAST cannot authenticate to applications that do not set an [authentication token](#authentication-tokens) during login. +- DAST cannot authenticate to applications that require more than two inputs to be filled out. Two inputs must be supplied, username and password. + +## Troubleshooting + +The [logs](#read-the-logs) provide insight into what DAST is doing and expecting during the authentication process. For more detailed +information, configure the [authentication report](#configure-the-authentication-report). + +For more information about particular error messages or situations see [known problems](#known-problems). + +The browser-based analyzer is used to authenticate the user. For advanced troubleshooting, see [browser-based troubleshooting](browser_based_troubleshooting.md). + +### Read the logs + +The console output of the DAST CI/CD job shows information about the authentication process using the `AUTH` log module. +For example, the following log shows failed authentication for a multi-step login form. +Authentication failed because a home page should be displayed after login. Instead, the login form was still present. + +```plaintext +2022-11-16T13:43:02.000 INF AUTH attempting to authenticate +2022-11-16T13:43:02.000 INF AUTH loading login page LoginURL=https://example.com/login +2022-11-16T13:43:10.000 INF AUTH multi-step authentication detected +2022-11-16T13:43:15.000 INF AUTH verifying if user submit was successful true_when="HTTP status code < 400" +2022-11-16T13:43:15.000 INF AUTH requirement is satisfied, no login HTTP message detected want="HTTP status code < 400" +2022-11-16T13:43:20.000 INF AUTH verifying if login attempt was successful true_when="HTTP status code < 400 and has authentication token and no login form found (no element found when searching using selector css:[id=email] or css:[id=password] or css:[id=submit])" +2022-11-24T14:43:20.000 INF AUTH requirement is satisfied, HTTP login request returned status code 200 url=https://example.com/user/login?error=invalid%20credentials want="HTTP status code < 400" +2022-11-16T13:43:21.000 INF AUTH requirement is unsatisfied, login form was found want="no login form found (no element found when searching using selector css:[id=email] or css:[id=password] or css:[id=submit])" +2022-11-16T13:43:21.000 INF AUTH login attempt failed error="authentication failed: failed to authenticate user" +``` + +### Configure the authentication report + +An authentication report can be saved as a CI/CD job artifact to assist with understanding the cause of an authentication failure. + +The report contains steps during the login process, HTTP requests and responses, the Document Object Model (DOM) and screenshots. + + + +An example configuration where the authentication debug report is exported may look like the following: + +```yaml +dast: + variables: + DAST_WEBSITE: "https://example.com" + DAST_AUTH_REPORT: "true" + artifacts: + paths: [gl-dast-debug-auth-report.html] + when: always +``` + +### Known problems + +#### Login form not found + +DAST failed to find a login form when loading the login page, often because the authentication URL could not be loaded. +The log reports a fatal error such as: + +```plaintext +2022-12-07T12:44:02.838 INF AUTH loading login page LoginURL=[authentication URL] +2022-12-07T12:44:11.119 FTL MAIN authentication failed: login form not found +``` + +Suggested actions: + +- Generate the [authentication report](#configure-the-authentication-report) to inspect HTTP response. +- Check the target application authentication is deployed and running. +- Check the `DAST_AUTH_URL` is correct. +- Check the GitLab Runner can access the `DAST_AUTH_URL`. +- Check the `DAST_BROWSER_PATH_TO_LOGIN_FORM` is valid if used. + +#### Scan doesn't crawl authenticated pages + +If DAST captures the wrong [authentication tokens](#authentication-tokens) during the authentication process then +the scan can't crawl authenticated pages. Names of cookies and storage authentication tokens are written to the log. For example: + +```plaintext +2022-11-24T14:42:31.492 INF AUTH authentication token cookies names=["sessionID"] +2022-11-24T14:42:31.492 INF AUTH authentication token storage events keys=["token"] +``` + +Suggested actions: + +- Generate the [authentication report](#configure-the-authentication-report) and look at the screenshot from the `Login submit` to verify that the login worked as expected. +- Verify the logged authentication tokens are those used by your application. +- If using cookies to store authentication tokens, set the names of the authentication token cookies using `DAST_AUTH_COOKIES`. + +#### Unable to find elements with selector + +DAST failed to find the username, password, first submit button, or submit button elements. The log reports a fatal error such as: + +```plaintext +2022-12-07T13:14:11.545 FTL MAIN authentication failed: unable to find elements with selector: css:#username +``` + +Suggested actions: + +- Generate the [authentication report](#configure-the-authentication-report) to use the screenshot from the `Login page` to verify that the page loaded correctly. +- Load the login page in a browser and verify the [selectors](#finding-an-elements-selector) configured in `DAST_USERNAME_FIELD`, `DAST_PASSWORD_FIELD`, `DAST_FIRST_SUBMIT_FIELD`, and `DAST_SUBMIT_FIELD` are correct. + +#### Failed to authenticate user + +DAST failed to authenticate due to a failed login verification check. The log reports a fatal error such as: + +```plaintext +2022-12-07T06:39:49.483 INF AUTH verifying if login attempt was successful true_when="HTTP status code < 400 and has authentication token and no login form found (no element found when searching using selector css:[name=username] or css:[name=password] or css:button[type=\"submit\"])" +2022-12-07T06:39:49.484 INF AUTH requirement is satisfied, HTTP login request returned status code 303 url=http://auth-manual:8090/login want="HTTP status code < 400" +2022-12-07T06:39:49.513 INF AUTH requirement is unsatisfied, login form was found want="no login form found (no element found when searching using selector css:[name=username] or css:[name=password] or css:button[type=\"submit\"])" +2022-12-07T06:39:49.589 INF AUTH login attempt failed error="authentication failed: failed to authenticate user" +2022-12-07T06:39:53.626 FTL MAIN authentication failed: failed to authenticate user +``` + +Suggested actions: + +- Look in the log for the `requirement is unsatisfied`. Respond to the appropriate error. + +#### Requirement unsatisfied, login form was found + +Applications typically display a dashboard when the user logs in and the login form with an error message when the +username or password is incorrect. + +This error occurs when DAST detects the login form on the page displayed after authenticating the user, +indicating that the login attempt failed. + +```plaintext +2022-12-07T06:39:49.513 INF AUTH requirement is unsatisfied, login form was found want="no login form found (no element found when searching using selector css:[name=username] or css:[name=password] or css:button[type=\"submit\"])" +``` + +Suggested actions: + +- Verify that the username and password/authentication credentials used are correct. +- Generate the [authentication report](#configure-the-authentication-report) and verify the `Request` for the `Login submit` is correct. +- It's possible that the authentication report `Login submit` request and response are empty. This occurs when there is no request that would result + in a full page reload, such as a request made when submitting a HTML form. This occurs when using websockets or AJAX to submit the login form. +- If the page displayed following user authentication genuinely has elements matching the login form selectors, configure `DAST_AUTH_VERIFICATION_URL` + or `DAST_AUTH_VERIFICATION_SELECTOR` to use an alternate method of verifying the login attempt. + +#### Requirement unsatisfied, selector returned no results + +DAST cannot find an element matching the selector provided in `DAST_AUTH_VERIFICATION_SELECTOR` on the page displayed following user login. + +```plaintext +2022-12-07T06:39:33.239 INF AUTH requirement is unsatisfied, searching DOM using selector returned no results want="has element css:[name=welcome]" +``` + +Suggested actions: + +- Generate the [authentication report](#configure-the-authentication-report) and look at the screenshot from the `Login submit` to verify that the expected page is displayed. +- Ensure the `DAST_AUTH_VERIFICATION_SELECTOR` [selector](#finding-an-elements-selector) is correct. + +#### Requirement unsatisfied, browser not at URL + +DAST detected that the page displayed following user login has a URL different to what was expected according to `DAST_AUTH_VERIFICATION_URL`. + +```plaintext +2022-12-07T11:28:00.241 INF AUTH requirement is unsatisfied, browser is not at URL browser_url="https://example.com/home" want="is at url https://example.com/user/dashboard" +``` + +Suggested actions: + +- Generate the [authentication report](#configure-the-authentication-report) and look at the screenshot from the `Login submit` to verify that the expected page is displayed. +- Ensure the `DAST_AUTH_VERIFICATION_URL` is correct. + +#### Requirement unsatisfied, HTTP login request status code + +The HTTP response when loading the login form or submitting the form had a status code of 400 (client error) +or 500 (server error). + +```plaintext +2022-12-07T06:39:53.626 INF AUTH requirement is unsatisfied, HTTP login request returned status code 502 url="https://example.com/user/login" want="HTTP status code < 400" +``` + +- Verify that the username and password/authentication credentials used are correct. +- Generate the [authentication report](#configure-the-authentication-report) and verify the `Request` for the `Login submit` is correct. +- Verify the target application works as expected. + +#### Requirement unsatisfied, no authentication token + +DAST could not detect an [authentication token](#authentication-tokens) created during the authentication process. + +```plaintext +2022-12-07T11:25:29.010 INF AUTH authentication token cookies names=[] +2022-12-07T11:25:29.010 INF AUTH authentication token storage events keys=[] +2022-12-07T11:25:29.010 INF AUTH requirement is unsatisfied, no basic authentication, cookie or storage event authentication token detected want="has authentication token" +``` + +Suggestion actions: + +- Generate the [authentication report](#configure-the-authentication-report) and look at the screenshot from the `Login submit` to verify that the login worked as expected. +- Using the browser's developer tools, investigate the cookies and local/session storage objects created while logging in. Ensure there is an authentication token created with sufficiently random value. +- If using cookies to store authentication tokens, set the names of the authentication token cookies using `DAST_AUTH_COOKIES`. diff --git a/doc/user/application_security/dast/browser_based.md b/doc/user/application_security/dast/browser_based.md index c0a97c0ff92..7377f31d0ce 100644 --- a/doc/user/application_security/dast/browser_based.md +++ b/doc/user/application_security/dast/browser_based.md @@ -10,39 +10,144 @@ type: reference, howto > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/323423) in GitLab 13.12. WARNING: -This product is in an early-access stage and is considered a [beta](../../../policy/alpha-beta-support.md#beta-features) feature. +This product is in an early-access stage and is considered a [beta](../../../policy/alpha-beta-support.md#beta-features) +feature. -GitLab DAST's browser-based analyzer was built by GitLab to test Single Page Applications (SPAs) and -traditional web applications. It both crawls the web application and analyzes the resulting output -for vulnerabilities. Analysis of modern applications, heavily reliant on JavaScript, is vital to -ensuring DAST coverage. +WARNING: +Do not run DAST scans against a production server. Not only can it perform *any* function that +a user can, such as clicking buttons or submitting forms, but it may also trigger bugs, leading to modification or loss of production data. Only run DAST scans against a test server. + +The DAST browser-based analyzer was built by GitLab to scan modern-day web applications for vulnerabilities. +Scans run in a browser to optimize testing applications heavily dependent on JavaScript, such as single-page applications. +See [how DAST scans an application](#how-dast-scans-an-application) for more information. + +To add the analyzer to your CI/CD pipeline, see [getting started](#getting-started). + +## How DAST scans an application + +A scan performs the following steps: + +1. [Authenticate](authentication.md), if configured. +1. [Crawl](#crawling-an-application) the target application to discover the surface area of the application by performing user actions such as following links, clicking buttons, and filling out forms. +1. [Passive scan](#passive-scans) to search for vulnerabilities in HTTP messages and pages discovered while crawling. +1. [Active scan](#active-scans) to search for vulnerabilities by injecting payloads into HTTP requests recorded during the crawl phase. + +### Crawling an application + +A "navigation" is an action a user might take on a page, such as clicking buttons, clicking anchor links, opening menu items, or filling out forms. +A "navigation path" is a sequence of navigation actions representing how a user might traverse an application. +DAST discovers the surface area of an application by crawling pages and content and identifying navigation paths. + +Crawling is initialized with a navigation path containing one navigation that loads the target application URL in a specially-instrumented Chromium browser. +DAST then crawls navigation paths until all have been crawled. + +To crawl a navigation path, DAST opens a browser window and instructs it to perform all the navigation actions in the navigation path. +When the browser has finished loading the result of the final action, DAST inspects the page for actions a user might take, +creates a new navigation for each found, and adds them to the navigation path to form new navigation paths. For example: + +- DAST processes navigation path `LoadURL[https://example.com]`. +- DAST finds two user actions, `LeftClick[class=menu]` and `LeftClick[id=users]`. +- DAST creates two new navigation paths, `LoadURL[https://example.com] -> LeftClick[class=menu]` and `LoadURL[https://example.com] -> LeftClick[id=users]`. +- Crawling begins on the two new navigation paths. + +It's common for an HTML element to exist in multiple places in an application, such as a menu visible on every page. +Duplicate elements can cause crawlers to crawl the same pages again or become stuck in a loop. +DAST uses an element uniqueness calculation based on HTML attributes to discard new navigation actions it has previously crawled. + +### Passive scans + +Passive scans check for vulnerabilities in the pages discovered during the crawl phase of the scan. +Passive scans are enabled by default. + +The checks search HTTP messages, cookies, storage events, console events, and DOM for vulnerabilities. +Examples of passive checks include searching for exposed credit cards, exposed secret tokens, missing content security policies, and redirection to untrusted locations. + +See [checks](checks/index.md) for more information about individual checks. + +### Active scans + +Active scans check for vulnerabilities by injecting attack payloads into HTTP requests recorded during the crawl phase of the scan. +Active scans are disabled by default due to the nature of their probing attacks. + +DAST analyzes each recorded HTTP request for injection locations, such as query values, header values, cookie values, form posts, and JSON string values. +Attack payloads are injected into the injection location, forming a new request. +DAST sends the request to the target application and uses the HTTP response to determine attack success. + +Active scans run two types of active check: + +- A match response attack analyzes the response content to determine attack success. For example, if an attack attempts to read the system password file, a finding is created when the response body contains evidence of the password file. +- A timing attack uses the response time to determine attack success. For example, if an attack attempts to force the target application to sleep, a finding is created when the application takes longer to respond than the sleep time. Timing attacks are repeated multiple times with different attack payloads to minimize false positives. + +A simplified timing attack works as follows: + +1. The crawl phase records the HTTP request `https://example.com?search=people`. +1. DAST analyzes the URL and finds a URL parameter injection location `https://example.com?search=[INJECT]`. +1. The active check defines a payload, `sleep 10`, that attempts to get a Linux host to sleep. +1. DAST send a new HTTP request to the target application with the injected payload `https://example.com?search=sleep%2010`. +1. The target application is vulnerable if it executes the query parameter value as a system command without validation, for example, `system(params[:search])` +1. DAST creates a finding if the response time takes longer than 10 seconds. + +## Getting started + +To run a DAST scan: + +- Read the [prerequisite](index.md#prerequisites) conditions for running a DAST scan. +- Create a [DAST job](#create-a-dast-cicd-job) in your CI/CD pipeline. +- [Authenticate](#authentication) as a user if your application requires it. + +### Create a DAST CI/CD job + +> - This template was [updated](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/62597) to DAST_VERSION: 2 in + GitLab 14.0. +> - This template was [updated](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/87183) to DAST_VERSION: 3 in + GitLab 15.0. + +To add DAST scanning to your application, use the DAST job defined +in the GitLab DAST CI/CD template file. Updates to the template are provided with GitLab +upgrades, allowing you to benefit from any improvements and additions. + +To create the CI/CD job: + +1. Include the appropriate CI/CD template: -The browser-based scanner works by loading the target application into a specially-instrumented -Chromium browser. A snapshot of the page is taken before a search to find any actions that a user -might perform, such as selecting on a link or filling in a form. For each action found, the -browser-based scanner executes it, takes a new snapshot, and determines what in the page changed -from the previous snapshot. Crawling continues by taking more snapshots and finding subsequent -actions. The benefit of scanning by following user actions in a browser is that the crawler can -interact with the target application much like a real user would, identifying complex flows that -traditional web crawlers don't understand. This results in better coverage of the website. + - [`DAST.gitlab-ci.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/ci/templates/Security/DAST.gitlab-ci.yml): + Stable version of the DAST CI/CD template. + - [`DAST.latest.gitlab-ci.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/ci/templates/Security/DAST.latest.gitlab-ci.yml): + Latest version of the DAST template. ([Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/254325) + in GitLab 13.8). -The browser-based scanner should provide greater coverage for most web applications, compared -with the current DAST AJAX crawler. While both crawlers are -used together with the current DAST scanner, the combination of the browser-based crawler with the -current DAST scanner is much more effective at finding and testing every page in an application. + WARNING: + The latest version of the template may include breaking changes. Use the + stable template unless you need a feature provided only in the latest template. -## Enable browser-based analyzer + For more information about template versioning, see the + [CI/CD documentation](../../../development/cicd/templates.md#latest-version). -To enable the browser-based analyzer: +1. Add a `dast` stage to your GitLab CI/CD stages configuration. -1. Ensure the DAST [prerequisites](index.md#prerequisites) are met. -1. Include the [DAST CI/CD template](proxy-based.md#include-the-dast-template). -1. Set the target website using the [`DAST_WEBSITE` CI/CD variable](proxy-based.md#available-cicd-variables). -1. Set the CI/CD variable `DAST_BROWSER_SCAN` to `true`. +1. Define the URL to be scanned by DAST by using one of these methods: -Example extract of `.gitlab-ci.yml` file: + - Set the `DAST_WEBSITE` [CI/CD variable](../../../ci/yaml/index.md#variables). + If set, this value takes precedence. + + - Adding the URL in an `environment_url.txt` file at your project's root is great for testing in + dynamic environments. To run DAST against an application dynamically created during a GitLab CI/CD + pipeline, write the application URL to an `environment_url.txt` file. DAST automatically reads the + URL to find the scan target. + + You can see an [example of this in our Auto DevOps CI YAML](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/ci/templates/Jobs/Deploy.gitlab-ci.yml). + +1. Set the `DAST_BROWSER_SCAN` [CI/CD variable](../../../ci/yaml/index.md#variables) to `"true"`. + +For example: ```yaml +stages: + - build + - test + - deploy + - dast + include: - template: DAST.gitlab-ci.yml @@ -52,36 +157,58 @@ dast: DAST_BROWSER_SCAN: "true" ``` +### Authentication + +The browser-based analyzer can authenticate a user prior to a scan. See [Authentication](authentication.md) for +configuration instructions. + ### Available CI/CD variables -The browser-based crawler can be configured using CI/CD variables. - -| CI/CD variable | Type | Example | Description | -|----------------------------------------------| ----------------| --------------------------------- | ------------| -| `DAST_WEBSITE` | URL | `http://www.site.com` | The URL of the website to scan. | -| `DAST_BROWSER_SCAN` | boolean | `true` | Configures DAST to use the browser-based crawler engine. | -| `DAST_BROWSER_ALLOWED_HOSTS` | List of strings | `site.com,another.com` | Hostnames included in this variable are considered in scope when crawled. By default the `DAST_WEBSITE` hostname is included in the allowed hosts list. | -| `DAST_BROWSER_EXCLUDED_HOSTS` | List of strings | `site.com,another.com` | Hostnames included in this variable are considered excluded and connections are forcibly dropped. | -| `DAST_BROWSER_EXCLUDED_ELEMENTS` | selector | `a[href='2.html'],css:.no-follow` | Comma-separated list of selectors that are ignored when scanning. | -| `DAST_BROWSER_IGNORED_HOSTS` | List of strings | `site.com,another.com` | Hostnames included in this variable are accessed but not reported against. | -| `DAST_BROWSER_MAX_ACTIONS` | number | `10000` | The maximum number of actions that the crawler performs. For example, selecting a link, or filling a form. | -| `DAST_BROWSER_MAX_DEPTH` | number | `10` | The maximum number of chained actions that the crawler takes. For example, `Click -> Form Fill -> Click` is a depth of three. | -| `DAST_BROWSER_NUMBER_OF_BROWSERS` | number | `3` | The maximum number of concurrent browser instances to use. For shared runners on GitLab.com, we recommended a maximum of three. Private runners with more resources may benefit from a higher number, but are likely to produce little benefit after five to seven instances. | -| `DAST_BROWSER_COOKIES` | dictionary | `abtesting_group:3,region:locked` | A cookie name and value to be added to every request. | -| `DAST_BROWSER_LOG` | List of strings | `brows:debug,auth:debug` | A list of modules and their intended log level. | -| `DAST_BROWSER_NAVIGATION_TIMEOUT` | [Duration string](https://pkg.go.dev/time#ParseDuration) | `15s` | The maximum amount of time to wait for a browser to navigate from one page to another. | -| `DAST_BROWSER_ACTION_TIMEOUT` | [Duration string](https://pkg.go.dev/time#ParseDuration) | `7s` | The maximum amount of time to wait for a browser to complete an action. | -| `DAST_BROWSER_STABILITY_TIMEOUT` | [Duration string](https://pkg.go.dev/time#ParseDuration) | `7s` | The maximum amount of time to wait for a browser to consider a page loaded and ready for analysis. | -| `DAST_BROWSER_NAVIGATION_STABILITY_TIMEOUT` | [Duration string](https://pkg.go.dev/time#ParseDuration) | `7s` | The maximum amount of time to wait for a browser to consider a page loaded and ready for analysis after a navigation completes. | -| `DAST_BROWSER_ACTION_STABILITY_TIMEOUT` | [Duration string](https://pkg.go.dev/time#ParseDuration) | `800ms` | The maximum amount of time to wait for a browser to consider a page loaded and ready for analysis after completing an action. | -| `DAST_BROWSER_SEARCH_ELEMENT_TIMEOUT` | [Duration string](https://pkg.go.dev/time#ParseDuration) | `3s` | The maximum amount of time to allow the browser to search for new elements or navigations. | -| `DAST_BROWSER_EXTRACT_ELEMENT_TIMEOUT` | [Duration string](https://pkg.go.dev/time#ParseDuration) | `5s` | The maximum amount of time to allow the browser to extract newly found elements or navigations. | -| `DAST_BROWSER_ELEMENT_TIMEOUT` | [Duration string](https://pkg.go.dev/time#ParseDuration) | `600ms` | The maximum amount of time to wait for an element before determining it is ready for analysis. | -| `DAST_BROWSER_PAGE_READY_SELECTOR` | selector | `css:#page-is-ready` | Selector that when detected as visible on the page, indicates to the analyzer that the page has finished loading and the scan can continue. Note: When this selector is set, but the element is not found, the scanner waits for the period defined in `DAST_BROWSER_STABILITY_TIMEOUT` before continuing the scan. This can significantly increase scanning time if the element is not present on multiple pages within the site. | - -The [DAST variables](proxy-based.md#available-cicd-variables) `SECURE_ANALYZERS_PREFIX`, `DAST_FULL_SCAN_ENABLED`, `DAST_AUTO_UPDATE_ADDONS`, `DAST_EXCLUDE_RULES`, `DAST_REQUEST_HEADERS`, `DAST_HTML_REPORT`, `DAST_MARKDOWN_REPORT`, `DAST_XML_REPORT`, -`DAST_AUTH_URL`, `DAST_USERNAME`, `DAST_PASSWORD`, `DAST_USERNAME_FIELD`, `DAST_PASSWORD_FIELD`, `DAST_FIRST_SUBMIT_FIELD`, `DAST_SUBMIT_FIELD`, `DAST_EXCLUDE_URLS`, `DAST_AUTH_VERIFICATION_URL`, `DAST_BROWSER_AUTH_VERIFICATION_SELECTOR`, `DAST_BROWSER_AUTH_VERIFICATION_LOGIN_FORM`, `DAST_BROWSER_AUTH_REPORT`, -`DAST_INCLUDE_ALPHA_VULNERABILITIES`, `DAST_PATHS_FILE`, `DAST_PATHS`, `DAST_ZAP_CLI_OPTIONS`, and `DAST_ZAP_LOG_CONFIGURATION` are also compatible with browser-based crawler scans. +These CI/CD variables are specific to the browser-based DAST analyzer. They can be used to customize the behavior of +DAST to your requirements. +For authentication CI/CD variables, see [Authentication](authentication.md). + +| CI/CD variable | Type | Example | Description | +|:--------------------------------------------|:---------------------------------------------------------|----------------------------------------|:------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| +| `DAST_ADVERTISE_SCAN` | boolean | `true` | Set to `true` to add a `Via` header to every request sent, advertising that the request was sent as part of a GitLab DAST scan. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/334947) in GitLab 14.1. | +| `DAST_BROWSER_ACTION_STABILITY_TIMEOUT` | [Duration string](https://pkg.go.dev/time#ParseDuration) | `800ms` | The maximum amount of time to wait for a browser to consider a page loaded and ready for analysis after completing an action. | +| `DAST_BROWSER_ACTION_TIMEOUT` | [Duration string](https://pkg.go.dev/time#ParseDuration) | `7s` | The maximum amount of time to wait for a browser to complete an action. | +| `DAST_BROWSER_ALLOWED_HOSTS` | List of strings | `site.com,another.com` | Hostnames included in this variable are considered in scope when crawled. By default the `DAST_WEBSITE` hostname is included in the allowed hosts list. | +| `DAST_BROWSER_COOKIES` | dictionary | `abtesting_group:3,region:locked` | A cookie name and value to be added to every request. | +| `DAST_BROWSER_CRAWL_GRAPH` | boolean | `true` | Set to `true` to generate an SVG graph of navigation paths visited during crawl phase of the scan. | +| `DAST_BROWSER_DEVTOOLS_LOG` | string | `Default:messageAndBody,truncate:2000` | Set to log protocol messages between DAST and the Chromium browser. | | +| `DAST_BROWSER_ELEMENT_TIMEOUT` | [Duration string](https://pkg.go.dev/time#ParseDuration) | `600ms` | The maximum amount of time to wait for an element before determining it is ready for analysis. | +| `DAST_BROWSER_EXCLUDED_ELEMENTS` | selector | `a[href='2.html'],css:.no-follow` | Comma-separated list of selectors that are ignored when scanning. | +| `DAST_BROWSER_EXCLUDED_HOSTS` | List of strings | `site.com,another.com` | Hostnames included in this variable are considered excluded and connections are forcibly dropped. | +| `DAST_BROWSER_EXTRACT_ELEMENT_TIMEOUT` | [Duration string](https://pkg.go.dev/time#ParseDuration) | `5s` | The maximum amount of time to allow the browser to extract newly found elements or navigations. | +| `DAST_BROWSER_FILE_LOG` | List of strings | `brows:debug,auth:debug` | A list of modules and their intended logging level for use in the file log. | +| `DAST_BROWSER_FILE_LOG_PATH` | string | `/output/browserker.log` | Set to the path of the file log. | +| `DAST_BROWSER_IGNORED_HOSTS` | List of strings | `site.com,another.com` | Hostnames included in this variable are accessed, not attacked, and not reported against. | +| `DAST_BROWSER_INCLUDE_ONLY_RULES` | List of strings | `16.1,16.2,16.3` | Comma-separated list of check identifiers to use for the scan. | +| `DAST_BROWSER_LOG` | List of strings | `brows:debug,auth:debug` | A list of modules and their intended logging level for use in the console log. | +| `DAST_BROWSER_LOG_CHROMIUM_OUTPUT` | boolean | `true` | Set to `true` to log Chromium `STDOUT` and `STDERR`. | +| `DAST_BROWSER_MAX_ACTIONS` | number | `10000` | The maximum number of actions that the crawler performs. For example, selecting a link, or filling a form. | +| `DAST_BROWSER_MAX_DEPTH` | number | `10` | The maximum number of chained actions that the crawler takes. For example, `Click -> Form Fill -> Click` is a depth of three. | +| `DAST_BROWSER_MAX_RESPONSE_SIZE_MB` | number | `15` | The maximum size of a HTTP response body. Responses with bodies larger than this are blocked by the browser. Defaults to 10 MB. | +| `DAST_BROWSER_NAVIGATION_STABILITY_TIMEOUT` | [Duration string](https://pkg.go.dev/time#ParseDuration) | `7s` | The maximum amount of time to wait for a browser to consider a page loaded and ready for analysis after a navigation completes. | +| `DAST_BROWSER_NAVIGATION_TIMEOUT` | [Duration string](https://pkg.go.dev/time#ParseDuration) | `15s` | The maximum amount of time to wait for a browser to navigate from one page to another. | +| `DAST_BROWSER_NUMBER_OF_BROWSERS` | number | `3` | The maximum number of concurrent browser instances to use. For shared runners on GitLab.com, we recommended a maximum of three. Private runners with more resources may benefit from a higher number, but are likely to produce little benefit after five to seven instances. | +| `DAST_BROWSER_PAGE_READY_SELECTOR` | selector | `css:#page-is-ready` | Selector that when detected as visible on the page, indicates to the analyzer that the page has finished loading and the scan can continue. | +| `DAST_BROWSER_SCAN` | boolean | `true` | Required to be `true` to run a browser-based scan. | +| `DAST_BROWSER_SEARCH_ELEMENT_TIMEOUT` | [Duration string](https://pkg.go.dev/time#ParseDuration) | `3s` | The maximum amount of time to allow the browser to search for new elements or user actions. | +| `DAST_BROWSER_STABILITY_TIMEOUT` | [Duration string](https://pkg.go.dev/time#ParseDuration) | `7s` | The maximum amount of time to wait for a browser to consider a page loaded and ready for analysis. | +| `DAST_EXCLUDE_RULES` | string | `10020,10026` | Set to a comma-separated list of ZAP Vulnerability Rule IDs to exclude them from running during the scan. Rule IDs are numbers and can be found from the DAST log or on the [ZAP project](https://www.zaproxy.org/docs/alerts/). | +| `DAST_EXCLUDE_URLS` | URLs | `https://example.com/.*/sign-out` | The URLs to skip during the authenticated scan; comma-separated. Regular expression syntax can be used to match multiple URLs. For example, `.*` matches an arbitrary character sequence. | +| `DAST_FULL_SCAN_ENABLED` | boolean | `true` | Set to `true` to run both passive and active checks. Default: `false` | +| `DAST_PATHS` | string | `/page1.html,/category1/page3.html` | Set to a comma-separated list of URL paths relative to `DAST_WEBSITE` for DAST to scan. | +| `DAST_PATHS_FILE` | string | `/builds/project/urls.txt` | Set to a file path containing a list of URL paths relative to `DAST_WEBSITE` for DAST to scan. The file must be plain text with one path per line. | +| `DAST_PKCS12_CERTIFICATE_BASE64` | string | `ZGZkZ2p5NGd...` | The PKCS12 certificate used for sites that require Mutual TLS. Must be encoded as base64 text. | +| `DAST_PKCS12_PASSWORD` | string | `password` | The password of the certificate used in `DAST_PKCS12_CERTIFICATE_BASE64`. Create sensitive [custom CI/CI variables](../../../ci/variables/index.md#custom-cicd-variables) using the GitLab UI. | +| `DAST_REQUEST_HEADERS` | string | `Cache-control:no-cache` | Set to a comma-separated list of request header names and values. Headers are added to every request made to `DAST_BROWSER_ALLOWED_HOSTS` by DAST. | +| `DAST_SKIP_TARGET_CHECK` | boolean | `true` | Set to `true` to prevent DAST from checking that the target is available before scanning. Default: `false`. | +| `DAST_TARGET_AVAILABILITY_TIMEOUT` | number | `60` | Time limit in seconds to wait for target availability. | +| `DAST_WEBSITE` | URL | `https://example.com` | The URL of the website to scan. | +| `SECURE_ANALYZERS_PREFIX` | URL | `registry.organization.com` | Set the Docker registry base address from which to download the analyzer. | ## Vulnerability detection @@ -150,53 +277,8 @@ dast: NOTE: Adjusting these values may impact scan time because they adjust how long each browser waits for various activities to complete. -## Debugging scans using logging - -Logging can be used to help you troubleshoot a scan. - -The CI/CD variable `DAST_BROWSER_LOG` configures the logging level for particular modules of the crawler. Each module represents a component of the browser-based crawler and is separated so that debug logs can be configured just for the area of the crawler that requires further inspection. For more details, see [Crawler modules](#crawler-modules). - -For example, the following job definition enables the browsing module and the authentication module to be logged in debug-mode: - -```yaml -include: - - template: DAST.gitlab-ci.yml - -dast: - variables: - DAST_WEBSITE: "https://my.site.com" - DAST_BROWSER_SCAN: "true" - DAST_BROWSER_LOG: "brows:debug,auth:debug" -``` - -### Log message format +## Artifacts -Log messages have the format `[time] [log level] [log module] [message] [additional properties]`. For example, the following log entry has level `INFO`, is part of the `CRAWL` log module, and has the message `Crawled path`. - -```txt -2021-04-21T00:34:04.000 INF CRAWL Crawled path nav_id=0cc7fd path="LoadURL [https://my.site.com:8090]" -``` - -### Crawler modules - -The modules that can be configured for logging are as follows: - -| Log module | Component overview | -| ---------- | ----------- | -| `AUTH` | Used for creating an authenticated scan. | -| `BROWS` | Used for querying the state or page of the browser. | -| `BPOOL` | The set of browsers that are leased out for crawling. | -| `CRAWL` | Used for the core crawler algorithm. | -| `DATAB` | Used for persisting data to the internal database. | -| `LEASE` | Used to create browsers to add them to the browser pool. | -| `MAIN` | Used for the flow of the main event loop of the crawler. | -| `NAVDB` | Used for persistence mechanisms to store navigation entries. | -| `REPT` | Used for generating reports. | -| `STAT` | Used for general statistics while running the scan. | - -### Artifacts - -DAST's browser-based analyzer generates artifacts that can help you understand how the scanner works. Using the latest version of the DAST [template](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/ci/templates/Security/DAST.latest.gitlab-ci.yml) these artifacts are exposed for download by default. The list of artifacts includes the following files: @@ -204,3 +286,7 @@ The list of artifacts includes the following files: - `gl-dast-debug-auth-report.html` - `gl-dast-debug-crawl-report.html` - `gl-dast-crawl-graph.svg` + +## Troubleshooting + +See [troubleshooting](browser_based_troubleshooting.md) for more information. diff --git a/doc/user/application_security/dast/browser_based_troubleshooting.md b/doc/user/application_security/dast/browser_based_troubleshooting.md new file mode 100644 index 00000000000..78f2723ee38 --- /dev/null +++ b/doc/user/application_security/dast/browser_based_troubleshooting.md @@ -0,0 +1,300 @@ +--- +stage: Secure +group: Dynamic Analysis +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments +type: reference, howto +--- + +# Troubleshooting DAST browser-based analyzer **(ULTIMATE)** + +The following troubleshooting scenarios have been collected from customer support cases. If you +experience a problem not addressed here, or the information here does not fix your problem, create a +support ticket. For more details, see the [GitLab Support](https://about.gitlab.com/support/) page. + +## When something goes wrong + +When something goes wrong with a DAST scan, if you have a particular error message then check [known problems](#known-problems). + +Otherwise, try to discover the problem by answering the following questions: + +- [What is the expected outcome?](#what-is-the-expected-outcome) +- [Is the outcome achievable by a human?](#is-the-outcome-achievable-by-a-human) +- [Any reason why DAST would not work?](#any-reason-why-dast-would-not-work) +- [How does your application work?](#how-does-your-application-work) +- [What is DAST doing?](#what-is-dast-doing) + +### What is the expected outcome? + +Many users who encounter issues with a DAST scan have a good high-level idea of what they think the scanner should be doing. For example, +it's not scanning particular pages, or it's not selecting a button on the page. + +As much as possible, try to isolate the problem to help narrow the search for a solution. For example, take the situation where DAST isn't scanning a particular page. +From where should DAST have found the page? What path did it take to navigate there? Were there elements on the referring page that DAST should have selected, but did not? + +### Is the outcome achievable by a human? + +DAST cannot scan an application if a human cannot manually traverse the application. + +Knowing the outcome you expect, try to replicate it manually using a browser on your machine. For example: + +- Open a new incognito/private browser window. +- Open Developer Tools. Keep an eye on the console for error messages. + - In Chrome: `View -> Developer -> Developer Tools`. + - In Firefox: `Tools -> Browser Tools -> Web Developer Tools`. +- If authenticating: + - Navigate to the `DAST_AUTH_URL`. + - Type in the `DAST_USERNAME` in the `DAST_USERNAME_FIELD`. + - Type in the `DAST_PASSWORD` in the `DAST_PASSWORD_FIELD`. + - Select the `DAST_SUBMIT_FIELD`. +- Select links and fill in forms. Navigate to the pages that aren't scanning correctly. +- Observe how your application behaves. Notice if there is anything that might cause problems for an automated scanner. + +### Any reason why DAST would not work? + +DAST cannot scan correctly when: + +- There is a CAPTCHA. Please turn these off in the testing environment for the application being scanned. +- It does not have access to the target application. Ensure the GitLab Runner can access the application using the URLs used in the DAST configuration. + +### How does your application work? + +Understanding how your application works is vital to figuring out why a DAST scan isn't working. For example, the following situations +may require additional configuration settings. + +- Is there a popup modal that hides elements? +- Does a loaded page change dramatically after a certain period of time? +- Is the application especially slow or fast to load? +- Is the target application jerky while loading? +- Does the application work differently based on the client's location? +- Is the application a single-page application? +- Does the application submit HTML forms, or does it use JavaScript and AJAX? +- Does the application use websockets? +- Does the application use a specific web framework? +- Does selecting buttons run JavaScript before continuing the form submit? Is it fast, slow? +- Is it possible DAST could be selecting or searching for elements before either the element or page is ready? + +### What is DAST doing? + +Logging remains the best way to understand what DAST is doing: + +- [Browser-based analyzer logging](#browser-based-analyzer-logging), useful for understanding what the analyzer is doing. +- [Chromium DevTools logging](#chromium-devtools-logging), useful to inspect the communication between DAST and Chromium. +- [Chromium Logs](#chromium-logs), useful for logging errors when Chromium crashes unexpectedly. + +## Browser-based analyzer logging + +The analyzer log is one of the most useful tools to help diagnose problems with a scan. Different parts of the analyzer can be logged at different levels. + +### Log message format + +Log messages have the format `[time] [log level] [log module] [message] [additional properties]`. + +For example, the following log entry has level `INFO`, is part of the `CRAWL` log module, has the message `Crawled path` and the additional properties `nav_id` and `path`. + +```txt +2021-04-21T00:34:04.000 INF CRAWL Crawled path nav_id=0cc7fd path="LoadURL [https://my.site.com:8090]" +``` + +### Log destination + +Logs are sent either to file or to console (the CI/CD job log). You can configure each destination to accept different logs using +the environment variables `DAST_BROWSER_LOG` for console logs and `DAST_BROWSER_FILE_LOG` for file logs. + +In the following example, the file log defaults to `DEBUG` level, the console log defaults to `INFO` level and logs the `AUTH` module at `DEBUG` level. + +```yaml +include: + - template: DAST.gitlab-ci.yml + +dast: + variables: + DAST_BROWSER_LOG: "auth:debug" + DAST_BROWSER_FILE_LOG: "loglevel:debug" + DAST_BROWSER_FILE_LOG_PATH: "/zap/wrk/dast-scan.log" + artifacts: + paths: + - dast-scan.log + when: always +``` + +### Log levels + +The log levels that can be configured are as follows: + +| Log module | Component overview | More | +|-------------------------|--------------------------------------------------------------------------|----------------------------------| +| `TRACE` | Used for specific, often noisy inner workings of a feature. | | +| `DEBUG` | Describes the inner-workings of a feature. Used for diagnostic purposes. | | +| `INFO` | Describes the high level flow of the scan and the results. | Default level if none specified. | +| `WARN` | Describes an error situation where DAST recovers and continues the scan. | | +| `FATAL`/`ERROR`/`PANIC` | Describes unrecoverable errors prior to exit. | | + +### Log modules + +`LOGLEVEL` configures the default log level for the log destination. If any of the following modules are configured, +DAST uses the log level for that module in preference to the default log level. + +The modules that can be configured for logging are as follows: + +| Log module | Component overview | +|------------|---------------------------------------------------------------------------------------------------| +| `ACTIV` | Used for active attacks. | +| `AUTH` | Used for creating an authenticated scan. | +| `BPOOL` | The set of browsers that are leased out for crawling. | +| `BROWS` | Used for querying the state or page of the browser. | +| `CACHE` | Used for reporting on cache hit and miss for cached HTTP resources. | +| `CHROM` | Used to log Chrome DevTools messages. | +| `CONTA` | Used for the container that collects parts of HTTP requests and responses from DevTools messages. | +| `CRAWL` | Used for the core crawler algorithm. | +| `DATAB` | Used for persisting data to the internal database. | +| `LEASE` | Used to create browsers to add them to the browser pool. | +| `MAIN` | Used for the flow of the main event loop of the crawler. | +| `NAVDB` | Used for persistence mechanisms to store navigation entries. | +| `REGEX` | Used for recording performance statistics when running regular expressions. | +| `REPT` | Used for generating reports. | +| `STAT` | Used for general statistics while running the scan. | +| `VLDFN` | Used for loading and parsing vulnerability definitions. | +| `WEBGW` | Used to log messages sent to the target application when running active checks. | + +### Example - log crawled paths + +Set the log module `CRAWL` to `DEBUG` to log navigation paths found during the crawl phase of the scan. This is useful for understanding +if DAST is crawling your target application correctly. + +```yaml +include: + - template: DAST.gitlab-ci.yml + +dast: + variables: + DAST_BROWSER_LOG: "crawl:debug" +``` + +For example, the following output shows that four anchor links we discovered during the crawl of the page at `https://example.com`. + +```plaintext +2022-11-17T11:18:05.578 DBG CRAWL executing step nav_id=6ec647d8255c729160dd31cb124e6f89 path="LoadURL [https://example.com]" step=1 +... +2022-11-17T11:18:11.900 DBG CRAWL found new navigations browser_id=2243909820020928961 nav_count=4 nav_id=6ec647d8255c729160dd31cb124e6f89 of=1 step=1 +2022-11-17T11:18:11.901 DBG CRAWL adding navigation action="LeftClick [a href=/page1.html]" nav=bd458cc1fc2d7c6fb984464b6d968866 parent_nav=6ec647d8255c729160dd31cb124e6f89 +2022-11-17T11:18:11.901 DBG CRAWL adding navigation action="LeftClick [a href=/page2.html]" nav=6dcb25f9f9ece3ee0071ac2e3166d8e6 parent_nav=6ec647d8255c729160dd31cb124e6f89 +2022-11-17T11:18:11.901 DBG CRAWL adding navigation action="LeftClick [a href=/page3.html]" nav=89efbb0c6154d6c6d85a63b61a7cdc6f parent_nav=6ec647d8255c729160dd31cb124e6f89 +2022-11-17T11:18:11.901 DBG CRAWL adding navigation action="LeftClick [a href=/page4.html]" nav=f29b4f4e0bdee70f5255de7fc080f04d parent_nav=6ec647d8255c729160dd31cb124e6f89 +``` + +## Chromium DevTools logging + +WARNING: +Logging DevTools messages is a security risk. The output contains secrets such as usernames, passwords and authentication tokens. +The output is uploaded to the GitLab server and may be visible in job logs. + +The DAST Browser-based scanner orchestrates a Chromium browser using the [Chrome DevTools Protocol](https://chromedevtools.github.io/devtools-protocol/). +Logging DevTools messages helps provide transparency into what the browser is doing. For example, if selecting a button does not work, a DevTools message might show that the cause is a CORS error in a browser console log. +Logs that contain DevTools messages can be very large in size. For this reason, it should only be enabled on jobs with a short duration. + +To log all DevTools messages, turn the `CHROM` log module to `trace` and configure logging levels. The following are examples of DevTools logs: + +```plaintext +2022-12-05T06:27:24.280 TRC CHROM event received {"method":"Fetch.requestPaused","params":{"requestId":"interception-job-3.0","request":{"url":"http://auth-auto:8090/font-awesome.min.css","method":"GET","headers":{"Accept":"text/css,*/*;q=0.1","Referer":"http://auth-auto:8090/login.html","User-Agent":"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) HeadlessChrome/105.0.5195.102 Safari/537.36"},"initialPriority":"VeryHigh","referrerPolicy":"strict-origin-when-cross-origin"},"frameId":"A706468B01C2FFAA2EB6ED365FF95889","resourceType":"Stylesheet","networkId":"39.3"}} method=Fetch.requestPaused +2022-12-05T06:27:24.280 TRC CHROM request sent {"id":47,"method":"Fetch.continueRequest","params":{"requestId":"interception-job-3.0","headers":[{"name":"Accept","value":"text/css,*/*;q=0.1"},{"name":"Referer","value":"http://auth-auto:8090/login.html"},{"name":"User-Agent","value":"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) HeadlessChrome/105.0.5195.102 Safari/537.36"}]}} id=47 method=Fetch.continueRequest +2022-12-05T06:27:24.281 TRC CHROM response received {"id":47,"result":{}} id=47 method=Fetch.continueRequest +``` + +### Customizing DevTools log levels + +Chrome DevTools requests, responses and events are namespaced by domain. DAST allows each domain and each domain with message to have different logging configuration. +The environment variable `DAST_BROWSER_DEVTOOLS_LOG` accepts a semi-colon separated list of logging configurations. +Logging configurations are declared using the structure `[domain/message]:[what-to-log][,truncate:[max-message-size]]`. + +- `domain/message` references what is being logged. + - `Default` can be used as a value to represent all domains and messages. + - Can be a domain, for example, `Browser`, `CSS`, `Page`, `Network`. + - Can be a domain with a message, for example, `Network.responseReceived`. + - If multiple configurations apply, the most specific configuration is used. +- `what-to-log` references whether and what to log. + - `message` logs that a message was received and does not log the message content. + - `messageAndBody` logs the message with the message content. Recommended to be used with `truncate`. + - `suppress` does not log the message. Used to silence noisy domains and messages. +- `truncate` is an optional configuration to limit the size of the message printed. + +### Example - log all DevTools messages + +Used to log everything when you're not sure where to start. + +```yaml +include: + - template: DAST.gitlab-ci.yml + +dast: + variables: + DAST_BROWSER_FILE_LOG: "chrom:trace" + DAST_BROWSER_FILE_LOG_PATH: "/zap/wrk/dast-scan.log" + DAST_BROWSER_DEVTOOLS_LOG: "Default:messageAndBody,truncate:2000" + artifacts: + paths: + - dast-scan.log + when: always +``` + +### Example - log HTTP messages + +Useful for when a resource isn't loading correctly. HTTP message events are logged, as is the decision to continue or +fail the request. Any errors in the browser console are also logged. + +```yaml +include: + - template: DAST.gitlab-ci.yml + +dast: + variables: + DAST_BROWSER_FILE_LOG: "chrom:trace" + DAST_BROWSER_FILE_LOG_PATH: "/zap/wrk/dast-scan.log" + DAST_BROWSER_DEVTOOLS_LOG: "Default:suppress;Fetch:messageAndBody,truncate:2000;Network:messageAndBody,truncate:2000;Log:messageAndBody,truncate:2000;Console:messageAndBody,truncate:2000" + artifacts: + paths: + - dast-scan.log + when: always +``` + +## Chromium logs + +In the rare event that Chromium crashes, it can be helpful to write the Chromium process `STDOUT` and `STDERR` to log. +Setting the environment variable `DAST_BROWSER_LOG_CHROMIUM_OUTPUT` to `true` achieves this purpose. + +DAST starts and stops many Chromium processes. DAST sends each process output to all log destinations with the log module `LEASE` and log level `INFO`. + +For example: + +```yaml +include: + - template: DAST.gitlab-ci.yml + +dast: + variables: + DAST_BROWSER_LOG_CHROMIUM_OUTPUT: "true" +``` + +## Known problems + +### Logs contain `response body exceeds allowed size` + +By default DAST processes HTTP requests where the HTTP response body is 10 MB or less. Otherwise, DAST blocks the response +which can cause scans to fail. This constraint is intended to reduce memory consumption during a scan. + +An example log is as follows, where DAST blocked the JavaScript file found at `https://example.com/large.js` as it's size is greater than the limit: + +```plaintext +2022-12-05T06:28:43.093 WRN BROWS response body exceeds allowed size allowed_size_bytes=1000000 browser_id=752944257619431212 nav_id=ae23afe2acbce2c537657a9112926f1a of=1 request_id=interception-job-2.0 response_size_bytes=9333408 step=1 url=https://example.com/large.js +2022-12-05T06:28:58.104 WRN CONTA request failed, attempting to continue scan error=net::ERR_BLOCKED_BY_RESPONSE index=0 requestID=38.2 url=https://example.com/large.js +``` + +This can be changed using the configuration `DAST_MAX_RESPONSE_SIZE_MB`. For example, + +```yaml +include: + - template: DAST.gitlab-ci.yml + +dast: + variables: + DAST_MAX_RESPONSE_SIZE_MB: "25" +``` diff --git a/doc/user/application_security/dast/checks/16.2.md b/doc/user/application_security/dast/checks/16.2.md index a317b9418a1..2051b118009 100644 --- a/doc/user/application_security/dast/checks/16.2.md +++ b/doc/user/application_security/dast/checks/16.2.md @@ -40,5 +40,5 @@ the `Server` header. - [CWE](https://cwe.mitre.org/data/definitions/16.html) - [Apache ServerTokens](https://blog.mozilla.org/security/2016/08/26/mitigating-mime-confusion-attacks-in-firefox/) -- [NGINX server_tokens](https://nginx.org/en/docs/http/ngx_http_core_module.html#server_tokens) +- [NGINX `server_tokens`](https://nginx.org/en/docs/http/ngx_http_core_module.html#server_tokens) - [IIS 10 Remove Server Header](https://learn.microsoft.com/en-us/iis/configuration/system.webserver/security/requestfiltering/#attributes) diff --git a/doc/user/application_security/dast/checks/16.3.md b/doc/user/application_security/dast/checks/16.3.md index d9e6f6f8d92..d1799baa517 100644 --- a/doc/user/application_security/dast/checks/16.3.md +++ b/doc/user/application_security/dast/checks/16.3.md @@ -32,4 +32,4 @@ information from the `X-Powered-By` header. ## Links - [CWE](https://cwe.mitre.org/data/definitions/16.html) -- [PHP expose_php](https://www.php.net/manual/en/ini.core.php#ini.expose-php) +- [PHP `expose_php`](https://www.php.net/manual/en/ini.core.php#ini.expose-php) diff --git a/doc/user/application_security/dast/checks/548.1.md b/doc/user/application_security/dast/checks/548.1.md index b6907db5928..6cef8ccdb63 100644 --- a/doc/user/application_security/dast/checks/548.1.md +++ b/doc/user/application_security/dast/checks/548.1.md @@ -41,5 +41,5 @@ indexing. - [CWE](https://cwe.mitre.org/data/definitions/548.html) - [Apache Options](https://httpd.apache.org/docs/2.4/mod/core.html#options) -- [NGINX autoindex](https://nginx.org/en/docs/http/ngx_http_autoindex_module.html) -- [IIS directoryBrowse element](https://learn.microsoft.com/en-us/iis/configuration/system.webserver/directorybrowse) +- [NGINX `autoindex`](https://nginx.org/en/docs/http/ngx_http_autoindex_module.html) +- [IIS `directoryBrowse` element](https://learn.microsoft.com/en-us/iis/configuration/system.webserver/directorybrowse) diff --git a/doc/user/application_security/dast/checks/798.33.md b/doc/user/application_security/dast/checks/798.33.md index 536faefdb51..4761ac9d157 100644 --- a/doc/user/application_security/dast/checks/798.33.md +++ b/doc/user/application_security/dast/checks/798.33.md @@ -4,11 +4,11 @@ group: Dynamic Analysis info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- -# Exposure of confidential secret or token Droneci Access Token +# Exposure of confidential secret or token Drone CI Access Token ## Description -The response body contains content that matches the pattern of a Droneci Access Token. +The response body contains content that matches the pattern of a Drone CI Access Token. Exposing this value could allow attackers to gain access to all resources granted by this token. ## Remediation diff --git a/doc/user/application_security/dast/checks/798.49.md b/doc/user/application_security/dast/checks/798.49.md index 7ea3a65fbfa..41a3e8ace3d 100644 --- a/doc/user/application_security/dast/checks/798.49.md +++ b/doc/user/application_security/dast/checks/798.49.md @@ -4,11 +4,11 @@ group: Dynamic Analysis info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- -# Exposure of confidential secret or token Freshbooks Access Token +# Exposure of confidential secret or token FreshBooks Access Token ## Description -The response body contains content that matches the pattern of a Freshbooks Access Token. +The response body contains content that matches the pattern of a FreshBooks Access Token. Exposing this value could allow attackers to gain access to all resources granted by this token. ## Remediation diff --git a/doc/user/application_security/dast/checks/798.65.md b/doc/user/application_security/dast/checks/798.65.md index f2ebfb988b2..083bfec3350 100644 --- a/doc/user/application_security/dast/checks/798.65.md +++ b/doc/user/application_security/dast/checks/798.65.md @@ -4,11 +4,11 @@ group: Dynamic Analysis info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- -# Exposure of confidential secret or token Launchdarkly Access Token +# Exposure of confidential secret or token LaunchDarkly Access Token ## Description -The response body contains content that matches the pattern of a Launchdarkly Access Token. +The response body contains content that matches the pattern of a LaunchDarkly Access Token. Exposing this value could allow attackers to gain access to all resources granted by this token. ## Remediation diff --git a/doc/user/application_security/dast/checks/798.97.md b/doc/user/application_security/dast/checks/798.97.md index d3035b05bbb..711288eba9c 100644 --- a/doc/user/application_security/dast/checks/798.97.md +++ b/doc/user/application_security/dast/checks/798.97.md @@ -4,11 +4,11 @@ group: Dynamic Analysis info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- -# Exposure of confidential secret or token Rubygem API token +# Exposure of confidential secret or token RubyGems API token ## Description -The response body contains content that matches the pattern of a Rubygem API token. +The response body contains content that matches the pattern of a RubyGems API token. Exposing this value could allow attackers to gain access to all resources granted by this token. ## Remediation diff --git a/doc/user/application_security/dast/checks/829.1.md b/doc/user/application_security/dast/checks/829.1.md index f18634b72d9..7df250c2047 100644 --- a/doc/user/application_security/dast/checks/829.1.md +++ b/doc/user/application_security/dast/checks/829.1.md @@ -20,7 +20,7 @@ applications users would be protected from the malicious alterations. All identified resources should be sourced from the same domain as the target application. If this is not possible, it is strongly recommended that all `script` tags that implement `src` values, or `link` tags that implement the `href` values include Sub-Resource Integrity. To generate SRI integrity values the -[srihash](https://www.srihash.org/) tool can be used, or by running one of the following commands: +[SRI hash](https://www.srihash.org/) tool can be used, or by running one of the following commands: - `cat FILENAME.js | openssl dgst -sha384 -binary | openssl base64 -A` - `shasum -b -a 384 FILENAME.js | awk '{ print $1 }' | xxd -r -p | base64` diff --git a/doc/user/application_security/dast/checks/829.2.md b/doc/user/application_security/dast/checks/829.2.md index 19490afe676..d9d3e5a6341 100644 --- a/doc/user/application_security/dast/checks/829.2.md +++ b/doc/user/application_security/dast/checks/829.2.md @@ -19,7 +19,7 @@ them with known good versions. All identified resources should be sourced from the same domain as the target application. If this is not possible, it is strongly recommended that all `script` tags that implement `src` values, or `link` tags that implement the `href` values include Sub-Resource Integrity. To generate SRI integrity values the -[srihash](https://www.srihash.org/) tool can be used, or by running one of the following commands: +[SRI hash](https://www.srihash.org/) tool can be used, or by running one of the following commands: - `cat FILENAME.js | openssl dgst -sha384 -binary | openssl base64 -A` - `shasum -b -a 384 FILENAME.js | awk '{ print $1 }' | xxd -r -p | base64` diff --git a/doc/user/application_security/dast/checks/index.md b/doc/user/application_security/dast/checks/index.md index 9466734f9cf..56406b24586 100644 --- a/doc/user/application_security/dast/checks/index.md +++ b/doc/user/application_security/dast/checks/index.md @@ -69,7 +69,7 @@ The [DAST browser-based crawler](../browser_based.md) provides a number of vulne | [798.30](798.30.md) | Exposure of confidential secret or token Dropbox API secret | High | Passive | | [798.31](798.31.md) | Exposure of confidential secret or token Dropbox long lived API token | High | Passive | | [798.32](798.32.md) | Exposure of confidential secret or token Dropbox short lived API token | High | Passive | -| [798.33](798.33.md) | Exposure of confidential secret or token Droneci Access Token | High | Passive | +| [798.33](798.33.md) | Exposure of confidential secret or token Drone CI Access Token | High | Passive | | [798.34](798.34.md) | Exposure of confidential secret or token Duffel API token | High | Passive | | [798.35](798.35.md) | Exposure of confidential secret or token Dynatrace API token | High | Passive | | [798.36](798.36.md) | Exposure of confidential secret or token EasyPost API token | High | Passive | @@ -84,7 +84,7 @@ The [DAST browser-based crawler](../browser_based.md) provides a number of vulne | [798.46](798.46.md) | Exposure of confidential secret or token Flutterwave Secret Key | High | Passive | | [798.47](798.47.md) | Exposure of confidential secret or token Flutterwave Encryption Key | High | Passive | | [798.48](798.48.md) | Exposure of confidential secret or token Frame.io API token | High | Passive | -| [798.49](798.49.md) | Exposure of confidential secret or token Freshbooks Access Token | High | Passive | +| [798.49](798.49.md) | Exposure of confidential secret or token FreshBooks Access Token | High | Passive | | [798.50](798.50.md) | Exposure of confidential secret or token GoCardless API token | High | Passive | | [798.52](798.52.md) | Exposure of confidential secret or token GitHub Personal Access Token | High | Passive | | [798.53](798.53.md) | Exposure of confidential secret or token GitHub OAuth Access Token | High | Passive | @@ -99,7 +99,7 @@ The [DAST browser-based crawler](../browser_based.md) provides a number of vulne | [798.62](798.62.md) | Exposure of confidential secret or token Kraken Access Token | High | Passive | | [798.63](798.63.md) | Exposure of confidential secret or token Kucoin Access Token | High | Passive | | [798.64](798.64.md) | Exposure of confidential secret or token Kucoin Secret Key | High | Passive | -| [798.65](798.65.md) | Exposure of confidential secret or token Launchdarkly Access Token | High | Passive | +| [798.65](798.65.md) | Exposure of confidential secret or token LaunchDarkly Access Token | High | Passive | | [798.66](798.66.md) | Exposure of confidential secret or token Linear API Token | High | Passive | | [798.67](798.67.md) | Exposure of confidential secret or token Linear Client Secret | High | Passive | | [798.68](798.68.md) | Exposure of confidential secret or token LinkedIn Client ID | High | Passive | @@ -126,7 +126,7 @@ The [DAST browser-based crawler](../browser_based.md) provides a number of vulne | [798.94](798.94.md) | Exposure of confidential secret or token Private Key | High | Passive | | [798.95](798.95.md) | Exposure of confidential secret or token Pulumi API token | High | Passive | | [798.96](798.96.md) | Exposure of confidential secret or token PyPI upload token | High | Passive | -| [798.97](798.97.md) | Exposure of confidential secret or token Rubygem API token | High | Passive | +| [798.97](798.97.md) | Exposure of confidential secret or token RubyGem API token | High | Passive | | [798.98](798.98.md) | Exposure of confidential secret or token RapidAPI Access Token | High | Passive | | [798.99](798.99.md) | Exposure of confidential secret or token Sendbird Access ID | High | Passive | | [798.100](798.100.md) | Exposure of confidential secret or token Sendbird Access Token | High | Passive | diff --git a/doc/user/application_security/dast/dast_troubleshooting.md b/doc/user/application_security/dast/dast_troubleshooting.md index 61a7520bf7c..0dcf203a3a9 100644 --- a/doc/user/application_security/dast/dast_troubleshooting.md +++ b/doc/user/application_security/dast/dast_troubleshooting.md @@ -5,7 +5,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w type: reference, howto --- -# Troubleshooting Dynamic Application Security Testing (DAST) **(ULTIMATE)** +# Troubleshooting DAST proxy-based analyzer **(ULTIMATE)** The following troubleshooting scenarios have been collected from customer support cases. If you experience a problem not addressed here, or the information here does not fix your problem, create a diff --git a/doc/user/application_security/dast/index.md b/doc/user/application_security/dast/index.md index d78a8fca98f..283e48ec499 100644 --- a/doc/user/application_security/dast/index.md +++ b/doc/user/application_security/dast/index.md @@ -25,7 +25,7 @@ whitepaper. GitLab provides the following DAST analyzers, one or more of which may be useful depending on the kind of application you're testing. -For scanning websites, use one of: +For scanning websites, use one of: - The [DAST proxy-based analyzer](proxy-based.md) for scanning traditional applications serving simple HTML. The proxy-based analyzer can be run automatically or on-demand. - The [DAST browser-based analyzer](browser_based.md) for scanning applications that make heavy use of JavaScript. This includes single page web applications. diff --git a/doc/user/application_security/dast/proxy-based.md b/doc/user/application_security/dast/proxy-based.md index ec98b809fb7..fc78018bdad 100644 --- a/doc/user/application_security/dast/proxy-based.md +++ b/doc/user/application_security/dast/proxy-based.md @@ -24,6 +24,40 @@ The analyzer uses the [OWASP Zed Attack Proxy](https://www.zaproxy.org/) (ZAP) t to attack your application and produce a more extensive security report. It can be very useful when combined with [Review Apps](../../../ci/review_apps/index.md). +## Templates + +> - The DAST latest template was [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/254325) in GitLab 13.8. +> - All DAST templates were [updated](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/62597) to DAST_VERSION: 2 in GitLab 14.0. +> - All DAST templates were [updated](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/87183) to DAST_VERSION: 3 in GitLab 15.0. + +GitLab DAST configuration is defined in CI/CD templates. Updates to the template are provided with +GitLab upgrades, allowing you to benefit from any improvements and additions. + +Available templates: + +- [`DAST.gitlab-ci.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/ci/templates/Security/DAST.gitlab-ci.yml): Stable version of the DAST CI/CD template. +- [`DAST.latest.gitlab-ci.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/ci/templates/Security/DAST.latest.gitlab-ci.yml): Latest version of the DAST template. + +WARNING: +The latest version of the template may include breaking changes. Use the stable template unless you +need a feature provided only in the latest template. + +For more information about template versioning, see the +[CI/CD documentation](../../../development/cicd/templates.md#latest-version). + +## DAST versions + +By default, the DAST template uses the latest major version of the DAST Docker image. You can choose +how DAST updates, using the `DAST_VERSION` variable: + +- Automatically update DAST with new features and fixes by pinning to a major + version (such as `1`). +- Only update fixes by pinning to a minor version (such as `1.6`). +- Prevent all updates by pinning to a specific version (such as `1.6.4`). + +Find the latest DAST versions on the [DAST releases](https://gitlab.com/gitlab-org/security-products/dast/-/releases) +page. + ## DAST run options You can use DAST to examine your web application: @@ -46,58 +80,32 @@ To enable DAST to run automatically, either: - Enable [Auto DAST](../../../topics/autodevops/stages.md#auto-dast) (provided by [Auto DevOps](../../../topics/autodevops/index.md)). -- [Include the DAST template](#include-the-dast-template) in your existing - `.gitlab-ci.yml` file. -- [Configure DAST using the UI](#configure-dast-using-the-ui). - -#### Include the DAST template +- [Edit the `.gitlab.ci.yml` file manually](#edit-the-gitlabciyml-file-manually). +- [Use an automatically configured merge request](#configure-dast-using-the-ui). -> - This template was [updated](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/62597) to DAST_VERSION: 2 in GitLab 14.0. -> - This template was [updated](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/87183) to DAST_VERSION: 3 in GitLab 15.0. +#### Edit the `.gitlab.ci.yml` file manually -If you want to manually add DAST to your application, the DAST job is defined -in a CI/CD template file. Updates to the template are provided with GitLab -upgrades, allowing you to benefit from any improvements and additions. +In this method you manually edit the existing `.gitlab-ci.yml` file. Use this method if your GitLab CI/CD configuration file is complex. To include the DAST template: -1. Select the CI/CD template you want to use: - - - [`DAST.gitlab-ci.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/ci/templates/Security/DAST.gitlab-ci.yml): - Stable version of the DAST CI/CD template. - - [`DAST.latest.gitlab-ci.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/ci/templates/Security/DAST.latest.gitlab-ci.yml): - Latest version of the DAST template. ([Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/254325) - in GitLab 13.8). - - WARNING: - The latest version of the template may include breaking changes. Use the - stable template unless you need a feature provided only in the latest template. - - For more information about template versioning, see the - [CI/CD documentation](../../../development/cicd/templates.md#latest-version). - -1. Add a `dast` stage to your GitLab CI stages configuration: - - ```yaml - stages: - - dast - ``` - -1. Add the template to GitLab, based on your version of GitLab: +1. On the top bar, select **Main menu > Projects** and find your project. +1. On the left sidebar, select **CI/CD > Editor**. +1. Copy and paste the following to the bottom of the `.gitlab-ci.yml` file. - - In GitLab 11.9 and later, [include](../../../ci/yaml/index.md#includetemplate) - the template by adding the following to your `.gitlab-ci.yml` file: + To use the DAST stable template: - ```yaml - include: - - template: <template_file.yml> + ```yaml + include: + - template: DAST.gitlab-ci.yml + ``` - variables: - DAST_WEBSITE: https://example.com - ``` + To use the DAST latest template: - - In GitLab 11.8 and earlier, add the contents of the template to your - `.gitlab_ci.yml` file. + ```yaml + include: + - template: DAST.latest.gitlab-ci.yml + ``` 1. Define the URL to be scanned by DAST by using one of these methods: @@ -125,9 +133,13 @@ To include the DAST template: You can see an example of this in our [Auto DevOps CI YAML](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/ci/templates/Jobs/Deploy.gitlab-ci.yml) file. +1. Select the **Validate** tab, then select **Validate pipeline**. + The message **Simulation completed successfully** indicates the file is valid. +1. Select the **Edit** tab. +1. Optional. In **Commit message**, customize the commit message. +1. Select **Commit changes**. -The included template creates a `dast` job in your CI/CD pipeline and scans -your project's running application for possible vulnerabilities. +Pipelines now include a DAST job. The results are saved as a [DAST report artifact](../../../ci/yaml/artifacts_reports.md#artifactsreportsdast) @@ -137,21 +149,12 @@ always take the latest DAST artifact available. Behind the scenes, the is used to run the tests on the specified URL and scan it for possible vulnerabilities. -By default, the DAST template uses the latest major version of the DAST Docker -image. Using the `DAST_VERSION` variable, you can choose how DAST updates: - -- Automatically update DAST with new features and fixes by pinning to a major - version (such as `1`). -- Only update fixes by pinning to a minor version (such as `1.6`). -- Prevent all updates by pinning to a specific version (such as `1.6.4`). - -Find the latest DAST versions on the [Releases](https://gitlab.com/gitlab-org/security-products/dast/-/releases) -page. - #### Configure DAST using the UI -You can enable or configure DAST settings using the UI. The generated settings are formatted so they -can be conveniently pasted into the `.gitlab-ci.yml` file. +In this method you select options in the UI. Based on your selections, a code +snippet is created that you paste into the `.gitlab-ci.yml` file. + +To configure DAST using the UI: 1. On the top bar, select **Main menu > Projects** and find your project. 1. On the left sidebar, select **Security & Compliance > Configuration**. @@ -168,102 +171,17 @@ can be conveniently pasted into the `.gitlab-ci.yml` file. 1. To add the snippet to your project's `.gitlab-ci.yml` file, select **Copy code and open `.gitlab-ci.yml` file**. The Pipeline Editor opens. 1. Paste the snippet into the `.gitlab-ci.yml` file. - 1. Select the **Lint** tab to confirm the edited `.gitlab-ci.yml` file is valid. - 1. Select the **Edit** tab, then select **Commit changes**. + 1. Select the **Validate** tab, then select **Validate pipeline**. + The message **Simulation completed successfully** indicates the file is valid. + 1. Select the **Edit** tab. + 1. Optional. In **Commit message**, customize the commit message. + 1. Select **Commit changes**. -When the snippet is committed to the `.gitlab-ci.yml` file, pipelines include a DAST job. +Pipelines now include a DAST job. ### API scan -> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/10928) in GitLab 12.10. -> - A new DAST API scanning engine was introduced in GitLab 13.10. - -Using an API specification as a scan's target is a useful way to seed URLs for scanning an API. -Vulnerability rules in an API scan are different than those in a normal website scan. - -A new DAST API scanning engine is available in GitLab 13.12 and later. For more details, see [DAST API scanning engine](../dast_api). The new scanning engine supports REST, SOAP, GraphQL, and generic APIs using forms, XML, and JSON. Testing can be performed using OpenAPI, Postman Collections, and HTTP Archive (HAR) documents. - -The target API instance's base URL is provided by using the `DAST_API_TARGET_URL` variable or an `environment_url.txt` file. - -#### Specification format - -API scans support OpenAPI V2 and OpenAPI V3 specifications. You can define these specifications using `JSON` or `YAML`. - -#### Import API specification from a URL - -If your API specification is accessible at a URL, you can pass that URL in directly as the target. -The specification does not have to be hosted on the same host as the API being tested. - -```yaml -include: - - template: DAST-API.gitlab-ci.yml - -variables: - DAST_API_SPECIFICATION: http://my.api/api-specification.yml -``` - -#### Import API specification from a file - -If your API specification file is in your repository, you can provide its filename as the target. - -```yaml -dast: - variables: - GIT_STRATEGY: fetch - DAST_API_SPECIFICATION: api-specification.yml -``` - -#### Full API scan - -API scans support full scanning, which can be enabled by using the `DAST_FULL_SCAN_ENABLED` -CI/CD variable. Domain validation is not supported for full API scans. - -#### Host override - -Specifications often define a host, which contains a domain name and a port. The -host referenced may be different than the host of the API's review instance. -This can cause incorrect URLs to be imported, or a scan on an incorrect host. -Use the `DAST_API_HOST_OVERRIDE` CI/CD variable to override these values. - -WARNING: -When using the API host override feature, you cannot use the `$DAST_WEBSITE` variable to override the hostname. -A host override is _only_ supported when importing the API specification from a URL. Attempts to override the -host throw an error when the API specification is imported from a file. This is due to a limitation in the -ZAP OpenAPI extension. - -For example, with a OpenAPI V3 specification containing: - -```yaml -servers: - - url: https://api.host.com -``` - -If the test version of the API is running at `https://api-test.host.com`, then -the following DAST configuration can be used: - -```yaml -include: - - template: DAST-API.gitlab-ci.yml - -variables: - DAST_API_SPECIFICATION: http://api-test.host.com/api-specification.yml - DAST_API_HOST_OVERRIDE: api-test.host.com -``` - -#### Authentication using headers - -Tokens in request headers are often used as a way to authenticate API requests. -You can achieve this by using the `DAST_REQUEST_HEADERS` CI/CD variable. -Headers are applied to every request DAST makes. - -```yaml -include: - - template: DAST-API.gitlab-ci.yml - -variables: - DAST_API_SPECIFICATION: http://api-test.api.com/api-specification.yml - DAST_REQUEST_HEADERS: "Authorization: Bearer my.token" -``` +- The [DAST API analyzer](../dast_api/index.md) is used for scanning web APIs. Web API technologies such as GraphQL, REST, and SOAP are supported. ### URL scan @@ -336,6 +254,10 @@ When using `DAST_PATHS` and `DAST_PATHS_FILE`, note the following: To perform a [full scan](#full-scan) on the listed paths, use the `DAST_FULL_SCAN_ENABLED` CI/CD variable. +## Authentication + +The proxy-based analyzer uses the browser-based analyzer to authenticate a user prior to a scan. See [Authentication](authentication.md) for configuration instructions. + ## Customize DAST settings You can customize the behavior of DAST using both CI/CD variables and command-line options. Use of CI/CD @@ -427,61 +349,49 @@ To enable Mutual TLS: #### Available CI/CD variables These CI/CD variables are specific to DAST. They can be used to customize the behavior of DAST to your requirements. +For authentication CI/CD variables, see [Authentication](authentication.md). WARNING: All customization of GitLab security scanning tools should be tested in a merge request before merging these changes to the default branch. Failure to do so can give unexpected results, including a large number of false positives. -| CI/CD variable | Type | Description | -|:-------------------------------------------------|:--------------|:------------------------------| -| `DAST_ADVERTISE_SCAN` | boolean | Set to `true` to add a `Via` header to every request sent, advertising that the request was sent as part of a GitLab DAST scan. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/334947) in GitLab 14.1. | -| `DAST_AGGREGATE_VULNERABILITIES` | boolean | Vulnerability aggregation is set to `true` by default. To disable this feature and see each vulnerability individually set to `false`. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/254043) in GitLab 14.0. | -| `DAST_API_HOST_OVERRIDE` <sup>1</sup> | string | Used to override domains defined in API specification files. Only supported when importing the API specification from a URL. Example: `example.com:8080`. | -| `DAST_API_SPECIFICATION` <sup>1</sup> | URL or string | The API specification to import. The specification can be hosted at a URL, or the name of a file present in the `/zap/wrk` directory. The variable `DAST_WEBSITE` must be specified if this is omitted. | -| `DAST_AUTH_REPORT` <sup>2</sup> | boolean | Used in combination with exporting the `gl-dast-debug-auth-report.html` artifact to aid in debugging authentication issues. | -| `DAST_AUTH_EXCLUDE_URLS` <sup>2</sup> | URLs | **{warning}** **[Removed](https://gitlab.com/gitlab-org/gitlab/-/issues/289959)** in GitLab 14.0. Replaced by `DAST_EXCLUDE_URLS`. The URLs to skip during the authenticated scan; comma-separated. Regular expression syntax can be used to match multiple URLs. For example, `.*` matches an arbitrary character sequence. Not supported for API scans. | -| `DAST_AUTH_URL` <sup>1,2</sup> | URL | The URL of the page containing the sign-in HTML form on the target website. `DAST_USERNAME` and `DAST_PASSWORD` are submitted with the login form to create an authenticated scan. Not supported for API scans. Example: `https://login.example.com`. | -| `DAST_AUTH_VERIFICATION_LOGIN_FORM` <sup>2</sup> | boolean | Verifies successful authentication by checking for the lack of a login form once the login form has been submitted. | -| `DAST_AUTH_VERIFICATION_SELECTOR` <sup>2</sup> | selector | Verifies successful authentication by checking for presence of a selector once the login form has been submitted. Example: `css:.user-photo`. | -| `DAST_AUTH_VERIFICATION_URL` <sup>1,2</sup> | URL | A URL only accessible to logged in users that DAST can use to confirm successful authentication. If provided, DAST exits if it cannot access the URL. Example: `"http://example.com/loggedin_page"`. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/207335) in GitLab 13.8. | -| `DAST_AUTO_UPDATE_ADDONS` | boolean | ZAP add-ons are pinned to specific versions in the DAST Docker image. Set to `true` to download the latest versions when the scan starts. Default: `false`. | -| `DAST_BROWSER_PATH_TO_LOGIN_FORM` <sup>1,2</sup> | selector | Comma-separated list of selectors that are selected prior to attempting to enter `DAST_USERNAME` and `DAST_PASSWORD` into the login form. Example: `"css:.navigation-menu,css:.login-menu-item"`. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/326633) in GitLab 14.1. | -| `DAST_DEBUG` <sup>1</sup> | boolean | Enable debug message output. Default: `false`. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/12652) in GitLab 13.1. | -| `DAST_EXCLUDE_RULES` | string | Set to a comma-separated list of Vulnerability Rule IDs to exclude them from running during the scan. Rule IDs are numbers and can be found from the DAST log or on the [ZAP project](https://www.zaproxy.org/docs/alerts/). For example, `HTTP Parameter Override` has a rule ID of `10026`. Cannot be used when `DAST_ONLY_INCLUDE_RULES` is set. **Note:** In earlier versions of GitLab the excluded rules were executed but vulnerabilities they generated were suppressed. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/118641) in GitLab 12.10. | -| `DAST_EXCLUDE_URLS` <sup>1,2</sup> | URLs | The URLs to skip during the authenticated scan; comma-separated. Regular expression syntax can be used to match multiple URLs. For example, `.*` matches an arbitrary character sequence. Not supported for API scans. Example, `http://example.com/sign-out`. | -| `DAST_FIRST_SUBMIT_FIELD` <sup>2</sup> | string | The `id` or `name` of the element that when selected submits the username form of a multi-page login process. For example, `css:button[type='user-submit']`. [Introduced](https://gitlab.com/gitlab-org/gitlab-ee/issues/9894) in GitLab 12.4. | -| `DAST_FULL_SCAN_DOMAIN_VALIDATION_REQUIRED` | boolean | **{warning}** **[Removed](https://gitlab.com/gitlab-org/gitlab/-/issues/293595)** in GitLab 14.0. Set to `true` to require domain validation when running DAST full scans. Not supported for API scans. Default: `false` | -| `DAST_FULL_SCAN_ENABLED` <sup>1</sup> | boolean | Set to `true` to run a [ZAP Full Scan](https://github.com/zaproxy/zaproxy/wiki/ZAP-Full-Scan) instead of a [ZAP Baseline Scan](https://github.com/zaproxy/zaproxy/wiki/ZAP-Baseline-Scan). Default: `false` | -| `DAST_HTML_REPORT` | string | The filename of the HTML report written at the end of a scan. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/12652) in GitLab 13.1. | -| `DAST_INCLUDE_ALPHA_VULNERABILITIES` | boolean | Set to `true` to include alpha passive and active scan rules. Default: `false`. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/12652) in GitLab 13.1. | -| `DAST_MARKDOWN_REPORT` | string | The filename of the Markdown report written at the end of a scan. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/12652) in GitLab 13.1. | -| `DAST_MASK_HTTP_HEADERS` | string | Comma-separated list of request and response headers to be masked (GitLab 13.1). Must contain **all** headers to be masked. Refer to [list of headers that are masked by default](#hide-sensitive-information). | -| `DAST_MAX_URLS_PER_VULNERABILITY` | number | The maximum number of URLs reported for a single vulnerability. `DAST_MAX_URLS_PER_VULNERABILITY` is set to `50` by default. To list all the URLs set to `0`. [Introduced](https://gitlab.com/gitlab-org/security-products/dast/-/merge_requests/433) in GitLab 13.12. | -| `DAST_ONLY_INCLUDE_RULES` | string | Set to a comma-separated list of Vulnerability Rule IDs to configure the scan to run only them. Rule IDs are numbers and can be found from the DAST log or on the [ZAP project](https://www.zaproxy.org/docs/alerts/). Cannot be used when `DAST_EXCLUDE_RULES` is set. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/250651) in GitLab 13.12. | -| `DAST_PASSWORD` <sup>1,2</sup> | string | The password to authenticate to in the website. Example: `P@55w0rd!` | -| `DAST_PASSWORD_FIELD` <sup>1,2</sup> | string | The selector of password field at the sign-in HTML form. Example: `id:password` | -| `DAST_PATHS` | string | Set to a comma-separated list of URLs for DAST to scan. For example, `/page1.html,/category1/page3.html,/page2.html`. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/214120) in GitLab 13.4. | -| `DAST_PATHS_FILE` | string | The file path containing the paths within `DAST_WEBSITE` to scan. The file must be plain text with one path per line. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/258825) in GitLab 13.6. | -| `DAST_PKCS12_CERTIFICATE_BASE64` | string | The PKCS12 certificate used for sites that require Mutual TLS. Must be encoded as base64 text. | -| `DAST_PKCS12_PASSWORD` | string | The password of the certificate used in `DAST_PKCS12_CERTIFICATE_BASE64`. | -| `DAST_REQUEST_HEADERS` <sup>1</sup> | string | Set to a comma-separated list of request header names and values. Headers are added to every request made by DAST. For example, `Cache-control: no-cache,User-Agent: DAST/1.0` | -| `DAST_SKIP_TARGET_CHECK` | boolean | Set to `true` to prevent DAST from checking that the target is available before scanning. Default: `false`. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/229067) in GitLab 13.8. | -| `DAST_SPIDER_MINS` <sup>1</sup> | number | The maximum duration of the spider scan in minutes. Set to `0` for unlimited. Default: One minute, or unlimited when the scan is a full scan. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/12652) in GitLab 13.1. | -| `DAST_SPIDER_START_AT_HOST` | boolean | Set to `false` to prevent DAST from resetting the target to its host before scanning. When `true`, non-host targets `http://test.site/some_path` is reset to `http://test.site` before scan. Default: `true`. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/258805) in GitLab 13.6. | -| `DAST_SUBMIT_FIELD` <sup>2</sup> | string | The `id` or `name` of the element that when selected submits the login form or the password form of a multi-page login process. For example, `css:button[type='submit']`. [Introduced](https://gitlab.com/gitlab-org/gitlab-ee/issues/9894) in GitLab 12.4. | -| `DAST_TARGET_AVAILABILITY_TIMEOUT` <sup>1</sup> | number | Time limit in seconds to wait for target availability. | -| `DAST_USE_AJAX_SPIDER` <sup>1</sup> | boolean | Set to `true` to use the AJAX spider in addition to the traditional spider, useful for crawling sites that require JavaScript. Default: `false`. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/12652) in GitLab 13.1. | -| `DAST_USERNAME` <sup>1,2</sup> | string | The username to authenticate to in the website. Example: `admin` | -| `DAST_USERNAME_FIELD` <sup>1,2</sup> | string | The selector of username field at the sign-in HTML form. Example: `name:username` | -| `DAST_XML_REPORT` | string | The filename of the XML report written at the end of a scan. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/12652) in GitLab 13.1. | -| `DAST_WEBSITE` <sup>1</sup> | URL | The URL of the website to scan. The variable `DAST_API_SPECIFICATION` must be specified if this is omitted. | -| `DAST_ZAP_CLI_OPTIONS` | string | ZAP server command-line options. For example, `-Xmx3072m` would set the Java maximum memory allocation pool size. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/12652) in GitLab 13.1. | -| `DAST_ZAP_LOG_CONFIGURATION` | string | Set to a semicolon-separated list of additional log4j properties for the ZAP Server. Example: `logger.httpsender.name=org.parosproxy.paros.network.HttpSender;logger.httpsender.level=debug;logger.sitemap.name=org.parosproxy.paros.model.SiteMap;logger.sitemap.level=debug;` | -| `SECURE_ANALYZERS_PREFIX` | URL | Set the Docker registry base address from which to download the analyzer. | +| CI/CD variable | Type | Description | +|:------------------------------------------------|:--------------|:------------------------------| +| `DAST_ADVERTISE_SCAN` | boolean | Set to `true` to add a `Via` header to every request sent, advertising that the request was sent as part of a GitLab DAST scan. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/334947) in GitLab 14.1. | +| `DAST_AGGREGATE_VULNERABILITIES` | boolean | Vulnerability aggregation is set to `true` by default. To disable this feature and see each vulnerability individually set to `false`. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/254043) in GitLab 14.0. | +| `DAST_API_HOST_OVERRIDE` <sup>1</sup> | string | **{warning}** **[Deprecated](https://gitlab.com/gitlab-org/gitlab/-/issues/383467)** in GitLab 15.7. Replaced by [DAST API scan](../dast_api/index.md#available-cicd-variables). Used to override domains defined in API specification files. Only supported when importing the API specification from a URL. Example: `example.com:8080`. | +| `DAST_API_SPECIFICATION` <sup>1</sup> | URL or string | **{warning}** **[Deprecated](https://gitlab.com/gitlab-org/gitlab/-/issues/383467)** in GitLab 15.7. Replaced by [DAST API scan](../dast_api/index.md#available-cicd-variables). The API specification to import. The specification can be hosted at a URL, or the name of a file present in the `/zap/wrk` directory. The variable `DAST_WEBSITE` must be specified if this is omitted. | +| `DAST_AUTH_EXCLUDE_URLS` | URLs | **{warning}** **[Removed](https://gitlab.com/gitlab-org/gitlab/-/issues/289959)** in GitLab 14.0. Replaced by `DAST_EXCLUDE_URLS`. The URLs to skip during the authenticated scan; comma-separated. Regular expression syntax can be used to match multiple URLs. For example, `.*` matches an arbitrary character sequence. | +| `DAST_AUTO_UPDATE_ADDONS` | boolean | ZAP add-ons are pinned to specific versions in the DAST Docker image. Set to `true` to download the latest versions when the scan starts. Default: `false`. | +| `DAST_DEBUG` <sup>1</sup> | boolean | Enable debug message output. Default: `false`. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/12652) in GitLab 13.1. | +| `DAST_EXCLUDE_RULES` | string | Set to a comma-separated list of Vulnerability Rule IDs to exclude them from running during the scan. Rule IDs are numbers and can be found from the DAST log or on the [ZAP project](https://www.zaproxy.org/docs/alerts/). For example, `HTTP Parameter Override` has a rule ID of `10026`. Cannot be used when `DAST_ONLY_INCLUDE_RULES` is set. **Note:** In earlier versions of GitLab the excluded rules were executed but vulnerabilities they generated were suppressed. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/118641) in GitLab 12.10. | +| `DAST_EXCLUDE_URLS` <sup>1</sup> | URLs | The URLs to skip during the authenticated scan; comma-separated. Regular expression syntax can be used to match multiple URLs. For example, `.*` matches an arbitrary character sequence. Example, `http://example.com/sign-out`. | +| `DAST_FULL_SCAN_DOMAIN_VALIDATION_REQUIRED` | boolean | **{warning}** **[Removed](https://gitlab.com/gitlab-org/gitlab/-/issues/293595)** in GitLab 14.0. Set to `true` to require domain validation when running DAST full scans. Default: `false` | +| `DAST_FULL_SCAN_ENABLED` <sup>1</sup> | boolean | Set to `true` to run a [ZAP Full Scan](https://github.com/zaproxy/zaproxy/wiki/ZAP-Full-Scan) instead of a [ZAP Baseline Scan](https://github.com/zaproxy/zaproxy/wiki/ZAP-Baseline-Scan). Default: `false` | +| `DAST_HTML_REPORT` | string | **{warning}** **[Deprecated](https://gitlab.com/gitlab-org/gitlab/-/issues/384340)** in GitLab 15.7. The filename of the HTML report written at the end of a scan. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/12652) in GitLab 13.1. | +| `DAST_INCLUDE_ALPHA_VULNERABILITIES` | boolean | Set to `true` to include alpha passive and active scan rules. Default: `false`. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/12652) in GitLab 13.1. | +| `DAST_MARKDOWN_REPORT` | string | **{warning}** **[Deprecated](https://gitlab.com/gitlab-org/gitlab/-/issues/384340)** in GitLab 15.7. The filename of the Markdown report written at the end of a scan. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/12652) in GitLab 13.1. | +| `DAST_MASK_HTTP_HEADERS` | string | Comma-separated list of request and response headers to be masked (GitLab 13.1). Must contain **all** headers to be masked. Refer to [list of headers that are masked by default](#hide-sensitive-information). | +| `DAST_MAX_URLS_PER_VULNERABILITY` | number | The maximum number of URLs reported for a single vulnerability. `DAST_MAX_URLS_PER_VULNERABILITY` is set to `50` by default. To list all the URLs set to `0`. [Introduced](https://gitlab.com/gitlab-org/security-products/dast/-/merge_requests/433) in GitLab 13.12. | +| `DAST_ONLY_INCLUDE_RULES` | string | Set to a comma-separated list of Vulnerability Rule IDs to configure the scan to run only them. Rule IDs are numbers and can be found from the DAST log or on the [ZAP project](https://www.zaproxy.org/docs/alerts/). Cannot be used when `DAST_EXCLUDE_RULES` is set. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/250651) in GitLab 13.12. | +| `DAST_PATHS` | string | Set to a comma-separated list of URLs for DAST to scan. For example, `/page1.html,/category1/page3.html,/page2.html`. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/214120) in GitLab 13.4. | +| `DAST_PATHS_FILE` | string | The file path containing the paths within `DAST_WEBSITE` to scan. The file must be plain text with one path per line. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/258825) in GitLab 13.6. | +| `DAST_PKCS12_CERTIFICATE_BASE64` | string | The PKCS12 certificate used for sites that require Mutual TLS. Must be encoded as base64 text. | +| `DAST_PKCS12_PASSWORD` | string | The password of the certificate used in `DAST_PKCS12_CERTIFICATE_BASE64`. | +| `DAST_REQUEST_HEADERS` <sup>1</sup> | string | Set to a comma-separated list of request header names and values. Headers are added to every request made by DAST. For example, `Cache-control: no-cache,User-Agent: DAST/1.0` | +| `DAST_SKIP_TARGET_CHECK` | boolean | Set to `true` to prevent DAST from checking that the target is available before scanning. Default: `false`. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/229067) in GitLab 13.8. | +| `DAST_SPIDER_MINS` <sup>1</sup> | number | The maximum duration of the spider scan in minutes. Set to `0` for unlimited. Default: One minute, or unlimited when the scan is a full scan. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/12652) in GitLab 13.1. | +| `DAST_SPIDER_START_AT_HOST` | boolean | Set to `false` to prevent DAST from resetting the target to its host before scanning. When `true`, non-host targets `http://test.site/some_path` is reset to `http://test.site` before scan. Default: `true`. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/258805) in GitLab 13.6. | +| `DAST_TARGET_AVAILABILITY_TIMEOUT` <sup>1</sup> | number | Time limit in seconds to wait for target availability. | +| `DAST_USE_AJAX_SPIDER` <sup>1</sup> | boolean | Set to `true` to use the AJAX spider in addition to the traditional spider, useful for crawling sites that require JavaScript. Default: `false`. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/12652) in GitLab 13.1. | +| `DAST_XML_REPORT` | string | **{warning}** **[Deprecated](https://gitlab.com/gitlab-org/gitlab/-/issues/384340)** in GitLab 15.7. The filename of the XML report written at the end of a scan. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/12652) in GitLab 13.1. | +| `DAST_WEBSITE` <sup>1</sup> | URL | The URL of the website to scan. | +| `DAST_ZAP_CLI_OPTIONS` | string | **{warning}** **[Deprecated](https://gitlab.com/gitlab-org/gitlab/-/issues/383467)** in GitLab 15.7. ZAP server command-line options. For example, `-Xmx3072m` would set the Java maximum memory allocation pool size. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/12652) in GitLab 13.1. | +| `DAST_ZAP_LOG_CONFIGURATION` | string | **{warning}** **[Deprecated](https://gitlab.com/gitlab-org/gitlab/-/issues/383467)** in GitLab 15.7. Set to a semicolon-separated list of additional log4j properties for the ZAP Server. Example: `logger.httpsender.name=org.parosproxy.paros.network.HttpSender;logger.httpsender.level=debug;logger.sitemap.name=org.parosproxy.paros.model.SiteMap;logger.sitemap.level=debug;` | +| `SECURE_ANALYZERS_PREFIX` | URL | Set the Docker registry base address from which to download the analyzer. | 1. Available to an on-demand DAST scan. -1. Used for authentication. ### Customize DAST using command-line options @@ -528,242 +438,6 @@ variables: DAST_ZAP_CLI_OPTIONS: "-config replacer.full_list(0).description=auth -config replacer.full_list(0).enabled=true -config replacer.full_list(0).matchtype=REQ_HEADER -config replacer.full_list(0).matchstr=Authorization -config replacer.full_list(0).regex=false -config replacer.full_list(0).replacement=TOKEN" ``` -## Authentication - -NOTE: -We highly recommend you configure the scanner to authenticate to the application. If you don't, it cannot check most of the application for security risks, as most -of your application is likely not accessible without authentication. We also recommend -you periodically confirm the scanner's authentication is still working, as this tends to break over -time due to authentication changes to the application. - -Create masked CI/CD variables to pass the credentials that DAST uses. -To create masked variables for the username and password, see [Create a custom variable in the UI](../../../ci/variables/index.md#custom-cicd-variables). -The key of the username variable must be `DAST_USERNAME`, -and the key of the password variable must be `DAST_PASSWORD`. - -After DAST has authenticated with the application, all cookies are collected from the web browser. -For each cookie a matching session token is created for use by ZAP. This ensures ZAP is recognized -by the application as correctly authenticated. - -Authentication supports single form logins, multi-step login forms, and authenticating to URLs outside of the configured target URL. - -WARNING: -**Never** run an authenticated scan against a production server. When an authenticated -scan is run, it may perform *any* function that the authenticated user can. This -includes actions like modifying and deleting data, submitting forms, and following links. -Only run an authenticated scan against a test server. - -### SSO - -DAST can authenticate to websites making use of SSO, with the following restrictions: - -- DAST cannot bypass a CAPTCHA if the authentication flow includes one. -- DAST cannot handle multi-factor authentication like one-time passwords (OTP) by using SMS or authenticator apps. -- DAST must get a cookie, or a local or session storage, with a sufficiently random value. - -The [authentication debug output](#configure-the-authentication-debug-output) can be helpful for troubleshooting SSO authentication -with DAST. - -### Log in using automatic detection of the login form - -By providing a `DAST_USERNAME`, `DAST_PASSWORD`, and `DAST_AUTH_URL`, DAST attempts to authenticate to the -target application by locating the login form based on a determination about whether or not the form contains username or password fields. - -Automatic detection is "best-effort", and depending on the application being scanned may provide either a resilient login experience or one that fails to authenticate the user. - -Login process: - -1. The `DAST_AUTH_URL` is loaded into the browser, and any forms on the page are located. - 1. If a form contains a username and password field, `DAST_USERNAME` and `DAST_PASSWORD` is inputted into the respective fields, the form submit button is selected and the user is logged in. - 1. If a form contains only a username field, it is assumed that the login form is multi-step. - 1. The `DAST_USERNAME` is inputted into the username field and the form submit button is selected. - 1. The subsequent pages loads where it is expected that a form exists and contains a password field. If found, `DAST_PASSWORD` is inputted, form submit button is selected and the user is logged in. - -### Log in using explicit selection of the login form - -By providing a `DAST_USERNAME_FIELD`, `DAST_PASSWORD_FIELD`, and `DAST_SUBMIT_FIELD`, in addition to the fields required for automatic login, -DAST attempts to authenticate to the target application by locating the login form based on the selectors provided. -Most applications benefit from this approach to authentication. - -Login process: - -1. The `DAST_AUTH_URL` is loaded into the browser, and any forms on the page are located. - 1. If the `DAST_FIRST_SUBMIT_FIELD` is not defined, then `DAST_USERNAME` is inputted into `DAST_USERNAME_FIELD`, `DAST_PASSWORD` is inputted into `DAST_PASSWORD_FIELD`, `DAST_SUBMIT_FIELD` is selected and the user is logged in. - 1. If the `DAST_FIRST_SUBMIT_FIELD` is defined, then it is assumed that the login form is multi-step. - 1. The `DAST_USERNAME` is inputted into the `DAST_USERNAME_FIELD` field and the `DAST_FIRST_SUBMIT_FIELD` is selected. - 1. The subsequent pages loads where the `DAST_PASSWORD` is inputted into the `DAST_PASSWORD_FIELD` field, the `DAST_SUBMIT_FIELD` is selected and the user is logged in. - -### Verifying successful login - -Once the login form has been submitted, DAST determines if the login was successful. Unsuccessful attempts at authentication cause the scan to halt. - -Following the submission of the login form, authentication is determined to be unsuccessful when: - -- A `400` or `500` series HTTP response status code is returned. -- A new cookie/browser storage value determined to be sufficiently random has not been set. - -In addition to these checks, the user can configure their own verification checks. -Each of the following checks can be used in conjunction with one another, if none are configured by default the presence of a login form is checked. - -#### Verifying based on the URL - -When `DAST_AUTH_VERIFICATION_URL` is configured, the URL displayed in the browser tab post login form submission is directly compared to the URL in the CI/CD variable. -If these are not exactly the same, authentication is deemed to be unsuccessful. - -For example: - -```yaml -include: - - template: DAST.gitlab-ci.yml - -dast: - variables: - DAST_WEBSITE: "https://example.com" - DAST_BROWSER_SCAN: "true" # use the browser-based GitLab DAST crawler - ... - DAST_AUTH_VERIFICATION_URL: "https://example.com/user/welcome" -``` - -#### Verify based on presence of an element - -When `DAST_AUTH_VERIFICATION_SELECTOR` is configured, the page displayed in the browser tab is searched for an element described by the selector in the CI/CD variable. -If no element is found, authentication is deemed to be unsuccessful. - -For example: - -```yaml -include: - - template: DAST.gitlab-ci.yml - -dast: - variables: - DAST_WEBSITE: "https://example.com" - DAST_BROWSER_SCAN: "true" # use the browser-based GitLab DAST crawler - ... - DAST_AUTH_VERIFICATION_SELECTOR: "css:.welcome-user" -``` - -#### Verify based on presence of a login form - -When `DAST_AUTH_VERIFICATION_LOGIN_FORM` is configured, the page displayed in the browser tab is searched for a form that is detected to be a login form. -If any such form is found, authentication is deemed to be unsuccessful. - -For example: - -```yaml -include: - - template: DAST.gitlab-ci.yml - -dast: - variables: - DAST_WEBSITE: "https://example.com" - DAST_BROWSER_SCAN: "true" # use the browser-based GitLab DAST crawler - ... - DAST_AUTH_VERIFICATION_LOGIN_FORM: "true" -``` - -### View the login form - -Many web applications show the user the login form in a pop-up (modal) window. -For these applications, navigating to the form requires both: - -- A starting URL. -- A list of elements to select to display the modal window. - -When `DAST_BROWSER_PATH_TO_LOGIN_FORM` is present, like in this example: - -```yaml -include: - - template: DAST.gitlab-ci.yml - -dast: - variables: - DAST_WEBSITE: "https://my.site.com" - DAST_BROWSER_SCAN: "true" # use the browser-based GitLab DAST crawler - ... - DAST_AUTH_URL: "https://my.site.com/admin" - DAST_BROWSER_PATH_TO_LOGIN_FORM: "css:.navigation-menu,css:.login-menu-item" -``` - -DAST performs these actions: - -1. Load the `DAST_AUTH_URL` page, such as `https://my.site.com/admin`. -1. After the page loads, DAST selects elements found by the selectors described - in `DAST_BROWSER_PATH_TO_LOGIN_FORM`. This example opens the navigation menu - and selects the login menu, to display the login modal window. -1. To continue the authentication process, DAST fills in the username and password - on the login form. - -### Configure the authentication debug output - -It is often difficult to understand the cause of an authentication failure when running DAST in a CI/CD pipeline. -To assist users in debugging authentication issues, a debug report can be generated and saved as a job artifact. -This HTML report contains all steps made during the login process, along with HTTP requests and responses, the Document Object Model (DOM) and screenshots. - - - -An example configuration where the authentication debug report is exported may look like the following: - -```yaml -dast: - variables: - DAST_WEBSITE: "https://example.com" - DAST_BROWSER_SCAN: "true" # use the browser-based GitLab DAST crawler - ... - DAST_AUTH_REPORT: "true" - artifacts: - paths: [gl-dast-debug-auth-report.html] - when: always -``` - -### Selectors - -Selectors are used by CI/CD variables to specify the location of an element displayed on a page in a browser. -Selectors have the format `type`:`search string`. The crawler searches for the selector using the search string based on the type. - -| Selector type | Example | Description | -| ------------- | ---------------------------------- | ----------- | -| `css` | `css:.password-field` | Searches for a HTML element having the supplied CSS selector. Selectors should be as specific as possible for performance reasons. | -| `id` | `id:element` | Searches for an HTML element with the provided element ID. | -| `name` | `name:element` | Searches for an HTML element with the provided element name. | -| `xpath` | `xpath://input[@id="my-button"]/a` | Searches for a HTML element with the provided XPath. Note that XPath searches are expected to be less performant than other searches. | -| None provided | `a.click-me` | Defaults to searching using a CSS selector. | - -#### Find selectors with Google Chrome - -Chrome DevTools element selector tool is an effective way to find a selector. - -1. Open Chrome and navigate to the page where you would like to find a selector, for example, the login page for your site. -1. Open the `Elements` tab in Chrome DevTools with the keyboard shortcut `Command + Shift + c` in macOS or `Ctrl + Shift + c` in Windows. -1. Select the `Select an element in the page to select it` tool. -  -1. Select the field on your page that you would like to know the selector for. -1. Once the tool is active, highlight a field you wish to view the details of. -  -1. Once highlighted, you can see the element's details, including attributes that would make a good candidate for a selector. - -In this example, the `id="user_login"` appears to be a good candidate. You can use this as a selector as the DAST username field by setting -`DAST_USERNAME_FIELD: "id:user_login"`. - -#### Choose the right selector - -Judicious choice of selector leads to a scan that is resilient to the application changing. - -In order of preference, it is recommended to choose as selectors: - -- `id` fields. These are generally unique on a page, and rarely change. -- `name` fields. These are generally unique on a page, and rarely change. -- `class` values specific to the field, such as the selector `"css:.username"` for the `username` class on the username field. -- Presence of field specific data attributes, such as the selector, `"css:[data-username]"` when the `data-username` field has any value on the username field. -- Multiple `class` hierarchy values, such as the selector `"css:.login-form .username"` when there are multiple elements with class `username` but only one nested inside the element with the class `login-form`. - -When using selectors to locate specific fields we recommend you avoid searching on: - -- Any `id`, `name`, `attribute`, `class` or `value` that is dynamically generated. -- Generic class names, such as `column-10` and `dark-grey`. -- XPath searches as they are less performant than other selector searches. -- Unscoped searches, such as those beginning with `css:*` and `xpath://*`. - ### Bleeding-edge vulnerability definitions ZAP first creates rules in the `alpha` class. After a testing period with @@ -804,8 +478,6 @@ An on-demand DAST scan: - Is associated with your project's default branch. - Is saved on creation so it can be run later. -### On-demand scan modes - An on-demand scan can be run in active or passive mode: - _Passive mode_ is the default and runs a ZAP Baseline Scan. @@ -814,35 +486,20 @@ An on-demand scan can be run in active or passive mode: ### View on-demand DAST scans -To view running completed and scheduled on-demand DAST scans for a project, go to -**Security & Compliance > On-demand Scans** in the left sidebar. +To view on-demand scans, from your project's home page, go to **Security & Compliance > On-demand +scans** in the left sidebar. -- To view both running and completed scans, select **All**. -- To view running scans only, select **Running**. -- To view finished scans, select **Finished**. A finished scan is a scan that either succeeded, - failed, or was canceled. -- To view scheduled scans, select **Scheduled**. It shows on-demand scans that have a schedule - set up. Those are _not_ included in the **All** tab. -- To view saved on-demand scan profiles, select **Scan library**. - Those are _not_ included in the **All** tab. +On-demand scans are grouped by their status. The scan library contains all available on-demand +scans. -#### Cancel an on-demand scan +From the **On-demand scans** page you can: -To cancel a pending or running on-demand scan, select **Cancel** (**{cancel}**) in the -on-demand scans list. - -#### Retry an on-demand scan - -To retry a scan that failed or succeeded with warnings, select **Retry** (**{retry}**) in the -on-demand scans list. - -#### View an on-demand scan's results - -To view a finished scan's results, select **View results** in the on-demand scans list. - -#### Edit an on-demand scan - -To edit an on-demand scan's settings, select **Edit** (**{pencil}**) in the **Scheduled** tab. +- [Run](#run-an-on-demand-dast-scan) an on-demand scan. +- View the results of an on-demand scan. +- Cancel (**{cancel}**) a pending or running on-demand scan. +- Retry (**{retry}**) a scan that failed, or succeeded with warnings. +- [Edit](#edit-an-on-demand-scan) (**{pencil}**) an on-demand scan's settings. +- [Delete](#delete-an-on-demand-scan) an on-demand scan. ### Run an on-demand DAST scan @@ -925,44 +582,37 @@ To schedule a scan: 1. To run the on-demand scan immediately, select **Save and run scan**. To [run](#run-a-saved-on-demand-scan) it according to the schedule you set, select **Save scan**. -#### List saved on-demand scans - -To list saved on-demand scans: - -1. From your project's home page, go to **Security & Compliance > On-demand Scans**. -1. Select the **Scan library** tab. - -#### View details of an on-demand scan +### View details of an on-demand scan To view details of an on-demand scan: -1. From your project's home page, go to **Security & Compliance > On-demand Scans**. +1. From your project's home page, go to **Security & Compliance > On-demand scans**. 1. Select the **Scan library** tab. 1. In the saved scan's row select **More actions** (**{ellipsis_v}**), then select **Edit**. -#### Edit an on-demand scan +### Edit an on-demand scan To edit an on-demand scan: -1. From your project's home page, go to **Security & Compliance > On-demand Scans**. +1. From your project's home page, go to **Security & Compliance > On-demand scans**. 1. Select the **Scan library** tab. 1. In the saved scan's row select **More actions** (**{ellipsis_v}**), then select **Edit**. 1. Edit the form. 1. Select **Save scan**. -#### Delete an on-demand scan +### Delete an on-demand scan To delete an on-demand scan: -1. From your project's home page, go to **Security & Compliance > On-demand Scans**. +1. From your project's home page, go to **Security & Compliance > On-demand scans**. 1. Select the **Scan library** tab. 1. In the saved scan's row select **More actions** (**{ellipsis_v}**), then select **Delete**. -1. Select **Delete** to confirm the deletion. +1. On the confirmation dialog, select **Delete**. ## Site profile -> - Scan method [Enabled on GitLab.com and self-managed](https://gitlab.com/gitlab-org/gitlab/-/issues/345837) in GitLab 15.6. -> - File URL [Enabled on GitLab.com and self-managed](https://gitlab.com/gitlab-org/gitlab/-/issues/345837) in GitLab 15.6. +> - Site profile features, scan method and file URL, were [enabled on GitLab.com and self-managed](https://gitlab.com/gitlab-org/gitlab/-/issues/345837) in GitLab 15.6. +> - GraphQL endpoint path feature was [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/378692) in GitLab 15.7. A site profile defines the attributes and configuration details of the deployed application, website, or API to be scanned by DAST. A site profile can be referenced in `.gitlab-ci.yml` and @@ -984,10 +634,11 @@ A site profile contains: - **Password form field**: The name of password field at the sign-in HTML form. - **Submit form field**: The `id` or `name` of the element that when selected submits the sign-in HTML form. -- **Scan method**: A type of method to perform API testing. The supported methods are OpenAPI, Postman Collections, and HTTP Archive (HAR) documents. -- **File URL**: The URL of the OpenAPI, Postman Collection, or HTTP Archive file. +- **Scan method**: A type of method to perform API testing. The supported methods are OpenAPI, Postman Collections, HTTP Archive (HAR), or GraphQL. + - **GraphQL endpoint path**: The path to the GraphQL endpoint. This path is concatenated with the target URL to provide the URI for the scan to test. The GraphQL endpoint must support introspection queries. + - **File URL**: The URL of the OpenAPI, Postman Collection, or HTTP Archive file. -When an API site type is selected, a [host override](#host-override) is used to ensure the API being scanned is on the same host as the target. This is done to reduce the risk of running an active scan against the wrong API. +When an API site type is selected, a host override is used to ensure the API being scanned is on the same host as the target. This is done to reduce the risk of running an active scan against the wrong API. When configured, request headers and password fields are encrypted using [`aes-256-gcm`](https://en.wikipedia.org/wiki/Advanced_Encryption_Standard) before being stored in the database. This data can only be read and decrypted with a valid secrets file. @@ -1235,9 +886,7 @@ and DAST site profiles are included in the [audit log](../../../administration/a The DAST tool outputs a `gl-dast-report.json` report file containing details of the scan and its results. This file is included in the job's artifacts. JSON is the default format, but you can output the report in Markdown, HTML, and XML formats. To specify an alternative -format, use a [CI/CD variable](#available-cicd-variables). You can also use a CI/CD variable -to configure the job to output the `gl-dast-debug-auth-report.html` file which helps when debugging -authentication issues. +format, use a [CI/CD variable](#available-cicd-variables). For details of the report's schema, see the [schema for DAST reports](https://gitlab.com/gitlab-org/security-products/security-report-schemas/-/blob/master/dist/dast-report-format.json). Example reports can be found in the [DAST repository](https://gitlab.com/gitlab-org/security-products/dast/-/tree/main/test/end-to-end/expect). diff --git a/doc/user/application_security/dast/run_dast_offline.md b/doc/user/application_security/dast/run_dast_offline.md index 05c6b74fbcd..7cb4eff8e68 100644 --- a/doc/user/application_security/dast/run_dast_offline.md +++ b/doc/user/application_security/dast/run_dast_offline.md @@ -39,7 +39,7 @@ process by which external resources can be imported or temporarily accessed. These scanners are [periodically updated](../index.md#vulnerability-scanner-maintenance) with new definitions, and you may be able to make occasional updates on your own. -For details on saving and transporting Docker images as a file, see Docker's documentation on +For details on saving and transporting Docker images as a file, see the Docker documentation on [`docker save`](https://docs.docker.com/engine/reference/commandline/save/), [`docker load`](https://docs.docker.com/engine/reference/commandline/load/), [`docker export`](https://docs.docker.com/engine/reference/commandline/export/), and diff --git a/doc/user/application_security/dast_api/index.md b/doc/user/application_security/dast_api/index.md index d77be0f0ca9..63276eba871 100644 --- a/doc/user/application_security/dast_api/index.md +++ b/doc/user/application_security/dast_api/index.md @@ -217,8 +217,9 @@ DAST API supports testing GraphQL endpoints multiple ways: - Test using a Postman Collection containing GraphQL queries. This section documents how to test using a GraphQL schema. The GraphQL schema support in -DAST API is able to query the schema from endpoints that support introspection. +DAST API is able to query the schema from endpoints that support [introspection](https://graphql.org/learn/introspection/). Introspection is enabled by default to allow tools like GraphiQL to work. +For details on how to enable introspection, see your GraphQL framework documentation. #### DAST API scanning with a GraphQL endpoint URL @@ -1046,6 +1047,8 @@ can be added, removed, and modified by creating a custom configuration. |[`DAST_API_EXCLUDE_URLS`](#exclude-urls) | Exclude API URL from testing. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/357195) in GitLab 14.10. | |[`DAST_API_EXCLUDE_PARAMETER_ENV`](#exclude-parameters) | JSON string containing excluded parameters. | |[`DAST_API_EXCLUDE_PARAMETER_FILE`](#exclude-parameters) | Path to a JSON file containing excluded parameters. | +|[`DAST_API_REQUEST_HEADERS`](#request-headers) | A comma-separated (`,`) list of headers to include on each scan request. Consider using `DAST_API_REQUEST_HEADERS_BASE64` when storing secret header values in a [masked variable](../../../ci/variables/index.md#mask-a-cicd-variable), which has character set restrictions. | +|[`DAST_API_REQUEST_HEADERS_BASE64`](#request-headers) | A comma-separated (`,`) list of headers to include on each scan request, Base64-encoded. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/378440) in GitLab 15.6. | |[`DAST_API_OPENAPI`](#openapi-specification) | OpenAPI specification file or URL. | |[`DAST_API_OPENAPI_RELAXED_VALIDATION`](#openapi-specification) | Relax document validation. Default is disabled. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/345950) in GitLab 14.7. | |[`DAST_API_OPENAPI_ALL_MEDIA_TYPES`](#openapi-specification) | Use all supported media types instead of one when generating requests. Causes test duration to be longer. Default is disabled. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/333304) in GitLab 14.10. | @@ -1488,6 +1491,61 @@ variables: In the previous sample, you could use the script `user-pre-scan-set-up.sh` to also install new runtimes or applications that later on you could use in our overrides command. +## Request Headers + +The request headers feature lets you specify fixed values for the headers during the scan session. For example, you can use the configuration variable `DAST_API_REQUEST_HEADERS` to set a fixed value in the `Cache-Control` header. If the headers you need to set include sensitive values like the `Authorization` header, use the [masked variable](../../../ci/variables/index.md#mask-a-cicd-variable) feature along with the [variable `DAST_API_REQUEST_HEADERS_BASE64`](#base64). + +Note that if the `Authorization` header or any other header needs to get updated while the scan is in progress, consider using the [overrides](#overrides) feature. + +The variable `DAST_API_REQUEST_HEADERS` lets you specify a comma-separated (`,`) list of headers. These headers are included on each request that the scanner performs. Each header entry in the list consists of a name followed by a colon (`:`) and then by its value. Whitespace before the key or value is ignored. For example, to declare a header name `Cache-Control` with the value `max-age=604800`, the header entry is `Cache-Control: max-age=604800`. To use two headers, `Cache-Control: max-age=604800` and `Age: 100`, set `DAST_API_REQUEST_HEADERS` variable to `Cache-Control: max-age=604800, Age: 100`. + +The order in which the different headers are provided into the variable `DAST_API_REQUEST_HEADERS` does not affect the result. Setting `DAST_API_REQUEST_HEADERS` to `Cache-Control: max-age=604800, Age: 100` produces the same result as setting it to `Age: 100, Cache-Control: max-age=604800`. + +### Base64 + +The `DAST_API_REQUEST_HEADERS_BASE64` variable accepts the same list of headers as `DAST_API_REQUEST_HEADERS`, with the only difference that the entire value of the variable must be Base64-encoded. For example, to set `DAST_API_REQUEST_HEADERS_BASE64` variable to `Authorization: QmVhcmVyIFRPS0VO, Cache-control: bm8tY2FjaGU=`, ensure you convert the list to its Base64 equivalent: `QXV0aG9yaXphdGlvbjogUW1WaGNtVnlJRlJQUzBWTywgQ2FjaGUtY29udHJvbDogYm04dFkyRmphR1U9`, and the Base64-encoded value must be used. This is useful when storing secret header values in a [masked variable](../../../ci/variables/index.md#mask-a-cicd-variable), which has character set restrictions. + +WARNING: +Base64 is used to support the [masked variable](../../../ci/variables/index.md#mask-a-cicd-variable) feature. Base64 encoding is not by itself a security measure, because sensitive values can be easily decoded. + +### Example: Adding a list of headers on each request using plain text + +In the following example of a `.gitlab-ci.yml`, `DAST_API_REQUEST_HEADERS` configuration variable is set to provide two header values as explained in [request headers](#request-headers). + +```yaml +stages: + - dast + +include: + - template: DAST-API.gitlab-ci.yml + +variables: + DAST_API_PROFILE: Quick + DAST_API_OPENAPI: test-api-specification.json + DAST_API_TARGET_URL: http://test-deployment/ + DAST_API_REQUEST_HEADERS: 'Cache-control: no-cache, Save-Data: on' +``` + +### Example: Using a masked CI/CD variable + +The following `.gitlab-ci.yml` sample assumes the [masked variable](../../../ci/variables/index.md#mask-a-cicd-variable) `SECRET_REQUEST_HEADERS_BASE64` is defined as a [group or instance level CI/CD variable defined in the UI](../../../ci/variables/index.md#add-a-cicd-variable-to-an-instance). The value of `SECRET_REQUEST_HEADERS_BASE64` is set to `WC1BQ01FLVNlY3JldDogc31jcnt0ISwgWC1BQ01FLVRva2VuOiA3MDVkMTZmNWUzZmI=`, which is the Base64-encoded text version of `X-ACME-Secret: s3cr3t!, X-ACME-Token: 705d16f5e3fb`. Then, it can be used as follows: + +```yaml +stages: + - dast + +include: + - template: DAST-API.gitlab-ci.yml + +variables: + DAST_API_PROFILE: Quick + DAST_API_OPENAPI: test-api-specification.json + DAST_API_TARGET_URL: http://test-deployment/ + DAST_API_REQUEST_HEADERS_BASE64: $SECRET_REQUEST_HEADERS_BASE64 +``` + +Consider using `DAST_API_REQUEST_HEADERS_BASE64` when storing secret header values in a [masked variable](../../../ci/variables/index.md#mask-a-cicd-variable), which has character set restrictions. + ## Exclude Paths > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/211892) in GitLab 14.0. @@ -1497,13 +1555,13 @@ When testing an API it can be useful to exclude certain paths. For example, you To verify the paths are excluded, review the `Tested Operations` and `Excluded Operations` portion of the job output. You should not see any excluded paths listed under `Tested Operations`. ```plaintext -2021-05-27 21:51:08 [INF] API Security: --[ Tested Operations ]------------------------- -2021-05-27 21:51:08 [INF] API Security: 201 POST http://target:7777/api/users CREATED -2021-05-27 21:51:08 [INF] API Security: ------------------------------------------------ -2021-05-27 21:51:08 [INF] API Security: --[ Excluded Operations ]----------------------- -2021-05-27 21:51:08 [INF] API Security: GET http://target:7777/api/messages -2021-05-27 21:51:08 [INF] API Security: POST http://target:7777/api/messages -2021-05-27 21:51:08 [INF] API Security: ------------------------------------------------ +2021-05-27 21:51:08 [INF] DAST API: --[ Tested Operations ]------------------------- +2021-05-27 21:51:08 [INF] DAST API: 201 POST http://target:7777/api/users CREATED +2021-05-27 21:51:08 [INF] DAST API: ------------------------------------------------ +2021-05-27 21:51:08 [INF] DAST API: --[ Excluded Operations ]----------------------- +2021-05-27 21:51:08 [INF] DAST API: GET http://target:7777/api/messages +2021-05-27 21:51:08 [INF] DAST API: POST http://target:7777/api/messages +2021-05-27 21:51:08 [INF] DAST API: ------------------------------------------------ ``` ### Examples @@ -1548,7 +1606,7 @@ variables: While testing an API you may might want to exclude a parameter (query string, header, or body element) from testing. This may be needed because a parameter always causes a failure, slows down testing, or for other reasons. To exclude parameters, you can set one of the following variables: `DAST_API_EXCLUDE_PARAMETER_ENV` or `DAST_API_EXCLUDE_PARAMETER_FILE`. -The `DAST_API_EXCLUDE_PARAMETER_ENV` allows providing a JSON string containing excluded parameters. This is a good option if the JSON is short and will not often change. Another option is the variable `DAST_API_EXCLUDE_PARAMETER_FILE`. This variable is set to a file path that can be checked into the repository, created by another job as an artifact, or generated at runtime with a pre script using `DAST_API_PRE_SCRIPT`. +The `DAST_API_EXCLUDE_PARAMETER_ENV` allows providing a JSON string containing excluded parameters. This is a good option if the JSON is short and will not often change. Another option is the variable `DAST_API_EXCLUDE_PARAMETER_FILE`. This variable is set to a file path that can be checked into the repository, created by another job as an artifact, or generated at runtime with a pre-script using `DAST_API_PRE_SCRIPT`. #### Exclude parameters using a JSON document @@ -1780,13 +1838,13 @@ As an alternative to excluding by paths, you can filter by any other component i In your job output you can check if any URLs matched any provided regular expression from `DAST_API_EXCLUDE_URLS`. Matching operations are listed in the **Excluded Operations** section. Operations listed in the **Excluded Operations** should not be listed in the **Tested Operations** section. For example the following portion of a job output: ```plaintext -2021-05-27 21:51:08 [INF] API Security: --[ Tested Operations ]------------------------- -2021-05-27 21:51:08 [INF] API Security: 201 POST http://target:7777/api/users CREATED -2021-05-27 21:51:08 [INF] API Security: ------------------------------------------------ -2021-05-27 21:51:08 [INF] API Security: --[ Excluded Operations ]----------------------- -2021-05-27 21:51:08 [INF] API Security: GET http://target:7777/api/messages -2021-05-27 21:51:08 [INF] API Security: POST http://target:7777/api/messages -2021-05-27 21:51:08 [INF] API Security: ------------------------------------------------ +2021-05-27 21:51:08 [INF] DAST API: --[ Tested Operations ]------------------------- +2021-05-27 21:51:08 [INF] DAST API: 201 POST http://target:7777/api/users CREATED +2021-05-27 21:51:08 [INF] DAST API: ------------------------------------------------ +2021-05-27 21:51:08 [INF] DAST API: --[ Excluded Operations ]----------------------- +2021-05-27 21:51:08 [INF] DAST API: GET http://target:7777/api/messages +2021-05-27 21:51:08 [INF] DAST API: POST http://target:7777/api/messages +2021-05-27 21:51:08 [INF] DAST API: ------------------------------------------------ ``` NOTE: @@ -2083,18 +2141,18 @@ The first step to resolving performance issues is to understand what is contribu The DAST API job output contains helpful information about how fast we are testing, how fast each operation being tested responds, and summary information. Let's take a look at some sample output to see how it can be used in tracking down performance issues: ```shell -API Security: Loaded 10 operations from: assets/har-large-response/large_responses.har -API Security: -API Security: Testing operation [1/10]: 'GET http://target:7777/api/large_response_json'. -API Security: - Parameters: (Headers: 4, Query: 0, Body: 0) -API Security: - Request body size: 0 Bytes (0 bytes) -API Security: -API Security: Finished testing operation 'GET http://target:7777/api/large_response_json'. -API Security: - Excluded Parameters: (Headers: 0, Query: 0, Body: 0) -API Security: - Performed 767 requests -API Security: - Average response body size: 130 MB -API Security: - Average call time: 2 seconds and 82.69 milliseconds (2.082693 seconds) -API Security: - Time to complete: 14 minutes, 8 seconds and 788.36 milliseconds (848.788358 seconds) +DAST API: Loaded 10 operations from: assets/har-large-response/large_responses.har +DAST API: +DAST API: Testing operation [1/10]: 'GET http://target:7777/api/large_response_json'. +DAST API: - Parameters: (Headers: 4, Query: 0, Body: 0) +DAST API: - Request body size: 0 Bytes (0 bytes) +DAST API: +DAST API: Finished testing operation 'GET http://target:7777/api/large_response_json'. +DAST API: - Excluded Parameters: (Headers: 0, Query: 0, Body: 0) +DAST API: - Performed 767 requests +DAST API: - Average response body size: 130 MB +DAST API: - Average call time: 2 seconds and 82.69 milliseconds (2.082693 seconds) +DAST API: - Time to complete: 14 minutes, 8 seconds and 788.36 milliseconds (848.788358 seconds) ``` This job console output snippet starts by telling us how many operations were found (10), followed by notifications that testing has started on a specific operation and a summary of the operation has been completed. The summary is the most interesting part of this log output. In the summary, we can see that it took DAST API 767 requests to fully test this operation and its related fields. We can also see that the average response time was 2 seconds and the time to complete was 14 minutes for this one operation. @@ -2281,7 +2339,7 @@ See the following documentation sections for assistance: See [Performance Tuning and Testing Speed](#performance-tuning-and-testing-speed) -### Error waiting for API Security 'http://127.0.0.1:5000' to become available +### Error waiting for DAST API 'http://127.0.0.1:5000' to become available A bug exists in versions of the DAST API analyzer prior to v1.6.196 that can cause a background process to fail under certain conditions. The solution is to update to a newer version of the DAST API analyzer. @@ -2294,6 +2352,11 @@ If the issue is occurring with versions v1.6.196 or greater, contact Support and 1. The `gl-api-security-scanner.log` file available as a job artifact. In the right-hand panel of the job details page, select the **Browse** button. 1. The `dast_api` job definition from your `.gitlab-ci.yml` file. +**Error message** + +- In [GitLab 15.6 and later](https://gitlab.com/gitlab-org/gitlab/-/issues/376078), `Error waiting for DAST API 'http://127.0.0.1:5000' to become available` +- In GitLab 15.5 and earlier, `Error waiting for API Security 'http://127.0.0.1:5000' to become available`. + ### `Failed to start scanner session (version header not found)` The DAST API engine outputs an error message when it cannot establish a connection with the scanner application component. The error message is shown in the job output window of the `dast_api` job. A common cause of this issue is changing the `DAST_API_API` variable from its default. @@ -2446,6 +2509,50 @@ DAST API uses the specified media types in the OpenAPI document to generate requ 1. Review supported media types in the [OpenAPI Specification](#openapi-specification) section. 1. Edit your OpenAPI document, allowing at least a given operation to accept any of the supported media types. Alternatively, a supported media type could be set in the OpenAPI document level and get applied to all operations. This step may require changes in your application to ensure the supported media type is accepted by the application. +### ``Error, error occurred trying to download `<URL>`: There was an error when retrieving content from Uri:' <URL>'. Error:The SSL connection could not be established, see inner exception.`` + +DAST API is compatible with a broad range of TLS configurations, including outdated protocols and ciphers. +Despite broad support, you might encounter connection errors. This error occurs because DAST API could not establish a secure connection with the server at the given URL. + +To resolve the issue: + +If the host in the error message supports non-TLS connections, change `https://` to `http://` in your configuration. +For example, if an error occurs with the following configuration: + +```yaml +stages: + - dast + +include: + - template: DAST-API.gitlab-ci.yml + +variables: + DAST_API_TARGET_URL: https://test-deployment/ + DAST_API_OPENAPI: https://specs/openapi.json +``` + +Change the prefix of `DAST_API_OPENAPI` from `https://` to `http://`: + +```yaml +stages: + - dast + +include: + - template: DAST-API.gitlab-ci.yml + +variables: + DAST_API_TARGET_URL: https://test-deployment/ + DAST_API_OPENAPI: http://specs/openapi.json +``` + +If you cannot use a non-TLS connection to access the URL, contact the Support team for help. + +You can expedite the investigation with the [testssl.sh tool](https://testssl.sh/). From a machine with a bash shell and connectivity to the affected server: + +1. Download the latest release `zip` or `tar.gz` file and extract from <https://github.com/drwetter/testssl.sh/releases>. +1. Run `./testssl.sh --log https://specs`. +1. Attach the log file to your support ticket. + ## Get support or request an improvement To get support for your particular problem, use the [getting help channels](https://about.gitlab.com/get-help/). diff --git a/doc/user/application_security/dependency_scanning/index.md b/doc/user/application_security/dependency_scanning/index.md index 4ed9ceedb4d..a4957c96db4 100644 --- a/doc/user/application_security/dependency_scanning/index.md +++ b/doc/user/application_security/dependency_scanning/index.md @@ -109,7 +109,7 @@ maximum of two directory levels from the repository's root. For example, the `gemnasium-dependency_scanning` job is enabled if a repository contains either `Gemfile`, `api/Gemfile`, or `api/client/Gemfile`, but not if the only supported dependency file is `api/v1/client/Gemfile`. -For Java and Python, when a supported depedency file is detected, Dependency Scanning attempts to build the project and execute some Java or Python commands to get the list of dependencies. For all other projects, the lock file is parsed to obtain the list of dependencies without needing to build the project first. +For Java and Python, when a supported dependency file is detected, Dependency Scanning attempts to build the project and execute some Java or Python commands to get the list of dependencies. For all other projects, the lock file is parsed to obtain the list of dependencies without needing to build the project first. When a supported dependency file is detected, all dependencies, including transitive dependencies are analyzed. There is no limit to the depth of nested or transitive dependencies that are analyzed. @@ -309,7 +309,7 @@ table.supported-languages ul { <li> <a id="notes-regarding-supported-languages-and-package-managers-3"></a> <p> - npm is only supported when <code>lockfileVersion = 1</code> or <code>lockfileVersion = 2</code>. Work to add support for <code>lockfileVersion = 3</code> is being tracked in issue <a href="https://gitlab.com/gitlab-org/gitlab/-/issues/365176">GitLab#365176</a>. + npm is supported for <code>lockfileVersion = 1</code>, <code>lockfileVersion = 2</code>, and <code>lockfileVersion = 3</code>. Support for <code>lockfileVersion = 3</code> was <a href="https://gitlab.com/gitlab-org/gitlab/-/issues/365176">introduced</a> in GitLab 15.7. </p> </li> <li> @@ -437,7 +437,7 @@ To support the following package managers, the GitLab analyzers proceed in two s <li> <a id="exported-dependency-information-notes-4"></a> <p> - Because of the implementation of <code>go build</code>, the Go build process requires network access, a pre-loaded modcache via <code>go mod download</code>, or vendored dependencies. For more information, + Because of the implementation of <code>go build</code>, the Go build process requires network access, a pre-loaded mod cache via <code>go mod download</code>, or vendored dependencies. For more information, refer to the Go documentation on <a href="https://pkg.go.dev/cmd/go#hdr-Compile_packages_and_dependencies">compiling packages and dependencies</a>. </p> </li> @@ -867,13 +867,8 @@ Here's an example dependency scanning report: ### CycloneDX Software Bill of Materials -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/350509) in GitLab 14.8 in [Beta](../../../policy/alpha-beta-support.md#beta-features). - -NOTE: -CycloneDX SBOMs are a [Beta](../../../policy/alpha-beta-support.md#beta-features) feature, -and the reports are subject to change during the beta period. Do not build integrations -that rely on the format of these SBOMs staying consistent, as the format might change -before the feature is made generally available. +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/350509) in GitLab 14.8 in [Beta](../../../policy/alpha-beta-support.md#beta-features). +> - Generally available in GitLab 15.7. In addition to the [JSON report file](#reports-json-format), the [Gemnasium](https://gitlab.com/gitlab-org/security-products/analyzers/gemnasium) Dependency Scanning tool outputs a [CycloneDX](https://cyclonedx.org/) Software Bill of Materials (SBOM) for @@ -996,7 +991,7 @@ process by which external resources can be imported or temporarily accessed. These scanners are [periodically updated](../index.md#vulnerability-scanner-maintenance) with new definitions, and you may be able to make occasional updates on your own. -For details on saving and transporting Docker images as a file, see Docker's documentation on +For details on saving and transporting Docker images as a file, see the Docker documentation on [`docker save`](https://docs.docker.com/engine/reference/commandline/save/), [`docker load`](https://docs.docker.com/engine/reference/commandline/load/), [`docker export`](https://docs.docker.com/engine/reference/commandline/export/), and [`docker import`](https://docs.docker.com/engine/reference/commandline/import/). @@ -1257,7 +1252,7 @@ file in the `CI_BUILDS_DIR` directory triggers the dependency scanning job. We recommend committing the lock files, which prevents this warning. -### I no longer get the latest Docker image after setting `DS_MAJOR_VERSION` or `DS_ANALYZER_IMAGE` +### You no longer get the latest Docker image after setting `DS_MAJOR_VERSION` or `DS_ANALYZER_IMAGE` If you have manually set `DS_MAJOR_VERSION` or `DS_ANALYZER_IMAGE` for specific reasons, and now must update your configuration to again get the latest patched versions of our diff --git a/doc/user/application_security/index.md b/doc/user/application_security/index.md index 5ddfa99fc81..6629c798cfa 100644 --- a/doc/user/application_security/index.md +++ b/doc/user/application_security/index.md @@ -240,9 +240,9 @@ reports are available to download. To download a report, select ### Ultimate -A merge request contains a security widget which displays a summary of the new results. New results are determined by comparing the current findings against existing findings in the target (default) branch (if there are prior findings). +A merge request contains a security widget which displays a summary of the new results. New results are determined by comparing the findings of the merge request against the findings of the most recent completed pipeline (`success`, `failed`, `canceled` or `skipped`) for the latest commit in the target branch. -We recommend you run a scan of the `default` branch before enabling feature branch scans for your developers. Otherwise, there is no base for comparison and all feature branches display the full scan results in the merge request security widget. +If security scans have not run for the most recent completed pipeline in the target branch there is no base for comparison. The vulnerabilities from the merge request findings will be listed as new in the merge request security widget. We recommend you run a scan of the `default` (target) branch before enabling feature branch scans for your developers. The merge request security widget displays only a subset of the vulnerabilities in the generated JSON artifact because it contains both new and existing findings. @@ -339,7 +339,7 @@ custom job: The above `.gitlab-ci.yml` causes a linting error: ```plaintext -Found errors in your .gitlab-ci.yml: +Unable to create pipeline - dependency_scanning job: chosen stage does not exist; available stages are .pre - unit-tests - .post @@ -590,7 +590,7 @@ like [`SAST.gitlab-ci.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/l the following error may occur, depending on your GitLab CI/CD configuration: ```plaintext -Found errors in your .gitlab-ci.yml: +Unable to create pipeline jobs:sast config key may not be used with `rules`: only/except ``` @@ -683,7 +683,7 @@ This can be used for offline setups or for anyone wishing to use [Auto DevOps](. Instructions are available in the [legacy template project](https://gitlab.com/gitlab-org/auto-devops-v12-10). -#### Vulnerabilities are found, but the job succeeds. How can I have a pipeline fail instead? +#### Vulnerabilities are found, but the job succeeds. How can you have a pipeline fail instead? In these circumstances, that the job succeeds is the default behavior. The job's status indicates success or failure of the analyzer itself. Analyzer results are displayed in the diff --git a/doc/user/application_security/offline_deployments/index.md b/doc/user/application_security/offline_deployments/index.md index 2db8e9522db..05e56560f95 100644 --- a/doc/user/application_security/offline_deployments/index.md +++ b/doc/user/application_security/offline_deployments/index.md @@ -117,7 +117,7 @@ This template should be used in a new, empty project, with a `.gitlab-ci.yml` fi ```yaml include: - - template: Secure-Binaries.gitlab-ci.yml + - template: Security/Secure-Binaries.gitlab-ci.yml ``` The pipeline downloads the Docker images needed for the Security Scanners and saves them as diff --git a/doc/user/application_security/policies/index.md b/doc/user/application_security/policies/index.md index f6d22ab28cd..a214d0d2cec 100644 --- a/doc/user/application_security/policies/index.md +++ b/doc/user/application_security/policies/index.md @@ -34,6 +34,10 @@ project and the security policy project, this is not recommended. Keeping the se project separate from the development project allows for complete separation of duties between security/compliance teams and development teams. +You should not link a security policy project to a development project and to the group +or sub-group the development project belongs to at the same time. Linking this way will result in +approval rules from the Scan Result Policy not being applied to merge requests in the development project. + All security policies are stored in the `.gitlab/security-policies/policy.yml` YAML file inside the linked security policy project. The format for this YAML is specific to the type of policy that is stored there. Examples and schema information are available for the following policy types: @@ -140,10 +144,10 @@ for more information on the product direction of security policies within GitLab ## Troubleshooting -### `Branch name does not follow the pattern 'update-policy-<timestamp>'` +### `Branch name 'update-policy-<timestamp>' does not follow the pattern '<branch_name_regex>'` When you create a new security policy or change an existing policy, a new branch is automatically created with the branch name following the pattern `update-policy-<timestamp>`. For example: `update-policy-1659094451`. -If you have group or instance push rules that do not allow branch name patterns that contain the text `update-policy-<timestamp>`, you will get an error that states `Branch name does not follow the pattern 'update-policy-<timestamp>'`. +If you have group or instance [push rules that do not allow branch name patterns](../../project/repository/push_rules.md#validate-branch-names) that contain the text `update-policy-<timestamp>`, you will get an error that states `Branch name 'update-policy-<timestamp>' does not follow the pattern '<branch_name_regex>'`. The workaround is to amend your group or instance push rules to allow branches following the pattern `update-policy-` followed by an integer timestamp. diff --git a/doc/user/application_security/policies/scan-execution-policies.md b/doc/user/application_security/policies/scan-execution-policies.md index f950d5116b1..c9c48c0c926 100644 --- a/doc/user/application_security/policies/scan-execution-policies.md +++ b/doc/user/application_security/policies/scan-execution-policies.md @@ -8,6 +8,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w > - Group-level security policies were [introduced](https://gitlab.com/groups/gitlab-org/-/epics/4425) in GitLab 15.2. > - Group-level security policies were [enabled on GitLab.com](https://gitlab.com/gitlab-org/gitlab/-/issues/356258) in GitLab 15.4. +> - Operational container scanning [introduced](https://gitlab.com/groups/gitlab-org/-/epics/3410) in GitLab 15.5 Group, subgroup, or project owners can use scan execution policies to require that security scans run on a specified schedule or with the project (or multiple projects if the policy is defined at a group or subgroup level) pipeline. Required scans are injected into the CI pipeline as new jobs @@ -86,16 +87,20 @@ This rule enforces the defined actions and schedules a scan on the provided date | Field | Type | Possible values | Description | |------------|------|-----------------|-------------| | `type` | `string` | `schedule` | The rule's type. | -| `branches` | `array` of `string` | `*` or the branch's name | The branch the given policy applies to (supports wildcard). | +| `branches` | `array` of `string` | `*` or the branch's name | The branch the given policy applies to (supports wildcard). This field is required if the `agents` field is not set. | | `cadence` | `string` | CRON expression (for example, `0 0 * * *`) | A whitespace-separated string containing five fields that represents the scheduled time. | -| `agents` | `object` | | The name of the [GitLab agents](../../clusters/agent/index.md) where [cluster image scanning](../../clusters/agent/vulnerabilities.md) will run. The object key is the name of the Kubernetes cluster configured for your project in GitLab. You can use the optional value of the object to select and scan specific Kubernetes resources. | +| `agents` | `object` | | The name of the [GitLab agents](../../clusters/agent/index.md) where [cluster image scanning](../../clusters/agent/vulnerabilities.md) will run. The object key is the name of the Kubernetes agent configured for your project in GitLab. This field is required if the `branches` field is not set. | GitLab supports the following types of CRON syntax for the `cadence` field: - A daily cadence of once per hour at a specified hour, for example: `0 18 * * *` - A weekly cadence of once per week on a specified day and at a specified hour, for example: `0 13 * * 0` -Other elements of the CRON syntax may work in the cadence field, however, GitLab does not officially test or support them. The CRON expression is evaluated in UTC by default. If you have a self-managed GitLab instance and have [changed the server timezone](../../../administration/timezone.md), the CRON expression is evaluated with the new timezone. +NOTE: +Other elements of the [CRON syntax]((https://docs.oracle.com/cd/E12058_01/doc/doc.1014/e12030/cron_expressions.htm)) may work in the cadence field if supported by the [cron](https://github.com/robfig/cron) we are using in our implementation, however, GitLab does not officially test or support them. + +NOTE: +If using the `agents` field, required for `Operational Container Scanning`, the CRON expression is evaluated in [UTC](https://www.timeanddate.com/worldclock/timezone/utc) using the system-time of the Kubernetes-agent pod. If not using the `agents` field, the CRON expression is evaluated in standard [UTC](https://www.timeanddate.com/worldclock/timezone/utc) time from GitLab.com. If you have a self-managed GitLab instance and have [changed the server timezone](../../../administration/timezone.md), the CRON expression is evaluated with the new timezone. ### `agent` schema @@ -108,20 +113,26 @@ Use this schema to define `agents` objects in the [`schedule` rule type](#schedu #### Policy example ```yaml -- name: Enforce Container Scanning in cluster connected through gitlab-agent for production and staging namespaces +- name: Enforce Container Scanning in cluster connected through my-gitlab-agent for default and kube-system namespaces enabled: true rules: - type: schedule cadence: '0 10 * * *' agents: - gitlab-agent: + <agent-name>: namespaces: - - 'production' - - 'staging' + - 'default' + - 'kube-system' actions: - scan: container_scanning ``` +The keys for a schedule rule are: + +- `cadence` (required): a [CRON expression](https://docs.oracle.com/cd/E12058_01/doc/doc.1014/e12030/cron_expressions.htm) for when the scans will be run +- `agents:<agent-name>` (required): The name of the agent to use for scanning +- `agents:<agent-name>:namespaces` (optional): The Kubernetes namespaces to scan. If omitted, all namespaces will be scanned. + ## `scan` action type This action executes the selected `scan` with additional parameters when conditions for at least one @@ -133,6 +144,7 @@ rule in the defined policy are met. | `site_profile` | `string` | Name of the selected [DAST site profile](../dast/proxy-based.md#site-profile). | The DAST site profile to execute the DAST scan. This field should only be set if `scan` type is `dast`. | | `scanner_profile` | `string` or `null` | Name of the selected [DAST scanner profile](../dast/proxy-based.md#scanner-profile). | The DAST scanner profile to execute the DAST scan. This field should only be set if `scan` type is `dast`.| | `variables` | `object` | | A set of CI variables, supplied as an array of `key: value` pairs, to apply and enforce for the selected scan. The `key` is the variable name, with its `value` provided as a string. This parameter supports any variable that the GitLab CI job supports for the specified scan. | +| `tags` | `array` of `string` | | A list of runner tags for the policy. The policy jobs will be run by runner with the specified tags. | Note the following: @@ -152,7 +164,6 @@ Note the following: mode when executed as part of a scheduled scan. - A container scanning scan that is configured for the `pipeline` rule type ignores the agent defined in the `agents` object. The `agents` object is only considered for `schedule` rule types. An agent with a name provided in the `agents` object must be created and configured for the project. -- The Dependency Scanning and SAST scans use the default templates and run in a [child pipeline](../../../ci/pipelines/downstream_pipelines.md#parent-child-pipelines). ## Example security policies project diff --git a/doc/user/application_security/policies/scan-result-policies.md b/doc/user/application_security/policies/scan-result-policies.md index 7482df18cc3..5df910efb15 100644 --- a/doc/user/application_security/policies/scan-result-policies.md +++ b/doc/user/application_security/policies/scan-result-policies.md @@ -184,3 +184,12 @@ It corresponds to a single object from the previous example: user_approvers: - adalberto.dare ``` + +## Example situations where scan result policies require additional approval + +There are several situations where the scan result policy will require an additional approval step. For example: + +- The number of security jobs is reduced in the working branch and no longer matches the number of security jobs in the target branch. Users can't skip the Scanning Result Policies by removing scanning jobs from the CI configuration. +- Someone stops a pipeline security job, and users can't skip the security scan. +- A job in a merge request fails and is configured with `allow_failure: false`. As a result, the pipeline is in a blocked state. +- A pipeline has a manual job that must run successfully for the entire pipeline to pass. diff --git a/doc/user/application_security/sast/customize_rulesets.md b/doc/user/application_security/sast/customize_rulesets.md index a0742eb79a7..3d8ad6c8bf6 100644 --- a/doc/user/application_security/sast/customize_rulesets.md +++ b/doc/user/application_security/sast/customize_rulesets.md @@ -11,251 +11,423 @@ info: To determine the technical writer assigned to the Stage/Group associated w > passthrough chains. Expanded to include additional passthrough types of `file`, `git`, and `url` in GitLab 14.6. > - [Added](https://gitlab.com/gitlab-org/gitlab/-/issues/235359) support for overriding rules in GitLab 14.8. -You can customize the default scanning rules provided by our SAST analyzers. -Ruleset customization supports the following that can be used -simultaneously: +You can customize the behavior of our SAST analyzers by [defining a ruleset configuration file](#create-the-configuration-file) in the +repository being scanned. There are two kinds of customization: -- [Disabling predefined rules](#disable-predefined-analyzer-rules). Available for all analyzers. -- [Overriding predefined rules](#override-predefined-analyzer-rules). Available for all analyzers. -- Modifying the default behavior of a given analyzer by [synthesizing and passing a custom configuration](#synthesize-a-custom-configuration). Available for only `nodejs-scan`, `gosec`, and `semgrep`. +- Modifying the behavior of **predefined rules**. This includes: + - [Disabling predefined rules](#disable-predefined-rules). Available for all analyzers. + - [Overriding predefined rules](#override-predefined-rules). Available for all analyzers. +- Replacing predefined rules by [synthesizing a custom configuration](#synthesize-a-custom-configuration) + using **passthroughs**. Available for only [nodejs-scan](https://gitlab.com/gitlab-org/security-products/analyzers/nodejs-scan) + and [semgrep](https://gitlab.com/gitlab-org/security-products/analyzers/semgrep). -To customize the default scanning rules, create a file containing custom rules. These rules -are passed through to the analyzer's underlying scanner tools. +## Disable predefined rules -To create a custom ruleset: +You can disable predefined rules for any SAST analyzer. Disabled rules won't appear +on the [Pipeline Security](../index.md#view-security-scan-information-in-the-pipeline-security-tab) +tab or the [Vulnerability Report](../index.md#view-security-scan-information-in-the-vulnerability-report). -1. Create a `.gitlab` directory at the root of your project, if one doesn't already exist. -1. Create a custom ruleset file named `sast-ruleset.toml` in the `.gitlab` directory. +Disabling rules has a retroactive effect. The analyzer continues to scan for the +vulnerability, but findings are omitted from the [`gl-sast-report.json` artifact](index.md#reports-json-format). + +See the [Schema](#schema) and [Examples](#examples) sections for information on how +to configure this behavior. + +## Override predefined rules + +Certain attributes of predefined rules can be overridden for any SAST analyzer. This +can be useful when adapting SAST to your existing workflow or tools. For example, you +might want to override the severity of a vulnerability based on organizational policy, +or choose a different message to display in the Vulnerability Report. + +See the [Schema](#schema) and [Examples](#examples) sections for information on how +to configure this behavior. + +## Synthesize a custom configuration + +You can completely replace the predefined rules of some SAST analyzers: + +- [nodejs-scan](https://gitlab.com/gitlab-org/security-products/analyzers/nodejs-scan) - you + can replace the default [njsscan configuration file](https://github.com/ajinabraham/njsscan#configure-njsscan) + with your own. +- [semgrep](https://gitlab.com/gitlab-org/security-products/analyzers/semgrep) - you can replace + the [GitLab-maintained ruleset](https://gitlab.com/gitlab-org/security-products/analyzers/semgrep/-/tree/main/rules) + with your own. -## Disable predefined analyzer rules +You provide your customizations via passthroughs, which are composed into a +passthrough chain at runtime and evaluated to produce a complete configuration. The +underlying scanner is then executed against this new configuration. -To disable analyzer rules: +There are multiple passthrough types that let you provide configuration in different +ways, such as using a file committed to your repository or inline in the ruleset +configuration file. You can also choose how subsequent passthroughs in the chain are +handled; they can overwrite or append to previous configuration. -1. Set the `disabled` flag to `true` in the context of a `ruleset` section +See the [Schema](#schema) and [Examples](#examples) sections for information on how +to configure this behavior. -1. In one or more `ruleset.identifier` sub sections, list the rules that you want disabled. Every `ruleset.identifier` section has: +## Create the configuration file -- a `type` field, to name the predefined rule identifier that the targeted analyzer uses. -- a `value` field, to name the rule to be disabled. +To create the ruleset configuration file: -### Example: Disable predefined rules of SAST analyzers +1. Create a `.gitlab` directory at the root of your project, if one doesn't already exist. +1. Create a file named `sast-ruleset.toml` in the `.gitlab` directory. + +## Schema + +### The top-level section + +The top-level section contains one or more _configuration sections_, defined as [TOML tables](https://toml.io/en/v1.0.0#table). + +| Setting | Description | +| --------| ----------- | +| `[$analyzer]` | Declares a configuration section for an analyzer. The name follows the snake-case names defined in the list of [SAST analyzers](analyzers.md#sast-analyzers). | -In the following example, the disabled rules are assigned to `eslint` -and `sobelow` by matching the `type` and `value` of identifiers: +Configuration example: ```toml -[eslint] - [[eslint.ruleset]] - disable = true - [eslint.ruleset.identifier] - type = "eslint_rule_id" - value = "security/detect-object-injection" +[semgrep] +... +``` - [[eslint.ruleset]] - disable = true - [eslint.ruleset.identifier] - type = "cwe" - value = "185" +Avoid creating configuration sections that modify existing rules _and_ synthesize a custom ruleset, as +the latter replaces predefined rules completely. -[sobelow] - [[sobelow.ruleset]] - disable = true - [sobelow.ruleset.identifier] - type = "sobelow_rule_id" - value = "sql_injection" +### The `[$analyzer]` configuration section + +The `[$analyzer]` section lets you customize the behavior of an analyzer. Valid properties +differ based on the kind of configuration you're making. + +| Setting | Applies to | Description | +| --------| -------------- | ----------- | +| `[[$analyzer.ruleset]]` | Predefined rules | Defines modifications to an existing rule. | +| `interpolate` | All | If set to `true`, you can use `$VAR` in the configuration to evaluate environment variables. Use this feature with caution, so you don't leak secrets or tokens. (Default: `false`) | +| `description` | Passthroughs | Description of the custom ruleset. | +| `targetdir` | Passthroughs | The directory where the final configuration should be persisted. If empty, a directory with a random name is created. The directory can contain up to 100MB of files. | +| `validate` | Passthroughs | If set to `true`, the content of each passthrough is validated. The validation works for `yaml`, `xml`, `json` and `toml` content. The proper validator is identified based on the extension used in the `target` parameter of the `[[$analyzer.passthrough]]` section. (Default: `false`) | +| `timeout` | Passthroughs | The maximum time to spend to evaluate the passthrough chain, before timing out. The timeout cannot exceed 300 seconds. (Default: 60) | + +#### `interpolate` + +WARNING: +To reduce the risk of leaking secrets, use this feature with caution. + +The example below shows a configuration that uses the `$GITURL` environment variable to access a +private repository. The variable contains a username and token (for example `https://user:token@url`), so +they're not explicitly stored in the configuration file. + +```toml +[semgrep] + description = "My private Semgrep ruleset" + interpolate = true + + [[semgrep.passthrough]] + type = "git" + value = "$GITURL" + ref = "refs/heads/main" ``` -Those vulnerabilities containing the provided type and value are now disabled, meaning -they won't be displayed in Merge Request nor the Vulnerability Report. +### The `[[$analyzer.ruleset]]` section -## Override predefined analyzer rules +The `[[$analyzer.ruleset]]` section targets and modifies a single predefined rule. You can define +one to many of these sections per analyzer. -To override analyzer rules: +| Setting | Description | +| --------| ----------- | +| `disable` | Whether the rule should be disabled. (Default: `false`) | +| `[$analyzer.ruleset.identifier]` | Selects the predefined rule to be modified. | +| `[$analyzer.ruleset.override]` | Defines the overrides for the rule. | -1. In one or more `ruleset.identifier` subsections, list the rules that you want to override. Every `ruleset.identifier` section has: +Configuration example: - - a `type` field, to name the predefined rule identifier that the targeted analyzer uses. - - a `value` field, to name the rule to be overridden. +```toml +[semgrep] + [[semgrep.ruleset]] + disable = true + ... +``` -1. In the `ruleset.override` context of a `ruleset` section, - provide the keys to override. Any combination of keys can be - overridden. Valid keys are: +### The `[$analyzer.ruleset.identifier]` section - - description - - message - - name - - severity (valid options are: Critical, High, Medium, Low, Unknown, Info) +The `[$analyzer.ruleset.identifier]` section defines the identifiers of the predefined +rule that you wish to modify. -### Example: Override predefined rules of SAST analyzers +| Setting | Description | +| --------| ----------- | +| `type` | The type of identifier used by the predefined rule. | +| `value` | The value of the identifier used by the predefined rule. | -Before adding a ruleset, we verify which vulnerability will be overwritten by viewing the [`gl-sast-report.json`](index.md#reports-json-format): +You can look up the correct values for `type` and `value` by viewing the +[`gl-sast-report.json`](index.md#reports-json-format) produced by the analyzer. +You can download this file as a job artifact from the analyzer's CI job. + +For example, the snippet below shows a finding from a `semgrep` rule with three +identifiers. The `type` and `value` keys in the JSON object correspond to the +values you should provide in this section. ```json -"identifiers": [ +... + "vulnerabilities": [ + { + "id": "7331a4b7093875f6eb9f6eb1755b30cc792e9fb3a08c9ce673fb0d2207d7c9c9", + "category": "sast", + "message": "Key Exchange without Entity Authentication", + "description": "Audit the use of ssh.InsecureIgnoreHostKey\n", + ... + "identifiers": [ { - "type": "gosec_rule_id", - "name": "Gosec Rule ID G307", - "value": "G307" + "type": "semgrep_id", + "name": "gosec.G106-1", + "value": "gosec.G106-1" }, { - "type": "CWE", - "name": "CWE-703", - "value": "703", - "url": "https://cwe.mitre.org/data/definitions/703.html" + "type": "cwe", + "name": "CWE-322", + "value": "322", + "url": "https://cwe.mitre.org/data/definitions/322.html" + }, + { + "type": "gosec_rule_id", + "name": "Gosec Rule ID G106", + "value": "G106" } ] + } + ... + ] +... ``` -In the following example, rules from `gosec` are matched by the `type` -and `value` of identifiers and then overridden: +Configuration example: ```toml -[gosec] - [[gosec.ruleset]] - [gosec.ruleset.identifier] - type = "CWE" - value = "703" - [gosec.ruleset.override] +[semgrep] + [[semgrep.ruleset]] + [semgrep.ruleset.identifier] + type = "semgrep_id" + value = "gosec.G106-1 + ... +``` + +### The `[$analyzer.ruleset.override]` section + +The `[$analyzer.ruleset.override]` section allows you to override attributes of a predefined rule. + +| Setting | Description | +| --------| ----------- | +| `description` | A detailed description of the issue. | +| `message` | (Deprecated) A description of the issue. | +| `name` | The name of the rule. | +| `severity` | The severity of the rule. Valid options are: `Critical`, `High`, `Medium`, `Low`, `Unknown`, `Info`) | + +NOTE: +While `message` is populated by the analyzers, it has been [deprecated](https://gitlab.com/gitlab-org/security-products/analyzers/report/-/blob/1d86d5f2e61dc38c775fb0490ee27a45eee4b8b3/vulnerability.go#L22) +in favor of `name` and `description`. + +Configuration example: + +```toml +[semgrep] + [[semgrep.ruleset]] + [semgrep.ruleset.override] severity = "Critical" + name = "Command injection" + ... ``` -If a vulnerability is found with a type `CWE` with a value of `703` then -the vulnerability severity is overwritten to `Critical`. +### The `[[$analyzer.passthrough]]` section -## Synthesize a custom configuration +NOTE: +This is currently supported by the `nodejs-scan` and `semgrep` analyzers only. -To create a custom configuration, you can use passthrough chains. +The `[[$analyzer.passthrough]]` section allows you to synthesize a custom configuration for an analyzer. You +can define up to 20 of these sections per analyzer. Passthroughs are composed into a _passthrough chain_ +that evaluates into a complete configuration that replaces the predefined rules of the analyzer. -A passthrough is a single step in a passthrough chain. The passthrough is evaluated -in a sequence to incrementally build a configuration. The configuration is then -passed to the target analyzer. +Passthroughs are evaluated in order. Passthroughs listed later in the chain have +a higher precedence and can overwrite or append to data yielded by previous +passthroughs (depending on the `mode`). This is useful for cases where you need +to use or modify an existing configuration. -A configuration section for an analyzer has the following -parameters: +The amount of data generated by a single passthrough is limited to 1MB. -| Parameter | Explanation | -| ------------- | ------ | -| `description` | Description about the analyzer configuration section. | -| `targetdir` | The `targetdir` parameter defines the directory where the final configuration is located. If `targetdir` is empty, the analyzer uses a random directory. The maximum size of `targetdir` is 100MB. | -| `validate` | If set to `true`, the target files for passthroughs (`raw`, `file` and `url`) are validated. The validation works for `yaml`, `xml`, `json` and `toml` files. The proper validator is identified based on the extension of the target file. By default, `validate` is set to `false`. | -| `interpolate` | If set to `true`, environment variable interpolation is enabled so that the configuration uses secrets/tokens. We advise using this feature with caution to not leak any secrets. By default, `interpolate` is set to `false`. | -| `timeout` | The total `timeout` for the evaluation of a passthrough chain is set to 60 seconds. If `timeout` is not set, the default timeout is 60 seconds. The timeout cannot exceed 300 seconds. | +| Setting | Applies to | Description | +| ------- | ---------- | ----------- | +| `type` | All | One of `file`, `raw`, `git` or `url`. | +| `target` | All | The target file to contain the data written by the passthrough evaluation. If empty, a random filename is used. | +| `mode` | All | If `overwrite`, the `target` file is overwritten. If `append`, new content is appended to the `target` file. Note that the `git` type only supports `overwrite`. (Default: `overwrite`) | +| `ref` | `type = "git"` | Contains the name of the branch or the SHA to pull. When using a branch name, specify it in the form `refs/heads/<branch>`, not `refs/remotes/<remote_name>/<branch>`. | +| `subdir` | `type = "git"` | Used to select a subdirectory of the Git repository as the configuration source. | +| `value` | All | For the `file`, `url`, and `git` types, defines the location of the file or Git repository. For the `raw` type, contains the inline configuration. | +| `validator` | All | Used to explicitly invoke validators (`xml`, `yaml`, `json`, `toml`) on the target file after the evaluation of a passthrough. | -A configuration section can include one or more passthrough sections. The maximum number of passthrough sections is 20. -There are several types of passthroughs: +#### Passthrough types | Type | Description | -| ------ | ------ | -| `file` | Use a file that is already available in the Git repository. | +| ------ | ----------- | +| `file` | Use a file that is present in the Git repository. | | `raw` | Provide the configuration inline. | | `git` | Pull the configuration from a remote Git repository. | -| `url` | Fetch the analyzer configuration through HTTP. | +| `url` | Fetch the configuration using HTTP. | -If multiple passthrough sections are defined in a passthrough chain, their -position in the chain defines the order in which they are evaluated. +WARNING: +When using the `raw` passthrough with a YAML snippet, it's recommended to format all indentation +in the `sast-ruleset.toml` file as spaces. The YAML specification mandates spaces over tabs, and the +analyzer will fail to parse your custom ruleset unless the indentation is represented accordingly. -- Passthroughs listed later in the chain sequence have a higher precedence. -- Passthroughs with a higher precedence overwrite (default) and append data - yielded by previous passthroughs. This is useful for cases where you need to - use or modify an existing configuration. +## Examples -Configure a passthrough these parameters: +### Disable predefined rules of SAST analyzers -| Parameter | Explanation | -| ------------ | ----------- | -| `type` | One of `file`, `raw`, `git` or `url`. | -| `target` | The target file that contains the data written by the passthrough evaluation. If no value is provided, a random target file is generated. | -| `mode` | `overwrite`: if `target` exists, overwrites the file; `append`: append to file instead. The default is `overwrite`. | -| `ref` | This option only applies to the `git` passthrough type and contains the name of the branch or the SHA to be used. When using a branch name, specify it in the form `refs/heads/<branch>`, not `refs/remotes/<remote_name>/<branch>`. | -| `subdir` | This option only applies to the `git` passthrough type and can be used to only consider a certain subdirectory of the source Git repository. | -| `value` | For the `file` `url` and `git` types, `value` defines the source location of the file/Git repository; for the `raw` type, `value` carries the raw content to be passed through. | -| `validator` | Can be used to explicitly invoke validators (`xml`, `yaml`, `json`, `toml`) on the target files after the application of a passthrough. Per default, no validator is set. | +With the following custom ruleset configuration, the following rules are omitted from the report: -The amount of data generated by a single passthrough is limited to 1MB. +- `semgrep` rules with a `semgrep_id` of `gosec.G106-1` or a `cwe` of `322`. +- `sobelow` rules with a `sobelow_rule_id` of `sql_injection`. +- `flawfinder` rules with a `flawfinder_func_name` of `memcpy`. -## Passthrough configuration examples +```toml +[semgrep] + [[semgrep.ruleset]] + disable = true + [semgrep.ruleset.identifier] + type = "semgrep_id" + value = "gosec.G106-1" -### Raw passthrough for nodejs-scan + [[semgrep.ruleset]] + disable = true + [semgrep.ruleset.identifier] + type = "cwe" + value = "322" -Define a custom analyzer configuration. In this example, customized rules are -defined for the `nodejs-scan` scanner: +[sobelow] + [[sobelow.ruleset]] + disable = true + [sobelow.ruleset.identifier] + type = "sobelow_rule_id" + value = "sql_injection" + +[flawfinder] + [[flawfinder.ruleset]] + disable = true + [flawfinder.ruleset.identifier] + type = "flawfinder_func_name" + value = "memcpy" +``` + +### Override predefined rules of SAST analyzers + +With the following custom ruleset configuration, vulnerabilities found with +`semgrep` with a type `CWE` and a value `322` will have their severity +overridden to `Critical`. + +```toml +[semgrep] + [[semgrep.ruleset]] + [semgrep.ruleset.identifier] + type = "CWE" + value = "322" + [semgrep.ruleset.override] + severity = "Critical" +``` + +### Synthesize a custom configuration using a raw passthrough for `nodejs-scan` + +With the following custom ruleset configuration, the predefined behavior +of the `nodejs-scan` analyzer is replaced with a custom configuration. + +The syntax used for the `value` follows the [njsscan config format](https://github.com/ajinabraham/njsscan#configure-njsscan). ```toml [nodejs-scan] - description = 'custom ruleset for nodejs-scan' + description = "My custom ruleset for nodejs-scan" [[nodejs-scan.passthrough]] type = "raw" value = ''' +--- - nodejs-extensions: - .js - + template-extensions: - .new - .hbs - '' - + ignore-filenames: -- skip.js - + - skip.js + ignore-paths: - __MACOSX - skip_dir - node_modules - + ignore-extensions: - .hbs - + ignore-rules: - regex_injection_dos - pug_jade_template - express_xss - ''' ``` -### File passthrough for Gosec +### Synthesize a custom configuration using a file passthrough for `semgrep` -Provide the name of the file containing a custom analyzer configuration. In -this example, customized rules for the `gosec` scanner are contained in the -file `gosec-config.json`: +With the following custom ruleset configuration, the predefined ruleset +of the `semgrep` analyzer is replaced with a custom ruleset contained in +a file called `my-semgrep-rules.yaml` in the repository being scanned. + +```yaml +# my-semgrep-rules.yml +--- +rules: +- id: my-custom-rule + pattern: print("Hello World") + message: | + Unauthorized use of Hello World. + severity: CRITICAL + languages: + - python +``` ```toml -[gosec] - description = 'custom ruleset for gosec' +[semgrep] + description = "My custom ruleset for Semgrep" - [[gosec.passthrough]] + [[semgrep.passthrough]] type = "file" - value = "gosec-config.json" + value = "my-semgrep-rules.yml" ``` -### Passthrough chain for Semgrep +### Synthesize a custom configuration using a passthrough chain for `semgrep` -In the below example, we generate a custom configuration under the `/sgrules` -target directory with a total `timeout` of 60 seconds. +With the following custom ruleset configuration, the predefined ruleset +of the `semgrep` analyzer is replaced with a custom ruleset produced by +evaluating a chain of four passthroughs. Each passthrough produces a file +that's written to the `/sgrules` directory within the container. A +`timeout` of 60 seconds is set in case any Git remotes are unresponsive. -Several passthrouh types generate a configuration for the target analyzer: +Different passthrough types are demonstrated in this example: -- Two `git` passthrough sections pull the head of branch - `refs/heads/test` from the `myrules` Git repository, and revision - `97f7686` from the `sast-rules` Git repository. From the `sast-rules` Git - repository, only data from the `go` subdirectory is considered. +- Two `git` passthroughs, the first pulling `refs/heads/test` from the + `myrules` Git repository, and the second pulling revision `97f7686` + from the `sast-rules` repository, and considering only files in the + `go` subdirectory. - The `sast-rules` entry has a higher precedence because it appears later in the configuration. - - If there is a filename collision between files in both repositories, files - from the `sast` repository overwrite files from the `myrules` repository, - as `sast-rules` has higher precedence. -- The `raw` entry creates a file named `insecure.yml` under `/sgrules`. The - full path is `/sgrules/insecure.yml`. -- The `url` entry fetches a configuration made available through a URL and - stores it in the `/sgrules/gosec.yml` file. + - If there's a filename collision between the two checkouts, files + from the `sast-rules` repository will overwrite files from the + `myrules` repository. +- A `raw` passthrough, which writes its `value` to `/sgrules/insecure.yml`. +- A `url` passthrough, which fetches a configuration hosted at a URL and + writes it to `/sgrules/gosec.yml`. Afterwards, Semgrep is invoked with the final configuration located under `/sgrules`. ```toml [semgrep] - description = 'semgrep custom rules configuration' + description = "My custom ruleset for Semgrep" targetdir = "/sgrules" timeout = 60 @@ -277,15 +449,15 @@ Afterwards, Semgrep is invoked with the final configuration located under rules: - id: "insecure" patterns: - - pattern: "func insecure() {...}" + - pattern: "func insecure() {...}" message: | Insecure function insecure detected metadata: cwe: "CWE-200: Exposure of Sensitive Information to an Unauthorized Actor" severity: "ERROR" languages: - - "go" - """ + - "go" +""" [[semgrep.passthrough]] type = "url" @@ -293,89 +465,93 @@ rules: target = "gosec.yml" ``` -### Interpolation - -The code snippet below shows an example configuration that uses an environment -variable `$GITURL` to access a private repositories with a Git URL. The variable contains -a username and token in the `value` field (for example `https://user:token@url`). -It does not explicitly store credentials in the configuration file. To reduce the risk of leaking secrets through created paths and files, use this feature with caution. - -```toml -[semgrep] - description = 'semgrep custom rules configuration' - targetdir = "/sgrules" - interpolate = true - - [[semgrep.passthrough]] - type = "git" - value = "$GITURL" - ref = "refs/heads/main" -``` - -### Configure the append mode for passthroughs - -To append data to previous passthroughs, use the `append` mode for the -passthrough types `file`, `url`, and `raw`. +### Configure the mode for passthroughs in a chain -Passthroughs in `override` mode overwrite files -created when preceding passthroughs in the chain find a naming -collision. If `mode` is set to `append`, a passthrough appends data to the -files created by its predecessors instead of overwriting. +You can choose how to handle filename conflicts that occur between +passthroughs in a chain. The default behavior is to overwrite +existing files with the same name, but you can choose `mode = append` +instead to append the content of later files onto earlier ones. -In the below Semgrep configuration,`/sgrules/insecure.yml` assembles two passthroughs. The rules are: +You can use the `append` mode for the `file`, `url`, and `raw` +passthrough types only. -- `insecure` -- `secret` - -These rules add a search pattern to the analyzer and extends Semgrep capabilities. - -For passthrough chains we recommend that you enable validation. To enable validation, -you can either: - -- set `validate` to `true` - -- set a passthrough `validator` to `xml`, `json`, `yaml`, or `toml`. +With the following custom ruleset configuration, two `raw` passthroughs +are used to iteratively assemble the `/sgrules/my-rules.yml` file, which +is then provided to Semgrep as the ruleset. Each passthrough appends a +single rule to the ruleset. The first passthrough is responsible for +initialising the top-level `rules` object, according to the +[Semgrep rule syntax](https://semgrep.dev/docs/writing-rules/rule-syntax/). ```toml [semgrep] - description = 'semgrep custom rules configuration' + description = "My custom ruleset for Semgrep" targetdir = "/sgrules" validate = true [[semgrep.passthrough]] type = "raw" - target = "insecure.yml" + target = "my-rules.yml" value = """ rules: - id: "insecure" patterns: - - pattern: "func insecure() {...}" + - pattern: "func insecure() {...}" message: | - Insecure function insecure detected + Insecure function 'insecure' detected metadata: - cwe: "... + cwe: "..." severity: "ERROR" languages: - - "go" + - "go" """ [[semgrep.passthrough]] type = "raw" mode = "append" - target = "insecure.yml" + target = "my-rules.yml" value = """ - id: "secret" patterns: - - pattern-either: - - pattern: "$MASK = \"...\"" - - metavariable-regex: - metavariable: "$MASK" - regex: "(password|pass|passwd|pwd|secret|token)" + - pattern-either: + - pattern: '$MASK = "..."' + - metavariable-regex: + metavariable: "$MASK" + regex: "(password|pass|passwd|pwd|secret|token)" message: | - Use of Hard-coded Password + Use of hard-coded password + metadata: cwe: "..." severity: "ERROR" languages: - - "go" + - "go" """ ``` + +```yaml +# /sgrules/my-rules.yml +rules: +- id: "insecure" + patterns: + - pattern: "func insecure() {...}" + message: | + Insecure function 'insecure' detected + metadata: + cwe: "..." + severity: "ERROR" + languages: + - "go" +- id: "secret" + patterns: + - pattern-either: + - pattern: '$MASK = "..."' + - metavariable-regex: + metavariable: "$MASK" + regex: "(password|pass|passwd|pwd|secret|token)" + message: | + Use of hard-coded password + metadata: + cwe: "..." + severity: "ERROR" + languages: + - "go" +``` diff --git a/doc/user/application_security/sast/index.md b/doc/user/application_security/sast/index.md index b1bc9794ced..94719224254 100644 --- a/doc/user/application_security/sast/index.md +++ b/doc/user/application_security/sast/index.md @@ -105,6 +105,7 @@ Check the [SAST direction page](https://about.gitlab.com/direction/secure/static | React | [Semgrep](https://gitlab.com/gitlab-org/security-products/analyzers/semgrep) with GitLab-managed rules | 13.10 | | Ruby | [brakeman](https://gitlab.com/gitlab-org/security-products/analyzers/brakeman) | 13.9 | | Ruby on Rails | [brakeman](https://gitlab.com/gitlab-org/security-products/analyzers/brakeman) | 10.3 | +| Scala (any build system) | [Semgrep](https://gitlab.com/gitlab-org/security-products/analyzers/semgrep) with GitLab-managed rules | 15.7 | | Scala<sup>2</sup> | [SpotBugs](https://gitlab.com/gitlab-org/security-products/analyzers/spotbugs) with the find-sec-bugs plugin | 11.0 (SBT) & 11.9 (Gradle, Maven) | | Swift (iOS) | [MobSF (beta)](https://gitlab.com/gitlab-org/security-products/analyzers/mobsf) | 13.5 | | TypeScript<sup>3</sup> | [ESLint security plugin](https://gitlab.com/gitlab-org/security-products/analyzers/eslint) | 11.9, [merged](https://gitlab.com/gitlab-org/gitlab/-/issues/36059) with ESLint in 13.2 | @@ -694,7 +695,7 @@ The process for importing Docker images into a local offline Docker registry dep process by which external resources can be imported or temporarily accessed. These scanners are [periodically updated](../index.md#vulnerability-scanner-maintenance) with new definitions, and you may be able to make occasional updates on your own. -For details on saving and transporting Docker images as a file, see Docker's documentation on +For details on saving and transporting Docker images as a file, see the Docker documentation on [`docker save`](https://docs.docker.com/engine/reference/commandline/save/), [`docker load`](https://docs.docker.com/engine/reference/commandline/load/), [`docker export`](https://docs.docker.com/engine/reference/commandline/export/), and [`docker import`](https://docs.docker.com/engine/reference/commandline/import/). diff --git a/doc/user/application_security/secret_detection/index.md b/doc/user/application_security/secret_detection/index.md index 8a066cf1be1..d955170ece2 100644 --- a/doc/user/application_security/secret_detection/index.md +++ b/doc/user/application_security/secret_detection/index.md @@ -82,43 +82,57 @@ To enable Secret Detection, either: - Enable [Auto DevOps](../../../topics/autodevops/index.md), which includes [Auto Secret Detection](../../../topics/autodevops/stages.md#auto-secret-detection). -- [Enable Secret Detection by including the template](#enable-secret-detection-by-including-the-template). +- [Edit the `.gitlab.ci.yml` file manually](#edit-the-gitlabciyml-file-manually). Use this method if + your `.gitlab-ci.yml` file is complex. -- [Enable Secret Detection using a merge request](#enable-secret-detection-using-a-merge-request). +- [Use an automatically configured merge request](#use-an-automatically-configured-merge-request). -### Enable Secret Detection by including the template +### Edit the `.gitlab.ci.yml` file manually -You should use this method if you have an existing GitLab CI/CD configuration file. +This method requires you to manually edit the existing `.gitlab-ci.yml` file. Use this method if +your GitLab CI/CD configuration file is complex. -Add the following extract to your `.gitlab-ci.yml` file: +1. On the top bar, select **Main menu > Projects** and find your project. +1. On the left sidebar, select **CI/CD > Editor**. +1. Copy and paste the following to the bottom of the `.gitlab-ci.yml` file: -```yaml -include: - - template: Jobs/Secret-Detection.gitlab-ci.yml -``` + ```yaml + include: + - template: Jobs/Secret-Detection.gitlab-ci.yml + ``` -Pipelines now include a Secret Detection job, and the results are included in the merge request -widget. +1. Select the **Validate** tab, then select **Validate pipeline**. + The message **Simulation completed successfully** indicates the file is valid. +1. Select the **Edit** tab. +1. Optional. In the **Commit message** text box, customize the commit message. +1. In the **Branch** text box, enter the name of the default branch. +1. Select **Commit changes**. -### Enable Secret Detection using a merge request +Pipelines now include a Secret Detection job. + +### Use an automatically configured merge request > - [Introduced](https://gitlab.com/groups/gitlab-org/-/epics/4496) in GitLab 13.11, deployed behind a feature flag, enabled by default. > - [Feature flag removed](https://gitlab.com/gitlab-org/gitlab/-/issues/329886) in GitLab 14.1. +This method automatically prepares a merge request, with the Secret Detection template included in +the `.gitlab-ci.yml` file. You then merge the merge request to enable Secret Detection. + NOTE: This method works best with no existing `.gitlab-ci.yml` file, or with a minimal configuration file. If you have a complex GitLab configuration file it may not be parsed successfully, and an -error may occur. +error may occur. In that case, use the [manual](#edit-the-gitlabciyml-file-manually) method instead. -To enable Secret Detection using a merge request: +To enable Secret Detection automatically: 1. On the top bar, select **Main menu > Projects** and find your project. 1. On the left sidebar, select **Security & Compliance > Configuration**. 1. In the **Secret Detection** row, select **Configure with a merge request**. +1. Optional. Complete the fields. +1. Select **Create merge request**. 1. Review and merge the merge request. -Pipelines now include a Secret Detection job, and the results are included in the merge request -widget. +Pipelines now include a Secret Detection job. ## Responding to a leaked secret diff --git a/doc/user/application_security/secret_detection/post_processing.md b/doc/user/application_security/secret_detection/post_processing.md index 8dbe459d4af..9c74467bce5 100644 --- a/doc/user/application_security/secret_detection/post_processing.md +++ b/doc/user/application_security/secret_detection/post_processing.md @@ -6,7 +6,13 @@ info: To determine the technical writer assigned to the Stage/Group associated w # Secret Detection post-processing and revocation **(FREE SAAS)** -> [Introduced](https://gitlab.com/groups/gitlab-org/-/epics/4639) in GitLab 13.6. +> - [Introduced](https://gitlab.com/groups/gitlab-org/-/epics/4639) in GitLab 13.6. +> - [Disabled by default for GitLab personal access tokens](https://gitlab.com/gitlab-org/gitlab/-/issues/371658) in GitLab 15.6 [with a flag](../../../administration/feature_flags.md) named `gitlab_pat_auto_revocation`. Available to GitLab.com only. + +FLAG: +By default, auto revocation of GitLab personal access tokens is not available. To opt-in on GitLab.com +during the [Beta period](https://about.gitlab.com/handbook/product/gitlab-the-product/#alpha-beta-ga), please +[let us know by completing this form](https://docs.google.com/forms/d/e/1FAIpQLSdRbFhvA5jvI-Rt_Qnl1PQ1znOXKK8m6lRtmM0uva4upetKvQ/viewform). GitLab supports running post-processing hooks after detecting a secret. These hooks can perform actions, like notifying the cloud service that issued the secret. @@ -16,7 +22,7 @@ The cloud provider can then confirm the credentials and take remediation actions - Reissuing a secret. - Notifying the creator of the secret. -GitLab SaaS supports post-processing for Amazon Web Services (AWS). +GitLab SaaS supports post-processing for [GitLab personal access tokens](../../profile/personal_access_tokens.md) and Amazon Web Services (AWS). Post-processing workflows vary by supported cloud providers. Post-processing is limited to a project's default branch. The epic diff --git a/doc/user/application_security/terminology/index.md b/doc/user/application_security/terminology/index.md index f156d60be8f..2666d91c5aa 100644 --- a/doc/user/application_security/terminology/index.md +++ b/doc/user/application_security/terminology/index.md @@ -83,11 +83,6 @@ once. A finding that doesn't exist but is incorrectly reported as existing. -### Feedback - -Feedback the user provides about a finding. Types of feedback include dismissal, creating an issue, -or creating a merge request. - ### Finding An asset that has the potential to be vulnerable, identified in a project by an analyzer. Assets @@ -96,6 +91,11 @@ applications, and infrastructure. Findings are all potential vulnerability items scanners identify in MRs/feature branches. Only after merging to default does a finding become a [vulnerability](#vulnerability). +You can interact with vulnerability findings in two ways. + +1. You can open an issue or merge request for the vulnerability finding. +1. You can dismiss the vulnerability finding. Dismissing the finding hides it from the default views. + ### Grouping A flexible and non-destructive way to visually organize vulnerabilities in groups when there are multiple findings @@ -164,15 +164,15 @@ table.package-managers-and-types ul { <tbody> <tr> <td>gem</td> - <td><a href="https://bundler.io/">bundler</a></td> + <td><a href="https://bundler.io/">Bundler</a></td> </tr> <tr> - <td>packagist</td> - <td><a href="https://getcomposer.org/">composer</a></td> + <td>Packagist</td> + <td><a href="https://getcomposer.org/">Composer</a></td> </tr> <tr> - <td>conan</td> - <td><a href="https://conan.io/">conan</a></td> + <td>Conan</td> + <td><a href="https://conan.io/">Conan</a></td> </tr> <tr> <td>go</td> @@ -180,10 +180,10 @@ table.package-managers-and-types ul { </tr> <tr> <td rowspan="3">maven</td> - <td><a href="https://gradle.org/">gradle</a></td> + <td><a href="https://gradle.org/">Gradle</a></td> </tr> <tr> - <td><a href="https://maven.apache.org/">maven</a></td> + <td><a href="https://maven.apache.org/">Maven</a></td> </tr> <tr> <td><a href="https://www.scala-sbt.org">sbt</a></td> @@ -196,12 +196,12 @@ table.package-managers-and-types ul { <td><a href="https://classic.yarnpkg.com/en">yarn</a></td> </tr> <tr> - <td>nuget</td> - <td><a href="https://www.nuget.org/">nuget</a></td> + <td>NuGet</td> + <td><a href="https://www.nuget.org/">NuGet</a></td> </tr> <tr> - <td rowspan="4">pypi</td> - <td><a href="https://setuptools.pypa.io/en/latest/">setuptools</a></td> + <td rowspan="4">PyPI</td> + <td><a href="https://setuptools.pypa.io/en/latest/">Setuptools</a></td> </tr> <tr> <td><a href="https://pip.pypa.io/en/stable">pip</a></td> diff --git a/doc/user/application_security/vulnerabilities/index.md b/doc/user/application_security/vulnerabilities/index.md index 9ddb1bb51e2..e86f9ff4673 100644 --- a/doc/user/application_security/vulnerabilities/index.md +++ b/doc/user/application_security/vulnerabilities/index.md @@ -59,6 +59,8 @@ To change a vulnerability's status from its Vulnerability Page: 1. From the **Status** dropdown list select a status, then select **Change status**. 1. Optionally, at the bottom of the page, add a comment to the log entry. +The Actions log records each status change along with which user changed the status and the time of the change. + ## Creating an issue for a vulnerability From a vulnerability's page you can create an issue to track all action taken to resolve or diff --git a/doc/user/application_security/vulnerability_report/index.md b/doc/user/application_security/vulnerability_report/index.md index 2b78dde4f63..dd919889b4d 100644 --- a/doc/user/application_security/vulnerability_report/index.md +++ b/doc/user/application_security/vulnerability_report/index.md @@ -260,7 +260,7 @@ To add a new vulnerability finding from your project level Vulnerability Report 1. On the top bar, select **Main menu > Projects** and find your project. 1. On the left sidebar, select **Security & Compliance > Vulnerability Report**. -1. Select **Submit Vulnerability**. +1. Select **Submit vulnerability**. 1. Complete the fields and submit the form. You will be brought to the newly created vulnerability's detail page. Manually created records appear in the diff --git a/doc/user/application_security/vulnerability_report/pipeline.md b/doc/user/application_security/vulnerability_report/pipeline.md index 9e20e4f6f78..57c18cb045e 100644 --- a/doc/user/application_security/vulnerability_report/pipeline.md +++ b/doc/user/application_security/vulnerability_report/pipeline.md @@ -79,6 +79,23 @@ incorporated once the pipeline finishes. | Resolved | No | Needs triage (Detected) | | N/A (i.e.: new vulnerability) | No | Needs triage (Detected) | +## Retention period for vulnerabilities + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/351524) in GitLab 15.5. + +GitLab has the following retention policies for vulnerabilities on non-default branches. Vulnerabilities are no longer available: + +- When the related CI job artifact expires. +- 90 days after the pipeline is created, even if the related CI job artifacts are locked. + +To view vulnerabilities, either: + +- Run a new pipeline. +- Download the related CI job artifacts if they are available. + +NOTE: +This does not apply for the vulnerabilities existing on the default branch. + ## Deduplication process When a pipeline contains jobs that produce multiple security reports of the same type, it is possible that the same diff --git a/doc/user/asciidoc.md b/doc/user/asciidoc.md index 3ceecf88c5e..6b642363dd6 100644 --- a/doc/user/asciidoc.md +++ b/doc/user/asciidoc.md @@ -9,6 +9,11 @@ info: To determine the technical writer assigned to the Stage/Group associated w GitLab uses the [Asciidoctor](https://asciidoctor.org) gem to convert AsciiDoc content to HTML5. Consult the [Asciidoctor User Manual](https://asciidoctor.org/docs/user-manual/) for a complete Asciidoctor reference. +You can use AsciiDoc in the following areas: + +- Wiki pages +- AsciiDoc documents (`.adoc` or `.asciidoc`) inside repositories + ## Syntax Here's a brief reference of the most commonly used AsciiDoc syntax. diff --git a/doc/user/clusters/agent/ci_cd_workflow.md b/doc/user/clusters/agent/ci_cd_workflow.md index 454be3c53c7..2a66549f9cb 100644 --- a/doc/user/clusters/agent/ci_cd_workflow.md +++ b/doc/user/clusters/agent/ci_cd_workflow.md @@ -60,6 +60,7 @@ Authorization configuration can take one or two minutes to propagate. > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/327850) in GitLab 14.4. > - [Changed](https://gitlab.com/gitlab-org/gitlab/-/issues/346566) to remove hierarchy restrictions in GitLab 15.6. +> - [Changed](https://gitlab.com/gitlab-org/gitlab/-/issues/356831) to allow authorizing projects in a user namespace in GitLab 15.7. To authorize the agent to access the GitLab project where you keep Kubernetes manifests: @@ -73,7 +74,7 @@ To authorize the agent to access the GitLab project where you keep Kubernetes ma - id: path/to/project ``` - - Authorized projects must have the same root group as the agent's configuration project. + - Authorized projects must have the same root group or user namespace as the agent's configuration project. - You can install additional agents into the same cluster to accommodate additional hierarchies. - You can authorize up to 100 projects. @@ -286,6 +287,35 @@ The identity can be specified with the following keys: See the [official Kubernetes documentation for details](https://kubernetes.io/docs/reference/access-authn-authz/authentication/#user-impersonation). +## Restrict project and group access to specific environments **(FREE)** + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/343885) in GitLab 15.7. + +By default, if your agent is [available to a project](#authorize-the-agent), all of the project's CI/CD jobs can use that agent. + +To restrict access to the agent to only jobs with specific environments, add `environments` to `ci_access.projects` or `ci_access.groups`. For example: + + ```yaml + ci_access: + projects: + - id: path/to/project-1 + - id: path/to/project-2 + environments: + - staging + - review/* + groups: + - id: path/to/group-1 + environments: + - production + ``` + +In this example: + +- All CI/CD jobs under `project-1` can access the agent. +- CI/CD jobs under `project-2` with `staging` or `review/*` environments can access the agent. + - `*` is a wildcard, so `review/*` matches all environments under `review`. +- CI/CD jobs for projects under `group-1` with `production` environments can access the agent. + ## Related topics - [Self-paced classroom workshop](https://gitlab-for-eks.awsworkshop.io) (Uses AWS EKS, but you can use for other Kubernetes clusters) diff --git a/doc/user/clusters/agent/gitops.md b/doc/user/clusters/agent/gitops.md index c0d4a5e088f..bd7dfb3abee 100644 --- a/doc/user/clusters/agent/gitops.md +++ b/doc/user/clusters/agent/gitops.md @@ -9,6 +9,8 @@ info: To determine the technical writer assigned to the Stage/Group associated w > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/259669) in GitLab 13.7. > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/332227) in GitLab 14.0, the `resource_inclusions` and `resource_exclusions` attributes were removed and `reconcile_timeout`, `dry_run_strategy`, `prune`, `prune_timeout`, `prune_propagation_policy`, and `inventory_policy` attributes were added. > - [Moved](https://gitlab.com/gitlab-org/gitlab/-/issues/346567) from GitLab Premium to GitLab Free in 15.3. +> - [Changed](https://gitlab.com/gitlab-org/gitlab/-/issues/346585) to make the `id` attribute optional in GitLab 15.7. +> - Specifying a branch, tag, or commit reference to fetch the Kubernetes manifest files [introduced](https://gitlab.com/groups/gitlab-org/-/epics/4516) in GitLab 15.7. With GitOps, you can manage containerized clusters and applications from a Git repository that: @@ -64,6 +66,10 @@ The following snippet shows an example of the possible keys and values for the G gitops: manifest_projects: - id: gitlab-org/cluster-integration/gitlab-agent + ref: # either `branch`, `tag` or `commit` can be specified + branch: production + # commit: <mysha> + # tag: v1.0 default_namespace: my-ns paths: # Read all YAML files from this directory. @@ -83,7 +89,11 @@ gitops: | Keyword | Description | |--|--| | `manifest_projects` | Projects where your Kubernetes manifests are stored. The agent monitors the files in the repositories in these projects. When manifest files change, the agent deploys the changes to the cluster. | -| `id` | Required. Path to a Git repository that has Kubernetes manifests in YAML or JSON format. No authentication mechanisms are currently supported. | +| `id` | Path to a Git repository that has Kubernetes manifests in YAML or JSON format. No authentication mechanisms are supported. Default is the agent configuration repository. | +| `ref` | Optional. Git reference in the configured Git repository to fetch the Kubernetes manifest files from. If not specified or empty, the default branch is used. If specified, it must contain either `branch`, `tag`, or `commit`. | +| `ref.branch` | Branch name in the configured Git repository to fetch the Kubernetes manifest files from. | +| `ref.tag` | Tag name in the configured Git repository to fetch the Kubernetes manifest files from. | +| `ref.commit` | Commit SHA in the configured Git repository to fetch the Kubernetes manifest files from. | | `default_namespace` | Namespace to use if not set explicitly in object manifest. Also used for inventory `ConfigMap` objects. | | `paths` | Repository paths to scan for manifest files. Directories with names that start with a dot `(.)` are ignored. | | `paths[].glob` | Required. See [doublestar](https://github.com/bmatcuk/doublestar#about) and [the match function](https://pkg.go.dev/github.com/bmatcuk/doublestar/v2#Match) for globbing rules. | diff --git a/doc/user/clusters/agent/gitops/helm.md b/doc/user/clusters/agent/gitops/helm.md index 0ec87376636..b9a59d37f5d 100644 --- a/doc/user/clusters/agent/gitops/helm.md +++ b/doc/user/clusters/agent/gitops/helm.md @@ -6,7 +6,8 @@ info: To determine the technical writer assigned to the Stage/Group associated w # Using Helm charts to update a Kubernetes cluster (Alpha) **(FREE)** -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/371019) in GitLab 15.4. +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/371019) in GitLab 15.4. +> - Specifying a branch, tag, or commit reference to fetch the Kubernetes manifest files [introduced](https://gitlab.com/groups/gitlab-org/-/epics/4516) in GitLab 15.7. You can deploy Helm charts to your Kubernetes cluster and keep the resources in your cluster in sync with your charts and values. To do this, you use the pull-based GitOps features of the agent for @@ -51,6 +52,8 @@ gitops: source: project: id: my-group/my-project-with-chart + ref: + branch: production path: dir-in-project/with/charts namespace: my-ns max_history: 1 @@ -68,6 +71,10 @@ gitops: | `max_history` | Optional. Maximum number of release [revisions to store in the cluster](https://helm.sh/docs/helm/helm_history/). | | `source` | Required. From where the chart should get installed. Only supports project sources. | | `source.project.id` | Required. ID of the project where Helm chart is committed. Authentication is not supported. | +| `source.project.ref` | Optional. Git reference in the configured Git repository to fetch the Chart from. If not specified or empty, the default branch is used. If specified, it must contain either `branch`, `tag`, or `commit`. | +| `source.project.ref.branch` | Branch name in the configured Git repository to fetch the Chart from. | +| `source.project.ref.tag` | Tag name in the configured Git repository to fetch the Chart from. | +| `source.project.ref.commit` | Commit SHA in the configured Git repository to fetch the Chart from. | | `source.project.path` | Optional. Path of the chart in the project repository. Root of the repository is used by default. Should be the directory with the `Chart.yaml` file. | ## Custom values diff --git a/doc/user/clusters/agent/install/index.md b/doc/user/clusters/agent/install/index.md index 2030052e3b0..62767f1dfd9 100644 --- a/doc/user/clusters/agent/install/index.md +++ b/doc/user/clusters/agent/install/index.md @@ -123,7 +123,7 @@ To install the agent on your cluster using Helm: 1. In your computer, open a terminal and [connect to your cluster](https://kubernetes.io/docs/tasks/access-application-cluster/access-cluster/). 1. Run the command you copied when you [registered your agent with GitLab](#register-the-agent-with-gitlab). -Optionally, you can [customize the Helm installation](#customize-the-helm-installation). +Optionally, you can [customize the Helm installation](#customize-the-helm-installation). If you install the agent on a production system, you should customize the Helm installation to skip creating the service account. ##### Customize the Helm installation diff --git a/doc/user/clusters/agent/vulnerabilities.md b/doc/user/clusters/agent/vulnerabilities.md index dcb3276deb5..d9a9981d211 100644 --- a/doc/user/clusters/agent/vulnerabilities.md +++ b/doc/user/clusters/agent/vulnerabilities.md @@ -12,17 +12,19 @@ info: To determine the technical writer assigned to the Stage/Group associated w To view cluster vulnerabilities, you can view the [vulnerability report](../../application_security/vulnerabilities/index.md). You can also configure your agent so the vulnerabilities are displayed with other agent information in GitLab. -## Enable operational container scanning **(ULTIMATE)** +## Enable operational container scanning -You can use operational container scanning -to scan container images in your cluster for security vulnerabilities. +You can use operational container scanning to scan container images in your cluster for security vulnerabilities. You +can enable the scanner to run on a cadence as configured via the agent, or setup scan execution policies within a +project that houses the agent. NOTE: In GitLab 15.0 and later, you do not need to install Starboard operator in the Kubernetes cluster. -To begin scanning all resources in your cluster, add a `container_scanning` -configuration block to your agent configuration with a `cadence` field -containing a CRON expression for when the scans will be run. +### Enable via agent configuration + +To enable scanning of all images within your Kubernetes cluster via the agent configuration, add a `container_scanning` configuration block to your agent +configuration with a `cadence` field containing a [CRON expression](https://docs.oracle.com/cd/E12058_01/doc/doc.1014/e12030/cron_expressions.htm) for when the scans will be run. ```yaml container_scanning: @@ -34,29 +36,67 @@ The `cadence` field is required. GitLab supports the following types of CRON syn - A daily cadence of once per hour at a specified hour, for example: `0 18 * * *` - A weekly cadence of once per week on a specified day and at a specified hour, for example: `0 13 * * 0` -It is possible that other elements of the CRON syntax will work in the cadence field, however, GitLab does not officially test or support them. +NOTE: +Other elements of the [CRON syntax](https://docs.oracle.com/cd/E12058_01/doc/doc.1014/e12030/cron_expressions.htm) may work in the cadence field if supported by the [cron](https://github.com/robfig/cron) we are using in our implementation, however, GitLab does not officially test or support them. + +NOTE: +The CRON expression is evaluated in [UTC](https://www.timeanddate.com/worldclock/timezone/utc) using the system-time of the Kubernetes-agent pod. By default, operational container scanning will attempt to scan the workloads in all -namespaces for vulnerabilities. The `vulnerability_report` block has a `namespaces` +namespaces for vulnerabilities. You can set the `vulnerability_report` block with the `namespaces` field which can be used to restrict which namespaces are scanned. For example, -if you would like to scan only the `development`, `staging`, and `production` -namespaces, you can use this configuration: +if you would like to scan only the `default`, `kube-system` namespaces, you can use this configuration: ```yaml container_scanning: cadence: '0 0 * * *' vulnerability_report: namespaces: - - development - - staging - - production + - default + - kube-system ``` -## View cluster vulnerabilities +## Enable via scan execution policies + +To enable scanning of all images within your Kubernetes cluster via scan execution policies, we can use the +[scan execution policy editor](../../application_security/policies/scan-execution-policies.md#scan-execution-policy-editor) +in order to create a new schedule rule. -Prerequisite: +NOTE: +The Kubernetes agent must be running in your cluster in order to scan running container images + +Here is an example of a policy which enables operational container scanning within the cluster the Kubernetes agent is attached to: + +```yaml +- name: Enforce Container Scanning in cluster connected through my-gitlab-agent for default and kube-system namespaces + enabled: true + rules: + - type: schedule + cadence: '0 10 * * *' + agents: + <agent-name>: + namespaces: + - 'default' + - 'kube-system' + actions: + - scan: container_scanning +``` -- You must have at least the Developer role. +The keys for a schedule rule are: + +- cadence (required): a [CRON expression](https://docs.oracle.com/cd/E12058_01/doc/doc.1014/e12030/cron_expressions.htm) for when the scans will be run +- agents:<agent-name> (required): The name of the agent to use for scanning +- agents:<agent-name>:namespaces (optional): The Kubernetes namespaces to scan. If omitted, all namespaces will be scanned + +NOTE: +Other elements of the [CRON syntax](https://docs.oracle.com/cd/E12058_01/doc/doc.1014/e12030/cron_expressions.htm) may work in the cadence field if supported by the [cron](https://github.com/robfig/cron) we are using in our implementation, however, GitLab does not officially test or support them. + +NOTE: +The CRON expression is evaluated in [UTC](https://www.timeanddate.com/worldclock/timezone/utc) using the system-time of the Kubernetes-agent pod. + +You can view the complete schema within the [scan execution policy documentation](../../application_security/policies/scan-execution-policies.md#scan-execution-policies-schema). + +## View cluster vulnerabilities To view vulnerability information in GitLab: @@ -68,3 +108,6 @@ To view vulnerability information in GitLab:  This information can also be found under [operational vulnerabilities](../../../user/application_security/vulnerability_report/index.md#operational-vulnerabilities). + +NOTE: +You must have at least the Developer role. diff --git a/doc/user/clusters/management_project_template.md b/doc/user/clusters/management_project_template.md index cbe577b9b74..a8d874ed608 100644 --- a/doc/user/clusters/management_project_template.md +++ b/doc/user/clusters/management_project_template.md @@ -102,7 +102,7 @@ The [built-in supported applications](https://gitlab.com/gitlab-org/project-temp - [Cert-manager](../infrastructure/clusters/manage/management_project_applications/certmanager.md) - [GitLab Runner](../infrastructure/clusters/manage/management_project_applications/runner.md) - [Ingress](../infrastructure/clusters/manage/management_project_applications/ingress.md) -- [Prometheus](../infrastructure/clusters/manage/management_project_applications/prometheus.md) +- [Prometheus](../../operations/metrics/index.md) - [Vault](../infrastructure/clusters/manage/management_project_applications/vault.md) Each application has an `applications/{app}/values.yaml` file. diff --git a/doc/user/compliance/license_compliance/index.md b/doc/user/compliance/license_compliance/index.md index fb5ce37c563..34434ef046a 100644 --- a/doc/user/compliance/license_compliance/index.md +++ b/doc/user/compliance/license_compliance/index.md @@ -177,7 +177,7 @@ directory of your project. Depending on your language, you may need to specify the path to the individual projects of a monorepo using the `LICENSE_FINDER_CLI_OPTS` variable. Passing in the project paths can significantly speed up builds over using the `--recursive` -license_finder option. +License Finder option. ```yaml include: @@ -660,7 +660,7 @@ The process for importing Docker images into a local offline Docker registry dep process by which external resources can be imported or temporarily accessed. Note that these scanners are [updated periodically](../../application_security/index.md#vulnerability-scanner-maintenance) with new definitions, so consider if you are able to make periodic updates yourself. -For details on saving and transporting Docker images as a file, see Docker's documentation on +For details on saving and transporting Docker images as a file, see the Docker documentation on [`docker save`](https://docs.docker.com/engine/reference/commandline/save/), [`docker load`](https://docs.docker.com/engine/reference/commandline/load/), [`docker export`](https://docs.docker.com/engine/reference/commandline/export/), and [`docker import`](https://docs.docker.com/engine/reference/commandline/import/). diff --git a/doc/user/free_user_limit.md b/doc/user/free_user_limit.md index 35777847947..4f569098d4d 100644 --- a/doc/user/free_user_limit.md +++ b/doc/user/free_user_limit.md @@ -6,7 +6,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w # Free user limit **(FREE SAAS)** -From October 19, 2022, a five-user limit will apply to top-level [namespaces](namespace/index.md) with private visibility on GitLab SaaS. These limits will roll out gradually, and impacted users will be notified in GitLab.com at least 60 days before the limit is applied. +A five-user limit applies to top-level [namespaces](namespace/index.md) with private visibility on GitLab SaaS. This limit is being rolled out gradually, and impacted users will be notified in GitLab.com at least 60 days before the limit is applied. When the five-user limit is applied, top-level private namespaces exceeding the user limit are placed in a read-only state. These namespaces cannot write new data to repositories, Git Large File Storage (LFS), packages, or registries. diff --git a/doc/user/gitlab_com/index.md b/doc/user/gitlab_com/index.md index d3d50ee1a8f..9bb1c4e968c 100644 --- a/doc/user/gitlab_com/index.md +++ b/doc/user/gitlab_com/index.md @@ -189,7 +189,7 @@ the default value [is the same as for self-managed instances](../admin_area/sett |-------------------------------|--------------------| | [Repository size including LFS](../admin_area/settings/account_and_limit_settings.md#repository-size-limit) | 10 GB | | [Maximum import size](../project/settings/import_export.md#maximum-import-file-size) | 5 GB | -| Maximum attachment size | 10 MB | +| Maximum attachment size | 100 MB | If you are near or over the repository size limit, you can either [reduce your repository size with Git](../project/repository/reducing_the_repo_size_using_git.md) @@ -314,7 +314,7 @@ The list of GitLab.com specific settings (and their defaults) is as follows: | `max_wal_senders` | 32 | 0 | | `max_wal_size` | 5GB | 1GB | | `shared_buffers` | 112896MB | Based on how much memory is available | -| `shared_preload_libraries` | pg_stat_statements | empty | +| `shared_preload_libraries` | `pg_stat_statements` | empty | | `shmall` | 30146560 | Based on the server's capabilities | | `shmmax` | 123480309760 | Based on the server's capabilities | | `wal_buffers` | 16MB | -1 | @@ -445,11 +445,18 @@ If more than the maximum number of allowed connections occur concurrently, they are dropped and users get [an `ssh_exchange_identification` error](../../topics/git/troubleshooting_git.md#ssh_exchange_identification-error). -### Import/export +### Group and project import by uploading export files -To help avoid abuse, project and group imports, exports, and export downloads -are rate limited. See [Project import/export rate limits](../../user/project/settings/import_export.md#rate-limits) and [Group import/export rate limits](../../user/group/settings/import_export.md#rate-limits) -for details. +To help avoid abuse, the following are rate limited: + +- Project and group imports. +- Group and project exports that use files. +- Export downloads. + +For more information, see: + +- [Project import/export rate limits](../../user/project/settings/import_export.md#rate-limits). +- [Group import/export rate limits](../../user/group/import/index.md#rate-limits). ### Non-configurable limits diff --git a/doc/user/group/access_and_permissions.md b/doc/user/group/access_and_permissions.md index 13a1fd31ee4..a7358db54df 100644 --- a/doc/user/group/access_and_permissions.md +++ b/doc/user/group/access_and_permissions.md @@ -228,7 +228,10 @@ projects in a group, allowing tighter control over project membership. For example, if you want to lock the group for an [Audit Event](../../administration/audit_events.md), you can guarantee that project membership cannot be modified during the audit. -You can still invite groups or to add members to groups, implicitly giving members access to projects in the **locked** group. +If group membership lock is enabled, the group owner can still: + +- Invite groups or add members to groups to give them access to projects in the **locked** group. +- Change the role of group members. The setting does not cascade. Projects in subgroups observe the subgroup configuration, ignoring the parent group. @@ -239,8 +242,10 @@ To prevent members from being added to projects in a group: 1. Under **Membership**, select **Users cannot be added to projects in this group**. 1. Select **Save changes**. -All users who previously had permissions can no longer add members to a group. -API requests to add a new user to a project are not possible. +After you lock the membership for a group: + +- All users who previously had permissions can no longer add members to a group. +- API requests to add a new user to a project are not possible. ## Manage group memberships via LDAP **(PREMIUM SELF)** diff --git a/doc/user/group/compliance_frameworks.md b/doc/user/group/compliance_frameworks.md index 7bd545003db..0e976cec866 100644 --- a/doc/user/group/compliance_frameworks.md +++ b/doc/user/group/compliance_frameworks.md @@ -21,7 +21,7 @@ Group owners can create, edit, and delete compliance frameworks: 1. Expand the **Compliance frameworks** section. 1. Create, edit, or delete compliance frameworks. -## Set a default compliance framework +## Default compliance frameworks > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/375036) in GitLab 15.6. @@ -29,6 +29,20 @@ Group owners can set a default compliance framework. The default framework is ap that are created within that group. It does not affect the framework applied to the existing projects. The default framework cannot be deleted. +A compliance framework that is set to default has a **default** label. + +### Set and remove as default + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/375038) in GitLab 15.7. + +Group owners can set a compliance framework as default (or remove the setting): + +1. On the top bar, select **Main menu > Groups > View all groups** and find your group. +1. On the left sidebar, select **Settings > General**. +1. Expand the **Compliance frameworks** section and locate the compliance framework to set (or remove) as default. +1. Select the vertical ellipsis (**{ellipsis_v}**) for the compliance frame and then select **Set default** (or + **Remove default**). + ### Example GraphQL mutations for setting a default compliance framework Creating a new compliance framework and setting it as the default framework for the group. @@ -191,7 +205,7 @@ When creating such an MR against a project with CF pipelines, the above snippet This is because in the context of the target project, `$CI_COMMIT_REF_NAME` evaluates to a non-existing branch name. To get the correct context, use `$CI_MERGE_REQUEST_SOURCE_PROJECT_PATH` instead of `$CI_PROJECT_PATH`. -This variable is only availabe in +This variable is only available in [merge request pipelines](../../ci/pipelines/merge_request_pipelines.md). For example, for a configuration that supports both merge request pipelines originating in project forks and branch pipelines, diff --git a/doc/user/group/contribution_analytics/index.md b/doc/user/group/contribution_analytics/index.md index b1efd2e9251..ddf468e39b0 100644 --- a/doc/user/group/contribution_analytics/index.md +++ b/doc/user/group/contribution_analytics/index.md @@ -24,8 +24,7 @@ To view Contribution Analytics: ## Using Contribution Analytics -There are three main bar graphs that illustrate the number of contributions per group -member for the following: +Three bar graphs illustrate the number of contributions made by each group member: - Push events - Merge requests diff --git a/doc/user/group/devops_adoption/index.md b/doc/user/group/devops_adoption/index.md index 67263f15f06..a81b61c50ce 100644 --- a/doc/user/group/devops_adoption/index.md +++ b/doc/user/group/devops_adoption/index.md @@ -12,7 +12,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w > - DAST and SAST metrics [added](https://gitlab.com/gitlab-org/gitlab/-/issues/328033) in GitLab 14.1. > - Fuzz Testing metrics [added](https://gitlab.com/gitlab-org/gitlab/-/issues/330398) in GitLab 14.2. > - Dependency Scanning metrics [added](https://gitlab.com/gitlab-org/gitlab/-/issues/328034) in GitLab 14.2. -> - Multiselect [added](https://gitlab.com/gitlab-org/gitlab/-/issues/333586) in GitLab 14.2. +> - Multi-select [added](https://gitlab.com/gitlab-org/gitlab/-/issues/333586) in GitLab 14.2. > - Overview table [added](https://gitlab.com/gitlab-org/gitlab/-/issues/335638) in GitLab 14.3. > - Adoption over time chart [added](https://gitlab.com/gitlab-org/gitlab/-/issues/337561) in GitLab 14.4. diff --git a/doc/user/group/epics/manage_epics.md b/doc/user/group/epics/manage_epics.md index 61aa5c5fe02..8e7b6fd82ad 100644 --- a/doc/user/group/epics/manage_epics.md +++ b/doc/user/group/epics/manage_epics.md @@ -494,7 +494,8 @@ The maximum number of direct child epics is 100. ### Child epics from other groups -> [Introduced](https://gitlab.com/groups/gitlab-org/-/epics/8502) in GitLab 15.6 [with a flag](../../../administration/feature_flags.md) named `child_epics_from_different_hierarchies`. Disabled by default. +> - [Introduced](https://gitlab.com/groups/gitlab-org/-/epics/8502) in GitLab 15.6 [with a flag](../../../administration/feature_flags.md) named `child_epics_from_different_hierarchies`. Disabled by default. +> - Minimum required role for the group [changed](https://gitlab.com/gitlab-org/gitlab/-/issues/382503) from Reporter to Guest in GitLab 15.7. FLAG: On self-managed GitLab, by default this feature is not available. To make it available per group, ask an administrator to [enable the feature flag](../../../administration/feature_flags.md) named `child_epics_from_different_hierarchies`. @@ -504,16 +505,18 @@ You can add a child epic that belongs to a group that is different from the pare Prerequisites: -- You must have at least the Reporter role for both the child and parent epics' groups. +- You must have at least the Guest role for both the child and parent epics' groups. - Multi-level child epics must be available for both the child and parent epics' groups. To add a child epic from another group, paste the epic's URL when [adding an existing epic](#add-a-child-epic-to-an-epic). ### Add a child epic to an epic +> Minimum required role for the group [changed](https://gitlab.com/gitlab-org/gitlab/-/issues/382503) from Reporter to Guest in GitLab 15.7. + Prerequisites: -- You must have at least the Reporter role for the parent epic's group. +- You must have at least the Guest role for the parent epic's group. To add a new epic as child epic: @@ -534,7 +537,8 @@ To add an existing epic as child epic: ### Move child epics between epics -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/33039) in GitLab 13.0. +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/33039) in GitLab 13.0. +> - Minimum required role for the group [changed](https://gitlab.com/gitlab-org/gitlab/-/issues/382503) from Reporter to Guest in GitLab 15.7. New child epics appear at the top of the list in the **Epics and Issues** tab. You can move child epics from one epic to another. @@ -543,7 +547,7 @@ Issues and child epics cannot be intermingled. Prerequisites: -- You must have at least the Reporter role for the parent epic's group. +- You must have at least the Guest role for the parent epic's group. To move child epics to another epic: @@ -552,14 +556,15 @@ To move child epics to another epic: ### Reorder child epics assigned to an epic -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/9367) in GitLab 12.5. +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/9367) in GitLab 12.5. +> - Minimum required role for the group [changed](https://gitlab.com/gitlab-org/gitlab/-/issues/382503) from Reporter to Guest in GitLab 15.7. New child epics appear at the top of the list in the **Epics and Issues** tab. You can reorder the list of child epics. Prerequisites: -- You must have at least the Reporter role for the parent epic's group. +- You must have at least the Guest role for the parent epic's group. To reorder child epics assigned to an epic: @@ -568,9 +573,11 @@ To reorder child epics assigned to an epic: ### Remove a child epic from a parent epic +> Minimum required role for the group [changed](https://gitlab.com/gitlab-org/gitlab/-/issues/382503) from Reporter to Guest in GitLab 15.7. + Prerequisites: -- You must have at least the Reporter role for the parent epic's group. +- You must have at least the Guest role for the parent epic's group. To remove a child epic from a parent epic: diff --git a/doc/user/group/import/index.md b/doc/user/group/import/index.md index 49515dd8a11..9a671ff6679 100644 --- a/doc/user/group/import/index.md +++ b/doc/user/group/import/index.md @@ -4,7 +4,23 @@ group: Import info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- -# Migrating groups with GitLab Migration **(FREE)** +# Migrating GitLab groups **(FREE)** + +You can migrate GitLab groups: + +- From self-managed GitLab to GitLab.com. +- From GitLab.com to self-managed GitLab. +- From one self-managed GitLab instance to another. +- Between groups in the same GitLab instance. + +You can migrate groups in two ways: + +- By direct transfer (recommended). +- By uploading an export file. + +If you migrate from GitLab.com to self-managed GitLab, an administrator can create users on the self-managed GitLab instance. + +## Migrate groups by direct transfer (recommended) > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/249160) in GitLab 13.7 for group resources [with a flag](../../feature_flags.md) named `bulk_import`. Disabled by default. > - Group items [enabled on GitLab.com and self-managed](https://gitlab.com/gitlab-org/gitlab/-/issues/338985) in GitLab 14.3. @@ -18,78 +34,103 @@ On self-managed GitLab, by default [migrating project items](#migrated-project-i this feature, ask an administrator to [enable the feature flag](../../../administration/feature_flags.md) named `bulk_import_projects`. On GitLab.com, migration of both groups and projects is available. -Users with the Owner role on a top-level group can use GitLab Migration to migrate the group to: +Prerequisites: + +- Network connection between instances or GitLab.com. Must support HTTPS. +- Owner role on the top-level group to migrate. + +You can import top-level groups to: - Another top-level group. - The subgroup of any existing top-level group. - Another GitLab instance, including GitLab.com. -Migrating groups using GitLab Migration is not the same as [migrating groups using file exports](../settings/import_export.md). -Importing and exporting groups using file exports requires you to export a group to a file and then import that file in -another GitLab instance. Migrating groups using GitLab Migration automates this step. +You can migrate: + +- By direct transfer using either the UI or the [API](../../../api/bulk_imports.md). +- Many groups at once. -## Import your groups into GitLab +When migrating a top-level group to GitLab.com, all its subgroups and projects are migrated too. -When you migrate a group, you connect to your GitLab instance and then choose -groups to import. Not all the data is migrated. See: +Not all group and project resources are imported. See list of migrated resources below: - [Migrated group items](#migrated-group-items). - [Migrated project items](#migrated-project-items). -Leave feedback about group migration in [the relevant issue](https://gitlab.com/gitlab-org/gitlab/-/issues/284495). +### Preparation + +GitLab maps users and their contributions correctly provided: + +- Users already exist on the target GitLab instance. +- Users have a public email on the source GitLab instance that matches their primary email on the target GitLab instance. +- Users' primary email addresses on the target GitLab instance are confirmed. Most users receives an email asking them to confirm their email address. +- When using an OmniAuth provider like SAML, GitLab and SAML accounts of users on GitLab must be linked. All users on the target GitLab instance must sign in + and verify their account on the target GitLab instance. -NOTE: You might need to reconfigure your firewall to prevent blocking the connection on the self-managed instance. -### Connect to the remote GitLab instance +If you use [SAML SSO for GitLab.com groups](../../group/saml_sso/index.md), +contributing users must have [linked their SAML identity to their GitLab.com account](../../group/saml_sso/index.md#linking-saml-to-your-existing-gitlabcom-account). -Before you begin, ensure that the target GitLab instance can communicate with the source over HTTPS -(HTTP is not supported). You might need to reconfigure your firewall to prevent blocking the connection on the self-managed -instance. +When migrating to GitLab.com, you must create users manually unless [SCIM](../../group/saml_sso/scim_setup.md) is used. Creating users with the API is only +available to self-managed instances because it requires administrator access. -Then create the group you want to import into, and connect: +### Connect to the source GitLab instance -1. Create a new group or subgroup: +Create the group you want to import to and connect the source: - - On the top bar, select `+` and then **New group**. - - Or, on an existing group's page, in the top right, select **New subgroup**. +1. Create either: -1. Select **Import group**. -1. Enter the source URL of your GitLab instance. + - A new group. On the top bar, select **{plus-square}**, then **New group**, and select **Import group**. + - A new subgroup. On existing group's page, either: + - Select **New subgroup**. + - On the top bar, Select **{plus-square}** and then **New subgroup**. Then on the left sidebar, select the **import an existing group** link. +1. Enter the URL of your source GitLab instance. 1. Generate or copy a [personal access token](../../../user/profile/personal_access_tokens.md) - with the `api` and `read_repository` scopes on your remote GitLab instance. -1. Enter the [personal access token](../../../user/profile/personal_access_tokens.md) for your remote GitLab instance. + with the `api` scope on your source GitLab instance. Both `api` and `read_repository` scopes are required when migrating from GitLab 15.0 and earlier. +1. Enter the [personal access token](../../../user/profile/personal_access_tokens.md) for your source GitLab instance. 1. Select **Connect instance**. ### Select the groups to import -After you have authorized access to the GitLab instance, you are redirected to the GitLab Group -Migration importer page. The remote groups you have the Owner role for are listed. +After you have authorized access to the source GitLab instance, you are redirected to the GitLab group +importer page. The top-level groups on the connected source instance you have the Owner role for are listed. -1. By default, the proposed group namespaces match the names as they exist in remote instance, but based on your permissions, you can choose to edit these names before you proceed to import any of them. +1. By default, the proposed group namespaces match the names as they exist in source instance, but based on your permissions, you can choose to edit these names before you proceed to import any of them. 1. Next to the groups you want to import, select **Import**. 1. The **Status** column shows the import status of each group. If you leave the page open, it updates in real-time. 1. After a group has been imported, select its GitLab path to open its GitLab URL.  -## Automate group and project import **(PREMIUM)** +### Group import history -For information on automating user, group, and project import API calls, see -[Automate group and project import](../../project/import/index.md#automate-group-and-project-import). +You can view all groups migrated by you by direct transfer listed on the group import history page. This list includes: + +- Paths of source groups. +- Paths of destination groups. +- Start date of each import. +- Status of each import. +- Error details if any errors occurred. -## Migrated group items +To view group import history: + +1. Sign in to GitLab. +1. On the top bar, select **Create new…** (**{plus-square}**). +1. Select **New group**. +1. Select **Import group**. +1. Select **History** in the upper right corner. +1. If there are any errors for a particular import, you can see them by selecting **Details**. + +### Migrated group items The [`import_export.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/import_export/group/import_export.yml) -file for groups lists many of the items migrated when migrating groups using group migration. View this file in the branch +file for groups lists many of the items imported when migrating groups by direct transfer. View this file in the branch for your version of GitLab to see the list of items relevant to you. For example, [`import_export.yml` on the `14-10-stable-ee` branch](https://gitlab.com/gitlab-org/gitlab/-/blob/14-10-stable-ee/lib/gitlab/import_export/group/import_export.yml). -Migrating projects with file exports uses the same export and import mechanisms as creating projects from templates at the [group](../custom_project_templates.md) and -[instance](../../admin_area/custom_project_templates.md) levels. Therefore, the list of exported items is the same. - -Items that are migrated to the target instance include: +Group items that are migrated to the target instance include: - Badges ([Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/292431) in 13.11) - Board Lists @@ -114,24 +155,24 @@ Items that are migrated to the target instance include: Any other items are **not** migrated. -## Migrated project items +### Migrated project items > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/267945) in GitLab 14.4 [with a flag](../../feature_flags.md) named `bulk_import_projects`. Disabled by default. > - [Enabled on GitLab.com](https://gitlab.com/gitlab-org/gitlab/-/issues/339941) in GitLab 15.6. FLAG: -On self-managed GitLab, migrating project resources are not available by default. To make them available, ask an administrator to [enable the feature flag](../../../administration/feature_flags.md) named `bulk_import_projects`. On GitLab.com, migration of project resources is available. +On self-managed GitLab, migrating project resources when migrating groups is not available by default. To make it available ask an administrator to [enable the feature flag](../../../administration/feature_flags.md) named `bulk_import_projects`. On GitLab.com, groups are migrated with all their projects by default. The [`import_export.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/import_export/project/import_export.yml) -file for projects lists many of the items migrated when migrating projects using group migration. View this file in the branch +file for projects lists many of the items imported when migrating projects using group migration. View this file in the branch for your version of GitLab to see the list of items relevant to you. For example, [`import_export.yml` on the `14-10-stable-ee` branch](https://gitlab.com/gitlab-org/gitlab/-/blob/14-10-stable-ee/lib/gitlab/import_export/project/import_export.yml). -Migrating projects with file exports uses the same export and import mechanisms as creating projects from templates at the [group](../../group/custom_project_templates.md) and -[instance](../../admin_area/custom_project_templates.md) levels. Therefore, the list of exported items is the same. +Project items that are migrated to the target instance include: - Projects ([Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/267945) in GitLab 14.4) - Auto DevOps ([Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/339410) in GitLab 14.6) + - Badges ([Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/75029) in GitLab 14.6) - Branches (including protected branches) ([Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/339414) in GitLab 14.7) - CI Pipelines ([Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/339407) in GitLab 14.6) - Designs ([Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/339421) in GitLab 15.1) @@ -141,6 +182,8 @@ Migrating projects with file exports uses the same export and import mechanisms - Issue resource milestone events ([Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/291983) in GitLab 15.4) - Issue resource iteration events ([Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/291983) in GitLab 15.4) - Merge request URL references ([Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/267947) in GitLab 15.6) + - Time Tracking ([Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/267946) in GitLab 14.4) + - Issue boards ([Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/71661) in GitLab 14.4) - Labels ([Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/339419) in GitLab 14.4) - LFS Objects ([Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/339405) in GitLab 14.8) - Members ([Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/341886) in GitLab 14.8) @@ -151,18 +194,33 @@ Migrating projects with file exports uses the same export and import mechanisms - Merge request resource state events ([Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/291983) in GitLab 15.4) - Merge request resource milestone events ([Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/291983) in GitLab 15.4) - Issue URL references ([Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/267947) in GitLab 15.6) - - Migrate Push Rules ([Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/339403) in GitLab 14.6) - - Pull Requests (including external pull requests) ([Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/339409) in GitLab 14.5) + - Time Tracking ([Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/339403) in GitLab 14.5) + - Push Rules ([Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/339403) in GitLab 14.6) + - Milestones ([Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/339417) in GitLab 14.5) + - External Pull Requests ([Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/339409) in GitLab 14.5) - Pipeline History ([Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/339412) in GitLab 14.6) - Pipeline Schedules ([Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/339408) in GitLab 14.8) + - Project Features ([Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/74722) in GitLab 14.6) - Releases ([Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/339422) in GitLab 15.1) - Release Evidences ([Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/360567) in GitLab 15.1) - Repositories ([Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/267945) in GitLab 14.4) - Snippets ([Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/343438) in GitLab 14.6) + - Settings ([Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/339416) in GitLab 14.6) + - Avatar ([Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/75249) in GitLab 14.6) + - Container Expiration Policy ([Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/75653) in GitLab 14.6) + - Project Properties ([Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/75898) in GitLab 14.6) + - Service Desk ([Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/75653) in GitLab 14.6) - Uploads ([Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/339401) in GitLab 14.5) - Wikis ([Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/345923) in GitLab 14.6) -## Troubleshooting Group Migration +Items excluded from migration, because they contain sensitive information: + +- Pipeline Triggers. + +Migrating projects with file exports uses the same export and import mechanisms as creating projects from templates at the [group](../../group/custom_project_templates.md) and +[instance](../../admin_area/custom_project_templates.md) levels. Therefore, the list of exported items is the same. + +### Troubleshooting In a [rails console session](../../../administration/operations/rails_console.md#starting-a-rails-console-session), you can find the failure or error messages for the group import attempt using: @@ -184,7 +242,10 @@ entities.map(&:failures).flatten entities.where(status: [-1]).pluck(:destination_name, :destination_namespace, :status) ``` -### Stale imports +You can also see all migrated entities with any failures related to them using an +[API endpoint](../../../api/bulk_imports.md#list-all-gitlab-migrations-entities). + +#### Stale imports > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/352985) in GitLab 14.10. @@ -199,10 +260,12 @@ import = BulkImports::Entity.where(namespace_id: Group.id).map(&:bulk_import) import.status #=> 3 means that the import timed out. ``` -### Error: `404 Group Not Found` +#### Error: `404 Group Not Found` + +If you attempt to import a group that has a path comprised of only numbers (for example, `5000`), GitLab attempts to +find the group by ID instead of the path. This causes a `404 Group Not Found` error in GitLab 15.4 and earlier. -If you attempt to import a group that has a path comprised of only numbers (for example, `5000`), GitLab attempts to find the group by ID instead of the -path. This causes a `404 Group Not Found` error. To solve this, the source group path must be changed to include a non-numerical character using either: +To solve this, you must change the source group path to include a non-numerical character using either: - The GitLab UI: @@ -212,3 +275,170 @@ path. This causes a `404 Group Not Found` error. To solve this, the source group 1. Under **Change group URL**, change the group URL to include non-numeric characters. - The [Groups API](../../../api/groups.md#update-group). + +### Provide feedback + +Please leave your feedback about migrating groups by direct transfer in +[the feedback issue](https://gitlab.com/gitlab-org/gitlab/-/issues/284495). + +## Migrate groups by uploading an export file (deprecated) + +> - [Introduced](https://gitlab.com/groups/gitlab-org/-/epics/2888) in GitLab 13.0 as an experimental feature. May change in future releases. +> - [Deprecated](https://gitlab.com/groups/gitlab-org/-/epics/4619) in GitLab 14.6. + +WARNING: +This feature was [deprecated](https://gitlab.com/groups/gitlab-org/-/epics/4619) in GitLab 14.6 and replaced by +[migrating groups by direct transfer](#migrate-groups-by-direct-transfer-recommended). To follow progress on a solution for +[offline environments](../../application_security/offline_deployments/index.md), see +[the relevant epic](https://gitlab.com/groups/gitlab-org/-/epics/8985). + +Prerequisites: + +- Owner role on the group to migrate. + +Using file exports, you can: + +- Export any group to a file and upload that file to another GitLab instance or to another location on the same instance. +- Use either the GitLab UI or the [API](../../../api/group_import_export.md). +- Migrate groups one by one, then export and import each project for the groups one by one. + +GitLab maps user contributions correctly when an admin access token is used to perform the import. GitLab does not map +user contributions correctly when you are importing from a self-managed instance to GitLab.com. Correct mapping of user +contributions when importing from a self-managed instance to GitLab.com can be preserved with paid involvement of +Professional Services team. + +Note the following: + +- Exports are stored in a temporary directory and are deleted every 24 hours by a specific worker. +- To preserve group-level relationships from imported projects, export and import groups first so that projects can + be imported into the desired group structure. +- Imported groups are given a `private` visibility level, unless imported into a parent group. +- If imported into a parent group, a subgroup inherits the same level of visibility unless otherwise restricted. +- You can export groups from the [Community Edition to the Enterprise Edition](https://about.gitlab.com/install/ce-or-ee/) + and vice versa. The Enterprise Edition retains some group data that isn't part of the Community Edition. If you're + exporting a group from the Enterprise Edition to the Community Edition, you may lose this data. For more information, + see [downgrading from EE to CE](../../../index.md). + +### Exported contents + +The [`import_export.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/import_export/group/import_export.yml) +file for groups lists items exported and imported when migrating groups using file exports. View this file in the branch +for your version of GitLab to see the list of items relevant to you. For example, +[`import_export.yml` on the `14-10-stable-ee` branch](https://gitlab.com/gitlab-org/gitlab/-/blob/14-10-stable-ee/lib/gitlab/import_export/group/import_export.yml). + +Group items that are exported include: + +- Milestones +- Labels +- Boards and Board Lists +- Badges +- Subgroups (including all the aforementioned data) +- Epics + - Epic resource state events ([Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/291983) in GitLab 15.4) +- Events +- [Wikis](../../project/wiki/group.md) + ([Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/53247) in GitLab 13.9) +- Iterations cadences ([Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/95372) in 15.4) + +Items that are **not** exported include: + +- Projects +- Runner tokens +- SAML discovery tokens + +### Preparation + +- To preserve the member list and their respective permissions on imported groups, review the users in these groups. Make +sure these users exist before importing the desired groups. +- Users must set a public email in the source GitLab instance that matches one of their verified emails in the target GitLab instance. + +### Enable export for a group + +Prerequisite: + +- You must have the Owner role for the group. + +To enable import and export for a group: + +1. On the top bar, select **Main menu > Admin**. +1. On the left sidebar, select **Settings > General**. +1. Expand **Visibility and access controls**. +1. In the **Import sources** section, select the checkboxes for the sources you want. + +### Export a group + +Prerequisites: + +- You must have the Owner role for the group. + +To export the contents of a group: + +1. On the top bar, select **Main menu > Groups** and find your group. +1. On the left sidebar, select **Settings > General**. +1. In the **Advanced** section, select **Export Group**. +1. After the export is generated, you should receive an email with a link to the [exported contents](#exported-contents) + in a compressed tar archive, with contents in NDJSON format. +1. Alternatively, you can download the export from the UI: + + 1. Return to your group's **Settings > General** page. + 1. In the **Advanced** section, select **Download export**. + You can also generate a new file by selecting **Regenerate export**. + +You can also export a group [using the API](../../../api/group_import_export.md). + +### Import the group + +1. Create a new group: + - On the top bar, select **Create new…** (**{plus-square}**) and then **New group**. + - On an existing group's page, select the **New subgroup** button. +1. Select **Import group**. +1. Enter your group name. +1. Accept or modify the associated group URL. +1. Select **Choose file**. +1. Select the file that you exported in the [Export a group](#export-a-group) section. +1. To begin importing, select **Import group**. + +Your newly imported group page appears after the operation completes. + +NOTE: +The maximum import file size can be set by the administrator, default is `0` (unlimited). +As an administrator, you can modify the maximum import file size. To do so, use the `max_import_size` option in the +[Application settings API](../../../api/settings.md#change-application-settings) or the +[Admin Area](../../admin_area/settings/account_and_limit_settings.md). +Default [modified](https://gitlab.com/gitlab-org/gitlab/-/issues/251106) from 50MB to 0 in GitLab 13.8. + +### Rate limits + +To help avoid abuse, by default, users are rate limited to: + +| Request Type | Limit | +| ---------------- | ---------------------------------------- | +| Export | 6 groups per minute | +| Download export | 1 download per group per minute | +| Import | 6 groups per minute | + +### Version history + +#### 14.0+ + +In GitLab 14.0, the JSON format is no longer supported for project and group exports. To allow for a +transitional period, you can still import any JSON exports. The new format for imports and exports +is NDJSON. + +#### 13.0+ + +GitLab can import bundles that were exported from a different GitLab deployment. +This ability is limited to two previous GitLab [minor](../../../policy/maintenance.md#versioning) +releases, which is similar to our process for [Security Releases](../../../policy/maintenance.md#security-releases). + +For example: + +| Current version | Can import bundles exported from | +|-----------------|----------------------------------| +| 13.0 | 13.0, 12.10, 12.9 | +| 13.1 | 13.1, 13.0, 12.10 | + +## Automate group and project import **(PREMIUM)** + +For information on automating user, group, and project import API calls, see +[Automate group and project import](../../project/import/index.md#automate-group-and-project-import). diff --git a/doc/user/group/manage.md b/doc/user/group/manage.md index a63e6c6dd7f..414b80d0f1d 100644 --- a/doc/user/group/manage.md +++ b/doc/user/group/manage.md @@ -78,7 +78,7 @@ If you don't want to wait, you can remove a group immediately. Prerequisites: -- You must have at least the Owner role for a group. +- You must have the Owner role for a group. - You have [marked the group for deletion](#remove-a-group). To immediately remove a group marked for deletion: @@ -172,7 +172,7 @@ Prerequisite: 1. On the left sidebar, select **Group information > Members**. 1. Select **Invite members**. 1. Fill in the fields. - - The role applies to all projects in the group. [Learn more about permissions](../permissions.md). + - The role applies to all projects in the group. For more information, see [permissions](../permissions.md). - On the **Access expiration date**, the user can no longer access projects in the group. 1. Select **Invite**. @@ -530,10 +530,10 @@ in a subgroup has access to the templates for that subgroup and any immediate parent groups. To learn how to create templates for issues and merge requests, see -[Description templates](../project/description_templates.md). +[description templates](../project/description_templates.md). Define project templates at a group level by setting a group as the template source. -[Learn more about group-level project templates](custom_project_templates.md). +For more information, see [group-level project templates](custom_project_templates.md). ### Enable group file template **(PREMIUM)** diff --git a/doc/user/group/saml_sso/group_sync.md b/doc/user/group/saml_sso/group_sync.md index 001c73b6979..80d145fc6bb 100644 --- a/doc/user/group/saml_sso/group_sync.md +++ b/doc/user/group/saml_sso/group_sync.md @@ -10,9 +10,9 @@ info: To determine the technical writer assigned to the Stage/Group associated w > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/363084) for self-managed instances in GitLab 15.1. WARNING: -Changing Group Sync configuration can remove users from the mapped GitLab group. +Adding or changing Group Sync configuration can remove users from the mapped GitLab group. Removal happens if there is any mismatch between the group names and the list of `groups` in the SAML response. -If changes must be made, ensure either the SAML response includes the `groups` attribute +Before making changes, ensure either the SAML response includes the `groups` attribute and the `AttributeValue` value matches the **SAML Group Name** in GitLab, or that all groups are removed from GitLab to disable Group Sync. @@ -21,17 +21,29 @@ For a demo of Group Sync using Azure, see [Demo: SAML Group Sync](https://youtu. ## Configure SAML Group Sync +NOTE: +You must include the SAML configuration block on all Sidekiq nodes in addition to Rails application nodes if you: + +- Use SAML Group Sync. +- Have multiple GitLab nodes, for example in a distributed or highly available architecture. + +WARNING: +To prevent users being accidentally removed from the GitLab group, follow these instructions closely before +enabling Group Sync in GitLab. + To configure SAML Group Sync: -- For GitLab self-managed: - 1. Configure the [SAML OmniAuth Provider](../../../integration/saml.md). - 1. Ensure your SAML identity provider sends an attribute statement with the same name as the value of the `groups_attribute` setting. -- For GitLab.com: - 1. See [SAML SSO for GitLab.com groups](index.md). - 1. Ensure your SAML identity provider sends an attribute statement named `Groups` or `groups`. +1. Configure the identity Provider: + - For self-managed GitLab, see the [SAML OmniAuth Provider documentation](../../../integration/saml.md). + - For GitLab.com, see the [SAML SSO for GitLab.com groups documentation](index.md). + +1. Capture [a SAML response](troubleshooting.md#saml-debugging-tools) during the sign-in process to confirm your SAML identity provider sends an attribute statement: + - For self-managed GitLab, with the same name as the value of the `groups_attribute` setting. + - For GitLab.com, named `Groups` or `groups`. NOTE: -The value for `Groups` or `groups` in the SAML response can be either the group name or the group ID. +The value for `Groups` or `groups` in the SAML response may be either the group name or an ID. +For example, Azure AD sends the Azure Group Object ID instead of the name. Use the ID value when configuring [SAML Group Links](#configure-saml-group-links). ```xml <saml:AttributeStatement> @@ -55,7 +67,7 @@ a SAML identity provider group name to a GitLab role. This can be done for a top To link the SAML groups: -1. In **SAML Group Name**, enter the value of the relevant `saml:AttributeValue`. +1. In **SAML Group Name**, enter the value of the relevant `saml:AttributeValue`. The value entered here must exactly match the value sent in the SAML response. For some IdPs, this may be a group ID or object ID (Azure AD) instead of a friendly group name. 1. Choose the role in **Access Level**. 1. Select **Save**. 1. Repeat to add additional group links if required. @@ -177,4 +189,4 @@ Because of a [known issue with Azure AD](https://support.esri.com/en/technical-a in the user's SAML assertion. To work around this issue, allow more than 150 group IDs to be sent in SAML token using configuration steps in the -[Azure AD documentation](https://support.esri.com/en/technical-article/000022190). +[Azure AD documentation](https://support.esri.com/en/technical-article/000022190). diff --git a/doc/user/group/saml_sso/index.md b/doc/user/group/saml_sso/index.md index 1c5e7ff0115..bd10560e138 100644 --- a/doc/user/group/saml_sso/index.md +++ b/doc/user/group/saml_sso/index.md @@ -53,7 +53,6 @@ GitLab.com uses the SAML NameID to identify users. The NameID element: also case-insensitive, which can result in users being unable to sign in. The relevant field name and recommended value for supported providers are in the [provider specific notes](#providers). -appropriate corresponding field. WARNING: Once users have signed into GitLab using the SSO SAML setup, changing the `NameID` breaks the configuration and potentially locks users out of the GitLab group. @@ -72,7 +71,7 @@ must be specified as an attribute named `email` or `mail`. You can configure the following attributes with GitLab.com Group SAML: - `username` or `nickname`. We recommend you configure only one of these. -- The [attributes available](../../../integration/saml.md#assertions) to self-managed GitLab instances. +- The [attributes available](../../../integration/saml.md#configure-assertions) to self-managed GitLab instances. ### Metadata configuration @@ -98,7 +97,7 @@ After you set up your identity provider to work with GitLab, you must configure  NOTE: -The certificate [fingerprint algorithm](../../../integration/saml.md#notes-on-configuring-your-identity-provider) must be in SHA1. When configuring the identity provider (such as [Google Workspace](#google-workspace-setup-notes)), use a secure signature algorithm. +The certificate [fingerprint algorithm](../../../integration/saml.md#configure-saml-on-your-idp) must be in SHA1. When configuring the identity provider (such as [Google Workspace](#google-workspace-setup-notes)), use a secure signature algorithm. ### Additional configuration information @@ -124,7 +123,7 @@ It can also help to compare the XML response from your provider with our [exampl > - [Improved](https://gitlab.com/gitlab-org/gitlab/-/issues/215155) in GitLab 15.5 [with a flag](../../../administration/feature_flags.md) named `transparent_sso_enforcement` to include transparent enforcement even when SSO enforcement is not enabled. Disabled on GitLab.com. FLAG: -On self-managed GitLab, transparent SSO enforcement is unavailable. On GitLab.com, transparent SSO enforcement is unavailable and can be configured by GitLab.com administrators only. +On self-managed GitLab, transparent SSO enforcement is unavailable. On GitLab.com, see the [Transparent SSO rollout](https://gitlab.com/gitlab-org/gitlab/-/issues/375788) issue for the current status. SSO is enforced when users access groups and projects in the organization's group hierarchy. Users can view other groups and projects without SSO sign in. @@ -178,13 +177,24 @@ When SSO is enforced, users are not immediately revoked. If the user: - Has an active session, they can continue accessing the group for up to 24 hours until the identity provider session times out. +### Selectively enable and disable transparent SSO enforcement + +There are two feature flags associated with this feature to allow precise control. If a customer has a problem with transparent SSO on GitLab.com, GitLab can help troubleshoot and override the feature flag as necessary. + +**`transparent_sso_enforcement`:** This feature flag should only be enabled or disabled by the Authentication and Authorization group +or in the case of a serious and widespread issue affecting many groups or users. See [issue 375788](https://gitlab.com/gitlab-org/gitlab/-/issues/375788) for the current GitLab.com rollout status. + +**`transparent_sso_enforcement_override`:** When the `transparent_sso_enforcement` feature flag is enabled, support or production teams can +turn off transparent SSO by enabling this feature flag for a specific customer group. **Enabling** this feature flag +disables transparent SSO enforcement. + ## Providers The SAML standard means that you can use a wide range of identity providers with GitLab. Your identity provider might have relevant documentation. It can be generic SAML documentation or specifically targeted for GitLab. When [configuring your identity provider](#configure-your-identity-provider), consider the notes below for specific providers to help avoid common issues and as a guide for terminology used. -For providers not listed below, you can refer to the [instance SAML notes on configuring an identity provider](../../../integration/saml.md#notes-on-configuring-your-identity-provider) +For providers not listed below, you can refer to the [instance SAML notes on configuring an identity provider](../../../integration/saml.md#configure-saml-on-your-idp) for additional guidance on information your identity provider may require. GitLab provides the following information for guidance only. @@ -338,10 +348,14 @@ When a user tries to sign in with Group SSO, GitLab attempts to find or create a ### Linking SAML to your existing GitLab.com account +> **Remember me** checkbox [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/121569) in GitLab 15.7. + To link SAML to your existing GitLab.com account: 1. Sign in to your GitLab.com account. [Reset your password](https://gitlab.com/users/password/new) if necessary. 1. Locate and visit the **GitLab single sign-on URL** for the group you're signing in to. A group owner can find this on the group's **Settings > SAML SSO** page. If the sign-in URL is configured, users can connect to the GitLab app from the identity provider. +1. Optional. Select the **Remember me** checkbox to stay signed in to GitLab for 2 weeks. You may still be asked to re-authenticate with your SAML provider + more frequently. 1. Select **Authorize**. 1. Enter your credentials on the identity provider if prompted. 1. You are then redirected back to GitLab.com and should now have access to the group. In the future, you can use SAML to sign in to GitLab.com. diff --git a/doc/user/group/saml_sso/troubleshooting.md b/doc/user/group/saml_sso/troubleshooting.md index 7dafd2c5075..f8075e62ecc 100644 --- a/doc/user/group/saml_sso/troubleshooting.md +++ b/doc/user/group/saml_sso/troubleshooting.md @@ -52,7 +52,7 @@ You can use one of the following to troubleshoot SAML: - A [quick start guide to start a Docker container](../../../administration/troubleshooting/test_environments.md#saml) with a plug and play SAML 2.0 identity provider if you only require a SAML provider. - A local environment by - [enabling SAML for groups on a self-managed instance](../../../integration/saml.md#configuring-group-saml-on-a-self-managed-gitlab-instance). + [enabling SAML for groups on a self-managed instance](../../../integration/saml.md#group-saml-on-a-self-managed-gitlab-instance). ## Verify configuration @@ -96,7 +96,14 @@ In these cases, use one of the [SAML debugging tools](#saml-debugging-tools), or a group owner can get a copy of the SAML response from when they select the "Verify SAML Configuration" button on the group SSO Settings page. -Use a base64 decoder to see a human-readable version of the SAML response. +Use a base64 decoder to see a human-readable version of the SAML response. To avoid pasting the SAML response online to decode it, you can use your +browser's console in the developers tools: + +```javascript +atob(decodeURI("<paste_SAML_response_here>")) +``` + +You should get the SAML response in XML format as output. ## Configuration errors @@ -138,7 +145,7 @@ Make sure this information is provided. Another issue that can result in this error is when the correct information is being sent by the identity provider, but the attributes don't match the names in the OmniAuth `info` hash. In this case, you must set `attribute_statements` in the SAML configuration to -[map the attribute names in your SAML Response to the corresponding OmniAuth `info` hash names](../../../integration/saml.md#attribute_statements). +[map the attribute names in your SAML Response to the corresponding OmniAuth `info` hash names](../../../integration/saml.md#map-saml-response-attribute-names). ## User sign in banner error messages @@ -221,7 +228,7 @@ If all users are receiving a `404` when attempting to sign in using SAML, confir [there is an active subscription](../../../subscriptions/gitlab_com/index.md#view-your-gitlab-saas-subscription) being used in this SAML SSO namespace. If you receive a `404` during setup when using "verify configuration", make sure you have used the correct -[SHA-1 generated fingerprint](../../../integration/saml.md#notes-on-configuring-your-identity-provider). +[SHA-1 generated fingerprint](../../../integration/saml.md#configure-saml-on-your-idp). If a user is trying to sign in for the first time and the GitLab single sign-on URL has not [been configured](index.md#configure-your-identity-provider), they may see a 404. As outlined in the [user access section](index.md#linking-saml-to-your-existing-gitlabcom-account), a group Owner needs to provide the URL to users. @@ -262,3 +269,15 @@ Pay particular attention to the following 403 errors: - `app_not_configured` - `app_not_configured_for_user` + +## SAML Name ID and email address do not match your user account **(PREMIUM SAAS)** + +If users encounter the error `SAML Name ID and email address do not match your user account. Contact an administrator.` +this means: + +- The NameID value sent by SAML does not match the existing SAML identity `extern_uid` value. +- Either the SAML response did not include an email address or the email address did not match the user's GitLab email address. + +A GitLab group Owner can use the [SAML API](../../../api/saml.md) to update the user's SAML `extern_uid`. +The `extern_uid` value must match the Name ID value sent by the SAML identity provider (IdP). Depending on the IdP configuration +this may be a generated unique ID, an email address, or other value. diff --git a/doc/user/group/settings/group_access_tokens.md b/doc/user/group/settings/group_access_tokens.md index 158e1654c6e..6e0caa633eb 100644 --- a/doc/user/group/settings/group_access_tokens.md +++ b/doc/user/group/settings/group_access_tokens.md @@ -68,6 +68,11 @@ To create a group access token: A group access token is displayed. Save the group access token somewhere safe. After you leave or refresh the page, you can't view it again. +WARNING: +Group access tokens are treated as [internal users](../../../development/internal_users.md). +If an internal user creates a group access token, that token is able to access all +groups that have visibility level set to [Internal](../../public_access.md). + ## Create a group access token using Rails console GitLab 14.6 and earlier doesn't support creating group access tokens using the UI diff --git a/doc/user/group/settings/import_export.md b/doc/user/group/settings/import_export.md index cec17688902..ff64a7dcd54 100644 --- a/doc/user/group/settings/import_export.md +++ b/doc/user/group/settings/import_export.md @@ -1,164 +1,11 @@ --- -stage: Manage -group: Import -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments +redirect_to: '../import/index.md' +remove_date: '2023-03-08' --- -# Migrating groups using file exports (deprecated) **(FREE)** +This document was moved to [another location](../import/index.md). -> - [Introduced](https://gitlab.com/groups/gitlab-org/-/epics/2888) in GitLab 13.0 as an experimental feature. May change in future releases. -> - [Deprecated](https://gitlab.com/groups/gitlab-org/-/epics/4619) in GitLab 14.6. - -WARNING: -This feature was [deprecated](https://gitlab.com/groups/gitlab-org/-/epics/4619) in GitLab 14.6 and replaced by -[a different migration method](../import/index.md). To follow progress on a solution for -[offline environments](../../application_security/offline_deployments/index.md), see -[the relevant issue](https://gitlab.com/gitlab-org/gitlab/-/issues/363406). - -You can export groups, with all their related data, from one GitLab instance to another. You can also: - -- [Migrate groups](../import/index.md) using the preferred method. -- [Migrate projects using file exports](../../project/settings/import_export.md). - -## Enable export for a group - -Prerequisite: - -- You must have the Owner role for the group. - -To enable import and export for a group: - -1. On the top bar, select **Main menu > Admin**. -1. On the left sidebar, select **Settings > General**. -1. Expand **Visibility and access controls**. -1. In the **Import sources** section, select the checkboxes for the sources you want. - -## Important Notes - -Note the following: - -- Exports are stored in a temporary directory and are deleted every 24 hours by a specific worker. -- To preserve group-level relationships from imported projects, run the Group Import/Export first, to allow projects to -be imported into the desired group structure. -- Imported groups are given a `private` visibility level, unless imported into a parent group. -- If imported into a parent group, a subgroup inherits the same level of visibility unless otherwise restricted. -- To preserve the member list and their respective permissions on imported groups, review the users in these groups. Make -sure these users exist before importing the desired groups. -- Users must set a public email in the source GitLab instance that matches one of their verified emails in the target GitLab instance. - -### Exported contents - -The [`import_export.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/import_export/group/import_export.yml) -file for groups lists many of the items exported and imported when migrating groups using file exports. View this file in the branch -for your version of GitLab to see the list of items relevant to you. For example, -[`import_export.yml` on the `14-10-stable-ee` branch](https://gitlab.com/gitlab-org/gitlab/-/blob/14-10-stable-ee/lib/gitlab/import_export/group/import_export.yml). - -Migrating projects with file exports uses the same export and import mechanisms as creating projects from templates at the [group](../custom_project_templates.md) and -[instance](../../admin_area/custom_project_templates.md) levels. Therefore, the list of exported items is the same. - -Items that are exported include: - -- Milestones -- Labels -- Boards and Board Lists -- Badges -- Subgroups (including all the aforementioned data) -- Epics - - Epic resource state events ([Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/291983) in GitLab 15.4) -- Events -- [Wikis](../../project/wiki/group.md) - ([Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/53247) in GitLab 13.9) -- Iterations cadences ([Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/95372) in 15.4) - -Items that are **not** exported include: - -- Projects -- Runner tokens -- SAML discovery tokens - -NOTE: -For more details on the specific data persisted in a group export, see the -[`import_export.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/import_export/group/import_export.yml) file. - -## Export a group - -Prerequisites: - -- You must have the Owner role for the group. - -To export the contents of a group: - -1. On the top bar, select **Main menu > Groups** and find your group. -1. On the left sidebar, select **Settings > General**. -1. In the **Advanced** section, select **Export Group**. -1. After the export is generated, you should receive an email with a link to the [exported contents](#exported-contents) - in a compressed tar archive, with contents in NDJSON format. -1. Alternatively, you can download the export from the UI: - - 1. Return to your group's **Settings > General** page. - 1. In the **Advanced** section, select **Download export**. - You can also generate a new file by selecting **Regenerate export**. - -NOTE: -The maximum import file size can be set by the Administrator, default is `0` (unlimited). -As an administrator, you can modify the maximum import file size. To do so, use the `max_import_size` option in the [Application settings API](../../../api/settings.md#change-application-settings) or the [Admin Area](../../admin_area/settings/account_and_limit_settings.md). Default [modified](https://gitlab.com/gitlab-org/gitlab/-/issues/251106) from 50MB to 0 in GitLab 13.8. - -You can also use the [group import/export API](../../../api/group_import_export.md). - -### Between CE and EE - -You can export groups from the [Community Edition to the Enterprise Edition](https://about.gitlab.com/install/ce-or-ee/) and vice versa. - -The Enterprise Edition retains some group data that isn't part of the Community Edition. If you're exporting a group from the Enterprise Edition to the Community Edition, you may lose this data. For more information, see [downgrading from EE to CE](../../../index.md). - -## Import the group - -1. Create a new group: - - On the top bar, select **New** (**{plus}**) and then **New group**. - - On an existing group's page, select the **New subgroup** button. -1. Select **Import group**. -1. Enter your group name. -1. Accept or modify the associated group URL. -1. Select **Choose file**. -1. Select the file that you exported in the [Export a group](#export-a-group) section. -1. To begin importing, select **Import group**. - -Your newly imported group page appears after the operation completes. - -## Automate group and project import **(PREMIUM)** - -For information on automating user, group, and project import API calls, see -[Automate group and project import](../../project/import/index.md#automate-group-and-project-import). - -## Version history - -### 14.0+ - -In GitLab 14.0, the JSON format is no longer supported for project and group exports. To allow for a -transitional period, you can still import any JSON exports. The new format for imports and exports -is NDJSON. - -### 13.0+ - -GitLab can import bundles that were exported from a different GitLab deployment. -This ability is limited to two previous GitLab [minor](../../../policy/maintenance.md#versioning) -releases, which is similar to our process for [Security Releases](../../../policy/maintenance.md#security-releases). - -For example: - -| Current version | Can import bundles exported from | -|-----------------|----------------------------------| -| 13.0 | 13.0, 12.10, 12.9 | -| 13.1 | 13.1, 13.0, 12.10 | - -## Rate Limits - -To help avoid abuse, by default, users are rate limited to: - -| Request Type | Limit | -| ---------------- | ---------------------------------------- | -| Export | 6 groups per minute | -| Download export | 1 download per group per minute | -| Import | 6 groups per minute | - -GitLab.com may have [different settings](../../gitlab_com/index.md#importexport) from the defaults. +<!-- This redirect file can be deleted after <2023-03-08>. --> +<!-- Redirects that point to other docs in the same project expire in three months. --> +<!-- Redirects that point to docs in a different project or site (for example, link is not relative and starts with `https:`) expire in one year. --> +<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/redirects.html --> diff --git a/doc/user/group/value_stream_analytics/index.md b/doc/user/group/value_stream_analytics/index.md index 980618ac3ae..1c02ca59e3d 100644 --- a/doc/user/group/value_stream_analytics/index.md +++ b/doc/user/group/value_stream_analytics/index.md @@ -223,7 +223,7 @@ Value stream analytics records the following times for each stage: - **Review**: 14:00 to 19:00: 5 hrs - **Staging**: 19:00 to 19:30: 30 minutes -There are some additional considerations for this example: +Keep in mind the following observations related to this example: - This example demonstrates that it doesn't matter if your first commit doesn't mention the issue number, you can do this later in any commit diff --git a/doc/user/img/markdown_logo.png b/doc/user/img/markdown_logo.png Binary files differindex 5184851b6cf..b5b08738106 100644 --- a/doc/user/img/markdown_logo.png +++ b/doc/user/img/markdown_logo.png diff --git a/doc/user/infrastructure/clusters/connect/new_gke_cluster.md b/doc/user/infrastructure/clusters/connect/new_gke_cluster.md index 6b8e2003b8d..7e6a0495d90 100644 --- a/doc/user/infrastructure/clusters/connect/new_gke_cluster.md +++ b/doc/user/infrastructure/clusters/connect/new_gke_cluster.md @@ -94,7 +94,7 @@ Use CI/CD environment variables to configure your project. 1. On the left sidebar, select **Settings > CI/CD**. 1. Expand **Variables**. 1. Set the variable `BASE64_GOOGLE_CREDENTIALS` to the `base64` encoded JSON file you just created. -1. Set the variable `TF_VAR_gcp_project` to your GCP's `project` name. +1. Set the variable `TF_VAR_gcp_project` to your GCP `project` name. 1. Set the variable `TF_VAR_agent_token` to the agent token displayed in the previous task. 1. Set the variable `TF_VAR_kas_address` to the agent server address displayed in the previous task. diff --git a/doc/user/infrastructure/clusters/manage/management_project_applications/prometheus.md b/doc/user/infrastructure/clusters/manage/management_project_applications/prometheus.md deleted file mode 100644 index 64d325dedc6..00000000000 --- a/doc/user/infrastructure/clusters/manage/management_project_applications/prometheus.md +++ /dev/null @@ -1,11 +0,0 @@ ---- -redirect_to: '../../../../../operations/metrics/index.md' -remove_date: '2022-11-03' ---- - -This document was moved to [another location](../../../../../operations/metrics/index.md). - -<!-- This redirect file can be deleted after <2022-11-03>. --> -<!-- Redirects that point to other docs in the same project expire in three months. --> -<!-- Redirects that point to docs in a different project or site (link is not relative and starts with `https:`) expire in one year. --> -<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/redirects.html --> diff --git a/doc/user/infrastructure/clusters/manage/management_project_applications/sentry.md b/doc/user/infrastructure/clusters/manage/management_project_applications/sentry.md deleted file mode 100644 index f42e9c83120..00000000000 --- a/doc/user/infrastructure/clusters/manage/management_project_applications/sentry.md +++ /dev/null @@ -1,11 +0,0 @@ ---- -redirect_to: '../../../../../operations/error_tracking.md' -remove_date: '2022-11-03' ---- - -This document was moved to [another location](../../../../../operations/error_tracking.md). - -<!-- This redirect file can be deleted after <2022-11-03>. --> -<!-- Redirects that point to other docs in the same project expire in three months. --> -<!-- Redirects that point to docs in a different project or site (link is not relative and starts with `https:`) expire in one year. --> -<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/redirects.html --> diff --git a/doc/user/infrastructure/iac/index.md b/doc/user/infrastructure/iac/index.md index 866b16652fa..f9891934067 100644 --- a/doc/user/infrastructure/iac/index.md +++ b/doc/user/infrastructure/iac/index.md @@ -72,10 +72,12 @@ To use a Terraform template: include: # To fetch the latest template, use: - template: Terraform.latest.gitlab-ci.yml + # To fetch the advanced latest template, use: + - template: Terraform/Base.latest.gitlab-ci.yml # To fetch the stable template, use: + - template: Terraform.gitlab-ci.yml + # To fetch the advanced stable template, use: - template: Terraform/Base.gitlab-ci.yml - # To fetch the advanced template, use: - - template: Terraform/Base.latest.gitlab-ci.yml ``` 1. Add the variables as described below: @@ -91,6 +93,10 @@ To use a Terraform template: 1. Optional. Override in your `.gitlab-ci.yml` file the attributes present in the template you fetched to customize your configuration. +### Terraform template recipes + +For GitLab-curated template recipes, see [Terraform template recipes](terraform_template_recipes.md). + ## Related topics - View [the images that contain the `gitlab-terraform` shell script](https://gitlab.com/gitlab-org/terraform-images). diff --git a/doc/user/infrastructure/iac/terraform_state.md b/doc/user/infrastructure/iac/terraform_state.md index a690cc78121..fc86210ed56 100644 --- a/doc/user/infrastructure/iac/terraform_state.md +++ b/doc/user/infrastructure/iac/terraform_state.md @@ -6,7 +6,11 @@ info: To determine the technical writer assigned to the Stage/Group associated w # GitLab-managed Terraform state **(FREE)** -> [Introduced](https://gitlab.com/groups/gitlab-org/-/epics/2673) in GitLab 13.0. +> - [Introduced](https://gitlab.com/groups/gitlab-org/-/epics/2673) in GitLab 13.0. +> - Support for state names that contain periods introduced in GitLab 15.7 [with a flag](../../../administration/feature_flags.md) named `allow_dots_on_tf_state_names`. Disabled by default. [Enabled on GitLab.com](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/106861) in GitLab 15.7. + +FLAG: +On self-managed GitLab, by default support for state names that contain periods is not available. To make it available, ask an administrator to [enable the feature flag](../../../administration/feature_flags.md) named `allow_dots_on_tf_state_names`. On GitLab.com, support for state names that contain periods is available. Requests for state files might generate HTTP 404 errors after enabling this feature. For more information, see [Troubleshooting the Terraform integration with GitLab](troubleshooting.md#state-not-found-if-the-state-name-contains-a-period). Terraform uses state files to store details about your infrastructure configuration. With Terraform remote [backends](https://www.terraform.io/language/settings/backends/configuration), diff --git a/doc/user/infrastructure/iac/terraform_template_recipes.md b/doc/user/infrastructure/iac/terraform_template_recipes.md new file mode 100644 index 00000000000..ab2c8c1c48a --- /dev/null +++ b/doc/user/infrastructure/iac/terraform_template_recipes.md @@ -0,0 +1,183 @@ +--- +stage: Configure +group: Configure +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments +--- + +# Terraform template recipes **(FREE)** + +You can customize your Terraform integration by adding the recipes on +this page to your pipeline. + +If you'd like to share your own Terraform configuration, consider +[contributing a recipe](https://gitlab.com/gitlab-org/gitlab/-/blob/master/doc/user/infrastructure/iac/terraform_template_recipes.md) +to this page. + +## Enable a `terraform destroy` job + +Add the following snippet to your `.gitlab-ci.yml`: + +```yaml +include: + - template: Terraform.latest.gitlab-ci.yml + +destroy: + extends: .terraform:destroy +``` + +The `destroy` job is part of the `cleanup` stage. Like the `deploy` +job, the `destroy` job is always `manual` and is not tied to the +default branch. + +## Run a custom `terraform` command in a job + +To define a job that runs a custom `terraform` command, the +`gitlab-terraform` wrapper can be used in any job: + +```yaml +include: + - template: Terraform.latest.gitlab-ci.yml + +state-list: + stage: validate # you can use any stage, just make sure to define it + script: gitlab-terraform state list +``` + +The `gitlab-terraform` command sets up a `terraform` command and runs +it with the given arguments. + +To run this job in the Terraform state-specific [resource group](../../../ci/resource_groups/index.md), +assign the job with `resource_group`: + +```yaml +include: + - template: Terraform.latest.gitlab-ci.yml + +state-list: + stage: validate # you can use any stage, just make sure to define it + resource_group: ${TF_STATE_NAME} + script: gitlab-terraform state list +``` + +## Add custom debug tools to jobs + +The default image used by Terraform template jobs contains only minimal tooling. +However, you might want to add additional tools for debugging. + +To add an additional tool: + +1. Install the tool in the `before_script` of a job or pipeline. +1. Use the tool in the `script` or `after_script` block. + - If you use the `script` block, be sure to re-add the template job commands. + +For example, the following snippet installs `bash` and `jq` in the `before_script` for all +jobs in the pipeline: + +```yaml +include: + - template: Terraform.latest.gitlab-ci.yml + +default: + before_script: apk add --update bash jq +``` + +To add it to only the `build` and `deploy` jobs, add it to those jobs directly: + +```yaml +include: + - template: Terraform.latest.gitlab-ci.yml + +build: + before_script: apk add --update bash jq + +deploy: + before_script: apk add --update bash jq +``` + +## Add custom container images + +For debug tools and simple installations, you should +[add a custom debug tool to your job](#add-custom-debug-tools-to-jobs). +If your tool is complex or benefits from caching, +you can create a custom container image based on the +[`gitlab-terraform`](https://gitlab.com/gitlab-org/terraform-images) images. +You can use your custom image in subsequent Terraform jobs. + +To define a custom container image: + +1. Define a new `Dockerfile` with custom tooling. For example, install `bash` and `jq` in `.gitlab/ci/Dockerfile`: + + ```dockerfile + FROM registry.gitlab.com/gitlab-org/terraform-images/stable:latest + + RUN apk add --update bash jq + ``` + +1. In a new job, define a `prepare` stage that builds the image whenever the `Dockerfile` changes. + - The built image is pushed to the [GitLab Container Registry](../../packages/container_registry). A tag is applied to indicate whether the image was built from a merge request or from the default branch. +1. Use your image in your Terraform jobs, such as `build` and `deploy`. + - You can combine your image with specialized `before_script` configurations to perform setup commands, like to generate inputs for Terraform. + +For example, a fully functioning pipeline configuration might look like: + +```yaml +include: + - template: Terraform.latest.gitlab-ci.yml + +variables: + IMAGE_TAG: latest + +workflow: + rules: + - if: $CI_MERGE_REQUEST_IID + changes: + - .gitlab/ci/Dockerfile + variables: + IMAGE_TAG: ${CI_COMMIT_REF_SLUG} + - when: always + +stages: + - prepare + - validate + - test + - build + - deploy + - cleanup + +prepare:image: + needs: [] + stage: prepare + image: + name: gcr.io/kaniko-project/executor:v1.9.0-debug + entrypoint: [""] + rules: + # Tag with the commit SHA if we're in an MR + - if: $CI_MERGE_REQUEST_IID + changes: + - .gitlab/ci/Dockerfile + variables: + DOCKER_TAG: $CI_COMMIT_REF_SLUG + # If we're on our main branch, tag with "latest" + - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH + changes: + - .gitlab/ci/Dockerfile + variables: + DOCKER_TAG: latest + before_script: + # Authenticate to the docker registry and dependency proxy + - echo "{\"auths\":{\"$CI_REGISTRY\":{\"auth\":\"$(printf "%s:%s" "${CI_REGISTRY_USER}" "${CI_REGISTRY_PASSWORD}" | base64 | tr -d '\n')\"}}}" > /kaniko/.docker/config.json + script: + - /kaniko/executor + --context "${CI_PROJECT_DIR}/.gitlab/ci" + --cache=true + --dockerfile "${CI_PROJECT_DIR}/.gitlab/ci/Dockerfile" + --destination "${CI_REGISTRY_IMAGE}:${DOCKER_TAG}" + +build: + image: ${CI_REGISTRY_IMAGE}:${IMAGE_TAG} + +deploy: + image: ${CI_REGISTRY_IMAGE}:${IMAGE_TAG} +``` + +For an example repository, see the [GitLab Terraform template usage project](https://gitlab.com/gitlab-org/configure/examples/terraform-template-usage). diff --git a/doc/user/infrastructure/iac/troubleshooting.md b/doc/user/infrastructure/iac/troubleshooting.md index 3921c6a7dc8..ad1821fbe10 100644 --- a/doc/user/infrastructure/iac/troubleshooting.md +++ b/doc/user/infrastructure/iac/troubleshooting.md @@ -131,3 +131,41 @@ To permit a user with the Developer role to run destructive commands, you need a 1. Set the value of `TF_USERNAME` to the username of your project access token. 1. Set the value of `TF_PASSWORD` to the password of your project access token. 1. Optional. Protect the variables to make them only available in pipelines that run on protected branches or protected tags. + +### State not found if the state name contains a period + +GitLab 15.6 and earlier returned 404 errors if the state name contained a period and Terraform attempted +a state lock. + +You could work around this limitation by adding `-lock=false` to your Terraform commands. The GitLab backend +accepted the request, but internally stripped the period and any characters that followed from the state name. +For example, a state named `foo.bar` would be stored as `foo`. However, this workaround wasn't recommended, +and could even cause state name collisions. + +In GitLab 15.7 and later, [state names with periods are supported](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/106861). If you use the `-lock=false` workaround and upgrade to GitLab 15.7 or later, +your jobs might fail. The failure is caused by the GitLab backend storing a new state with the full state name, which diverges from the existing state name. + +To fix the failing jobs, rename your state names to exclude the period and any characters that follow it. For example, +if you use the Terraform template: + +```yaml +include: + - template: Terraform.gitlab-ci.yml + +variables: + TF_STATE_NAME: foo +``` + +If your `TF_HTTP_ADDRESS`, `TF_HTTP_LOCK_ADDRESS` and `TF_HTTP_UNLOCK_ADDRESS` are set, be sure +to update the state names there. + +Alternatively, you can [migrate your terraform state](terraform_state.md#migrate-to-a-gitlab-managed-terraform-state). + +#### Self-managed GitLab instances + +By default, support for state names with periods is not enabled on self-managed GitLab. +You can enable it from the Rails console: + +```ruby +Feature.enable(:allow_dots_on_tf_state_names) +``` diff --git a/doc/user/markdown.md b/doc/user/markdown.md index b6f3ba1cfdd..17b91eb9483 100644 --- a/doc/user/markdown.md +++ b/doc/user/markdown.md @@ -1,6 +1,6 @@ --- -stage: Create -group: Source Code +stage: Plan +group: Project Management info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- @@ -211,7 +211,7 @@ You can use it to point out a :bug: or warn about :speak_no_evil: patches. And i If you're new to this, don't be :fearful:. You can join the emoji :family:. Just look up one of the supported codes. -Consult the [Emoji Cheat Sheet](https://www.emojicopy.com) for a list of all supported emoji codes. :thumbsup: +Consult the [Emoji Cheat Sheet](https://www.webfx.com/tools/emoji-cheat-sheet/) for a list of all supported emoji codes. :thumbsup: ``` Sometimes you want to <img src="https://gitlab.com/gitlab-org/gitlab-foss/raw/master/public/-/emojis/2/monkey.png" width="20px" height="20px" style="display:inline;margin:0;border: 0"> around a bit and add some <img src="https://gitlab.com/gitlab-org/gitlab-foss/raw/master/public/-/emojis/2/star2.png" width="20px" height="20px" style="display:inline;margin:0;border: 0"> to your <img src="https://gitlab.com/gitlab-org/gitlab-foss/raw/master/public/-/emojis/2/speech_balloon.png" width="20px" height="20px" style="display:inline;margin:0;border: 0">. Well we have a gift for you: @@ -374,12 +374,12 @@ a^2+b^2=c^2 #### LaTeX-compatible fencing -> Introduced in GitLab 15.4 [with a flag](../administration/feature_flags.md) named `markdown_dollar_math`. Disabled by default. +> Introduced in GitLab 15.4 [with a flag](../administration/feature_flags.md) named `markdown_dollar_math`. Disabled by default. Enabled on GitLab.com. [View this topic in GitLab](https://gitlab.com/gitlab-org/gitlab/-/blob/master/doc/user/markdown.md#latex-compatible-fencing). FLAG: -On self-managed GitLab, by default this feature is not available. To make it available, +On self-managed GitLab, by default this feature is not available. To make it available per group, ask an administrator to [enable the feature flag](../administration/feature_flags.md) named `markdown_dollar_math`. On GitLab.com, this feature is available. The feature is not ready for production use. @@ -1017,8 +1017,25 @@ Do not change to a reference style link.  -In the rare case where you must set a specific height or width for an image, -you can use the `img` HTML tag instead of Markdown and set its `height` and +#### Change the image dimensions + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/28118) in GitLab 15.7. + +You can control the width and height of an image by following the image with +an attribute list. +The value must an integer with a unit of either `px` (default) or `%`. + +For example + +```markdown +{width=100 height=100px} + +{width=75%} +``` + +{width=100 height=100px} + +You can also use the `img` HTML tag instead of Markdown and set its `height` and `width` parameters. #### Videos @@ -1283,7 +1300,7 @@ Some text to show that the reference links can follow later. Using header ID anchors: -- This line links to [a section on a different Markdown page, using a "#" and the header ID](permissions.md#project-features-permissions) +- This line links to [a section on a different Markdown page, using a "#" and the header ID](permissions.md#project-members-permissions) - This line links to [a different section on the same page, using a "#" and the header ID](#header-ids-and-links) Using references: @@ -1530,6 +1547,7 @@ Tables are not part of the core Markdown specification, but are part of GitLab F by pipes (`|`). - You **can** have blank cells. 1. Column widths are calculated dynamically based on the content of the cells. +1. To use the pipe character (`|`) in the text and not as table delimiter, you must escape it with a backslash (`\|`). Example: @@ -1581,7 +1599,8 @@ use `<br>` tags to force a cell to have multiple lines: | Item2 | This item has:<br>- Multiple items<br>- That we want listed separately | You can use HTML formatting in GitLab itself to add [task lists](#task-lists) with checkboxes, -but they do not render properly on `docs.gitlab.com`: +but they do not render properly on `docs.gitlab.com`. Note that these tasks will not save their +state when selected, like regular GitLab task lists. ```markdown | header 1 | header 2 | @@ -1590,6 +1609,31 @@ but they do not render properly on `docs.gitlab.com`: | cell 3 | <ul><li> - [ ] Task one </li><li> - [ ] Task two </li></ul> | ``` +To have fully functioning task lists in a table, create an HTML table with Markdown in the cells: + +```html +<table> +<thead> +<tr><th>header 1</th><th>header 2</th></tr> +</thead> +<tbody> +<tr> +<td>cell 1</td> +<td>cell 2</td> +</tr> +<tr> +<td>cell 3</td> +<td> + +- [ ] Task one +- [ ] Task two + +</td> +</tr> +</tbody> +</table> +``` + ##### Copy and paste from a spreadsheet > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/27205) in GitLab 12.7. diff --git a/doc/user/namespace/index.md b/doc/user/namespace/index.md index 36e048868ad..7d26600cc38 100644 --- a/doc/user/namespace/index.md +++ b/doc/user/namespace/index.md @@ -9,20 +9,24 @@ info: To determine the technical writer assigned to the Stage/Group associated w In GitLab, a *namespace* provides one place to organize your related projects. Projects in one namespace are separate from projects in other namespaces, which means you can use the same name for projects in different namespaces. +## Types of namespaces + GitLab has two types of namespaces: -- A *Personal* namespace, which is based on your username and provided to you when you create your account. - - If you change your username, the project and namespace URLs in your account also change. Before you change your username, - read about [repository redirects](../project/repository/index.md#what-happens-when-a-repository-path-changes). +- A *personal* namespace, which is based on your username and provided to you when you create your account. - You cannot create subgroups in a personal namespace. - Groups in your namespace do not inherit your namespace permissions and group features. - - All the *Personal Projects* created will fall under the scope of this namespace. + - All the projects you create are under the scope of this namespace. + - If you change your username, the project and namespace URLs in your account also change. Before you change your username, + read about [repository redirects](../project/repository/index.md#what-happens-when-a-repository-path-changes). -- A *group* or *subgroup* namespace: +- A *group* or *subgroup* namespace, which is based on the group or subgroup name: - You can create multiple subgroups to manage multiple projects. - - You can change the URL of group and subgroup namespaces. - You can configure settings specifically for each subgroup and project in the namespace. - When you create a subgroup, it inherits some of the parent group settings. You can view these in the subgroup **Settings**. + - You can change the URL of group and subgroup namespaces. + +## Determine which type of namespace you're viewing To determine whether you're viewing a group or personal namespace, you can view the URL. For example: diff --git a/doc/user/operations_dashboard/index.md b/doc/user/operations_dashboard/index.md index 0a65ea0f64d..619d822adf2 100644 --- a/doc/user/operations_dashboard/index.md +++ b/doc/user/operations_dashboard/index.md @@ -1,6 +1,6 @@ --- -stage: none -group: unassigned +stage: Release +group: Release info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- diff --git a/doc/user/packages/conan_repository/index.md b/doc/user/packages/conan_repository/index.md index 3d3fe35fd65..dd6605c2f01 100644 --- a/doc/user/packages/conan_repository/index.md +++ b/doc/user/packages/conan_repository/index.md @@ -270,8 +270,9 @@ Prerequisites: ``` NOTE: -If you try to install the package you just created in this tutorial, the package -already exists on your local computer, so this command has no effect. +If you try installing the package you created in this tutorial, the install command +will have no effect because the package already exists. +Delete `~/.conan/data` to clean up the packages stored in the cache. ## Remove a Conan package diff --git a/doc/user/packages/container_registry/index.md b/doc/user/packages/container_registry/index.md index 65e7918d827..4b4d6190dc2 100644 --- a/doc/user/packages/container_registry/index.md +++ b/doc/user/packages/container_registry/index.md @@ -6,11 +6,11 @@ info: To determine the technical writer assigned to the Stage/Group associated w # GitLab Container Registry **(FREE)** -> - The group-level Container Registry was [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/23315) in GitLab 12.10. -> - Searching by image repository name was [introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/31322) in GitLab 13.0. +> Searching by image repository name was [introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/31322) in GitLab 13.0. NOTE: -If you pull container images from Docker Hub, you can also use the [GitLab Dependency Proxy](../dependency_proxy/index.md#use-the-dependency-proxy-for-docker-images) to avoid running into rate limits and speed up your pipelines. +If you pull container images from Docker Hub, you can use the [GitLab Dependency Proxy](../dependency_proxy/index.md#use-the-dependency-proxy-for-docker-images) +to avoid rate limits and speed up your pipelines. With the Docker Container Registry integrated into GitLab, every GitLab project can have its own space to store its Docker images. @@ -28,26 +28,26 @@ You can view the Container Registry for a project or group. 1. Go to your project or group. 1. Go to **Packages and registries > Container Registry**. -You can search, sort, filter, and [delete](#delete-images-from-within-gitlab) +You can search, sort, filter, and [delete](#delete-images-using-the-gitlab-ui) containers on this page. You can share a filtered view by copying the URL from your browser. Only members of the project or group can access a private project's Container Registry. +Images downloaded from a private registry may be [available to other users in a shared runner](https://docs.gitlab.com/runner/security/index.html#usage-of-private-docker-images-with-if-not-present-pull-policy). If a project is public, so is the Container Registry. ### View the tags of a specific image -You can view a list of tags associated with a given container image: +You can use the Container Registry **Tag Details** page to view a list of tags associated with a given container image: 1. Go to your project or group. 1. Go to **Packages and registries > Container Registry**. 1. Select the container image you are interested in. -This brings up the Container Registry **Tag Details** page. You can view details about each tag, -such as when it was published, how much storage it consumes, and the manifest and configuration -digests. +You can view details about each tag, such as when it was published, how much storage it consumes, +and the manifest and configuration digests. -You can search, sort (by tag name), filter, and [delete](#delete-images-from-within-gitlab) +You can search, sort (by tag name), filter, and [delete](#delete-images-using-the-gitlab-ui) tags on this page. You can share a filtered view by copying the URL from your browser. ## Use images from the Container Registry @@ -67,7 +67,7 @@ To download and run a container image hosted in the GitLab Container Registry: docker run [options] registry.example.com/group/project/image [arguments] ``` -[Authentication](#authenticate-with-the-container-registry) is needed to download images from private repository. +[Authentication](#authenticate-with-the-container-registry) is needed to download images from a private repository. For more information on running Docker containers, visit the [Docker documentation](https://docs.docker.com/get-started/). @@ -85,7 +85,7 @@ then your image must be named `gitlab.example.com/mynamespace/myproject` at a mi You can append additional names to the end of an image name, up to two levels deep. -For example, these are all valid image names for images within the project named `myproject`: +For example, these are all valid image names for images in the project named `myproject`: ```plaintext registry.example.com/mynamespace/myproject:some-tag @@ -192,12 +192,12 @@ You can configure your `.gitlab-ci.yml` file to build and push images to the Con - Before building, use `docker build --pull` to fetch changes to base images. It takes slightly longer, but it ensures your image is up-to-date. - Before each `docker run`, do an explicit `docker pull` to fetch - the image that was just built. This is especially important if you are + the image that was just built. This step is especially important if you are using multiple runners that cache images locally. If you use the Git SHA in your image tag, each job is unique and you should never have a stale image. However, it's still possible to have a - stale image if you re-build a given commit after a dependency has changed. + stale image if you rebuild a given commit after a dependency has changed. - Don't build directly to the `latest` tag because multiple jobs may be happening simultaneously. @@ -234,12 +234,12 @@ build: - docker push $IMAGE_TAG ``` -Here, `$CI_REGISTRY_IMAGE` would be resolved to the address of the registry tied -to this project. Since `$CI_COMMIT_REF_NAME` resolves to the branch or tag name, -and your branch name can contain forward slashes (for example, `feature/my-feature`), it is -safer to use `$CI_COMMIT_REF_SLUG` as the image tag. This is due to that image tags -cannot contain forward slashes. We also declare our own variable, `$IMAGE_TAG`, -combining the two to save us some typing in the `script` section. +In this example, `$CI_REGISTRY_IMAGE` resolves to the address of the registry tied +to this project. `$CI_COMMIT_REF_NAME` resolves to the branch or tag name, which +can contain forward slashes. Image tags can't contain forward slashes. Use +`$CI_COMMIT_REF_SLUG` as the image tag. You can declare the variable, `$IMAGE_TAG`, +combining `$CI_REGISTRY_IMAGE` and `$CI_REGISTRY_IMAGE` to save some typing in the +`script` section. Here's a more elaborate example that splits up the tasks into 4 pipeline stages, including two tests that run in parallel. The `build` is stored in the container @@ -385,17 +385,18 @@ unreferenced, administrators must run [garbage collection](../../../administrati On GitLab.com, the latest version of the Container Registry includes an automatic online garbage collector. For more information, see [this blog post](https://about.gitlab.com/blog/2021/10/25/gitlab-com-container-registry-update/). -This is an instance-wide feature, rolling out gradually to a subset of the user base, so some new image repositories created -from GitLab 14.5 onwards are served by this new version of the Container Registry. In this new -version of the Container Registry, layers that aren't referenced by any image manifest, and image -manifests that have no tags and aren't referenced by another manifest (such as multi-architecture -images), are automatically scheduled for deletion after 24 hours if left unreferenced. +The automatic online garbage collector is an instance-wide feature, rolling out gradually to a subset +of the user base. Some new image repositories created from GitLab 14.5 onward are served by this +new version of the Container Registry. In this new version of the Container Registry, layers that aren't +referenced by any image manifest, and image manifests that have no tags and aren't referenced by another +manifest (such as multi-architecture images), are automatically scheduled for deletion after 24 hours if +left unreferenced. -### Delete images from within GitLab +### Delete images using the GitLab UI -To delete images from within GitLab: +To delete images using the GitLab UI: -1. Navigate to your project's or group's **Packages and registries > Container Registry**. +1. Go to your project's or group's **Packages and registries > Container Registry**. 1. From the **Container Registry** page, you can select what you want to delete, by either: @@ -419,7 +420,7 @@ information, see the following endpoints: ### Delete images using GitLab CI/CD WARNING: -GitLab CI/CD doesn't provide a built-in way to remove your images, but this example +GitLab CI/CD doesn't provide a built-in way to remove your images. This example uses a third-party tool called [reg](https://github.com/genuinetools/reg) that talks to the GitLab Registry API. You are responsible for your own actions. For assistance with this tool, see @@ -455,21 +456,18 @@ build_image: - main delete_image: - image: docker:20.10.16 + before_script: + - curl --fail --show-error --location "https://github.com/genuinetools/reg/releases/download/v$REG_VERSION/reg-linux-amd64" --output ./reg + - echo "$REG_SHA256 ./reg" | sha256sum -c - + - chmod a+x ./reg + image: curlimages/curl:7.86.0 + script: + - ./reg rm -d --auth-url $CI_REGISTRY -u $CI_REGISTRY_USER -p $CI_REGISTRY_PASSWORD $IMAGE_TAG stage: clean - services: - - docker:20.10.16-dind variables: IMAGE_TAG: $CI_PROJECT_PATH:$CI_COMMIT_REF_SLUG REG_SHA256: ade837fc5224acd8c34732bf54a94f579b47851cc6a7fd5899a98386b782e228 REG_VERSION: 0.16.1 - before_script: - - apk add --no-cache curl - - curl --fail --show-error --location "https://github.com/genuinetools/reg/releases/download/v$REG_VERSION/reg-linux-amd64" --output /usr/local/bin/reg - - echo "$REG_SHA256 /usr/local/bin/reg" | sha256sum -c - - - chmod a+x /usr/local/bin/reg - script: - - /usr/local/bin/reg rm -d --auth-url $CI_REGISTRY -u $CI_REGISTRY_USER -p $CI_REGISTRY_PASSWORD $IMAGE_TAG only: - branches except: @@ -487,16 +485,14 @@ defined in the `delete_image` job. You can create a per-project [cleanup policy](reduce_container_registry_storage.md#cleanup-policy) to ensure older tags and images are regularly removed from the Container Registry. -## Limitations +## Known issues -- Moving or renaming existing Container Registry repositories is not supported - once you have pushed images, because the images are stored in a path that matches - the repository path. To move or rename a repository with a - Container Registry, you must delete all existing images. - Community suggestions to work around this limitation have been shared in - [issue 18383](https://gitlab.com/gitlab-org/gitlab/-/issues/18383#possible-workaround). -- Prior to GitLab 12.10, any tags that use the same image ID as the `latest` tag - are not deleted by the cleanup policy. +Moving or renaming existing Container Registry repositories is not supported +after you have pushed images. The images are stored in a path that matches +the repository path. To move or rename a repository with a +Container Registry, you must delete all existing images. +Community suggestions to work around this known issue have been shared in +[issue 18383](https://gitlab.com/gitlab-org/gitlab/-/issues/18383#possible-workaround). ## Disable the Container Registry for a project @@ -530,7 +526,7 @@ for more details about the permissions that this setting grants to users. is internal or private, the Container Registry is also internal or private. - **Only Project Members**: The Container Registry is visible only to project members with - Reporter role or higher. This is similar to the behavior of a private project with Container + Reporter role or higher. This visibility is similar to the behavior of a private project with Container Registry visibility set to **Everyone With Access**. 1. Select **Save changes**. @@ -541,7 +537,7 @@ The ability to view the Container Registry and pull images is controlled by the visibility permissions. You can change this through the [visibility setting on the UI](#change-visibility-of-the-container-registry) or the [API](../../../api/container_registry.md#change-the-visibility-of-the-container-registry). [Other permissions](../../permissions.md) -such as updating the Container Registry, pushing or deleting images, and so on are not affected by +such as updating the Container Registry and pushing or deleting images are not affected by this setting. However, disabling the Container Registry disables all Container Registry operations. | | | Anonymous<br/>(Everyone on internet) | Guest | Reporter, Developer, Maintainer, Owner | @@ -553,188 +549,3 @@ this setting. However, disabling the Container Registry disables all Container R | Private project with Container Registry visibility <br/> set to **Everyone With Access** (UI) or `enabled` (API) | View Container Registry <br/> and pull images | No | No | Yes | | Private project with Container Registry visibility <br/> set to **Only Project Members** (UI) or `private` (API) | View Container Registry <br/> and pull images | No | No | Yes | | Any project with Container Registry `disabled` | All operations on Container Registry | No | No | No | - -## Troubleshooting the GitLab Container Registry - -### Migrating OCI container images to GitLab Container Registry - -Migrating built container images to the GitLab registry is not a current feature. However, an [epic](https://gitlab.com/groups/gitlab-org/-/epics/5210) is open to track the work on this feature. - -Some third-party tools can help migrate container images, for example, [skopeo](https://github.com/containers/skopeo), which can [copy container images](https://github.com/containers/skopeo#copying-images) between various storage mechanisms. You can use skopeo to copy from container registries, container storage backends, local directories, and local OCI-layout directories to the GitLab Container Registry. - -### Docker connection error - -A Docker connection error can occur when there are special characters in either the group, -project or branch name. Special characters can include: - -- Leading underscore -- Trailing hyphen/dash - -To get around this, you can [change the group path](../../group/manage.md#change-a-groups-path), -[change the project path](../../project/settings/index.md#rename-a-repository) or change the branch -name. - -You may also get a `404 Not Found` or `Unknown Manifest` message if you are using -a Docker Engine version earlier than 17.12. Later versions of Docker Engine use -[the v2 API](https://docs.docker.com/registry/spec/manifest-v2-2/). - -The images in your GitLab Container Registry must also use the Docker v2 API. -For information on how to update your images, see the [Docker help](https://docs.docker.com/registry/spec/deprecated-schema-v1). - -### `Blob unknown to registry` error when pushing a manifest list - -When [pushing a Docker manifest list](https://docs.docker.com/engine/reference/commandline/manifest/#create-and-push-a-manifest-list) -to the GitLab Container Registry, you may receive the error -`manifest blob unknown: blob unknown to registry`. This is likely caused by having multiple images -with different architectures, spread out over several repositories instead of the same repository. - -For example, you may have two images, each representing an architecture: - -- The `amd64` platform -- The `arm64v8` platform - -To build a multi-arch image with these images, you must push them to the same repository as the -multi-arch image. - -To address the `Blob unknown to registry` error, include the architecture in the tag name of -individual images. For example, use `mygroup/myapp:1.0.0-amd64` and `mygroup/myapp:1.0.0-arm64v8`. -You can then tag the manifest list with `mygroup/myapp:1.0.0`. - -### Troubleshoot as a GitLab server administrator - -Troubleshooting the GitLab Container Registry, most of the times, requires -you to sign in to GitLab server with administrator access. - -[Read how to troubleshoot the Container Registry](../../../administration/packages/container_registry.md#troubleshooting). - -### Unable to change path or transfer a project - -If you try to change a project's path or transfer a project to a new namespace, -you may receive one of the following errors: - -- "Project cannot be transferred, because tags are present in its container registry." -- "Namespace cannot be moved because at least one project has tags in container registry." - -This issue occurs when the project has images in the Container Registry. -You must delete or move these images before you can change the path or transfer -the project. - -The following procedure uses these sample project names: - -- For the current project: `gitlab.example.com/org/build/sample_project/cr:v2.9.1` -- For the new project: `gitlab.example.com/new_org/build/new_sample_project/cr:v2.9.1` - -Use your own URLs to complete the following steps: - -1. Download the Docker images on your computer: - - ```shell - docker login gitlab.example.com - docker pull gitlab.example.com/org/build/sample_project/cr:v2.9.1 - ``` - - NOTE: - For container registry authentication, use either a - [personal access token](../../profile/personal_access_tokens.md) or a - [deploy token](../../project/deploy_tokens/index.md). - -1. Rename the images to match the new project name: - - ```shell - docker tag gitlab.example.com/org/build/sample_project/cr:v2.9.1 gitlab.example.com/new_org/build/new_sample_project/cr:v2.9.1 - ``` - -1. Delete the images in the old project by using the [UI](#delete-images) or [API](../../../api/packages.md#delete-a-project-package). - There may be a delay while the images are queued and deleted. -1. Change the path or transfer the project by going to **Settings > General** - and expanding **Advanced**. -1. Restore the images: - - ```shell - docker push gitlab.example.com/new_org/build/new_sample_project/cr:v2.9.1 - ``` - -Follow [this issue](https://gitlab.com/gitlab-org/gitlab/-/issues/18383) for details. - -### Tags on S3 backend remain after successful deletion requests - -With S3 as your storage backend, tags may remain even though: - -- In the UI, you see that the tags are scheduled for deletion. -- In the API, you get an HTTP `200` response. -- The registry log shows a successful `Delete` request. - -An example `DELETE` request in the registry log: - -```shell -{"content_type":"","correlation_id":"01FQGNSKVMHQEAVE21KYTJN2P4","duration_ms":62,"host":"localhost:5000","level":"info","method":"DELETE","msg":"access","proto":"HTTP/1.1","referrer":"","remote_addr":"127.0.0.1:47498","remote_ip":"127.0.0.1","status":202,"system":"http","time":"2021-12-22T08:58:15Z","ttfb_ms":62,"uri":"/v2/<path to repo>/tags/reference/<tag_name>","user_agent":"GitLab/<version>","written_bytes":0} -``` - -There may be some errors not properly cached. Follow these steps to investigate further: - -1. In your configuration file, set the registry's log level to `debug`, and the S3 driver's log - level to `logdebugwithhttpbody`. For Omnibus, make these edits in the `gitlab.rb` file: - - ```shell - # Change the registry['log_level'] to debug - registry['log_level'] = 'debug' - - # Set log level for registry log from storage side - registry['storage'] = { - 's3' => { - 'bucket' => 'your-s3-bucket', - 'region' => 'your-s3-region' - }, - - 'loglevel' = "logdebugwithhttpbody" - } - ``` - - Then save and reconfigure GitLab: - - ```shell - sudo gitlab-ctl reconfigure - ``` - -1. Attempt to delete one or more tags using the GitLab UI or API. - -1. Inspect the registry logs and look for a response from S3. Although the response could be - `200 OK`, the body might have the error `AccessDenied`. This indicates a permission problem from - the S3 side. - -1. Ensure your S3 configuration has the `deleteObject` permission scope. Here's an - [example role for an S3 bucket](../../../administration/object_storage.md#iam-permissions). - Once adjusted, trigger another tag deletion. You should be able to successfully delete tags. - -Follow [this issue](https://gitlab.com/gitlab-org/container-registry/-/issues/551) for details. - -### Tags temporarily cannot be marked for deletion - -GitLab is [migrating to the next generation of the Container Registry](https://gitlab.com/groups/gitlab-org/-/epics/5523). -During the migration, you may encounter difficulty deleting tags. -If you encounter an error, it's likely that your image repository is in the process of being migrated. -Wait a few minutes and try again. - -### `unauthorized: authentication required` when pushing large images - -When pushing large images, you might get an error like the following: - -```shell -docker push gitlab.example.com/myproject/docs:latest -The push refers to a repository [gitlab.example.com/myproject/docs] -630816f32edb: Preparing -530d5553aec8: Preparing -... -4b0bab9ff599: Waiting -d1c800db26c7: Waiting -42755cf4ee95: Waiting -unauthorized: authentication required -``` - -On self-managed GitLab instances, by default, tokens for the Container Registry expire every five minutes. -When pushing larger images, or images that take longer than five minutes to push, -you might encounter this error. On GitLab.com, the expiration time is 15 minutes. - -If you are using self-managed GitLab, you can ask an administrator to -[increase the token duration](../../../administration/packages/container_registry.md#increase-token-duration) -if necessary. diff --git a/doc/user/packages/container_registry/reduce_container_registry_data_transfer.md b/doc/user/packages/container_registry/reduce_container_registry_data_transfer.md index 74cbcba2ffc..0ce9635e05a 100644 --- a/doc/user/packages/container_registry/reduce_container_registry_data_transfer.md +++ b/doc/user/packages/container_registry/reduce_container_registry_data_transfer.md @@ -44,7 +44,7 @@ Consider using a smaller base image, such as [Alpine Linux](https://alpinelinux. An Alpine image is around 5MB, which is several times smaller than popular base images such as [Debian](https://hub.docker.com/_/debian). If your application is distributed as a self-contained static binary, such as for Go applications, -you can also consider using Docker's [scratch](https://hub.docker.com/_/scratch/) +you can also consider using the Docker [scratch](https://hub.docker.com/_/scratch/) base image. If you need to use a specific base image OS, look for `-slim` or `-minimal` variants, as this helps diff --git a/doc/user/packages/container_registry/reduce_container_registry_storage.md b/doc/user/packages/container_registry/reduce_container_registry_storage.md index 23d835ddf5f..cbf9af633ac 100644 --- a/doc/user/packages/container_registry/reduce_container_registry_storage.md +++ b/doc/user/packages/container_registry/reduce_container_registry_storage.md @@ -4,17 +4,18 @@ group: Container Registry info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- -# Reduce Container Registry Storage **(FREE)** +# Reduce Container Registry storage **(FREE)** -Container registries become large over time without cleanup. When a large number of images or tags are added: +Container registries can grow in size over time if you don't manage your registry usage. For example, +if you add a large number of images or tags: -- Fetching the list of available tags or images becomes slower. +- Retrieving the list of available tags or images becomes slower. - They take up a large amount of storage space on the server. -We recommend deleting unnecessary images and tags and setting up a [cleanup policy](#cleanup-policy) +You should delete unnecessary images and tags and set up a [cleanup policy](#cleanup-policy) to automatically manage your container registry usage. -## Check Container Registry Storage Use +## Check Container Registry storage use The Usage Quotas page (**Settings > Usage Quotas > Storage**) displays storage usage for Packages. This page includes the Container Registry usage, which is only available on GitLab.com. @@ -23,10 +24,16 @@ metadata database. Support for improvements is proposed in epic [5523](https://g You cannot use the Container Registry in self-managed instances, but epic [5521](https://gitlab.com/groups/gitlab-org/-/epics/5521) proposes to change this behavior. Image layers stored in the Container Registry are deduplicated at the root namespace level. -If you tag the same image more than once in the same repository or across distinct -repositories under the same root namespace, it is only counted once. -If an image layer is shared across multiple images, in the same -container repository, project, group, or across different repositories, it is only counted once. + +An image is only counted once if: + +- You tag the same image more than once in the same repository. +- You tag the same image across distinct repositories under the same root namespace. + +An image layer is only counted once if: + +- You share the image layer across multiple images in the same container repository, project, or group. +- You share the image layer across different repositories. Only layers that are referenced by tagged images are accounted for. Untagged images and any layers referenced exclusively by them are subject to [online garbage collection](index.md#delete-images). @@ -50,7 +57,7 @@ To delete the underlying layers and images that aren't associated with any tags, ### Enable the cleanup policy -Cleanup policies can be run on all projects, with these exceptions: +You can run cleanup policies on all projects with these exceptions: - For self-managed GitLab instances, the project must have been created in GitLab 12.8 or later. However, an administrator can enable the cleanup policy @@ -63,7 +70,7 @@ Cleanup policies can be run on all projects, with these exceptions: ApplicationSetting.last.update(container_expiration_policies_enable_historic_entries: true) ``` - Enabling cleanup policies on all project can impact performance, especially if you + Enabling cleanup policies on all projects can impact performance, especially if you are using an [external registry](#use-with-external-container-registries). WARNING: @@ -72,34 +79,34 @@ GitLab.com that don't have a container image. ### How the cleanup policy works -The cleanup policy collects all tags in the Container Registry and excludes tags -until only the tags to be deleted remain. +The cleanup policy collects all tags in the Container Registry and excludes tags until the only +tags you want to delete remain. The cleanup policy searches for images based on the tag name. Support for full path matching is tracked in issue [281071](https://gitlab.com/gitlab-org/gitlab/-/issues/281071). The cleanup policy: 1. Collects all tags for a given repository in a list. -1. Excludes the tag named `latest` from the list. -1. Evaluates the `name_regex` (tags to expire), excluding non-matching names from the list. -1. Excludes from the list any tags matching the `name_regex_keep` value (tags to preserve). +1. Excludes the tag named `latest`. +1. Evaluates the `name_regex` (tags to expire), excluding non-matching names. +1. Excludes any tags matching the `name_regex_keep` value (tags to preserve). 1. Excludes any tags that do not have a manifest (not part of the options in the UI). 1. Orders the remaining tags by `created_date`. -1. Excludes from the list the N tags based on the `keep_n` value (Number of tags to retain). -1. Excludes from the list the tags more recent than the `older_than` value (Expiration interval). -1. Finally, the remaining tags in the list are deleted from the Container Registry. +1. Excludes the N tags based on the `keep_n` value (Number of tags to retain). +1. Excludes the tags more recent than the `older_than` value (Expiration interval). +1. Deletes the remaining tags in the list from the Container Registry. WARNING: On GitLab.com, the execution time for the cleanup policy is limited. Some tags may remain in the Container Registry after the policy runs. The next time the policy runs, the remaining tags are included. -It may take multiple runs for all tags to be deleted. +It may take multiple runs to delete all tags. WARNING: GitLab self-managed installations support third-party container registries that comply with the [Docker Registry HTTP API V2](https://docs.docker.com/registry/spec/api/) -specification. However, this specification does not include a tag delete operation. Therefore, when -interacting with third-party container registries, GitLab uses a workaround to delete tags. See the -[related issue](https://gitlab.com/gitlab-org/gitlab/-/issues/15737) +specification. However, this specification does not include a tag delete operation. Therefore, GitLab uses a +workaround to delete tags when interacting with third-party container registries. Refer to +issue [15737](https://gitlab.com/gitlab-org/gitlab/-/issues/15737) for more information. Due to possible implementation variations, this workaround is not guaranteed to work with all third-party registries in the same predictable way. If you use the GitLab Container Registry, this workaround is not required because we implemented a special tag delete operation. In @@ -115,18 +122,18 @@ To create a cleanup policy in the UI: 1. Expand the **Clean up image tags** section. 1. Complete the fields. - | Field | Description | - |---------------------------------------------------------------------------|-------------------------------------------------------------------------------------------------------------------| - | **Toggle** | Turn the policy on or off. | - | **Run cleanup** | How often the policy should run. | - | **Keep the most recent** | How many tags to _always_ keep for each image. | - | **Keep tags matching** | The regex pattern that determines which tags to preserve. The `latest` tag is always preserved. For all tags, use `.*`. See other [regex pattern examples](#regex-pattern-examples). | - | **Remove tags older than** | Remove only tags older than X days. | - | **Remove tags matching** | The regex pattern that determines which tags to remove. This value cannot be blank. For all tags, use `.*`. See other [regex pattern examples](#regex-pattern-examples). | + | Field | Description | + |----------------------------|-------------------------------------------------| + | **Toggle** | Turn the policy on or off. | + | **Run cleanup** | How often the policy should run. | + | **Keep the most recent** | How many tags to _always_ keep for each image. | + | **Keep tags matching** | A regex pattern that determines which tags to preserve. The `latest` tag is always preserved. For all tags, use `.*`. See other [regex pattern examples](#regex-pattern-examples). | + | **Remove tags older than** | Remove only tags older than X days. | + | **Remove tags matching** | A regex pattern that determines which tags to remove. This value cannot be blank. For all tags, use `.*`. See other [regex pattern examples](#regex-pattern-examples). | 1. Select **Save**. -Depending on the interval you chose, the policy is scheduled to run. +The policy runs on the scheduled interval you selected. NOTE: If you edit the policy and select **Save** again, the interval is reset. @@ -135,7 +142,8 @@ If you edit the policy and select **Save** again, the interval is reset. Cleanup policies use regex patterns to determine which tags should be preserved or removed, both in the UI and the API. -Regex patterns are automatically surrounded with `\A` and `\Z` anchors. Do not include any `\A`, `\Z`, `^` or `$` token in the regex patterns as they are not necessary. +Regex patterns are automatically surrounded with `\A` and `\Z` anchors. Therefore, you do not need to include any +`\A`, `\Z`, `^` or `$` tokens in the regex patterns. Here are some examples of regex patterns you can use: @@ -180,17 +188,17 @@ Here are some examples of regex patterns you can use: Cleanup policies are executed as a background process. This process is complex, and depending on the number of tags to delete, the process can take time to finish. -To prevent server resource starvation, the following application settings are available: +You can use the following application settings to prevent server resource starvation: - `container_registry_expiration_policies_worker_capacity`: the maximum number of cleanup workers - running concurrently. This must be greater than or equal to `0`. We recommend starting with a low - number and increasing it after monitoring the resources used by the background workers. To remove + running concurrently. This value must be greater than or equal to `0`. You should start with a low + number and increase it after monitoring the resources used by the background workers. To remove all workers and not execute the cleanup policies, set this to `0`. The default value is `4`. - `container_registry_delete_tags_service_timeout`: the maximum time (in seconds) that the cleanup process can take to delete a batch of tags. The default value is `250`. - `container_registry_cleanup_tags_service_max_list_size`: the maximum number of tags that can be - deleted in a single execution. Additional tags must be deleted in another execution. We recommend - starting with a low number and increasing it after monitoring that container images are properly + deleted in a single execution. Additional tags must be deleted in another execution. You should + start with a low number and increase it after verifying that container images are properly deleted. The default value is `200`. - `container_registry_expiration_policies_caching`: enable or disable tag creation timestamp caching during execution of policies. Cached timestamps are stored in [Redis](../../../development/architecture.md#redis). @@ -213,7 +221,8 @@ You can set, update, and disable the cleanup policies using the GitLab API. Examples: -- Select all tags, keep at least 1 tag per image, clean up any tag older than 14 days, run once a month, preserve any images with the name `main` and the policy is enabled: +- Select all tags, keep at least 1 tag per image, clean up any tag older than 14 days, run once a month, preserve +any images with the name `main`, and the policy is enabled: ```shell curl --request PUT --header 'Content-Type: application/json;charset=UTF-8' --header "PRIVATE-TOKEN: <your_access_token>" \ @@ -251,14 +260,14 @@ See the API documentation for further details: [Edit project API](../../../api/p When using an [external container registry](../../../administration/packages/container_registry.md#use-an-external-container-registry-with-gitlab-as-an-auth-endpoint), running a cleanup policy on a project may have some performance risks. -If a project runs a policy to remove thousands of tags +If a project runs a policy to remove thousands of tags, the GitLab background jobs may get backed up or fail completely. -For projects created before GitLab 12.8, we recommend you enable container cleanup policies +For projects created before GitLab 12.8, you should enable container cleanup policies only if the number of tags being cleaned up is minimal. ## More Container Registry storage reduction options -Here are some other options to reduce your project's use of Container Registry storage: +Here are some other options you can use to reduce the Container Registry storage used by your project: - Use the [GitLab UI](index.md#delete-images) to delete individual image tags or the entire repository containing all the tags. @@ -330,6 +339,10 @@ the tags. To create the list and delete the tags: 1. Remove any tags that you want to keep from the `list_o_tags.out` file. For example, you can use `sed` to parse the file and remove the tags. + ::Tabs + + :::TabTitle Linux + ```shell # Remove the `latest` tag from the file sed -i '/latest/d' list_o_tags.out @@ -344,12 +357,24 @@ the tags. To create the list and delete the tags: sed -i '/_v3$/d' list_o_tags.out ``` - If you are running macOS, you must add `.bak` to the commands. For example: + :::TabTitle macOS ```shell + # Remove the `latest` tag from the file sed -i .bak '/latest/d' list_o_tags.out + + # Remove the first N tags from the file + sed -i .bak '1,Nd' list_o_tags.out + + # Remove the tags starting with `Av` from the file + sed -i .bak '/^Av/d' list_o_tags.out + + # Remove the tags ending with `_v3` from the file + sed -i .bak '/_v3$/d' list_o_tags.out ``` + ::EndTabs + 1. Double-check the `list_o_tags.out` file to make sure it contains only the tags that you want to delete. diff --git a/doc/user/packages/container_registry/troubleshoot_container_registry.md b/doc/user/packages/container_registry/troubleshoot_container_registry.md new file mode 100644 index 00000000000..eac7e0fcacd --- /dev/null +++ b/doc/user/packages/container_registry/troubleshoot_container_registry.md @@ -0,0 +1,129 @@ +--- +stage: Package +group: Container Registry +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments +--- + +# Troubleshooting the GitLab Container Registry + +You must sign in to GitLab with administrator rights to troubleshoot most issues with the GitLab Container Registry. + +You can find [additional troubleshooting information](../../../administration/packages/container_registry.md#troubleshooting) in the GitLab Container Registry administration documentation. + +## Migrating OCI container images to GitLab Container Registry + +Migrating container images to the GitLab registry is not supported, but [epic](https://gitlab.com/groups/gitlab-org/-/epics/5210) proposes to change this behavior. + +You can use third-party tools to migrate container images. For example, [skopeo](https://github.com/containers/skopeo), can [copy container images](https://github.com/containers/skopeo#copying-images) between various storage mechanisms. You can use skopeo to copy from container registries, container storage backends, local directories, and local OCI-layout directories to the GitLab Container Registry. + +## Docker connection error + +A Docker connection error can occur when there are special characters in either the group, +project, or branch name. Special characters include: + +- A leading underscore. +- A trailing hyphen or dash. + +To resolve this error, you can change the [group path](../../group/manage.md#change-a-groups-path), +the [project path](../../project/settings/index.md#rename-a-repository) or the branch name. + +You may get a `404 Not Found` or `Unknown Manifest` error message if you use +Docker Engine 17.11 or earlier. Current versions of Docker Engine use +the [v2 API](https://docs.docker.com/registry/spec/manifest-v2-2/). + +The images in your GitLab Container Registry must use the Docker v2 API. +For information on how to update version 1 images to version 2, see the [Docker documentation](https://docs.docker.com/registry/spec/deprecated-schema-v1). + +## `Blob unknown to registry` error when pushing a manifest list + +When [pushing a Docker manifest list](https://docs.docker.com/engine/reference/commandline/manifest/#create-and-push-a-manifest-list) +to the GitLab Container Registry, you may receive the error +`manifest blob unknown: blob unknown to registry`. This error is likely caused by having multiple images +with different architectures spread out over several repositories instead of the same repository. + +For example, you may have two images, each representing an architecture: + +- The `amd64` platform. +- The `arm64v8` platform. + +To build a multi-arch image with these images, you must push them to the same repository as the +multi-arch image. + +To address the `Blob unknown to registry` error, include the architecture in the tag name of +individual images. For example, use `mygroup/myapp:1.0.0-amd64` and `mygroup/myapp:1.0.0-arm64v8`. +You can then tag the manifest list with `mygroup/myapp:1.0.0`. + +## Unable to change project path or transfer a project + +If you try to change a project path or transfer a project to a new namespace, +you may receive one of the following errors: + +- Project cannot be transferred because tags are present in its container registry. +- Namespace cannot be moved because at least one project has tags in the container registry. + +This error occurs when the project has images in the Container Registry. +You must delete or move these images before you change the path or transfer +the project. + +The following procedure uses these sample project names: + +- For the current project: `gitlab.example.com/org/build/sample_project/cr:v2.9.1`. +- For the new project: `gitlab.example.com/new_org/build/new_sample_project/cr:v2.9.1`. + +1. Download the Docker images on your computer: + + ```shell + docker login gitlab.example.com + docker pull gitlab.example.com/org/build/sample_project/cr:v2.9.1 + ``` + + NOTE: + Use either a [personal access token](../../profile/personal_access_tokens.md) or a + [deploy token](../../project/deploy_tokens/index.md) to authenticate your user account. + +1. Rename the images to match the new project name: + + ```shell + docker tag gitlab.example.com/org/build/sample_project/cr:v2.9.1 gitlab.example.com/new_org/build/new_sample_project/cr:v2.9.1 + ``` + +1. Delete the images in the old project by using the [UI](index.md#delete-images) or [API](../../../api/packages.md#delete-a-project-package). + There may be a delay while the images are queued and deleted. +1. Change the path or transfer the project: + + 1. On the top bar, select **Main menu > Projects** and find your project. + 1. On the left sidebar, select **Settings > General**. + 1. Expand the **Advanced** section. + 1. In the **Change path** text box, edit the path. + 1. Select **Change path**. + +1. Restore the images: + + ```shell + docker push gitlab.example.com/new_org/build/new_sample_project/cr:v2.9.1 + ``` + +See this [issue](https://gitlab.com/gitlab-org/gitlab/-/issues/18383) for details. + +## `unauthorized: authentication required` when pushing large images + +When pushing large images, you may see an authentication error like the following: + +```shell +docker push gitlab.example.com/myproject/docs:latest +The push refers to a repository [gitlab.example.com/myproject/docs] +630816f32edb: Preparing +530d5553aec8: Preparing +... +4b0bab9ff599: Waiting +d1c800db26c7: Waiting +42755cf4ee95: Waiting +unauthorized: authentication required +``` + +This error happens when your authentication token expires before the image push is complete. By default, tokens for +the Container Registry on self-managed GitLab instances expire every five minutes. On GitLab.com, the token expiration +time is set to 15 minutes. + +If you are using self-managed GitLab, you can ask an administrator to +[increase the token duration](../../../administration/packages/container_registry.md#increase-token-duration). diff --git a/doc/user/packages/generic_packages/index.md b/doc/user/packages/generic_packages/index.md index 563f35f2f4f..9b49f946984 100644 --- a/doc/user/packages/generic_packages/index.md +++ b/doc/user/packages/generic_packages/index.md @@ -118,7 +118,7 @@ API or the UI. > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/293755) in GitLab 13.12. > - [Required permissions](https://gitlab.com/gitlab-org/gitlab/-/issues/350682) changed from developer to maintainer in GitLab 15.0. -To prevent users from publishing duplicate generic packages, you can use the [GraphQl API](../../../api/graphql/reference/index.md#packagesettings) +To prevent users from publishing duplicate generic packages, you can use the [GraphQL API](../../../api/graphql/reference/index.md#packagesettings) or the UI. In the UI: diff --git a/doc/user/packages/gradle_repository/index.md b/doc/user/packages/gradle_repository/index.md new file mode 100644 index 00000000000..4247c13297d --- /dev/null +++ b/doc/user/packages/gradle_repository/index.md @@ -0,0 +1,372 @@ +--- +stage: Package +group: Package Registry +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments +--- + +# Maven packages in the Package Registry **(FREE)** + +Publish [Maven](https://maven.apache.org) artifacts in your project's Package Registry using Gradle. +Then, install the packages whenever you need to use them as a dependency. + +For documentation of the specific API endpoints that the Maven package manager +client uses, see the [Maven API documentation](../../../api/packages/maven.md). + +Learn how to build a [Gradle](../workflows/build_packages.md#gradle) package. + +## Publish to the GitLab Package Registry + +### Tokens + +You need a token to publish a package. Different tokens are available depending on what you're trying to +achieve. For more information, review the [guidance on tokens](../package_registry/index.md#authenticate-with-the-registry). + +- If your organization uses two-factor authentication (2FA), you must use a personal access token with the scope set to `api`. +- If you publish a package via CI/CD pipelines, you must use a CI job token. + +Create a token and save it to use later in the process. + +## Authenticate to the Package Registry with Gradle + +### Authenticate with a personal access token or deploy token in Gradle + +In [your `GRADLE_USER_HOME` directory](https://docs.gradle.org/current/userguide/directory_layout.html#dir:gradle_user_home), +create a file `gradle.properties` with the following content: + +```properties +gitLabPrivateToken=REPLACE_WITH_YOUR_TOKEN +``` + +Your token name depends on which token you use. + +| Token type | Token name | +| --------------------- | --------------- | +| Personal access token | `Private-Token` | +| Deploy token | `Deploy-Token` | + +Add a `repositories` section to your +[`build.gradle`](https://docs.gradle.org/current/userguide/tutorial_using_tasks.html) +file: + +```groovy +repositories { + maven { + url "https://gitlab.example.com/api/v4/groups/<group>/-/packages/maven" + name "GitLab" + credentials(HttpHeaderCredentials) { + name = 'REPLACE_WITH_TOKEN_NAME' + value = gitLabPrivateToken + } + authentication { + header(HttpHeaderAuthentication) + } + } +} +``` + +Or add it to your `build.gradle.kts` file if you are using Kotlin DSL: + +```kotlin +repositories { + maven { + url = uri("https://gitlab.example.com/api/v4/groups/<group>/-/packages/maven") + name = "GitLab" + credentials(HttpHeaderCredentials::class) { + name = "REPLACE_WITH_TOKEN_NAME" + value = findProperty("gitLabPrivateToken") as String? + } + authentication { + create("header", HttpHeaderAuthentication::class) + } + } +} +``` + +### Authenticate with a CI job token in Gradle + +To authenticate with a CI job token, add a `repositories` section to your +[`build.gradle`](https://docs.gradle.org/current/userguide/tutorial_using_tasks.html) +file: + +```groovy +repositories { + maven { + url "${CI_API_V4_URL}/groups/<group>/-/packages/maven" + name "GitLab" + credentials(HttpHeaderCredentials) { + name = 'Job-Token' + value = System.getenv("CI_JOB_TOKEN") + } + authentication { + header(HttpHeaderAuthentication) + } + } +} +``` + +Or add it to your `build.gradle.kts` file if you are using Kotlin DSL: + +```kotlin +repositories { + maven { + url = uri("$CI_API_V4_URL/groups/<group>/-/packages/maven") + name = "GitLab" + credentials(HttpHeaderCredentials::class) { + name = "Job-Token" + value = System.getenv("CI_JOB_TOKEN") + } + authentication { + create("header", HttpHeaderAuthentication::class) + } + } +} +``` + +### Naming convention + +You can use one of three API endpoints to install a Maven package. You must publish a package to a project, but note which endpoint +you use to install the package. The option you choose determines the settings you add to your `pom.xml` file for publishing. + +The three endpoints are: + +- **Project-level**: Use when you have a few Maven packages that are not in the same GitLab group. +- **Group-level**: Use when installing packages from many different projects in the same GitLab group. GitLab does not guarantee the uniqueness of package names in the group. You can have two projects with the same package name and package version. As a result, GitLab serves whichever one is more recent. +- **Instance-level**: Use when installing many packages from different GitLab groups or in their own namespace. + +**Only packages with the same path as the project** are exposed by the instance-level endpoint. + +| Project | Package | Instance-level endpoint available | +| ------------------- | -------------------------------- | --------------------------------- | +| `foo/bar` | `foo/bar/1.0-SNAPSHOT` | Yes | +| `gitlab-org/gitlab` | `foo/bar/1.0-SNAPSHOT` | No | +| `gitlab-org/gitlab` | `gitlab-org/gitlab/1.0-SNAPSHOT` | Yes | + +#### Endpoint URLs + +| Endpoint | Endpoint URL | Additional information | +| -------- | ------------------------------------------------------------------------ | ---------------------------------------------------------------------------------------------------------------------------------- | +| Project | `https://gitlab.example.com/api/v4/projects/<project_id>/packages/maven` | Replace `gitlab.example.com` with your domain name. Replace `<project_id>` with your project ID found on your project's homepage. | +| Group | `https://gitlab.example.com/api/v4/groups/<group_id>/-/packages/maven` | Replace `gitlab.example.com` with your domain name. Replace `<group_id>` with your group ID found on your group's homepage. | +| Instance | `https:///gitlab.example.com/api/v4/packages/maven` | Replace `gitlab.example.com` with your domain name. | + +In all cases, to publish a package, you need: + +- A project-specific URL in the `distributionManagement` section. +- A `repository` and `distributionManagement` section. + +### Edit the Groovy DSL or Kotlin DSL + +The Gradle Groovy DSL `repositories` section should look like this: + +```groovy +repositories { + maven { + url "<your_endpoint_url>" + name "GitLab" + } +} +``` + +In Kotlin DSL: + +```kotlin +repositories { + maven { + url = uri("<your_endpoint_url>") + name = "GitLab" + } +} +``` + +- Replace `<your_endpoint_url>` with the [endpoint](#endpoint-urls) you chose. + +## Publish using Gradle + +Your token name depends on which token you use. + +| Token type | Token name | +| --------------------- | --------------- | +| Personal access token | `Private-Token` | +| Deploy token | `Deploy-Token` | + +To publish a package by using Gradle: + +1. Add the Gradle plugin [`maven-publish`](https://docs.gradle.org/current/userguide/publishing_maven.html) to the plugins section: + + In Groovy DSL: + + ```groovy + plugins { + id 'java' + id 'maven-publish' + } + ``` + + In Kotlin DSL: + + ```kotlin + plugins { + java + `maven-publish` + } + ``` + +1. Add a `publishing` section: + + In Groovy DSL: + + ```groovy + publishing { + publications { + library(MavenPublication) { + from components.java + } + } + repositories { + maven { + url "https://gitlab.example.com/api/v4/projects/<PROJECT_ID>/packages/maven" + credentials(HttpHeaderCredentials) { + name = "REPLACE_WITH_TOKEN_NAME" + value = gitLabPrivateToken // the variable resides in $GRADLE_USER_HOME/gradle.properties + } + authentication { + header(HttpHeaderAuthentication) + } + } + } + } + ``` + + In Kotlin DSL: + + ```kotlin + publishing { + publications { + create<MavenPublication>("library") { + from(components["java"]) + } + } + repositories { + maven { + url = uri("https://gitlab.example.com/api/v4/projects/<PROJECT_ID>/packages/maven") + credentials(HttpHeaderCredentials::class) { + name = "REPLACE_WITH_TOKEN_NAME" + value = + findProperty("gitLabPrivateToken") as String? // the variable resides in $GRADLE_USER_HOME/gradle.properties + } + authentication { + create("header", HttpHeaderAuthentication::class) + } + } + } + } + ``` + +1. Replace `PROJECT_ID` with your project ID, which you can find on your project's home page. + +1. Run the publish task: + + ```shell + gradle publish + ``` + +Go to your project's **Packages and registries** page and view the published packages. + +## Install a package + +To install a package from the GitLab Package Registry, you must configure +the [remote and authenticate](#authenticate-to-the-package-registry-with-gradle). +After configuring the remote and authenticate, you can install a package from a project, group, or namespace. + +If multiple packages have the same name and version, when you install +a package, the most recently-published package is retrieved. + +Add a [dependency](https://docs.gradle.org/current/userguide/declaring_dependencies.html) to `build.gradle` in the dependencies section: + +```groovy +dependencies { + implementation 'com.mycompany.mydepartment:my-project:1.0-SNAPSHOT' +} +``` + +Or to `build.gradle.kts` if you are using Kotlin DSL: + +```kotlin +dependencies { + implementation("com.mycompany.mydepartment:my-project:1.0-SNAPSHOT") +} +``` + +## Helpful hints + +For the complete list of helpful hints, see the [Maven documentation](../maven_repository/index.md#helpful-hints). + +### Create Maven packages with GitLab CI/CD by using Gradle + +You can create a package each time the `main` branch +is updated. + +1. Authenticate with [a CI job token in Gradle](#authenticate-with-a-ci-job-token-in-gradle). + +1. Add a `deploy` job to your `.gitlab-ci.yml` file: + + ```yaml + deploy: + image: gradle:6.5-jdk11 + script: + - 'gradle publish' + only: + - main + ``` + +1. Commit files to your repository. + +When the pipeline is successful, the Maven package is created. + +### Publishing a package with the same name or version + +When you publish a package with the same name and version as an existing package, the new package +files are added to the existing package. You can still use the UI or API to access and view the +existing package's older assets. + +Consider using the Packages API or the UI to delete older package versions. + +### Do not allow duplicate Maven packages + +To prevent users from publishing duplicate Maven packages, you can use the [GraphQl API](../../../api/graphql/reference/index.md#packagesettings) or the UI. + +In the UI: + +1. For your group, go to **Settings > Packages and registries**. +1. Expand the **Package Registry** section. +1. Turn on the **Do not allow duplicates** toggle. +1. Optional. To allow some duplicate packages, in the **Exceptions** box, enter a regex pattern that matches the names and/or versions of packages you want to allow. + +Your changes are automatically saved. + +### Request forwarding to Maven Central + +FLAG: +By default, this feature is not available for self-managed. To make it available, ask an administrator to [enable the feature flag](../../../administration/feature_flags.md) named `maven_central_request_forwarding`. +This feature is not available for SaaS users. + +When a Maven package is not found in the Package Registry, the request is forwarded +to [Maven Central](https://search.maven.org/). + +When the feature flag is enabled, administrators can disable this behavior in the +[Continuous Integration settings](../../admin_area/settings/continuous_integration.md). + +There are many ways to configure your Maven project to request packages +in Maven Central from GitLab. Maven repositories are queried in a +[specific order](https://maven.apache.org/guides/mini/guide-multiple-repositories.html#repository-order). +By default, maven-central is usually checked first through the +[Super POM](https://maven.apache.org/guides/introduction/introduction-to-the-pom.html#Super_POM), so +GitLab needs to be configured to be queried before maven-central. + +[Using GitLab as a mirror of the central proxy](../maven_repository/index.md#setting-gitlab-as-a-mirror-for-the-central-proxy) is one +way to force GitLab to be queried in place of maven-central. + +Maven forwarding is restricted to only the project level and +group level [endpoints](#naming-convention). The instance-level endpoint +has naming restrictions that prevent it from being used for packages that don't follow that convention and also +introduces too much security risk for supply-chain style attacks. diff --git a/doc/user/packages/helm_repository/index.md b/doc/user/packages/helm_repository/index.md index bba68494c2d..785ef344c8e 100644 --- a/doc/user/packages/helm_repository/index.md +++ b/doc/user/packages/helm_repository/index.md @@ -72,7 +72,7 @@ Once built, a chart can be uploaded to the desired channel with `curl` or `helm ### Release channels You can publish Helm charts to channels in GitLab. Channels are a method you can use to differentiate Helm chart repositories. -For example, you can use `stable` and `devel` as channels to allow users to add the `stable` repo while `devel` charts are isolated. +For example, you can use `stable` and `devel` as channels to allow users to add the `stable` repository while `devel` charts are isolated. ## Use CI/CD to publish a Helm package diff --git a/doc/user/packages/maven_repository/index.md b/doc/user/packages/maven_repository/index.md index 2aa71e111fb..c6c2f238564 100644 --- a/doc/user/packages/maven_repository/index.md +++ b/doc/user/packages/maven_repository/index.md @@ -4,9 +4,7 @@ group: Package Registry info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- -# Maven packages in the Package Repository **(FREE)** - -> [Moved](https://gitlab.com/gitlab-org/gitlab/-/issues/221259) from GitLab Premium to GitLab Free in 13.3. +# Maven packages in the Package Registry **(FREE)** Publish [Maven](https://maven.apache.org) artifacts in your project's Package Registry. Then, install the packages whenever you need to use them as a dependency. @@ -14,75 +12,29 @@ Then, install the packages whenever you need to use them as a dependency. For documentation of the specific API endpoints that the Maven package manager client uses, see the [Maven API documentation](../../../api/packages/maven.md). -Learn how to build a [Maven](../workflows/build_packages.md#maven) or [Gradle](../workflows/build_packages.md#gradle) package. - -## Authenticate to the Package Registry with Maven - -To authenticate to the Package Registry, you need one of the following: +Learn how to build a [Maven](../workflows/build_packages.md#maven) package. -- A [personal access token](../../../user/profile/personal_access_tokens.md) with the scope set to `api`. -- A [deploy token](../../project/deploy_tokens/index.md) with the scope set to `read_package_registry`, `write_package_registry`, or both. -- A [CI_JOB_TOKEN](#authenticate-with-a-ci-job-token-in-maven). +## Publish to the GitLab Package Registry -### Authenticate with a personal access token in Maven - -To use a personal access token, add this section to your -[`settings.xml`](https://maven.apache.org/settings.html) file. +### Authenticate to the Package Registry -The `name` must be `Private-Token`. - -```xml -<settings> - <servers> - <server> - <id>gitlab-maven</id> - <configuration> - <httpHeaders> - <property> - <name>Private-Token</name> - <value>REPLACE_WITH_YOUR_PERSONAL_ACCESS_TOKEN</value> - </property> - </httpHeaders> - </configuration> - </server> - </servers> -</settings> -``` +You need an token to publish a package. There are different tokens available depending on what you're trying to achieve. For more information, review the [guidance on tokens](../package_registry/index.md#authenticate-with-the-registry). -### Authenticate with a deploy token in Maven +Create a token and save it to use later in the process. -> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/213566) deploy token authentication in GitLab 13.0. -> - [Moved](https://gitlab.com/gitlab-org/gitlab/-/issues/221259) from GitLab Premium to GitLab Free in 13.3. +### Edit the `settings.xml` -To use a deploy token, add this section to your +Add the following section to your [`settings.xml`](https://maven.apache.org/settings.html) file. -The `name` must be `Deploy-Token`. - -```xml -<settings> - <servers> - <server> - <id>gitlab-maven</id> - <configuration> - <httpHeaders> - <property> - <name>Deploy-Token</name> - <value>REPLACE_WITH_YOUR_DEPLOY_TOKEN</value> - </property> - </httpHeaders> - </configuration> - </server> - </servers> -</settings> -``` - -### Authenticate with a CI job token in Maven - -To authenticate with a CI job token, add this section to your -[`settings.xml`](https://maven.apache.org/settings.html) file. +NOTE: +The `<name>` field must be named to match the token you chose. -The `name` must be `Job-Token`. +| Token type | Name must be | Token | +| --------------------- | --------------- | ---------------------------------------------------------------------- | +| Personal access token | `Private-Token` | Paste token as-is, or define an environment variable to hold the token | +| Deploy token | `Deploy-Token` | Paste token as-is, or define an environment variable to hold the token | +| CI Job token | `Job-Token` | `${CI_JOB_TOKEN}` | ```xml <settings> @@ -92,8 +44,8 @@ The `name` must be `Job-Token`. <configuration> <httpHeaders> <property> - <name>Job-Token</name> - <value>${CI_JOB_TOKEN}</value> + <name>REPLACE_WITH_NAME</name> + <value>REPLACE_WITH_TOKEN</value> </property> </httpHeaders> </configuration> @@ -102,361 +54,70 @@ The `name` must be `Job-Token`. </settings> ``` -Read more about [how to create Maven packages using GitLab CI/CD](#create-maven-packages-with-gitlab-cicd). - -## Authenticate to the Package Registry with Gradle - -To authenticate to the Package Registry, you need either a personal access token or deploy token. - -- If you use a [personal access token](../../../user/profile/personal_access_tokens.md), set the scope to `api`. -- If you use a [deploy token](../../project/deploy_tokens/index.md), set the scope to `read_package_registry`, `write_package_registry`, or both. - -### Authenticate with a personal access token in Gradle - -In [your `GRADLE_USER_HOME` directory](https://docs.gradle.org/current/userguide/directory_layout.html#dir:gradle_user_home), -create a file `gradle.properties` with the following content: - -```properties -gitLabPrivateToken=REPLACE_WITH_YOUR_PERSONAL_ACCESS_TOKEN -``` - -Add a `repositories` section to your -[`build.gradle`](https://docs.gradle.org/current/userguide/tutorial_using_tasks.html) -file: - -```groovy -repositories { - maven { - url "https://gitlab.example.com/api/v4/groups/<group>/-/packages/maven" - name "GitLab" - credentials(HttpHeaderCredentials) { - name = 'Private-Token' - value = gitLabPrivateToken - } - authentication { - header(HttpHeaderAuthentication) - } - } -} -``` - -Or add it to your `build.gradle.kts` file if you are using Kotlin DSL: - -```kotlin -repositories { - maven { - url = uri("https://gitlab.example.com/api/v4/groups/<group>/-/packages/maven") - name = "GitLab" - credentials(HttpHeaderCredentials::class) { - name = "Private-Token" - value = findProperty("gitLabPrivateToken") as String? - } - authentication { - create("header", HttpHeaderAuthentication::class) - } - } -} -``` +### Naming convention -### Authenticate with a deploy token in Gradle - -To authenticate with a deploy token, add a `repositories` section to your -[`build.gradle`](https://docs.gradle.org/current/userguide/tutorial_using_tasks.html) -file: - -```groovy -repositories { - maven { - url "https://gitlab.example.com/api/v4/groups/<group>/-/packages/maven" - name "GitLab" - credentials(HttpHeaderCredentials) { - name = 'Deploy-Token' - value = '<deploy-token>' - } - authentication { - header(HttpHeaderAuthentication) - } - } -} -``` +You can use one of three endpoints to install a Maven package. You must publish a package to a project, but the endpoint you choose determines the settings you add to your `pom.xml` file for publishing. -Or add it to your `build.gradle.kts` file if you are using Kotlin DSL: - -```kotlin -repositories { - maven { - url = uri("https://gitlab.example.com/api/v4/groups/<group>/-/packages/maven") - name = "GitLab" - credentials(HttpHeaderCredentials::class) { - name = "Deploy-Token" - value = "<deploy-token>" - } - authentication { - create("header", HttpHeaderAuthentication::class) - } - } -} -``` +The three endpoints are: -### Authenticate with a CI job token in Gradle - -To authenticate with a CI job token, add a `repositories` section to your -[`build.gradle`](https://docs.gradle.org/current/userguide/tutorial_using_tasks.html) -file: - -```groovy -repositories { - maven { - url "${CI_API_V4_URL}/groups/<group>/-/packages/maven" - name "GitLab" - credentials(HttpHeaderCredentials) { - name = 'Job-Token' - value = System.getenv("CI_JOB_TOKEN") - } - authentication { - header(HttpHeaderAuthentication) - } - } -} -``` +- **Project-level**: Use when you have a few Maven packages and they are not in the same GitLab group. +- **Group-level**: Use when you want to install packages from many different projects in the same GitLab group. GitLab does not guarantee the uniqueness of package names within the group. You can have two projects with the same package name and package version. As a result, GitLab serves whichever one is more recent. +- **Instance-level**: Use when you have many packages in different GitLab groups or in their own namespace. -Or add it to your `build.gradle.kts` file if you are using Kotlin DSL: - -```kotlin -repositories { - maven { - url = uri("$CI_API_V4_URL/groups/<group>/-/packages/maven") - name = "GitLab" - credentials(HttpHeaderCredentials::class) { - name = "Job-Token" - value = System.getenv("CI_JOB_TOKEN") - } - authentication { - create("header", HttpHeaderAuthentication::class) - } - } -} -``` +**Only packages that have the same path as the project** are exposed by the instance-level endpoint. -## Use the GitLab endpoint for Maven packages +| Project | Package | Instance-level endpoint available | +| ------------------- | -------------------------------- | --------------------------------- | +| `foo/bar` | `foo/bar/1.0-SNAPSHOT` | Yes | +| `gitlab-org/gitlab` | `foo/bar/1.0-SNAPSHOT` | No | +| `gitlab-org/gitlab` | `gitlab-org/gitlab/1.0-SNAPSHOT` | Yes | -To use the GitLab endpoint for Maven packages, choose an option: +#### Endpoint URLs -- **Project-level**: To publish Maven packages to a project, use a project-level endpoint. - To install Maven packages, use a project-level endpoint when you have few Maven packages - and they are not in the same GitLab group. -- **Group-level**: Use a group-level endpoint when you want to install packages from - many different projects in the same GitLab group. -- **Instance-level**: Use an instance-level endpoint when you want to install many - packages from different GitLab groups or in their own namespace. +| Endpoint | Endpoint URL for `pom.xml` | Additional information | +| -------- | ------------------------------------------------------------------------ | ---------------------------------------------------------------------------------------------------------------------------------- | +| Project | `https://gitlab.example.com/api/v4/projects/<project_id>/packages/maven` | Replace `gitlab.example.com` with your domain name. Replace `<project_id>` with your project ID, found on your project's homepage. | +| Group | `https://gitlab.example.com/api/v4/groups/<group_id>/-/packages/maven` | Replace `gitlab.example.com` with your domain name. Replace `<group_id>` with your group ID, found on your group's homepage. | +| Instance | `https:///gitlab.example.com/api/v4/packages/maven` | Replace `gitlab.example.com` with your domain name. | -The option you choose determines the settings you add to your `pom.xml` file. +### Edit the `pom.xml` for publishing -In all cases, to publish a package, you need: +No matter which endpoint you choose, you must have: - A project-specific URL in the `distributionManagement` section. - A `repository` and `distributionManagement` section. -### Project-level Maven endpoint - -The relevant `repository` section of your `pom.xml` -in Maven should look like this: +The relevant `repository` section of your `pom.xml` in Maven should look like this: ```xml <repositories> <repository> <id>gitlab-maven</id> - <url>https://gitlab.example.com/api/v4/projects/PROJECT_ID/packages/maven</url> + <url><your_endpoint_url></url> </repository> </repositories> <distributionManagement> <repository> <id>gitlab-maven</id> - <url>https://gitlab.example.com/api/v4/projects/PROJECT_ID/packages/maven</url> + <url>https://gitlab.example.com/api/v4/projects/<project_id>/packages/maven</url> </repository> <snapshotRepository> <id>gitlab-maven</id> - <url>https://gitlab.example.com/api/v4/projects/PROJECT_ID/packages/maven</url> + <url>https://gitlab.example.com/api/v4/projects/<project_id>/packages/maven</url> </snapshotRepository> </distributionManagement> ``` -The corresponding section in Gradle Groovy DSL would be: - -```groovy -repositories { - maven { - url "https://gitlab.example.com/api/v4/projects/PROJECT_ID/packages/maven" - name "GitLab" - } -} -``` - -In Kotlin DSL: - -```kotlin -repositories { - maven { - url = uri("https://gitlab.example.com/api/v4/projects/PROJECT_ID/packages/maven") - name = "GitLab" - } -} -``` - -- The `id` is what you [defined in `settings.xml`](#authenticate-to-the-package-registry-with-maven). -- The `PROJECT_ID` is your project ID, which you can view on your project's home page. -- Replace `gitlab.example.com` with your domain name. -- For retrieving artifacts, use either the - [URL-encoded](../../../api/index.md#namespaced-path-encoding) path of the project - (like `group%2Fproject`) or the project's ID (like `42`). However, only the - project's ID can be used for publishing. - -### Group-level Maven endpoint - -> [Moved](https://gitlab.com/gitlab-org/gitlab/-/issues/221259) from GitLab Premium to GitLab Free in 13.3. - -If you rely on many packages, it might be inefficient to include the `repository` section -with a unique URL for each package. Instead, you can use the group-level endpoint for -all the Maven packages stored within one GitLab group. Only packages you have access to -are available for download. - -The group-level endpoint works with any package names, so you -have more flexibility in naming, compared to the [instance-level endpoint](#instance-level-maven-endpoint). -However, GitLab does not guarantee the uniqueness of package names within -the group. You can have two projects with the same package name and package -version. As a result, GitLab serves whichever one is more recent. - -This example shows the relevant `repository` section of your `pom.xml` file. -You still need a project-specific URL for publishing a package in -the `distributionManagement` section: - -```xml -<repositories> - <repository> - <id>gitlab-maven</id> - <url>https://gitlab.example.com/api/v4/groups/GROUP_ID/-/packages/maven</url> - </repository> -</repositories> -<distributionManagement> - <repository> - <id>gitlab-maven</id> - <url>https://gitlab.example.com/api/v4/projects/PROJECT_ID/packages/maven</url> - </repository> - <snapshotRepository> - <id>gitlab-maven</id> - <url>https://gitlab.example.com/api/v4/projects/PROJECT_ID/packages/maven</url> - </snapshotRepository> -</distributionManagement> -``` - -For Gradle, the corresponding `repositories` section in Groovy DSL would look like: - -```groovy -repositories { - maven { - url "https://gitlab.example.com/api/v4/groups/GROUP_ID/-/packages/maven" - name "GitLab" - } -} -``` - -In Kotlin DSL: - -```kotlin -repositories { - maven { - url = uri("https://gitlab.example.com/api/v4/groups/GROUP_ID/-/packages/maven") - name = "GitLab" - } -} -``` - -- For the `id`, use what you [defined in `settings.xml`](#authenticate-to-the-package-registry-with-maven). -- For `GROUP_ID`, use your group ID, which you can view on your group's home page. -- For `PROJECT_ID`, use your project ID, which you can view on your project's home page. +- The `id` is what you [defined in `settings.xml`](#edit-the-settingsxml). +- The `<your_endpoint_url>` depends on which [endpoint](#endpoint-urls) you choose. - Replace `gitlab.example.com` with your domain name. -- For retrieving artifacts, use either the - [URL-encoded](../../../api/index.md#namespaced-path-encoding) path of the group - (like `group%2Fsubgroup`) or the group's ID (like `12`). - -### Instance-level Maven endpoint - -> [Moved](https://gitlab.com/gitlab-org/gitlab/-/issues/221259) from GitLab Premium to GitLab Free in 13.3. - -If you rely on many packages, it might be inefficient to include the `repository` section -with a unique URL for each package. Instead, you can use the instance-level endpoint for -all Maven packages stored in GitLab. All packages you have access to are available -for download. - -**Only packages that have the same path as the project** are exposed by -the instance-level endpoint. - -| Project | Package | Instance-level endpoint available | -| ------------------- | -------------------------------- | --------------------------------- | -| `foo/bar` | `foo/bar/1.0-SNAPSHOT` | Yes | -| `gitlab-org/gitlab` | `foo/bar/1.0-SNAPSHOT` | No | -| `gitlab-org/gitlab` | `gitlab-org/gitlab/1.0-SNAPSHOT` | Yes | - -This example shows how relevant `repository` section of your `pom.xml`. -You still need a project-specific URL in the `distributionManagement` section. - -```xml -<repositories> - <repository> - <id>gitlab-maven</id> - <url>https://gitlab.example.com/api/v4/packages/maven</url> - </repository> -</repositories> -<distributionManagement> - <repository> - <id>gitlab-maven</id> - <url>https://gitlab.example.com/api/v4/projects/PROJECT_ID/packages/maven</url> - </repository> - <snapshotRepository> - <id>gitlab-maven</id> - <url>https://gitlab.example.com/api/v4/projects/PROJECT_ID/packages/maven</url> - </snapshotRepository> -</distributionManagement> -``` - -The corresponding repositories section in Gradle Groovy DSL would look like: - -```groovy -repositories { - maven { - url "https://gitlab.example.com/api/v4/packages/maven" - name "GitLab" - } -} -``` - -In Kotlin DSL: - -```kotlin -repositories { - maven { - url = uri("https://gitlab.example.com/api/v4/packages/maven") - name = "GitLab" - } -} -``` - -- The `id` is what you [defined in `settings.xml`](#authenticate-to-the-package-registry-with-maven). -- The `PROJECT_ID` is your project ID, which you can view on your project's home page. -- Replace `gitlab.example.com` with your domain name. -- For retrieving artifacts, use either the - [URL-encoded](../../../api/index.md#namespaced-path-encoding) path of the project - (like `group%2Fproject`) or the project's ID (like `42`). However, only the - project's ID can be used for publishing. ## Publish a package -After you have set up the [remote and authentication](#authenticate-to-the-package-registry-with-maven) -and [configured your project](#use-the-gitlab-endpoint-for-maven-packages), +After you have set up the [authentication](#authenticate-to-the-package-registry) +and [chosen an endpoint for publishing](#naming-convention), publish a Maven package to your project. -### Publish by using Maven - To publish a package by using Maven: ```shell @@ -474,122 +135,13 @@ If the deploy is successful, the build success message should be displayed: The message should also show that the package was published to the correct location: ```shell -Uploading to gitlab-maven: https://gitlab.example.com/api/v4/projects/PROJECT_ID/packages/maven/com/mycompany/mydepartment/my-project/1.0-SNAPSHOT/my-project-1.0-20200128.120857-1.jar +Uploading to gitlab-maven: https://example.com/api/v4/projects/PROJECT_ID/packages/maven/com/mycompany/mydepartment/my-project/1.0-SNAPSHOT/my-project-1.0-20200128.120857-1.jar ``` -### Publish by using Gradle - -To publish a package by using Gradle: - -1. Add the Gradle plugin [`maven-publish`](https://docs.gradle.org/current/userguide/publishing_maven.html) to the plugins section: - - In Groovy DSL: - - ```groovy - plugins { - id 'java' - id 'maven-publish' - } - ``` - - In Kotlin DSL: - - ```kotlin - plugins { - java - `maven-publish` - } - ``` - -1. Add a `publishing` section: - - In Groovy DSL: - - ```groovy - publishing { - publications { - library(MavenPublication) { - from components.java - } - } - repositories { - maven { - url "https://gitlab.example.com/api/v4/projects/<PROJECT_ID>/packages/maven" - credentials(HttpHeaderCredentials) { - name = "Private-Token" - value = gitLabPrivateToken // the variable resides in $GRADLE_USER_HOME/gradle.properties - } - authentication { - header(HttpHeaderAuthentication) - } - } - } - } - ``` - - In Kotlin DSL: - - ```kotlin - publishing { - publications { - create<MavenPublication>("library") { - from(components["java"]) - } - } - repositories { - maven { - url = uri("https://gitlab.example.com/api/v4/projects/<PROJECT_ID>/packages/maven") - credentials(HttpHeaderCredentials::class) { - name = "Private-Token" - value = - findProperty("gitLabPrivateToken") as String? // the variable resides in $GRADLE_USER_HOME/gradle.properties - } - authentication { - create("header", HttpHeaderAuthentication::class) - } - } - } - } - ``` - -1. Replace `PROJECT_ID` with your project ID, which can be found on your project's home page. - -1. Run the publish task: - - ```shell - gradle publish - ``` - -Now navigate to your project's **Packages and registries** page and view the published artifacts. - -### Publishing a package with the same name or version - -When you publish a package with the same name and version as an existing package, the new package -files are added to the existing package. You can still use the UI or API to access and view the -existing package's older files. - -To delete these older package versions, consider using the Packages API or the UI. - -#### Do not allow duplicate Maven packages - -> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/296895) in GitLab 13.9. -> - [Required permissions](https://gitlab.com/gitlab-org/gitlab/-/issues/350682) changed from developer to maintainer in GitLab 15.0. - -To prevent users from publishing duplicate Maven packages, you can use the [GraphQl API](../../../api/graphql/reference/index.md#packagesettings) or the UI. - -In the UI: - -1. For your group, go to **Settings > Packages and registries**. -1. Expand the **Package Registry** section. -1. Turn on the **Reject duplicates** toggle. -1. Optional. To allow some duplicate packages, in the **Exceptions** box, enter a regex pattern that matches the names and/or versions of packages you want to allow. - -Your changes are automatically saved. - ## Install a package To install a package from the GitLab Package Registry, you must configure -the [remote and authenticate](#authenticate-to-the-package-registry-with-maven). +the [remote and authenticate](#authenticate-to-the-package-registry). When this is completed, you can install a package from a project, group, or namespace. @@ -633,8 +185,8 @@ You can install packages by using the Maven `dependency:get` [command](https://m mvn dependency:get -Dartifact=com.nickkipling.app:nick-test-app:1.1-SNAPSHOT -DremoteRepositories=gitlab-maven::::<gitlab endpoint url> -s <path to settings.xml> ``` - - `<gitlab endpoint url>` is the URL of the GitLab [endpoint](#use-the-gitlab-endpoint-for-maven-packages). - - `<path to settings.xml>` is the path to the `settings.xml` file that contains the [authentication details](#authenticate-to-the-package-registry-with-maven). + - `<gitlab endpoint url>` is the URL of the GitLab [endpoint](#endpoint-urls). + - `<path to settings.xml>` is the path to the `settings.xml` file that contains the [authentication details](#edit-the-settingsxml). NOTE: The repository IDs in the command(`gitlab-maven`) and the `settings.xml` file must match. @@ -645,34 +197,34 @@ The message should show that the package is downloading from the Package Registr Downloading from gitlab-maven: http://gitlab.example.com/api/v4/projects/PROJECT_ID/packages/maven/com/mycompany/mydepartment/my-project/1.0-SNAPSHOT/my-project-1.0-20200128.120857-1.pom ``` -NOTE: -In the GitLab UI, on the Package Registry page for Maven, you can view and copy these commands. +## Helpful hints -### Use Gradle +### Publishing a package with the same name or version -Add a [dependency](https://docs.gradle.org/current/userguide/declaring_dependencies.html) to `build.gradle` in the dependencies section: +When you publish a package with the same name and version as an existing package, the new package +files are added to the existing package. You can still use the UI or API to access and view the +existing package's older assets. -```groovy -dependencies { - implementation 'com.mycompany.mydepartment:my-project:1.0-SNAPSHOT' -} -``` +To delete older package versions, consider using the Packages API or the UI. -Or to `build.gradle.kts` if you are using Kotlin DSL: +### Do not allow duplicate Maven packages -```kotlin -dependencies { - implementation("com.mycompany.mydepartment:my-project:1.0-SNAPSHOT") -} -``` +To prevent users from publishing duplicate Maven packages, you can use the [GraphQl API](../../../api/graphql/reference/index.md#packagesettings) or the UI. -### Request forwarding to Maven Central +In the UI: + +1. For your group, go to **Settings > Packages and registries**. +1. Expand the **Package Registry** section. +1. Turn on the **Do not allow duplicates** toggle. +1. Optional. To allow some duplicate packages, in the **Exceptions** box, enter a regex pattern that matches the names and/or versions of packages you want to allow. + +Your changes are automatically saved. -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/362657) behind a [feature flag](../../feature_flags.md), disabled by default in GitLab 15.4 +### Request forwarding to Maven Central FLAG: -On self-managed GitLab, by default this feature is not available. To make it available, ask an administrator to [enable the feature flag](../../../administration/feature_flags.md) named `maven_central_request_forwarding`. -On GitLab.com, this feature is not available. +By default this feature is not available for self-managed. To make it available, ask an administrator to [enable the feature flag](../../../administration/feature_flags.md) named `maven_central_request_forwarding`. +This feature is not available for SaaS users. When a Maven package is not found in the Package Registry, the request is forwarded to [Maven Central](https://search.maven.org/). @@ -690,8 +242,8 @@ GitLab needs to be configured to be queried before maven-central. [Using GitLab as a mirror of the central proxy](#setting-gitlab-as-a-mirror-for-the-central-proxy) is one way to force GitLab to be queried in place of maven-central. -Maven forwarding is restricted to only the [project level](#project-level-maven-endpoint) and -[group level](#group-level-maven-endpoint) endpoints. The [instance level endpoint](#instance-level-maven-endpoint) +Maven forwarding is restricted to only the project level and +group level [endpoints](#naming-convention). The instance level endpoint has naming restrictions that prevent it from being used for packages that don't follow that convention and also introduces too much security risk for supply-chain style attacks. @@ -710,7 +262,7 @@ section to your `settings.xml`: <httpHeaders> <property> <name>Private-Token</name> - <value>{personal_access_token}</value> + <value><personal_access_token></value> </property> </httpHeaders> </configuration> @@ -720,25 +272,19 @@ section to your `settings.xml`: <mirror> <id>central-proxy</id> <name>GitLab proxy of central repo</name> - <url>https://gitlab.example.com/api/v4/projects/{project_id}/packages/maven</url> + <url>https://gitlab.example.com/api/v4/projects/<project_id>/packages/maven</url> <mirrorOf>central</mirrorOf> </mirror> </mirrors> </settings> ``` -## Remove a package - -For your project, go to **Packages and registries > Package Registry**. - -To remove a package, select the red trash icon or, from the package details, the **Delete** button. - -## Create Maven packages with GitLab CI/CD +### Create Maven packages with GitLab CI/CD After you have configured your repository to use the Package Repository for Maven, you can configure GitLab CI/CD to build new packages automatically. -### Create Maven packages with GitLab CI/CD by using Maven +### Create Maven packages with GitLab CI/CD using Maven You can create a new package each time the `main` branch is updated. @@ -808,37 +354,51 @@ user's home location. In this example: - The user is `root`, because the job runs in a Docker container. - Maven uses the configured CI/CD variables. -### Create Maven packages with GitLab CI/CD by using Gradle +### Version validation -You can create a package each time the `main` branch -is updated. +The version string is validated by using the following regex. -1. Authenticate with [a CI job token in Gradle](#authenticate-with-a-ci-job-token-in-gradle). +```ruby +\A(?!.*\.\.)[\w+.-]+\z +``` -1. Add a `deploy` job to your `.gitlab-ci.yml` file: +You can experiment with the regex and try your version strings on [this regular expression editor](https://rubular.com/r/rrLQqUXjfKEoL6). - ```yaml - deploy: - image: gradle:6.5-jdk11 - script: - - 'gradle publish' - only: - - main - ``` +### Useful Maven command-line options -1. Commit files to your repository. +There are some [Maven command-line options](https://maven.apache.org/ref/current/maven-embedder/cli.html) +that you can use when performing tasks with GitLab CI/CD. -When the pipeline is successful, the package is created. +- File transfer progress can make the CI logs hard to read. + Option `-ntp,--no-transfer-progress` was added in + [3.6.1](https://maven.apache.org/docs/3.6.1/release-notes.html#User_visible_Changes). + Alternatively, look at `-B,--batch-mode` + [or lower level logging changes.](https://stackoverflow.com/questions/21638697/disable-maven-download-progress-indication) -### Version validation +- Specify where to find the `pom.xml` file (`-f,--file`): -The version string is validated by using the following regex. + ```yaml + package: + script: + - 'mvn --no-transfer-progress -f helloworld/pom.xml package' + ``` -```ruby -\A(?!.*\.\.)[\w+.-]+\z -``` +- Specify where to find the user settings (`-s,--settings`) instead of + [the default location](https://maven.apache.org/settings.html). There's also a `-gs,--global-settings` option: + + ```yaml + package: + script: + - 'mvn -s settings/ci.xml package' + ``` -You can play around with the regex and try your version strings on [this regular expression editor](https://rubular.com/r/rrLQqUXjfKEoL6). +### Supported CLI commands + +The GitLab Maven repository supports the following Maven CLI commands: + +- `mvn deploy`: Publish your package to the Package Registry. +- `mvn install`: Install packages specified in your Maven project. +- `mvn dependency:get`: Install a specific package. ## Troubleshooting @@ -870,34 +430,6 @@ mvn deploy \ WARNING: When you set these options, all network requests are logged and a large amount of output is generated. -### Useful Maven command-line options - -There are some [Maven command-line options](https://maven.apache.org/ref/current/maven-embedder/cli.html) -that you can use when performing tasks with GitLab CI/CD. - -- File transfer progress can make the CI logs hard to read. - Option `-ntp,--no-transfer-progress` was added in - [3.6.1](https://maven.apache.org/docs/3.6.1/release-notes.html#User_visible_Changes). - Alternatively, look at `-B,--batch-mode` - [or lower level logging changes.](https://stackoverflow.com/questions/21638697/disable-maven-download-progress-indication) - -- Specify where to find the `pom.xml` file (`-f,--file`): - - ```yaml - package: - script: - - 'mvn --no-transfer-progress -f helloworld/pom.xml package' - ``` - -- Specify where to find the user settings (`-s,--settings`) instead of - [the default location](https://maven.apache.org/settings.html). There's also a `-gs,--global-settings` option: - - ```yaml - package: - script: - - 'mvn -s settings/ci.xml package' - ``` - ### Verify your Maven settings If you encounter issues within CI/CD that relate to the `settings.xml` file, try adding @@ -916,11 +448,3 @@ package: - 'mvn help:system' - 'mvn package' ``` - -## Supported CLI commands - -The GitLab Maven repository supports the following Maven CLI commands: - -- `mvn deploy`: Publish your package to the Package Registry. -- `mvn install`: Install packages specified in your Maven project. -- `mvn dependency:get`: Install a specific package. diff --git a/doc/user/packages/npm_registry/index.md b/doc/user/packages/npm_registry/index.md index 5d2efc52ba9..c62999100c1 100644 --- a/doc/user/packages/npm_registry/index.md +++ b/doc/user/packages/npm_registry/index.md @@ -6,250 +6,97 @@ info: To determine the technical writer assigned to the Stage/Group associated w # npm packages in the Package Registry **(FREE)** -> [Moved](https://gitlab.com/gitlab-org/gitlab/-/issues/221259) from GitLab Premium to GitLab Free in 13.3. - -Publish npm packages in your project's Package Registry. Then install the -packages whenever you need to use them as a dependency. - -Only [scoped](https://docs.npmjs.com/misc/scope/) packages are supported. - -For documentation of the specific API endpoints that the npm package manager -client uses, see the [npm API documentation](../../../api/packages/npm.md). - -WARNING: -Never hardcode GitLab tokens (or any tokens) directly in `.npmrc` files or any other files that can -be committed to a repository. +For documentation of the specific API endpoints that the npm package manager client uses, see the [npm API documentation](../../../api/packages/npm.md). Learn how to build an [npm](../workflows/build_packages.md#npm) or [yarn](../workflows/build_packages.md#yarn) package. -## Use the GitLab endpoint for npm packages - -To use the GitLab endpoint for npm packages, choose an option: +Watch a [video demo](https://youtu.be/yvLxtkvsFDA) of how to publish npm packages to the GitLab Package Registry. -- **Project-level**: Use when you have few npm packages and they are not in - the same GitLab group. The [package naming convention](#package-naming-convention) is not enforced at this level. - Instead, you should use a [scope](https://docs.npmjs.com/cli/v6/using-npm/scope/) for your package. - When you use a scope, the registry URL is [updated](#authenticate-to-the-package-registry) only for that scope. -- **Instance-level**: Use when you have many npm packages in different - GitLab groups or in their own namespace. Be sure to comply with the [package naming convention](#package-naming-convention). +## Publish to GitLab Package Registry -Some features such as [publishing](#publish-an-npm-package) a package is only available on the project-level endpoint. +### Authentication to the Package Registry -## Authenticate to the Package Registry +You need an token to publish a package. There are different tokens available depending on what you're trying to achieve. For more information, review the [guidance on tokens](../../../user/packages/package_registry/index.md#authenticate-with-the-registry). -You must authenticate with the Package Registry when the project -is private. Public projects do not require authentication. +- If your organization uses two factor authentication (2FA), you must use a personal access token with the scope set to `api`. +- If you are publishing a package via CI/CD pipelines, you must use a CI job token. -To authenticate, use one of the following: +Create a token and save it to use later in the process. -- A [personal access token](../../../user/profile/personal_access_tokens.md) - (required for two-factor authentication (2FA)), with the scope set to `api`. -- A [deploy token](../../project/deploy_tokens/index.md), with the scope set to `read_package_registry`, `write_package_registry`, or both. -- It's not recommended, but you can use [OAuth tokens](../../../api/oauth2.md#resource-owner-password-credentials-flow). - Standard OAuth tokens cannot authenticate to the GitLab npm Registry. You must use a personal access token with OAuth headers. -- A [CI job token](#authenticate-with-a-ci-job-token). -- Your npm package name must be in the format of [`@scope/package-name`](#package-naming-convention). - It must match exactly, including the case. +### Naming convention -### Authenticate with a personal access token or deploy token +Depending on how the package will be installed, you may need to adhere to the naming convention. -To authenticate with the Package Registry, you need a [personal access token](../../profile/personal_access_tokens.md) or [deploy token](../../project/deploy_tokens/index.md). +You can use one of two API endpoints to install packages: -#### Project-level npm endpoint +- **Instance-level**: Use when you have many npm packages in different GitLab groups or in their own namespace. +- **Project-level**: Use when you have few npm packages and they are not in the same GitLab group. -To use the [project-level](#use-the-gitlab-endpoint-for-npm-packages) npm endpoint, set your npm configuration: +If you plan to install a package through the [project level](#install-from-the-project-level), then you do not have to adhere to the naming convention. -```shell -# Set URL for your scoped packages. -# For example package with name `@foo/bar` will use this URL for download -npm config set @foo:registry https://gitlab.example.com/api/v4/projects/<your_project_id>/packages/npm/ - -# Add the token for the scoped packages URL. Replace <your_project_id> -# with the project where your package is located. -npm config set -- '//gitlab.example.com/api/v4/projects/<your_project_id>/packages/npm/:_authToken' "<your_token>" -``` - -- `<your_project_id>` is your project ID, found on the project's home page. -- `<your_token>` is your personal access token or deploy token. -- Replace `gitlab.example.com` with your domain name. +If you plan to install a package through the [instance level](#install-from-the-instance-level), then you must name your package with a [scope](https://docs.npmjs.com/misc/scope/). Scoped packages begin with a `@` have the format of `@owner/package-name`. You can set up the scope for your package in the `.npmrc` file and by using the `publishConfig` option in the `package.json`. -You should now be able to publish and install npm packages in your project. +- The value used for the `@scope` is the root of the project that will host the packages and not the root + of the project with the source code of the package itself. The scope should be lowercase. +- The package name can be anything you want -If you encounter an error with [Yarn](https://classic.yarnpkg.com/en/), view -[troubleshooting steps](#troubleshooting). +| Project URL | Package Registry in | Scope | Full package name | +| ------------------------------------------------------- | ------------------- | --------- | ---------------------- | +| `https://gitlab.com/my-org/engineering-group/analytics` | Analytics | `@my-org` | `@my-org/package-name` | -#### Instance-level npm endpoint - -NOTE: -Note: Using `CI_JOB_TOKEN` to install npm packages with dependencies in another project will give you 404 errors. You can use a [personal access token](../../profile/personal_access_tokens.md) as a workaround. [GitLab-#352962](https://gitlab.com/gitlab-org/gitlab/-/issues/352962) proposes a fix to this bug. - -To use the [instance-level](#use-the-gitlab-endpoint-for-npm-packages) npm endpoint, set your npm configuration: +Make sure that the name of your package in the `package.json` file matches this convention: ```shell -# Set URL for your scoped packages. -# For example package with name `@foo/bar` will use this URL for download -npm config set @foo:registry https://gitlab.example.com/api/v4/packages/npm/ - -# Add the token for the scoped packages URL. This will allow you to download -# `@foo/` packages from private projects. -npm config set -- '//gitlab.example.com/api/v4/packages/npm/:_authToken' "<your_token>" -``` - -- `<your_token>` is your personal access token or deploy token. -- Replace `gitlab.example.com` with your domain name. - -You should now be able to install npm packages in your project. - -If you encounter an error with [Yarn](https://classic.yarnpkg.com/en/), view -[troubleshooting steps](#troubleshooting). - -### Authenticate with a CI job token - -> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/9104) in GitLab 12.5. -> - [Moved](https://gitlab.com/gitlab-org/gitlab/-/issues/221259) from GitLab Premium to GitLab Free in 13.3. - -If you're using npm with GitLab CI/CD, a CI job token can be used instead of a personal access token or deploy token. -The token inherits the permissions of the user that generates the pipeline. - -#### Project-level npm endpoint - -To use the [project-level](#use-the-gitlab-endpoint-for-npm-packages) npm endpoint, add a corresponding section to your `.npmrc` file: - -```ini -@foo:registry=https://gitlab.example.com/api/v4/projects/${CI_PROJECT_ID}/packages/npm/ -//gitlab.example.com/api/v4/projects/${CI_PROJECT_ID}/packages/npm/:_authToken=${CI_JOB_TOKEN} +"name": "@my-org/package-name" ``` -#### Instance-level npm endpoint - -To use the [instance-level](#use-the-gitlab-endpoint-for-npm-packages) npm endpoint, add a corresponding section to your `.npmrc` file: - -```ini -@foo:registry=https://gitlab.example.com/api/v4/packages/npm/ -//gitlab.example.com/api/v4/packages/npm/:_authToken=${CI_JOB_TOKEN} -``` +## Publishing a package via the command line -#### Use variables to avoid hard-coding auth token values +### Authenticating via the `.npmrc` -To avoid hard-coding the `authToken` value, you may use a variable in its place: +Create or edit the `.npmrc` file in the same directory as your `package.json`. Include the following lines in the `.npmrc` file: ```shell -npm config set -- '//gitlab.example.com/api/v4/projects/<your_project_id>/packages/npm/:_authToken' "${NPM_TOKEN}" -npm config set -- '//gitlab.example.com/api/v4/packages/npm/:_authToken' "${NPM_TOKEN}" +@scope:registry=https://your_domain_name/api/v4/projects/your_project_id/packages/npm/ +//your_domain_name/api/v4/projects/your_project_id/packages/npm/:_authToken="${NPM_TOKEN}" ``` -Then, you can run `npm publish` either locally or by using GitLab CI/CD. +- Replace `@scope` with the [root level group](#naming-convention) of the project you're publishing to the package to. +- Replace `your_domain_name` with your domain name, for example, `gitlab.com`. +- Replace `your_project_id` is your project ID, found on the project's home page. +- `"${NPM_TOKEN}"` will be associated with the token you created later in the process. -- **Locally:** Export `NPM_TOKEN` before publishing: - - ```shell - NPM_TOKEN=<your_token> npm publish - ``` - -- **GitLab CI/CD:** Set an `NPM_TOKEN` [CI/CD variable](../../../ci/variables/index.md) - under your project's **Settings > CI/CD > Variables**. +WARNING: +Never hardcode GitLab tokens (or any tokens) directly in `.npmrc` files or any other files that can +be committed to a repository. -## Working with private registries +### Publishing a package via the command line -When working with private repositories, you may want to configure additional settings to ensure a secure communication channel: +Associate your [token](#authentication-to-the-package-registry) with the `"${NPM_TOKEN}"` in the `.npmrc`. Replace `your_token` with a deploy token, group access token, project access token, or personal access token. ```shell -# Force npm to always require authentication when accessing the registry, even for GET requests. -npm config set always-auth true -``` - -## Package naming convention - -When you use the [instance-level endpoint](#use-the-gitlab-endpoint-for-npm-packages), only the packages with names in the format of `@scope/package-name` are available. - -- The `@scope` is the root namespace of the GitLab project. To follow npm's convention, it should be - lowercase. However, the GitLab package registry allows for uppercase. Before GitLab 13.10, the - `@scope` had to be a case-sensitive match of the GitLab project's root namespace. This was - problematic because the npm public registry does not allow uppercase letters. GitLab 13.10 relaxes - this requirement and translates uppercase in the GitLab `@scope` to lowercase for npm. For - example, a package `@MyScope/package-name` in GitLab becomes `@myscope/package-name` for npm. -- The `package-name` can be whatever you want. - -NOTE: -The value used for the `@scope` is the root of the project that will end up hosting the packages and not the root -of the project with the source code of the package itself. For example, assume your package source code is located -at `source-code-group/package-code` and deployed to a package registry inside `registries-group/registry-project`. -In this case, the `@scope` needs to be `@registries-group` and not `@source-code-group`. - -For example, if your project is `https://gitlab.example.com/my-org/engineering-group/team-amazing/analytics`, -the root namespace is `my-org`. When you publish a package, it must have `my-org` as the scope. - -| Project | Package | Supported | -| ------------------- | -------------------- | --------- | -| `my-org/bar` | `@my-org/bar` | Yes | -| `my-org/bar/baz` | `@my-org/baz` | Yes | -| `My-Org/Bar/baz` | `@my-org/Baz` | Yes | -| `My-Org/Bar/baz` | `@My-Org/Baz` | Yes | -| `my-org/bar/buz` | `@my-org/anything` | Yes | -| `gitlab-org/gitlab` | `@gitlab-org/gitlab` | Yes | -| `gitlab-org/gitlab` | `@foo/bar` | No | - -In GitLab, this regex validates all package names from all package managers: - -```plaintext -/\A\@?(([\w\-\.\+]*)\/)*([\w\-\.]+)@?(([\w\-\.\+]*)\/)*([\w\-\.]*)\z/ +NPM_TOKEN=your_token npm publish ``` -This regex allows almost all of the characters that npm allows, with a few exceptions (for example, `~` is not allowed). - -The regex also allows for capital letters, while npm does not. +Your package should now publish to the Package Registry. -## Limitations +## Publishing a package via a CI/CD pipeline -When you update the path of a user or group, or transfer a subgroup or project, -you must remove any npm packages first. You cannot update the root namespace -of a project with npm packages. Make sure you update your `.npmrc` files to follow -the naming convention and run `npm publish` if necessary. +### Authenticating via the `.npmrc` -## Publish an npm package - -Prerequisites: - -- [Authenticate](#authenticate-to-the-package-registry) to the Package Registry. -- Set a [project-level npm endpoint](#use-the-gitlab-endpoint-for-npm-packages). - -To upload an npm package to your project, run this command: +Create or edit the `.npmrc` file in the same directory as your `package.json` in a GitLab project. Include the following lines in the `.npmrc` file: ```shell -npm publish +@scope:registry=https://your_domain_name/api/v4/projects/your_project_id/packages/npm/ +//your_domain_name/api/v4/projects/${CI_PROJECT_ID}/packages/npm/:_authToken=${CI_JOB_TOKEN} ``` -To view the package, go to your project's **Packages and registries**. +- Replace `@scope` with the [root level group](#naming-convention) of the project you're publishing to the package to. +- The `${CI_PROJECT_ID}` and `${CI_JOB_TOKEN}` are [predefined variables](../../../ci/variables/predefined_variables.md) that are available in the pipeline and do not need to be replaced. -You can also define `"publishConfig"` for your project in `package.json`. For example: +### Publishing a package via a CI/CD pipeline -```json -{ - "publishConfig": { - "@foo:registry": " https://gitlab.example.com/api/v4/projects/<your_project_id>/packages/npm/" - } -} -``` - -This forces the package to publish only to the specified registry. - -If you try to publish a package [with a name that already exists](#publishing-packages-with-the-same-name-or-version) within -a given scope, you get a `403 Forbidden!` error. - -## Publish an npm package by using CI/CD - -Prerequisites: - -- [Authenticate](#authenticate-to-the-package-registry) to the Package Registry. -- Set a [project-level npm endpoint](#use-the-gitlab-endpoint-for-npm-packages). -- Your npm package name must be in the format of [`@scope/package-name`](#package-naming-convention). - It must match exactly, including the case. This is different than the - npm naming convention, but it is required to work with the GitLab Package Registry. - -To work with npm commands within [GitLab CI/CD](../../../ci/index.md), you can use -`CI_JOB_TOKEN` in place of the personal access token or deploy token in your commands. - -An example `.gitlab-ci.yml` file for publishing npm packages: +In the GitLab project that houses your `.npmrc` and `package.json`, edit or create a `.gitlab-ci.yml` file. For example: ```yaml image: node:latest @@ -262,143 +109,105 @@ deploy: script: - echo "//${CI_SERVER_HOST}/api/v4/projects/${CI_PROJECT_ID}/packages/npm/:_authToken=${CI_JOB_TOKEN}">.npmrc - npm publish - environment: production ``` -See the -[Publish npm packages to the GitLab Package Registry using semantic-release](../../../ci/examples/semantic-release.md) -step-by-step guide and demo project for a complete example. - -## Configure the GitLab npm registry with Yarn 2 - -You can get started with Yarn 2 by following the [Yarn documentation](https://yarnpkg.com/getting-started/install/). +Your package should now publish to the Package Registry when the pipeline runs. -To publish and install with the project-level npm endpoint, set the following configuration in -`.yarnrc.yml`: +## Install a package -```yaml -npmScopes: - foo: - npmRegistryServer: 'https://gitlab.example.com/api/v4/projects/<your_project_id>/packages/npm/' - npmPublishRegistry: 'https://gitlab.example.com/api/v4/projects/<your_project_id>/packages/npm/' - -npmRegistries: - //gitlab.example.com/api/v4/projects/<your_project_id>/packages/npm/: - npmAlwaysAuth: true - npmAuthToken: '<your_token>' -``` +If multiple packages have the same name and version, when you install a package, the most recently-published package is retrieved. -For the instance-level npm endpoint, use this Yarn 2 configuration in `.yarnrc.yml`: +You can install a package from a GitLab project or instance: -```yaml -npmScopes: - foo: - npmRegistryServer: 'https://gitlab.example.com/api/v4/packages/npm/' - -npmRegistries: - //gitlab.example.com/api/v4/packages/npm/: - npmAlwaysAuth: true - npmAuthToken: '<your_token>' -``` +- **Instance-level**: Use when you have many npm packages in different GitLab groups or in their own namespace. +- **Project-level**: Use when you have few npm packages and they are not in the same GitLab group. -In this configuration: +### Install from the instance level -- Replace `<your_token>` with your personal access token or deploy token. -- Replace `<your_project_id>` with your project's ID, which you can find on the project's home page. -- Replace `gitlab.example.com` with your domain name. -- Your scope is `foo`, without `@`. +WARNING: +In order to install a package from the instance level, the package must have been published following the scoped [naming convention](#naming-convention). -## Publishing packages with the same name or version +1. Authenticate to the Package Registry -You cannot publish a package if a package of the same name and version already exists. -You must delete the existing package first. + If you would like to install a package from a private project, you will need to authenticate to the Package Registry. Skip this step if the project is not private. -This rule has a different impact depending on the package name: + ```shell + npm config set -- //your_domain_name/api/v4/packages/npm/:_authToken=your_token + ``` -- For packages following the [naming convention](#package-naming-convention), you can't publish a - package with a duplicate name and version to the root namespace. -- For packages not following the [naming convention](#package-naming-convention), you can't publish - a package with a duplicate name and version to the project you target with the upload. + - Replace `your_domain_name` with your domain name, for example, `gitlab.com`. + - Replace `your_token` with a deploy token, group access token, project access token, or personal access token. -This aligns with npmjs.org's behavior. However, npmjs.org does not ever let you publish -the same version more than once, even if it has been deleted. +1. Set the registry -## `package.json` limitations + ```shell + npm config set @scope:registry https://your_domain_name.com/api/v4/packages/npm/ + ``` -You can't publish a package if its `package.json` file exceeds 20,000 characters. + - Replace `@scope` with the [root level group](#naming-convention) of the project you're installing to the package from. + - Replace `your_domain_name` with your domain name, for example `gitlab.com`. + - Replace `your_token` with a deploy token, group access token, project access token, or personal access token. -## Install a package +1. Install the package -npm packages are commonly-installed by using the `npm` or `yarn` commands -in a JavaScript project. You can install a package from the scope of a project or instance. + ```shell + npm install @scope/my-package + ``` -If multiple packages have the same name and version, when you install a package, the most recently-published package is retrieved. +### Install from the project level -1. Set the URL for scoped packages. +1. Authenticate to the Package Registry - For [instance-level endpoints](#use-the-gitlab-endpoint-for-npm-packages) run: + If you would like to install a package from a private project, you will need to authenticate to the Package Registry. Skip this step if the project is not private. ```shell - npm config set @foo:registry https://gitlab.example.com/api/v4/packages/npm/ + npm config set -- //your_domain_name/api/v4/projects/your_project_id/packages/npm/:_authToken=your_token ``` - - Replace `@foo` with your scope. - - Replace `gitlab.example.com` with your domain name. + - Replace `your_domain_name` with your domain name, for example, `gitlab.com`. + - Replace `your_project_id` is your project ID, found on the project's home page. + - Replace `your_token` with a deploy token, group access token, project access token, or personal access token. - For [project-level endpoints](#use-the-gitlab-endpoint-for-npm-packages) run: +1. Set the registry ```shell - npm config set @foo:registry https://gitlab.example.com/api/v4/projects/<your_project_id>/packages/npm/ + npm config set @scope:registry=https://your_domain_name/api/v4/projects/your_project_id/packages/npm/ ``` - - Replace `@foo` with your scope. - - Replace `gitlab.example.com` with your domain name. - - Replace `<your_project_id>` with your project ID, found on the project's home page. + - Replace `@scope` with the [root level group](#naming-convention) of the project you're installing to the package from. + - Replace `your_domain_name` with your domain name, for example, `gitlab.com`. + - Replace `your_project_id` is your project ID, found on the project's home page. -1. Ensure [authentication](#authenticate-to-the-package-registry) is configured. - -1. To install a package in your project, run: +1. Install the package ```shell - npm install @my-scope/my-package + npm install @scope/my-package ``` - Or if you're using Yarn: +## Helpful hints - ```shell - yarn add @my-scope/my-package - ``` +### Package forwarding to npmjs.com -In [GitLab 12.9 and later](https://gitlab.com/gitlab-org/gitlab/-/issues/55344), -when an npm package is not found in the Package Registry, the request is forwarded to [npmjs.com](https://www.npmjs.com/). +When an npm package is not found in the Package Registry, the request is forwarded to [npmjs.com](https://www.npmjs.com/). Administrators can disable this behavior in the [Continuous Integration settings](../../admin_area/settings/continuous_integration.md). +Group owners can disable this behavior in the group Packages and Registries settings. + ### Install npm packages from other organizations You can route package requests to organizations and users outside of GitLab. -To do this, add lines to your `.npmrc` file. Replace `my-org` with the namespace or group that owns your project's repository, +To do this, add lines to your `.npmrc` file. Replace `@my-other-org` with the namespace or group that owns your project's repository, and use your organization's URL. The name is case-sensitive and must match the name of your group or namespace exactly. -Use environment variables to set up your tokens: `export MY_TOKEN="<your token>"`. - ```shell -@foo:registry=https://gitlab.example.com/api/v4/packages/npm/ -//gitlab.example.com/api/v4/packages/npm/:_authToken=${MY_TOKEN} -//gitlab.example.com/api/v4/projects/<your_project_id>/packages/npm/:_authToken=${MY_TOKEN} - -@my-other-org:registry=https://gitlab.example.com/api/v4/packages/npm/ -//gitlab.example.com/api/v4/packages/npm/:_authToken=${MY_TOKEN} -//gitlab.example.com/api/v4/projects/<your_project_id>/packages/npm/:_authToken=${MY_TOKEN} +@scope:registry=https://my_domain_name.com/api/v4/packages/npm/ +@my-other-org:registry=https://my_domain_name.example.com/api/v4/packages/npm/ ``` ### npm metadata -> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/11867) in GitLab 12.6. -> - [Moved](https://gitlab.com/gitlab-org/gitlab/-/issues/221259) from GitLab Premium to GitLab Free in 13.3. -> - [Improved](https://gitlab.com/gitlab-org/gitlab/-/issues/330929) in GitLab 14.5. - The GitLab Package Registry exposes the following attributes to the npm client. These are similar to the [abbreviated metadata format](https://github.com/npm/registry/blob/9e368cf6aaca608da5b2c378c0d53f475298b916/docs/responses/package-metadata.md#abbreviated-metadata-format): @@ -417,10 +226,7 @@ These are similar to the [abbreviated metadata format](https://github.com/npm/re - `engines` - `_hasShrinkwrap` -## Add npm distribution tags - -> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/9425) in GitLab 12.8. -> - [Moved](https://gitlab.com/gitlab-org/gitlab/-/issues/221259) from GitLab Premium to GitLab Free in 13.3. +### Add npm distribution tags You can add [distribution tags](https://docs.npmjs.com/cli/dist-tag/) to newly-published packages. Tags are optional and can be assigned to only one package at a time. @@ -443,87 +249,46 @@ View [this issue](https://gitlab.com/gitlab-org/gitlab/-/issues/258835) for deta Due to a bug in npm 6.9.0, deleting distribution tags fails. Make sure your npm version is 6.9.1 or later. -## Troubleshooting - -When troubleshooting npm issues, first run the same command with the `--verbose` flag to confirm -what registry you are hitting. - -To improve performance, npm caches files related to a package. Note that npm doesn't remove data by -itself. The cache grows as new packages are installed. If you encounter issues, clear the cache with -this command: - -```shell -npm cache clean --force -``` - -### Error running Yarn with the Package Registry for npm registry - -If you are using [Yarn](https://classic.yarnpkg.com/en/) with the npm registry, you may get -an error message like: - -```shell -yarn install v1.15.2 -warning package.json: No license field -info No lockfile found. -warning XXX: No license field -[1/4] 🔍 Resolving packages... -[2/4] 🚚 Fetching packages... -error An unexpected error occurred: "https://gitlab.example.com/api/v4/projects/XXX/packages/npm/XXX/XXX/-/XXX/XXX-X.X.X.tgz: Request failed \"404 Not Found\"". -info If you think this is a bug, please open a bug report with the information provided in "/Users/XXX/gitlab-migration/module-util/yarn-error.log". -info Visit https://classic.yarnpkg.com/en/docs/cli/install for documentation about this command -``` - -In this case, try adding this to your `.npmrc` file (and replace `<your_token>` -with your personal access token or deploy token): +### Supported CLI commands -```plaintext -//gitlab.example.com/api/v4/projects/:_authToken=<your_token> -``` +The GitLab npm repository supports the following commands for the npm CLI (`npm`) and yarn CLI +(`yarn`): -You can also use `yarn config` instead of `npm config` when setting your auth-token dynamically: +- `npm install`: Install npm packages. +- `npm publish`: Publish an npm package to the registry. +- `npm dist-tag add`: Add a dist-tag to an npm package. +- `npm dist-tag ls`: List dist-tags for a package. +- `npm dist-tag rm`: Delete a dist-tag. +- `npm ci`: Install npm packages directly from your `package-lock.json` file. +- `npm view`: Show package metadata. -```shell -yarn config set '//gitlab.example.com/api/v4/projects/<your_project_id>/packages/npm/:_authToken' "<your_token>" -yarn config set '//gitlab.example.com/api/v4/packages/npm/:_authToken' "<your_token>" -``` +## Troubleshooting ### `npm publish` targets default npm registry (`registry.npmjs.org`) Ensure that your package scope is set consistently in your `package.json` and `.npmrc` files. -For example, if your project name in GitLab is `foo/my-package`, then your `package.json` file +For example, if your project name in GitLab is `@scope/my-package`, then your `package.json` file should look like: ```json { - "name": "@foo/my-package", - "version": "1.0.0", - "description": "Example package for GitLab npm registry" + "name": "@scope/my-package" } ``` And the `.npmrc` file should look like: -```ini -//gitlab.example.com/api/v4/projects/<your_project_id>/packages/npm/:_authToken=<your_token> -//gitlab.example.com/api/v4/packages/npm/:_authToken=<your_token> -@foo:registry=https://gitlab.example.com/api/v4/packages/npm/ -``` - -### `npm install` returns `Error: Failed to replace env in config: ${npm_TOKEN}` - -You do not need a token to run `npm install` unless your project is private. The token is only required to publish. If the `.npmrc` file was checked in with a reference to `$npm_TOKEN`, you can remove it. If you prefer to leave the reference in, you must set a value prior to running `npm install` or set the value by using [GitLab CI/CD variables](../../../ci/variables/index.md): - ```shell -NPM_TOKEN=<your_token> npm install +@scope:registry=https://your_domain_name/api/v4/projects/your_project_id/packages/npm/ +//your_domain_name/api/v4/projects/your_project_id/packages/npm/:_authToken="${NPM_TOKEN}" ``` ### `npm install` returns `npm ERR! 403 Forbidden` If you get this error, ensure that: -- The Package Registry is enabled in your project settings. Although the Package Registry is enabled - by default, it's possible to [disable it](../package_registry/index.md#disable-the-package-registry). +- The Package Registry is enabled in your project settings. Although the Package Registry is enabled by default, it's possible to [disable it](../package_registry/index.md#disable-the-package-registry). - Your token is not expired and has appropriate permissions. - A package with the same name or version doesn't already exist within the given scope. - The scoped packages URL includes a trailing slash: @@ -534,30 +299,25 @@ If you get this error, ensure that: If you get this error, one of the following problems could be causing it. -#### Package name does not meet the naming convention +### Package name does not meet the naming convention -Your package name may not meet the -[`@scope/package-name` package naming convention](#package-naming-convention). +Your package name may not meet the [`@scope/package-name` package naming convention](#naming-convention). -Ensure the name meets the convention exactly, including the case. -Then try to publish again. +Ensure the name meets the convention exactly, including the case. Then try to publish again. -#### Package already exists +### Package already exists -Your package has already been published to another project in the same -root namespace and therefore cannot be published again using the same name. +Your package has already been published to another project in the same root namespace and therefore cannot be published again using the same name. -This is also true even if the prior published package shares the same name, -but not the version. +This is also true even if the prior published package shares the same name, but not the version. -#### Package JSON file is too large +### Package JSON file is too large -Make sure that your `package.json` file does not [exceed `20,000` characters](#packagejson-limitations). +Make sure that your `package.json` file does not exceed `20,000` characters. ### `npm publish` returns `npm ERR! 500 Internal Server Error - PUT` -This is a [known issue](https://gitlab.com/gitlab-org/gitlab/-/issues/238950) in GitLab -13.3.x and later. The error in the logs will appear as: +This is a [known issue](https://gitlab.com/gitlab-org/gitlab/-/issues/238950) in GitLab 13.3.x and later. The error in the logs will appear as: ```plaintext >NoMethodError - undefined method `preferred_language' for #<Rack::Response @@ -572,22 +332,7 @@ This might be accompanied by another error: This is usually a permissions issue with either: - `'packages_storage_path'` default `/var/opt/gitlab/gitlab-rails/shared/packages/`. -- The remote bucket if [object storage](../../../administration/packages/index.md#using-object-storage) +- The remote bucket if [object storage](../../../administration/packages/index.md#use-object-storage) is used. In the latter case, ensure the bucket exists and GitLab has write access to it. - -## Supported CLI commands - -The GitLab npm repository supports the following commands for the npm CLI (`npm`) and yarn CLI -(`yarn`): - -- `npm install`: Install npm packages. -- `npm publish`: Publish an npm package to the registry. -- `npm dist-tag add`: Add a dist-tag to an npm package. -- `npm dist-tag ls`: List dist-tags for a package. -- `npm dist-tag rm`: Delete a dist-tag. -- `npm ci`: Install npm packages directly from your `package-lock.json` file. -- `npm view`: Show package metadata. -- `yarn add`: Install an npm package. -- `yarn update`: Update your dependencies. diff --git a/doc/user/packages/nuget_repository/img/visual_studio_adding_nuget_source.png b/doc/user/packages/nuget_repository/img/visual_studio_adding_nuget_source.png Binary files differdeleted file mode 100644 index 7397403f4bf..00000000000 --- a/doc/user/packages/nuget_repository/img/visual_studio_adding_nuget_source.png +++ /dev/null diff --git a/doc/user/packages/nuget_repository/img/visual_studio_nuget_source_added.png b/doc/user/packages/nuget_repository/img/visual_studio_nuget_source_added.png Binary files differdeleted file mode 100644 index 8c14a14e304..00000000000 --- a/doc/user/packages/nuget_repository/img/visual_studio_nuget_source_added.png +++ /dev/null diff --git a/doc/user/packages/nuget_repository/index.md b/doc/user/packages/nuget_repository/index.md index 956202bb990..540db463f0a 100644 --- a/doc/user/packages/nuget_repository/index.md +++ b/doc/user/packages/nuget_repository/index.md @@ -118,27 +118,25 @@ A project-level endpoint is also required to install NuGet packages from a proje To use the [project-level](#use-the-gitlab-endpoint-for-nuget-packages) NuGet endpoint, add the Package Registry as a source with Visual Studio: 1. Open [Visual Studio](https://visualstudio.microsoft.com/vs/). -1. In Windows, select **File > Options**. On macOS, select **Visual Studio > Preferences**. +1. In Windows, select **Tools > Options**. On macOS, select **Visual Studio > Preferences**. 1. In the **NuGet** section, select **Sources** to view a list of all your NuGet sources. 1. Select **Add**. 1. Complete the following fields: - **Name**: Name for the source. - - **Location**: `https://gitlab.example.com/api/v4/projects/<your_project_id>/packages/nuget/index.json`, + - **Source**: `https://gitlab.example.com/api/v4/projects/<your_project_id>/packages/nuget/index.json`, where `<your_project_id>` is your project ID, and `gitlab.example.com` is your domain name. - - **Username**: Your GitLab username or deploy token username. - - **Password**: Your personal access token or deploy token. - -  1. Select **Save**. +1. When you access the package, you must enter your **Username** and **Password**: -The source is displayed in your list. + - **Username**: Your GitLab username or deploy token username. + - **Password**: Your personal access token or deploy token. - +The source is displayed in your list. -If you get a warning, ensure that the **Location**, **Username**, and +If you get a warning, ensure that the **Source**, **Username**, and **Password** are correct. #### Group-level endpoint @@ -148,27 +146,25 @@ To install a package from a group, use a group-level endpoint. To use the [group-level](#use-the-gitlab-endpoint-for-nuget-packages) NuGet endpoint, add the Package Registry as a source with Visual Studio: 1. Open [Visual Studio](https://visualstudio.microsoft.com/vs/). -1. In Windows, select **File > Options**. On macOS, select **Visual Studio > Preferences**. +1. In Windows, select **Tools > Options**. On macOS, select **Visual Studio > Preferences**. 1. In the **NuGet** section, select **Sources** to view a list of all your NuGet sources. 1. Select **Add**. 1. Complete the following fields: - **Name**: Name for the source. - - **Location**: `https://gitlab.example.com/api/v4/groups/<your_group_id>/-/packages/nuget/index.json`, + - **Source**: `https://gitlab.example.com/api/v4/groups/<your_group_id>/-/packages/nuget/index.json`, where `<your_group_id>` is your group ID, and `gitlab.example.com` is your domain name. - - **Username**: Your GitLab username or deploy token username. - - **Password**: Your personal access token or deploy token. - -  1. Select **Save**. +1. When you access the package, you must enter your **Username** and **Password**. -The source is displayed in your list. + - **Username**: Your GitLab username or deploy token username. + - **Password**: Your personal access token or deploy token. - +The source is displayed in your list. -If you get a warning, ensure that the **Location**, **Username**, and +If you get a warning, ensure that the **Source**, **Username**, and **Password** are correct. ### Add a source with the .NET CLI diff --git a/doc/user/packages/package_registry/index.md b/doc/user/packages/package_registry/index.md index 1aeb98fd48a..caa305999c5 100644 --- a/doc/user/packages/package_registry/index.md +++ b/doc/user/packages/package_registry/index.md @@ -9,7 +9,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w > [Moved](https://gitlab.com/gitlab-org/gitlab/-/issues/221259) from GitLab Premium to GitLab Free in 13.3. With the GitLab Package Registry, you can use GitLab as a private or public registry for a variety -of [supported package managers](#supported-package-managers). +of [supported package managers](supported_package_managers.md). You can publish and share packages, which can be consumed as a dependency in downstream projects. ## Package workflows @@ -60,6 +60,9 @@ For most package types, the following credential types are valid: allows access to packages in the project running the job for the users running the pipeline. Access to other external projects can be configured. +- If your organization uses two factor authentication (2FA), you must use a personal access token with the scope set to `api`. +- If you are publishing a package via CI/CD pipelines, you must use a CI job token. + NOTE: If you have not activated the "Packages" feature for your project at **Settings > General > Project features**, you will receive a 403 Forbidden response. Accessing package registry via deploy token is not available when external authorization is enabled. @@ -78,7 +81,7 @@ Learn more about using the GitLab Package Registry with CI/CD: - [Conan](../conan_repository/index.md#publish-a-conan-package-by-using-cicd) - [Generic](../generic_packages/index.md#publish-a-generic-package-by-using-cicd) - [Maven](../maven_repository/index.md#create-maven-packages-with-gitlab-cicd) -- [npm](../npm_registry/index.md#publish-an-npm-package-by-using-cicd) +- [npm](../npm_registry/index.md#publishing-a-package-via-a-cicd-pipeline) - [NuGet](../nuget_repository/index.md#publish-a-nuget-package-by-using-cicd) - [PyPI](../pypi_repository/index.md#authenticate-with-a-ci-job-token) - [RubyGems](../rubygems_registry/index.md#authenticate-with-a-ci-job-token) @@ -117,50 +120,40 @@ The **Packages and registries > Package Registry** entry is removed from the sid [Project-level permissions](../../permissions.md) determine actions such as downloading, pushing, or deleting packages. -The visibility of the Package Registry is independent of the repository and can't be controlled from +The visibility of the Package Registry is independent of the repository and can be controlled from your project's settings. For example, if you have a public project and set the repository visibility -to **Only Project Members**, the Package Registry is then public. However, disabling the Package +to **Only Project Members**, the Package Registry is then public. Disabling the Package Registry disables all Package Registry operations. -[GitLab-#329253](https://gitlab.com/gitlab-org/gitlab/-/issues/329253) -proposes adding the ability to control Package Registry visibility from the UI. - -| | | Anonymous<br/>(everyone on internet) | Guest | Reporter, Developer, Maintainer, Owner | -| -------------------- | --------------------- | --------- | ----- | ------------------------------------------ | -| Public project with Package Registry enabled | View Package Registry <br/> and pull packages | Yes | Yes | Yes | -| Internal project with Package Registry enabled | View Package Registry <br/> and pull packages | No | Yes | Yes | -| Private project with Package Registry enabled | View Package Registry <br/> and pull packages | No | No | Yes | -| Any project with Package Registry disabled | All operations on Package Registry | No | No | No | - -## Supported package managers +| Project visibility | Action | [Role](../../permissions.md#roles) required | +|--------------------|-----------------------|---------------------------------------------------------| +| Public | View Package Registry | `n/a`, everyone on the internet can perform this action | +| Public | Publish a package | Developer or higher | +| Public | Pull a package | `n/a`, everyone on the internet can perform this action | +| Internal | View Package Registry | Guest or higher | +| Internal | Publish a package | Developer or higher | +| Internal | Pull a package | Guest or higher(1) | +| Private | View Package Registry | Reporter or higher | +| Private | Publish a package | Developer or higher | +| Private | Pull a package | Reporter or higher(1) | -WARNING: -Not all package manager formats are ready for production use. To view each format's status, see the -table's **Status** column. +### Allow anyone to pull from Package Registry -The Package Registry supports the following formats: +> Introduced in GitLab 15.7 [with a flag](../../../administration/feature_flags.md) named `package_registry_access_level`. Enabled by default. -| Package type | GitLab version | Status | -| ------------ | -------------- |------- | -| [Maven](../maven_repository/index.md) | 11.3+ | GA | -| [npm](../npm_registry/index.md) | 11.7+ | GA | -| [NuGet](../nuget_repository/index.md) | 12.8+ | GA | -| [PyPI](../pypi_repository/index.md) | 12.10+ | GA | -| [Generic packages](../generic_packages/index.md) | 13.5+ | GA | -| [Composer](../composer_repository/index.md) | 13.2+ | [Beta](https://gitlab.com/groups/gitlab-org/-/epics/6817) | -| [Conan](../conan_repository/index.md) | 12.6+ | [Beta](https://gitlab.com/groups/gitlab-org/-/epics/6816) | -| [Helm](../helm_repository/index.md) | 14.1+ | [Beta](https://gitlab.com/groups/gitlab-org/-/epics/6366) | -| [Debian](../debian_repository/index.md) | 14.2+ | [Alpha](https://gitlab.com/groups/gitlab-org/-/epics/6057) | -| [Go](../go_proxy/index.md) | 13.1+ | [Alpha](https://gitlab.com/groups/gitlab-org/-/epics/3043) | -| [Ruby gems](../rubygems_registry/index.md) | 13.10+ | [Alpha](https://gitlab.com/groups/gitlab-org/-/epics/3200) | +FLAG: +On self-managed GitLab, by default this feature is available. To disable it, +ask an administrator to [disable the feature flag](../../../administration/feature_flags.md) named `package_registry_access_level`. -[Status](../../../policy/alpha-beta-support.md): +If you want to allow anyone (everyone on the internet) to pull from the Package Registry, no matter what the project visibility is, you can use the additional toggle `Allow anyone to pull from Package Registry` that appears when the project visibility is Private or Internal. -- Alpha: behind a feature flag and not officially supported. -- Beta: several known issues that may prevent expected use. -- GA (Generally Available): ready for production use at any scale. +Several known issues exist when you allow anyone to pull from the Package Registry: -You can also use the [API](../../../api/packages.md) to administer the Package Registry. +- Project-level endpoints are supported. Group-level and instance-level endpoints are not supported. Support for group-level endpoints is proposed in [issue 383537](https://gitlab.com/gitlab-org/gitlab/-/issues/383537). +- It does not work with the [Composer](../composer_repository/index.md#install-a-composer-package), because Composer only has a group endpoint. +- It does not work with the [Debian](../debian_repository/index.md#install-a-package) repository. Support for the Debian repository is proposed in [issue 385258](https://gitlab.com/gitlab-org/gitlab/-/issues/385258). +- It does not work with the [Ruby gems](../rubygems_registry/index.md#install-a-ruby-gem) repository. Support for the Ruby gems repository is proposed in [issue 385259](https://gitlab.com/gitlab-org/gitlab/-/issues/385259). +- It will work with Conan, but using [`conan search`](../conan_repository/index.md#search-for-conan-packages-in-the-package-registry) does not work. ## Accepting contributions @@ -170,8 +163,8 @@ guides you through the process. <!-- vale gitlab.Spelling = NO --> -| Format | Status | -| ------ | ------ | +| Format | Status | +| --------- | ------------------------------------------------------------- | | Chef | [#36889](https://gitlab.com/gitlab-org/gitlab/-/issues/36889) | | CocoaPods | [#36890](https://gitlab.com/gitlab-org/gitlab/-/issues/36890) | | Conda | [#36891](https://gitlab.com/gitlab-org/gitlab/-/issues/36891) | diff --git a/doc/user/packages/package_registry/supported_hash_types.md b/doc/user/packages/package_registry/supported_hash_types.md new file mode 100644 index 00000000000..6d7dbf87468 --- /dev/null +++ b/doc/user/packages/package_registry/supported_hash_types.md @@ -0,0 +1,25 @@ +--- +stage: Package +group: Package Registry +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments +--- + +# Supported hash types **(FREE)** + +Hash values are used to ensure you are using the correct package. You can view these values in the user interface or with the [API](../../../api/packages.md). + +The Package Registry supports the following hash types: + +| Package type | Supported hashes | +|--------------------------------------------------|----------------------------------| +| [Maven](../maven_repository/index.md) | MD5, SHA1 | +| [npm](../npm_registry/index.md) | SHA1 | +| [NuGet](../nuget_repository/index.md) | not applicable | +| [PyPI](../pypi_repository/index.md) | MD5, SHA256 | +| [Generic packages](../generic_packages/index.md) | SHA256 | +| [Composer](../composer_repository/index.md) | not applicable | +| [Conan](../conan_repository/index.md) | MD5, SHA1 | +| [Helm](../helm_repository/index.md) | not applicable | +| [Debian](../debian_repository/index.md) | MD5, SHA1, SHA256 | +| [Go](../go_proxy/index.md) | MD5, SHA1, SHA256 | +| [Ruby gems](../rubygems_registry/index.md) | MD5, SHA1, SHA256 (gemspec only) | diff --git a/doc/user/packages/package_registry/supported_package_managers.md b/doc/user/packages/package_registry/supported_package_managers.md new file mode 100644 index 00000000000..75b8c95a0fa --- /dev/null +++ b/doc/user/packages/package_registry/supported_package_managers.md @@ -0,0 +1,34 @@ +--- +stage: Package +group: Package Registry +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments +--- + +# Supported package managers **(FREE)** + +WARNING: +Not all package manager formats are ready for production use. + +The Package Registry supports the following package manager types: + +| Package type | GitLab version | Status | +| ------------------------------------------------ | -------------- | ---------------------------------------------------------- | +| [Maven](../maven_repository/index.md) | 11.3+ | GA | +| [npm](../npm_registry/index.md) | 11.7+ | GA | +| [NuGet](../nuget_repository/index.md) | 12.8+ | GA | +| [PyPI](../pypi_repository/index.md) | 12.10+ | GA | +| [Generic packages](../generic_packages/index.md) | 13.5+ | GA | +| [Composer](../composer_repository/index.md) | 13.2+ | [Beta](https://gitlab.com/groups/gitlab-org/-/epics/6817) | +| [Conan](../conan_repository/index.md) | 12.6+ | [Beta](https://gitlab.com/groups/gitlab-org/-/epics/6816) | +| [Helm](../helm_repository/index.md) | 14.1+ | [Beta](https://gitlab.com/groups/gitlab-org/-/epics/6366) | +| [Debian](../debian_repository/index.md) | 14.2+ | [Alpha](https://gitlab.com/groups/gitlab-org/-/epics/6057) | +| [Go](../go_proxy/index.md) | 13.1+ | [Alpha](https://gitlab.com/groups/gitlab-org/-/epics/3043) | +| [Ruby gems](../rubygems_registry/index.md) | 13.10+ | [Alpha](https://gitlab.com/groups/gitlab-org/-/epics/3200) | + +[Status](../../../policy/alpha-beta-support.md): + +- Alpha: behind a feature flag and not officially supported. +- Beta: several known issues that may prevent expected use. +- GA (Generally Available): ready for production use at any scale. + +You can also use the [API](../../../api/packages.md) to administer the Package Registry. diff --git a/doc/user/packages/pypi_repository/index.md b/doc/user/packages/pypi_repository/index.md index f6ed9654882..0e2fc7ca7da 100644 --- a/doc/user/packages/pypi_repository/index.md +++ b/doc/user/packages/pypi_repository/index.md @@ -283,6 +283,32 @@ characters are removed. A `pip install` request for `my.package` looks for packages that match any of the three characters, such as `my-package`, `my_package`, and `my....package`. +## Using `requirements.txt` + +If you want pip to access your private registry, add the `--extra-index-url` parameter along with the URL for your registry to your `requirements.txt` file. + +```plaintext +--extra-index-url https://gitlab.example.com/api/v4/projects/<project_id>/packages/pypi/simple +package-name==1.0.0 +``` + +If this is a private registry, you can authenticate in a couple of ways. For example: + +- Using your `requirements.txt` file: + +```plaintext +--extra-index-url https://__token__:<your_personal_token>@gitlab.example.com/api/v4/projects/<project_id>/packages/pypi/simple +package-name==1.0.0 +``` + +- Using a `~/.netrc` file: + +```plaintext +machine gitlab.example.com +login __token__ +password <your_personal_token> +``` + ## Troubleshooting To improve performance, the pip command caches files related to a package. Note that pip doesn't remove data by @@ -293,6 +319,18 @@ this command: pip cache purge ``` +### Multiple `index-url` or `extra-index-url` parameters + +You can define multiple `index-url` and `extra-index-url` parameters. + +If you use the same domain name (such as `gitlab.example.com`) multiple times with different authentication +tokens, `pip` may not be able to find your packages. This problem is due to how `pip` +[registers and stores your tokens](https://github.com/pypa/pip/pull/10904#issuecomment-1126690115) during commands executions. + +To workaround this issue, you can use a [group deploy token](../../project/deploy_tokens/index.md) with the +scope `read_package_registry` from a common parent group for all projects or groups targeted by the +`index-url` and `extra-index-url` values. + ## Supported CLI commands The GitLab PyPI repository supports the following CLI commands: diff --git a/doc/user/packages/terraform_module_registry/index.md b/doc/user/packages/terraform_module_registry/index.md index 2b99ff807ec..9b09d846034 100644 --- a/doc/user/packages/terraform_module_registry/index.md +++ b/doc/user/packages/terraform_module_registry/index.md @@ -50,7 +50,7 @@ error `{"error":"404 Not Found"}`. Example request using a personal access token: ```shell -curl --header "PRIVATE-TOKEN: <your_access_token>" \ +curl --fail-with-body --header "PRIVATE-TOKEN: <your_access_token>" \ --upload-file path/to/file.tgz \ "https://gitlab.example.com/api/v4/projects/<your_project_id>/packages/terraform/modules/my-module/my-system/0.0.1/file" ``` @@ -66,7 +66,7 @@ Example response: Example request using a deploy token: ```shell -curl --header "DEPLOY-TOKEN: <deploy_token>" \ +curl --fail-with-body --header "DEPLOY-TOKEN: <deploy_token>" \ --upload-file path/to/file.tgz \ "https://gitlab.example.com/api/v4/projects/<your_project_id>/packages/terraform/modules/my-module/my-system/0.0.1/file" ``` @@ -127,14 +127,14 @@ upload: script: - TERRAFORM_MODULE_NAME=$(echo "${TERRAFORM_MODULE_NAME}" | tr " _" -) # module-name must not have spaces or underscores, so translate them to hyphens - tar -vczf ${TERRAFORM_MODULE_NAME}-${TERRAFORM_MODULE_SYSTEM}-${TERRAFORM_MODULE_VERSION}.tgz -C ${TERRAFORM_MODULE_DIR} --exclude=./.git . - - 'curl --location --header "JOB-TOKEN: ${CI_JOB_TOKEN}" + - 'curl --fail-with-body --location --header "JOB-TOKEN: ${CI_JOB_TOKEN}" --upload-file ${TERRAFORM_MODULE_NAME}-${TERRAFORM_MODULE_SYSTEM}-${TERRAFORM_MODULE_VERSION}.tgz ${CI_API_V4_URL}/projects/${CI_PROJECT_ID}/packages/terraform/modules/${TERRAFORM_MODULE_NAME}/${TERRAFORM_MODULE_SYSTEM}/${TERRAFORM_MODULE_VERSION}/file' rules: - if: $CI_COMMIT_TAG ``` -To trigger this upload job, add a Git tag to your commit. Ensure the tag follows the [Semantic Versioning Specification](https://semver.org/) that Terraform requires. The `rules:if: $CI_COMMIT_TAG` ensures that only tagged commits to your repo trigger the module upload job. +To trigger this upload job, add a Git tag to your commit. Ensure the tag follows the [Semantic Versioning Specification](https://semver.org/) that Terraform requires. The `rules:if: $CI_COMMIT_TAG` ensures that only tagged commits to your repository trigger the module upload job. For other ways to control jobs in your CI/CD pipeline, refer to the [`.gitlab-ci.yml`](../../../ci/yaml/index.md) keyword reference. ## Example projects diff --git a/doc/user/packages/workflows/build_packages.md b/doc/user/packages/workflows/build_packages.md index ec971195ea9..eab1e4392e3 100644 --- a/doc/user/packages/workflows/build_packages.md +++ b/doc/user/packages/workflows/build_packages.md @@ -302,7 +302,7 @@ The npm version is shown in the output: ``` 1. Enter responses to the questions. Ensure the **package name** follows - the [naming convention](../npm_registry/index.md#package-naming-convention) and is scoped to the project or group where the registry exists. + the [naming convention](../npm_registry/index.md#naming-convention) and is scoped to the project or group where the registry exists. ## Yarn @@ -334,7 +334,7 @@ The Yarn version is shown in the output: ``` 1. Enter responses to the questions. Ensure the **package name** follows - the [naming convention](../npm_registry/index.md#package-naming-convention) and is scoped to the + the [naming convention](../npm_registry/index.md#naming-convention) and is scoped to the project or group where the registry exists. A `package.json` file is created. diff --git a/doc/user/packages/workflows/project_registry.md b/doc/user/packages/workflows/project_registry.md index df4b087f6d5..371ab83a4fb 100644 --- a/doc/user/packages/workflows/project_registry.md +++ b/doc/user/packages/workflows/project_registry.md @@ -50,15 +50,15 @@ If you're using npm, create an `.npmrc` file. Add the appropriate URL for publis packages to your project. Finally, add a section to your `package.json` file. Follow the instructions in the -[GitLab Package Registry npm documentation](../npm_registry/index.md#authenticate-to-the-package-registry). After +[GitLab Package Registry npm documentation](../npm_registry/index.md#authentication-to-the-package-registry). After you do this, you can publish your npm package to your project using `npm publish`, as described in the -[publishing packages](../npm_registry/index.md#publish-an-npm-package) section. +[publishing packages](../npm_registry/index.md#publish-to-gitlab-package-registry) section. ### Maven If you are using Maven, you update your `pom.xml` file with distribution sections. These updates include the -appropriate URL for your project, as described in the [GitLab Maven Repository documentation](../maven_repository/index.md#project-level-maven-endpoint). -Then, you need to add a `settings.xml` file and [include your access token](../maven_repository/index.md#authenticate-with-a-personal-access-token-in-maven). +appropriate URL for your project, as described in the [GitLab Maven Repository documentation](../maven_repository/index.md#naming-convention). +Then, you need to add a `settings.xml` file and [include your access token](../maven_repository/index.md#authenticate-to-the-package-registry). Now you can [publish Maven packages](../maven_repository/index.md#publish-a-package) to your project. ### Conan diff --git a/doc/user/packages/yarn_repository/index.md b/doc/user/packages/yarn_repository/index.md new file mode 100644 index 00000000000..7e2f45019cd --- /dev/null +++ b/doc/user/packages/yarn_repository/index.md @@ -0,0 +1,248 @@ +--- +stage: Package +group: Package Registry +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments +--- + +# Publish packages with Yarn + +Publish npm packages in your project's Package Registry using Yarn. Then install the +packages whenever you need to use them as a dependency. + +Learn how to build a [yarn](../workflows/build_packages.md#yarn) package. + +You can get started with Yarn 2 by following the [Yarn documentation](https://yarnpkg.com/getting-started/install/). + +## Publish to GitLab Package Registry + +### Authentication to the Package Registry + +You need a token to publish a package. Different tokens are available depending on what you're trying to +achieve. For more information, review the [guidance on tokens](../../../user/packages/package_registry/index.md#authenticate-with-the-registry). + +- If your organization uses two-factor authentication (2FA), you must use a personal access token with the scope set to `api`. +- If you publish a package via CI/CD pipelines, you must use a CI job token. + +Create a token and save it to use later in the process. + +### Naming convention + +Depending on how you install the package, you may need to adhere to the naming convention. + +You can use one of two API endpoints to install packages: + +- **Instance-level**: Use when you have many npm packages in different GitLab groups or in their own namespace. +- **Project-level**: Use when you have a few npm packages, and they are not in the same GitLab group. + +If you plan to install a package through the [project level](#install-from-the-project-level), you do not have to +adhere to the naming convention. + +If you plan to install a package through the [instance level](#install-from-the-instance-level), then you must name +your package with a [scope](https://docs.npmjs.com/misc/scope/). Scoped packages begin with a `@` and have the +`@owner/package-name` format. You can set up the scope for your package in the `.yarnrc.yml` file and by using the +`publishConfig` option in the `package.json`. + +- The value used for the `@scope` is the root of the project that hosts the packages and not the root + of the project with the package's source code. The scope should be lowercase. +- The package name can be anything you want + +| Project URL | Package Registry in | Scope | Full package name | +| ------------------------------------------------------- | ------------------- | --------- | ---------------------- | +| `https://gitlab.com/my-org/engineering-group/analytics` | Analytics | `@my-org` | `@my-org/package-name` | + +### Configuring `.yarnrc.yml` to publish from the project level + +To publish with the project-level npm endpoint, set the following configuration in +`.yarnrc.yml`: + +```yaml +npmScopes: + foo: + npmRegistryServer: 'https://<your_domain>/api/v4/projects/<your_project_id>/packages/npm/' + npmPublishRegistry: 'https://<your_domain>/api/v4/projects/<your_project_id>/packages/npm/' + +npmRegistries: + //gitlab.example.com/api/v4/projects/<your_project_id>/packages/npm/: + npmAlwaysAuth: true + npmAuthToken: '<your_token>' +``` + +In this configuration: + +- Replace `<your_domain>` with your domain name. +- Replace `<your_project_id>` with your project's ID, which you can find on the project's home page. +- Replace `<your_token>` with a deploy token, group access token, project access token, or personal access token. + +### Configuring `.yarnrc.yml` to publish from the instance level + +For the instance-level npm endpoint, use this Yarn 2 configuration in `.yarnrc.yml`: + +```yaml +npmScopes: + <scope>: + npmRegistryServer: 'https://<your_domain>/api/v4/packages/npm/' + +npmRegistries: + //gitlab.example.com/api/v4/packages/npm/: + npmAlwaysAuth: true + npmAuthToken: '<your_token>' +``` + +In this configuration: + +- Replace `<your_domain>` with your domain name. +- Your scope is `<scope>`, without `@`. +- Replace `<your_token>` with a deploy token, group access token, project access token, or personal access token. + +### Publishing a package via the command line + +Publish a package: + +```shell +npm publish +``` + +Your package should now publish to the Package Registry. + +### Publishing via a CI/CD pipeline + +In the GitLab project that houses your `yarnrc.yml`, edit or create a `.gitlab-ci.yml` file. For example: + +```yaml +image: node:latest + +stages: + - deploy + +deploy: + stage: deploy + script: + - npm publish +``` + +Your package should now publish to the Package Registry when the pipeline runs. + +## Install a package + +If multiple packages have the same name and version, the most recently-published package is retrieved when you install a package. + +You can install a package from a GitLab project or instance: + +- **Instance-level**: Use when you have many npm packages in different GitLab groups or in their own namespace. +- **Project-level**: Use when you have a few npm packages, and they are not in the same GitLab group. + +### Install from the instance level + +WARNING: +You must use packages published with the scoped [naming convention](#naming-convention) when you install a package from the instance level. + +1. Authenticate to the Package Registry + + If you install a package from a private project, you must authenticate to the Package Registry. Skip this step if the project is not private. + + ```shell + npm config set -- //your_domain_name/api/v4/packages/npm/:_authToken=your_token + ``` + + - Replace `your_domain_name` with your domain name, for example, `gitlab.com`. + - Replace `your_token` with a deploy token, group access token, project access token, or personal access token. + +1. Set the registry + + ```shell + npm config set @scope:registry https://your_domain_name.com/api/v4/packages/npm/ + ``` + + - Replace `@scope` with the [root level group](#naming-convention) of the project you're installing to the package from. + - Replace `your_domain_name` with your domain name, for example, `gitlab.com`. + - Replace `your_token` with a deploy token, group access token, project access token, or personal access token. + +1. Install the package + + ```shell + yarn add @scope/my-package + ``` + +### Install from the project level + +1. Authenticate to the Package Registry + + If you install a package from a private project, you must authenticate to the Package Registry. Skip this step if the project is not private. + + ```shell + npm config set -- //your_domain_name/api/v4/projects/your_project_id/packages/npm/:_authToken=your_token + ``` + + - Replace `your_domain_name` with your domain name, for example, `gitlab.com`. + - Replace `your_project_id` is your project ID, found on the project's home page. + - Replace `your_token` with a deploy token, group access token, project access token, or personal access token. + +1. Set the registry + + ```shell + npm config set @scope:registry=https://your_domain_name/api/v4/projects/your_project_id/packages/npm/ + ``` + + - Replace `@scope` with the [root level group](#naming-convention) of the project you're installing to the package from. + - Replace `your_domain_name` with your domain name, for example, `gitlab.com`. + - Replace `your_project_id` is your project ID, found on the project's home page. + +1. Install the package + + ```shell + yarn add @scope/my-package + ``` + +## Helpful hints + +For full helpful hints information, refer to the [npm documentation](../npm_registry/index.md#helpful-hints). + +### Supported CLI commands + +The GitLab npm repository supports the following commands for the npm CLI (`npm`) and yarn CLI +(`yarn`): + +- `npm install`: Install npm packages. +- `npm publish`: Publish an npm package to the registry. +- `npm dist-tag add`: Add a dist-tag to an npm package. +- `npm dist-tag ls`: List dist-tags for a package. +- `npm dist-tag rm`: Delete a dist-tag. +- `npm ci`: Install npm packages directly from your `package-lock.json` file. +- `npm view`: Show package metadata. +- `yarn add`: Install an npm package. +- `yarn update`: Update your dependencies. + +## Troubleshooting + +For full troubleshooting information, refer to the [npm documentation](../npm_registry/index.md#troubleshooting). + +### Error running Yarn with the Package Registry for the npm registry + +If you are using [Yarn](https://classic.yarnpkg.com/en/) with the npm registry, you may get +an error message like: + +```shell +yarn install v1.15.2 +warning package.json: No license field +info No lockfile found. +warning XXX: No license field +[1/4] 🔍 Resolving packages... +[2/4] 🚚 Fetching packages... +error An unexpected error occurred: "https://gitlab.example.com/api/v4/projects/XXX/packages/npm/XXX/XXX/-/XXX/XXX-X.X.X.tgz: Request failed \"404 Not Found\"". +info If you think this is a bug, please open a bug report with the information provided in "/Users/XXX/gitlab-migration/module-util/yarn-error.log". +info Visit https://classic.yarnpkg.com/en/docs/cli/install for documentation about this command +``` + +In this case, try adding this to your `.npmrc` file (and replace `<your_token>` +with your personal access token or deploy token): + +```plaintext +//gitlab.example.com/api/v4/projects/:_authToken=<your_token> +``` + +You can also use `yarn config` instead of `npm config` when setting your auth-token dynamically: + +```shell +yarn config set '//gitlab.example.com/api/v4/projects/<your_project_id>/packages/npm/:_authToken' "<your_token>" +yarn config set '//gitlab.example.com/api/v4/packages/npm/:_authToken' "<your_token>" +``` diff --git a/doc/user/permissions.md b/doc/user/permissions.md index 8e152a8c190..f3702b848fa 100644 --- a/doc/user/permissions.md +++ b/doc/user/permissions.md @@ -6,35 +6,35 @@ info: To determine the technical writer assigned to the Stage/Group associated w # Permissions and roles **(FREE)** -Users have different abilities depending on the role they have in a -particular group or project. If a user is both in a project's group and the -project itself, the highest role is used. +When you add a user to a project or group, you assign them a role. +The role determines which actions they can take in GitLab. -On [public and internal projects](../api/projects.md#project-visibility-level), the Guest role -(not to be confused with [Guest user](#free-guest-users)) is not enforced. +If you add a user to both a project's group and the +project itself, the higher role is used. -When a member leaves a team's project, all the assigned [issues](project/issues/index.md) and -[merge requests](project/merge_requests/index.md) are automatically unassigned. +GitLab [administrators](../administration/index.md) have all permissions. -GitLab [administrators](../administration/index.md) receive all permissions. +## Roles -To add or import a user, you can follow the -[project members documentation](project/members/index.md). +The available roles are: -## Principles behind permissions +- Guest (This role applies to [private and internal projects](../user/public_access.md) only.) +- Reporter +- Developer +- Maintainer +- Owner -See our [product handbook on permissions](https://about.gitlab.com/handbook/product/gitlab-the-product/#permissions-in-gitlab). +A user assigned the Guest role has the least permissions, +and the Owner has the most. -## Instance-wide user permissions - -By default, users can create top-level groups and change their -usernames. A GitLab administrator can configure the GitLab instance to -[modify this behavior](../administration/user_settings.md). +By default, all users can create top-level groups and change their +usernames. A GitLab administrator can [change this behavior](../administration/user_settings.md) +for the GitLab instance. ## Project members permissions -- [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/219299) in GitLab 14.8, personal namespace owners appear with Owner role in new projects in their namespace. Introduced [with a flag](../administration/feature_flags.md) named `personal_project_owner_with_owner_access`. Disabled by default. -- [Generally available](https://gitlab.com/gitlab-org/gitlab/-/issues/351919) in GitLab 14.9. Feature flag `personal_project_owner_with_owner_access` [removed](https://gitlab.com/gitlab-org/gitlab/-/issues/219299). +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/219299) in GitLab 14.8, personal namespace owners appear with Owner role in new projects in their namespace. Introduced [with a flag](../administration/feature_flags.md) named `personal_project_owner_with_owner_access`. Disabled by default. +> - [Generally available](https://gitlab.com/gitlab-org/gitlab/-/issues/351919) in GitLab 14.9. Feature flag `personal_project_owner_with_owner_access` [removed](https://gitlab.com/gitlab-org/gitlab/-/issues/219299). A user's role determines what permissions they have on a project. The Owner role provides all permissions but is available only: @@ -79,14 +79,17 @@ The following table lists project permissions available for each role: | [GitLab Pages](project/pages/index.md):<br>Manage | | | | ✓ | ✓ | | [GitLab Pages](project/pages/index.md):<br>Manage GitLab Pages domains and certificates | | | | ✓ | ✓ | | [GitLab Pages](project/pages/index.md):<br>Remove GitLab Pages | | | | ✓ | ✓ | -| [Incident Management](../operations/incident_management/index.md):<br>View [alerts](../operations/incident_management/alerts.md) | | ✓ | ✓ | ✓ | ✓ | | [Incident Management](../operations/incident_management/index.md):<br>Assign an alert | ✓ | ✓ | ✓ | ✓ | ✓ | -| [Incident Management](../operations/incident_management/index.md):<br>[Change an alert status](../operations/incident_management/alerts.md#change-an-alerts-status) | | ✓ | ✓ | ✓ | ✓ | -| [Incident Management](../operations/incident_management/index.md):<br>View [incident](../operations/incident_management/incidents.md) | ✓ | ✓ | ✓ | ✓ | ✓ | -| [Incident Management](../operations/incident_management/index.md):<br>Create [incident](../operations/incident_management/incidents.md) | | ✓ | ✓ | ✓ | ✓ | -| [Incident Management](../operations/incident_management/index.md):<br>View [on-call schedules](../operations/incident_management/oncall_schedules.md) | | ✓ | ✓ | ✓ | ✓ | | [Incident Management](../operations/incident_management/index.md):<br>Participate in on-call rotation | ✓ | ✓ | ✓ | ✓ | ✓ | +| [Incident Management](../operations/incident_management/index.md):<br>View [incident](../operations/incident_management/incidents.md) | ✓ | ✓ | ✓ | ✓ | ✓ | +| [Incident Management](../operations/incident_management/index.md):<br>Change [alert status](../operations/incident_management/alerts.md#change-an-alerts-status) | | ✓ | ✓ | ✓ | ✓ | +| [Incident Management](../operations/incident_management/index.md):<br>Change [incident severity](../operations/incident_management/manage_incidents.md#change-severity) | | ✓ | ✓ | ✓ | ✓ | +| [Incident Management](../operations/incident_management/index.md):<br>Create [incident](../operations/incident_management/incidents.md) | | ✓ | ✓ | ✓ | ✓ | +| [Incident Management](../operations/incident_management/index.md):<br>View [alerts](../operations/incident_management/alerts.md) | | ✓ | ✓ | ✓ | ✓ | | [Incident Management](../operations/incident_management/index.md):<br>View [escalation policies](../operations/incident_management/escalation_policies.md) | | ✓ | ✓ | ✓ | ✓ | +| [Incident Management](../operations/incident_management/index.md):<br>View [on-call schedules](../operations/incident_management/oncall_schedules.md) | | ✓ | ✓ | ✓ | ✓ | +| [Incident Management](../operations/incident_management/index.md):<br>Change [incident escalation status](../operations/incident_management/manage_incidents.md#change-status) | | | ✓ | ✓ | ✓ | +| [Incident Management](../operations/incident_management/index.md):<br>Change [incident escalation policy](../operations/incident_management/manage_incidents.md#change-escalation-policy) | | | ✓ | ✓ | ✓ | | [Incident Management](../operations/incident_management/index.md):<br>Manage [on-call schedules](../operations/incident_management/oncall_schedules.md) | | | | ✓ | ✓ | | [Incident Management](../operations/incident_management/index.md):<br>Manage [escalation policies](../operations/incident_management/escalation_policies.md) | | | | ✓ | ✓ | | [Issue boards](project/issue_board.md):<br>Create or delete lists | | ✓ | ✓ | ✓ | ✓ | @@ -161,7 +164,7 @@ The following table lists project permissions available for each role: | [Projects](project/index.md):<br>Edit comments (posted by any user) | | | | ✓ | ✓ | | [Projects](project/index.md):<br>Edit project badges | | | | ✓ | ✓ | | [Projects](project/index.md):<br>Edit project settings | | | | ✓ | ✓ | -| [Projects](project/index.md):<br>Export project | | | | ✓ | ✓ | +| [Projects](project/index.md):<br>[Export project](project/settings/import_export.md) | | | | ✓ | ✓ | | [Projects](project/index.md):<br>Manage [project access tokens](project/settings/project_access_tokens.md) (*11*) | | | | ✓ (*20*) | ✓ | | [Projects](project/index.md):<br>Manage [Project Operations](../operations/index.md) | | | | ✓ | ✓ | | [Projects](project/index.md):<br>Rename project | | | | ✓ | ✓ | @@ -204,7 +207,7 @@ The following table lists project permissions available for each role: | [Security dashboard](application_security/security_dashboard/index.md):<br>Use security dashboard | | | ✓ | ✓ | ✓ | | [Security dashboard](application_security/security_dashboard/index.md):<br>View vulnerability | | | ✓ | ✓ | ✓ | | [Security dashboard](application_security/security_dashboard/index.md):<br>View vulnerability findings in [dependency list](application_security/dependency_list/index.md) | | | ✓ | ✓ | ✓ | -| [Tasks](tasks.md):<br>Create (*17*) | ✓ | ✓ | ✓ | ✓ | ✓ | +| [Tasks](tasks.md):<br>Create (*17*) | | ✓ | ✓ | ✓ | ✓ | | [Tasks](tasks.md):<br>Edit | | ✓ | ✓ | ✓ | ✓ | | [Tasks](tasks.md):<br>Remove from issue | | ✓ | ✓ | ✓ | ✓ | | [Tasks](tasks.md):<br>Delete (*21*) | | | | | ✓ | @@ -218,7 +221,7 @@ The following table lists project permissions available for each role: <!-- markdownlint-disable MD029 --> 1. On self-managed GitLab instances, guest users are able to perform this action only on - public and internal projects (not on private projects). [External users](#external-users) + public and internal projects (not on private projects). [External users](admin_area/external_users.md) must be given explicit access even if the project is internal. For GitLab.com, see the [GitLab.com visibility settings](gitlab_com/index.md#visibility-settings). 2. Guest users can only view the [confidential issues](project/issues/confidential_issues.md) they created themselves or are assigned to. @@ -327,29 +330,7 @@ This table shows granted privileges for jobs triggered by specific types of user | Push source and LFS | | | | | 1. Only if the triggering user is not an external one. -1. Only if the triggering user is a member of the project. - -### Wiki and issues - -Project features like [wikis](project/wiki/index.md) and issues can be hidden from users depending on -which visibility level you select on project settings. - -- Disabled: disabled for everyone -- Only team members: only team members can see even if your project is public or internal -- Everyone with access: everyone can see depending on your project's visibility level -- Everyone: enabled for everyone (only available for GitLab Pages) - -### Protected branches - -Additional restrictions can be applied on a per-branch basis with [protected branches](project/protected_branches.md). -Additionally, you can customize permissions to allow or prevent project Developers or Maintainers -from pushing to a protected branch. Read through the documentation on -[protected branches](project/protected_branches.md) to learn more. - -### Value stream analytics permissions - -Find the current permissions on the value stream analytics dashboard, as described in -[related documentation](analytics/value_stream_analytics.md#access-permissions-for-value-stream-analytics). +1. Only if the triggering user is a member of the project. See also [Usage of private Docker images with `if-not-present` pull policy](http://docs.gitlabl.com/runner/security/index.html#usage-of-private-docker-images-with-if-not-present-pull-policy). ### File Locking permissions **(PREMIUM)** @@ -380,6 +361,7 @@ The following table lists group permissions available for each role: | Action | Guest | Reporter | Developer | Maintainer | Owner | |-----------------------------------------------------------------------------------------|-------|----------|-----------|------------|-------| +| Add/remove [child epics](group/epics/manage_epics.md#multi-level-child-epics) | ✓ (8) | ✓ | ✓ | ✓ | ✓ | | Add an issue to an [epic](group/epics/index.md) | ✓ (7) | ✓ (7) | ✓ (7) | ✓ (7) | ✓ (7) | | Browse group | ✓ | ✓ | ✓ | ✓ | ✓ | | Pull a container image using the dependency proxy | ✓ | ✓ | ✓ | ✓ | ✓ | @@ -397,6 +379,7 @@ The following table lists group permissions available for each role: | Pull [packages](packages/index.md) | | ✓ | ✓ | ✓ | ✓ | | Delete [packages](packages/index.md) | | | | ✓ | ✓ | | Create/edit/delete [Maven and generic package duplicate settings](packages/generic_packages/index.md#do-not-allow-duplicate-generic-packages) | | | | ✓ | ✓ | +| Enable/disable package request forwarding | | | | ✓ | ✓ | | Pull a Container Registry image | ✓ (6) | ✓ | ✓ | ✓ | ✓ | | Remove a Container Registry image | | | ✓ | ✓ | ✓ | | View [Group DevOps Adoption](group/devops_adoption/index.md) | | ✓ | ✓ | ✓ | ✓ | @@ -448,6 +431,7 @@ The following table lists group permissions available for each role: 5. In addition, if your group is public or internal, all users who can see the group can also see group wiki pages. 6. Users can only view events based on their individual actions. 7. You must have permission to [view the epic](group/epics/manage_epics.md#who-can-view-an-epic) and edit the issue. +8. You must have permission to [view](group/epics/manage_epics.md#who-can-view-an-epic) the parent and child epics. <!-- markdownlint-enable MD029 --> @@ -460,109 +444,6 @@ nested groups if you have membership in one of its parents. To learn more, read through the documentation on [subgroups memberships](group/subgroups/index.md#subgroup-membership). -## External users **(FREE SELF)** - -In cases where it is desired that a user has access only to some internal or -private projects, there is the option of creating **External Users**. This -feature may be useful when for example a contractor is working on a given -project and should only have access to that project. - -External users: - -- Cannot create project, groups, and snippets in their personal namespaces. -- Can only create projects (including forks), subgroups, and snippets within top-level groups to which they are explicitly granted access. -- Can only access public projects and projects to which they are explicitly granted access, - thus hiding all other internal or private ones from them (like being - logged out). -- Can only access public groups and groups to which they are explicitly granted access, - thus hiding all other internal or private ones from them (like being - logged out). -- Can only access public snippets. - -Access can be granted by adding the user as member to the project or group. -Like usual users, they receive a role in the project or group with all -the abilities that are mentioned in the [permissions table above](#project-members-permissions). -For example, if an external user is added as Guest, and your project is internal or -private, they do not have access to the code; you need to grant the external -user access at the Reporter level or above if you want them to have access to the code. You should -always take into account the -[project's visibility and permissions settings](project/settings/index.md#configure-project-visibility-features-and-permissions) -as well as the permission level of the user. - -NOTE: -External users still count towards a license seat. - -An administrator can flag a user as external by either of the following methods: - -- [Through the API](../api/users.md#user-modification). -- Using the GitLab UI: - 1. On the top bar, select **Main menu > Admin**. - 1. On the left sidebar, select **Overview > Users** to create a new user or edit an existing one. - There, you can find the option to flag the user as external. - -Additionally, users can be set as external users using: - -- [SAML groups](../integration/saml.md#external-groups). -- [LDAP groups](../administration/auth/ldap/ldap_synchronization.md#external-groups). - -### Setting new users to external - -By default, new users are not set as external users. This behavior can be changed -by an administrator: - -1. On the top bar, select **Main menu > Admin**. -1. On the left sidebar, select **Settings > General**. -1. Expand the **Account and limit** section. - -If you change the default behavior of creating new users as external, you -have the option to narrow it down by defining a set of internal users. -The **Internal users** field allows specifying an email address regex pattern to -identify default internal users. New users whose email address matches the regex -pattern are set to internal by default rather than an external collaborator. - -The regex pattern format is in Ruby, but it needs to be convertible to JavaScript, -and the ignore case flag is set (`/regex pattern/i`). Here are some examples: - -- Use `\.internal@domain\.com$` to mark email addresses ending with - `.internal@domain.com` as internal. -- Use `^(?:(?!\.ext@domain\.com).)*$\r?` to mark users with email addresses - not including `.ext@domain.com` as internal. - -WARNING: -Be aware that this regex could lead to a -[regular expression denial of service (ReDoS) attack](https://en.wikipedia.org/wiki/ReDoS). - -## Free Guest users **(ULTIMATE)** - -When a user is given the Guest role on a project, group, or both, and holds no -higher permission level on any other project or group on the GitLab instance, -the user is considered a guest user by GitLab and does not consume a license seat. -There is no other specific "guest" designation for newly created users. - -If the user is assigned a higher role on any projects or groups, the user -takes a license seat. If a user creates a project, the user becomes a Maintainer -on the project, resulting in the use of a license seat. Also, note that if your -project is internal or private, Guest users have all the abilities that are -mentioned in the [permissions table above](#project-members-permissions) (they -are unable to browse the project's repository, for example). - -NOTE: -To prevent a guest user from creating projects, as an administrator, you can edit the -user's profile to mark the user as [external](#external-users). -Beware though that even if a user is external, if they already have Reporter or -higher permissions in any project or group, they are **not** counted as a -free guest user. - -## Auditor users **(PREMIUM SELF)** - -Auditor users are given read-only access to all projects, groups, and other -resources on the GitLab instance. - -An Auditor user should be able to access all projects and groups of a GitLab instance -with the permissions described on the documentation on [auditor users permissions](../administration/auditor_users.md#auditor-user-permissions-and-restrictions). - -[Read more about Auditor users.](../administration/auditor_users.md) - ## Users with minimal access **(PREMIUM)** > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/40942) in GitLab 13.4. @@ -571,6 +452,11 @@ Owners can add members with a "minimal access" role to a parent group. Such user projects and subgroups underneath. Owners must explicitly add these "minimal access" users to the specific subgroups and projects. +You can use minimal access to give the same member more than one role in a group: + +1. Add the member to the parent group with a minimal access role. +1. Invite the member as a direct member with a specific role in any subgroup or project in that group. + Because of an [outstanding issue](https://gitlab.com/gitlab-org/gitlab/-/issues/267996), when minimal access users: - Sign in with standard web authentication, they receive a `404` error when accessing the parent group. @@ -584,16 +470,6 @@ Users with even a "minimal access" role are counted against your number of licen requirement does not apply for [GitLab Ultimate](https://about.gitlab.com/pricing/) subscriptions. -## Project features - -Project features like wiki and issues can be hidden from users depending on -which visibility level you select on project settings. - -- Disabled: disabled for everyone. -- Only team members: only team members can see, even if your project is public or internal. -- Everyone with access: everyone can see depending on your project visibility level. -- Everyone: enabled for everyone (only available for GitLab Pages). - ## Release permissions with protected tags [The permission to create tags](project/protected_tags.md) is used to define if a user can @@ -602,12 +478,12 @@ create, edit, and delete [Releases](project/releases/index.md). See [Release permissions](project/releases/index.md#release-permissions) for more information. -## LDAP users permissions - -LDAP user permissions can be manually overridden by an administrator. -Read through the documentation on [LDAP users permissions](group/access_and_permissions.md#manage-group-memberships-via-ldap) to learn more. - -## Project aliases +## Related topics -Project aliases can only be read, created and deleted by a GitLab administrator. -Read through the documentation on [Project aliases](../user/project/import/index.md#project-aliases) to learn more. +- [The GitLab principles behind permissions](https://about.gitlab.com/handbook/product/gitlab-the-product/#permissions-in-gitlab) +- [Members](project/members/index.md) +- Customize permissions on [protected branches](project/protected_branches.md) +- [LDAP users permissions](group/access_and_permissions.md#manage-group-memberships-via-ldap) +- [Value stream analytics permissions](analytics/value_stream_analytics.md#access-permissions-for-value-stream-analytics) +- [Project aliases](../user/project/import/index.md#project-aliases) +- [Auditor users](../administration/auditor_users.md) diff --git a/doc/user/product_analytics/index.md b/doc/user/product_analytics/index.md index 8e340fff32a..46f8b57a64c 100644 --- a/doc/user/product_analytics/index.md +++ b/doc/user/product_analytics/index.md @@ -4,25 +4,51 @@ group: Product Analytics info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- -# Product analytics **(ULTIMATE)** **Alpha** +# Product analytics **(ULTIMATE)** -> Introduced in GitLab 15.4 [with a flag](../../administration/feature_flags.md) named `cube_api_proxy`. Disabled by default. +> Introduced in GitLab 15.4 as an [Alpha](../../policy/alpha-beta-support.md#alpha-features) feature [with a flag](../../administration/feature_flags.md) named `cube_api_proxy`. Disabled by default. FLAG: On self-managed GitLab, by default this feature is not available. To make it available per project or for your entire instance, ask an administrator to [enable the feature flag](../../administration/feature_flags.md) named `cube_api_proxy`. On GitLab.com, this feature is not available. This feature is not ready for production use. -## Overview +This page is a work in progress, and we're updating the information as we add more features. +For more information, visit the [Product Analytics group direction page](https://about.gitlab.com/direction/analytics/product-analytics/). -You can view the [product category](https://about.gitlab.com/direction/analytics/product-analytics/) page for more information about our direction. This page is a work in progress and will be updated as we add more features. +## Enable product analytics + +You can enable and configure product analytics to track events +within your project applications on a self-managed instance. + +Prerequisite: + +- You must be an administrator of a self-managed GitLab instance. + +1. On the top bar, select **Main menu > Admin**. +1. On the left sidebar, select **Settings > General**. +1. Expand the **Product analytics** section. +1. Select **Enable product analytics** and enter the configuration values. + The following table shows the required configuration parameters and example values: + + | Name | Value | + |------------------------------|------------------------------------------------------------| + | Jitsu host | `https://jitsu.gitlab.com` | + | Jitsu project ID | `g0maofw84gx5sjxgse2k` | + | Jitsu administrator email | `jitsu.admin@gitlab.com` | + | Jitsu administrator password | `<your_password>` | + | Clickhouse URL | `https://<username>:<password>@clickhouse.gitlab.com:8123` | + | Cube API URL | `https://cube.gitlab.com` | + | Cube API key | `25718201b3e9...ae6bbdc62dbb` | + +1. Select **Save changes**. ## Product analytics dashboards Each project can define an unlimited number of dashboards. These dashboards are defined using our YAML schema and stored -in the `.gitlab/product_analytics/dashboards/` directory. The name of the file is the name of the dashboard, and visualizations are shared across dashboards.. +in the `.gitlab/product_analytics/dashboards/` directory of a project repository. The name of the file is the name of the dashboard, and visualizations are shared across dashboards. -Project maintainers can enforce approval rules on dashboard changes, and dashboards can be versioned in source control. +Project maintainers can enforce approval rules on dashboard changes using features such as code owners and approval rules. Dashboards are versioned in source control with the rest of a project's code. ### Define a dashboard @@ -30,8 +56,8 @@ To define a dashboard: 1. In `.gitlab/product_analytics/dashboards/`, create a directory named like the dashboard. Each dashboard should have its own directory. 1. In the new directory, create a `.yaml` file with the same name as the directory. This file contains the dashboard definition, and must conform to the JSON schema defined in `ee/app/validators/json_schemas/product_analytics_dashboard.json`. -1. In the `.gitlab/product_analytics/dashboards/visualizations/` directory, create a `yaml` file. This file defines the visualization type for the dashboard, and must conform to the schema in -`ee/app/validators/json_schemas/product_analytics_visualization.json`. +1. In the `.gitlab/product_analytics/dashboards/visualizations/` directory, create a `yaml` file. This file defines the visualization type for the dashboard, and must conform to the schema in + `ee/app/validators/json_schemas/product_analytics_visualization.json`. The example below includes three dashboards and one visualization that applies to all dashboards. diff --git a/doc/user/profile/account/delete_account.md b/doc/user/profile/account/delete_account.md index 5e2908a26e1..18b4e53fb31 100644 --- a/doc/user/profile/account/delete_account.md +++ b/doc/user/profile/account/delete_account.md @@ -56,6 +56,7 @@ When deleting users, you can either: - Issues. - Merge requests. - Notes and comments. + - Releases. - Personal access tokens. - Snippets. diff --git a/doc/user/profile/active_sessions.md b/doc/user/profile/active_sessions.md index 430d1c3dc9f..7a837258cb2 100644 --- a/doc/user/profile/active_sessions.md +++ b/doc/user/profile/active_sessions.md @@ -40,7 +40,7 @@ To revoke an active session: NOTE: When any session is revoked all **Remember me** tokens for all -devices are revoked. See [Why do I keep getting signed out?](index.md#why-do-i-keep-getting-signed-out) +devices are revoked. See [Why do you keep getting signed out?](index.md#why-do-you-keep-getting-signed-out) for more information about the **Remember me** feature. <!-- ## Troubleshooting diff --git a/doc/user/profile/contributions_calendar.md b/doc/user/profile/contributions_calendar.md index 6df7ad56c5e..d66e555970a 100644 --- a/doc/user/profile/contributions_calendar.md +++ b/doc/user/profile/contributions_calendar.md @@ -21,50 +21,20 @@ The contribution calendar only displays contributions from the last 12 months, b GitLab tracks the following contribution events: -- `approved` - - Merge request -- `closed` - - [Epic](../group/epics/index.md) - - Issue - - Merge request - - Milestone -- `commented` on any `Noteable` record. - - Alert - - Commit - - Design - - Issue - - Merge request - - Snippet -- `created` - - Design - - [Epic](../group/epics/index.md) - - Issue - - Merge request - - Milestone - - Project - - Wiki page -- `destroyed` - - Design - - Milestone - - Wiki page -- `expired` - - Project membership -- `joined` - - Project membership -- `left` - - Project membership -- `merged` - - Merge request -- `pushed` commits to (or deleted commits from) a repository, individually or in bulk. - - Project -- `reopened` - - [Epic](../group/epics/index.md) - - Issue - - Merge request - - Milestone -- `updated` - - Design - - Wiki page +| Event | Contribution | +| ----- | ------------ | +| `approved` | Merge request | +| `closed` | [Epic](../group/epics/index.md), Issue, Merge request, Milestone, Work item | +| `commented` on any `Noteable` record. | Alert, Commit, Design, Issue, Merge request, Snippet | +| `created` | Design, Epic, Issue, Merge request, Milestone, Project, Wiki page, Work item | +| `destroyed` | Design, Milestone, Wiki page | +| `expired` | Project membership | +| `joined` | Project membership | +| `left` | Project membership | +| `merged` | Merge request | +| `pushed` commits to (or deleted commits from) a repository, individually or in bulk. | Project | +| `reopened` | Epic, Issue, Merge request, Milestone | +| `updated` | Design, Wiki page | ### View daily contributions diff --git a/doc/user/profile/index.md b/doc/user/profile/index.md index 4adf6c351df..d6c5bd6c108 100644 --- a/doc/user/profile/index.md +++ b/doc/user/profile/index.md @@ -328,7 +328,7 @@ To view a user's activity in a top-level Activity view: ## Troubleshooting -### Why do I keep getting signed out? +### Why do you keep getting signed out? When you sign in to the main GitLab application, a `_gitlab_session` cookie is set. When you close your browser, the cookie is cleared client-side @@ -368,7 +368,7 @@ GitLab uses both session and persistent cookies: - Session cookie: Session cookies are normally removed at the end of the browser session when the browser is closed. The `_gitlab_session` cookie has no fixed expiration date. However, - it expires based on its [`session_expire_delay`](#why-do-i-keep-getting-signed-out). + it expires based on its [`session_expire_delay`](#why-do-you-keep-getting-signed-out). - Persistent cookie: The `remember_user_token` is a cookie with an expiration date of two weeks. GitLab activates this cookie if you select **Remember Me** when you sign in. diff --git a/doc/user/profile/notifications.md b/doc/user/profile/notifications.md index 1deb4842107..d0a420a4bbd 100644 --- a/doc/user/profile/notifications.md +++ b/doc/user/profile/notifications.md @@ -76,7 +76,7 @@ For each project and group you can select one of the following levels: | Participate | Receive notifications for threads you have participated in. | | On mention | Receive notifications when you are [mentioned](../discussions/index.md#mentions) in a comment. | | Disabled | Receive no notifications. | -| Custom | Receive notifications for selected events. | +| Custom | Receive notifications for selected events and threads you have participated in. | ### Global notification settings diff --git a/doc/user/profile/preferences.md b/doc/user/profile/preferences.md index dce8684d993..664d22959a2 100644 --- a/doc/user/profile/preferences.md +++ b/doc/user/profile/preferences.md @@ -96,7 +96,7 @@ A diff compares the old/removed content with the new/added content (for example, [Markdown inline diff](../markdown.md#inline-diff)). Typically, the colors red and green are used for removed and added lines in diffs. The exact colors depend on the selected [syntax highlighting theme](#syntax-highlighting-theme). -The colors may lead to difficulties in case of red–green color blindness. +The colors may lead to difficulties in case of red-green color blindness. For this reason, you can customize the following colors: @@ -203,6 +203,22 @@ NOTE: This feature is experimental, and choosing absolute times might break certain layouts. Open an issue if you notice that using absolute times breaks a layout. +## Web IDE + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/370139) in GitLab 15.7 [with a flag](../../administration/feature_flags.md) named `vscode_web_ide`. Disabled by default. + +FLAG: +On self-managed GitLab, by default this feature is not available. To make it available, ask an administrator to [enable the feature flag](../../administration/feature_flags.md) named `vscode_web_ide`. On GitLab.com, this feature is available. + +The [Web IDE Beta](../project/web_ide_beta/index.md) is +the default editing environment when the `vscode_web_ide` feature +flag is enabled. + +To stop using the Web IDE Beta: + +1. In the **Web IDE** section, select the **Opt out of the Web IDE Beta** checkbox. +1. Select **Save changes**. + ## Integrations Configure your preferences with third-party services which provide enhancements to your GitLab experience. diff --git a/doc/user/profile/user_passwords.md b/doc/user/profile/user_passwords.md index b8dbdcdd956..9c1ba8852d2 100644 --- a/doc/user/profile/user_passwords.md +++ b/doc/user/profile/user_passwords.md @@ -61,12 +61,8 @@ Self-managed installations can configure the following additional password requi ## Block weak passwords > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/23610) in GitLab 15.4 [with a flag](../../administration/feature_flags.md) named `block_weak_passwords`, weak passwords aren't accepted. Disabled by default on self-managed. -> - [Enabled](https://gitlab.com/gitlab-org/gitlab/-/issues/363445) on GitLab.com. - -FLAG: -On self-managed GitLab, by default blocking weak passwords is not available. To make it available, ask an administrator -to [enable the feature flag](../../administration/feature_flags.md) named `block_weak_passwords`. On GitLab.com, this -feature is available but can be configured by GitLab.com administrators only. +> - [Enabled](https://gitlab.com/gitlab-org/gitlab/-/issues/363445) on GitLab.com in GitLab 15.6. +> - [Generally available](https://gitlab.com/gitlab-org/gitlab/-/issues/363445) and enabled on self-managed in GitLab 15.7. Feature flag `block_weak_passwords` removed. GitLab disallows weak passwords. Your password is considered weak when it: diff --git a/doc/user/project/badges.md b/doc/user/project/badges.md index 5d1d10fc37d..dc650bd9482 100644 --- a/doc/user/project/badges.md +++ b/doc/user/project/badges.md @@ -89,6 +89,8 @@ which are evaluated when displaying the badge. The following placeholders are available: - `%{project_path}`: Path of a project including the parent groups +- `%{project_title}`: Title of a project +- `%{project_name}`: Name of a project - `%{project_id}`: Database ID associated with a project - `%{default_branch}`: Default branch name configured for a project's repository - `%{commit_sha}`: ID of the most recent commit to the default branch of a diff --git a/doc/user/project/canary_deployments.md b/doc/user/project/canary_deployments.md index 3e6a9acc304..95f38c7e354 100644 --- a/doc/user/project/canary_deployments.md +++ b/doc/user/project/canary_deployments.md @@ -68,7 +68,7 @@ Here's an example setup flow from scratch: If it isn't, follow the documentation to specify the image version. 1. [Run a new Auto DevOps pipeline](../../ci/pipelines/index.md#run-a-pipeline-manually) and make sure that the `production` job succeeds and creates a production environment. -1. Configure a [`canary` deployment job for Auto DevOps pipelines](../../topics/autodevops/customize.md#deploy-policy-for-canary-environments). +1. Configure a [`canary` deployment job for Auto DevOps pipelines](../../topics/autodevops/cicd_variables.md#deploy-policy-for-canary-environments). 1. [Run a new Auto DevOps pipeline](../../ci/pipelines/index.md#run-a-pipeline-manually) and make sure that the `canary` job succeeds and creates a canary deployment with Canary Ingress. diff --git a/doc/user/project/clusters/add_eks_clusters.md b/doc/user/project/clusters/add_eks_clusters.md index b3d381c3148..52288af101a 100644 --- a/doc/user/project/clusters/add_eks_clusters.md +++ b/doc/user/project/clusters/add_eks_clusters.md @@ -195,7 +195,7 @@ If a default Storage Class doesn't already exist and is desired, follow Amazon's to create one. Alternatively, disable PostgreSQL by setting the project variable -[`POSTGRES_ENABLED`](../../../topics/autodevops/customize.md#cicd-variables) to `false`. +[`POSTGRES_ENABLED`](../../../topics/autodevops/cicd_variables.md#cicd-variables) to `false`. ## Deploy the app to EKS diff --git a/doc/user/project/clusters/cluster_access.md b/doc/user/project/clusters/cluster_access.md index c9b3596d92f..529e7a6da12 100644 --- a/doc/user/project/clusters/cluster_access.md +++ b/doc/user/project/clusters/cluster_access.md @@ -49,16 +49,16 @@ GitLab creates the following resources for RBAC clusters. | Name | Type | Details | Created when | |:----------------------|:---------------------|:-----------------------------------------------------------------------------------------------------------|:-----------------------| | `gitlab` | `ServiceAccount` | `default` namespace | Creating a new cluster | -| `gitlab-admin` | `ClusterRoleBinding` | [`cluster-admin`](https://kubernetes.io/docs/reference/access-authn-authz/rbac/#user-facing-roles) roleRef | Creating a new cluster | +| `gitlab-admin` | `ClusterRoleBinding` | [`cluster-admin`](https://kubernetes.io/docs/reference/access-authn-authz/rbac/#user-facing-roles) role | Creating a new cluster | | `gitlab-token` | `Secret` | Token for `gitlab` ServiceAccount | Creating a new cluster | | Environment namespace | `Namespace` | Contains all environment-specific resources | Deploying to a cluster | | Environment namespace | `ServiceAccount` | Uses namespace of environment | Deploying to a cluster | | Environment namespace | `Secret` | Token for environment ServiceAccount | Deploying to a cluster | -| Environment namespace | `RoleBinding` | [`admin`](https://kubernetes.io/docs/reference/access-authn-authz/rbac/#user-facing-roles) roleRef | Deploying to a cluster | +| Environment namespace | `RoleBinding` | [`admin`](https://kubernetes.io/docs/reference/access-authn-authz/rbac/#user-facing-roles) role | Deploying to a cluster | The environment namespace `RoleBinding` was [updated](https://gitlab.com/gitlab-org/gitlab/-/issues/31113) in GitLab 13.6 -to `admin` roleRef. Previously, the `edit` roleRef was used. +to `admin` role. Previously, the `edit` role was used. ## ABAC cluster resources diff --git a/doc/user/project/deploy_tokens/index.md b/doc/user/project/deploy_tokens/index.md index 3dd6f14ea70..5f279ddda5b 100644 --- a/doc/user/project/deploy_tokens/index.md +++ b/doc/user/project/deploy_tokens/index.md @@ -216,19 +216,3 @@ Prerequisites: - A deploy token with `read_registry` and `write_registry` scopes. Follow the dependency proxy [authentication instructions](../../packages/dependency_proxy/index.md). - -## Troubleshooting - -### Error: `api error: Repository or object not found:` - -When using a group deploy token to clone from LFS objects, you might get `404 Not Found` responses -and this error message. This occurs because of a bug, documented in -[issue 235398](https://gitlab.com/gitlab-org/gitlab/-/issues/235398). - -```plaintext -api error: Repository or object not found: -https://<URL-with-token>.git/info/lfs/objects/batch -Check that it exists and that you have proper access to it -``` - -The workaround is to use a project deploy token. diff --git a/doc/user/project/description_templates.md b/doc/user/project/description_templates.md index 40c36236932..ffbe7447aa8 100644 --- a/doc/user/project/description_templates.md +++ b/doc/user/project/description_templates.md @@ -40,7 +40,7 @@ To create an issue description template: where `mytemplate` is the name of your issue template. 1. Commit to your default branch. -To check if this has worked correctly, [create a new issue](issues/managing_issues.md#create-an-issue) +To check if this has worked correctly, [create a new issue](issues/create_issues.md) and see if you can find your description template in the **Choose a template** dropdown list. ## Create a merge request template @@ -81,7 +81,23 @@ To discard any changes to the description you've made after selecting the templa NOTE: You can create shortcut links to create an issue using a designated template. -For example: `https://gitlab.com/gitlab-org/gitlab/-/issues/new?issuable_template=Feature%20proposal`. Read more about [creating issues using a URL with prefilled values](issues/managing_issues.md#using-a-url-with-prefilled-values). +For example: `https://gitlab.com/gitlab-org/gitlab/-/issues/new?issuable_template=Feature%20proposal`. Read more about [creating issues using a URL with prefilled values](issues/create_issues.md#using-a-url-with-prefilled-values). + +### Supported variables in merge request templates + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/89810) in GitLab 15.7. + +When you save a merge request for the first time, GitLab replaces these variables in +your merge request template with their values: + +| Variable | Description | Output example | +|----------|-------------|----------------| +| `%{all_commits}` | Messages from all commits in the merge request. Limited to 100 most recent commits. Skips commit bodies exceeding 100 KiB and merge commit messages. | `* Feature introduced` <br><br> `This commit implements feature` <br> `Changelog:added` <br><br> `* Bug fixed` <br><br> `* Documentation improved` <br><br>`This commit introduced better docs.` | +| `%{co_authored_by}` | Names and emails of commit authors in a `Co-authored-by` Git commit trailer format. Limited to authors of 100 most recent commits in merge request. | `Co-authored-by: Zane Doe <zdoe@example.com>` <br> `Co-authored-by: Blake Smith <bsmith@example.com>` | +| `%{first_commit}` | Full message of the first commit in merge request diff. | `Update README.md` | +| `%{first_multiline_commit}` | Full message of the first commit that's not a merge commit and has more than one line in message body. Merge request title if all commits aren't multiline. | `Update README.md`<br><br>`Improved project description in readme file.` | +| `%{source_branch}` | The name of the branch being merged. | `my-feature-branch` | +| `%{target_branch}` | The name of the branch that the changes are applied to. | `main` | ### Set instance-level description templates **(PREMIUM SELF)** diff --git a/doc/user/project/import/bitbucket.md b/doc/user/project/import/bitbucket.md index 3f1a2dcfe2b..98b46650b42 100644 --- a/doc/user/project/import/bitbucket.md +++ b/doc/user/project/import/bitbucket.md @@ -104,7 +104,7 @@ current Bitbucket public name, and reconnect if there's a mismatch: 1. [Use the API to get the currently authenticated user](../../../api/users.md#for-normal-users-1). -1. In the API's response, the `identities` attribute contains the Bitbucket account that exists in +1. In the API response, the `identities` attribute contains the Bitbucket account that exists in the GitLab database. If the `extern_uid` doesn't match the current Bitbucket public name, the user should reconnect their Bitbucket account in the [GitLab profile service sign-in](https://gitlab.com/-/profile/account). diff --git a/doc/user/project/import/bitbucket_server.md b/doc/user/project/import/bitbucket_server.md index 1f34c6d4ad9..d7fa1338c55 100644 --- a/doc/user/project/import/bitbucket_server.md +++ b/doc/user/project/import/bitbucket_server.md @@ -63,6 +63,10 @@ The following items are changed when they are imported: ## User assignment +Prerequisite: + +- Authentication token with administrator access. + When issues and pull requests are importing, the importer tries to find the author's email address with a confirmed email address in the GitLab user database. If no such user is available, the project creator is set as the author. The importer appends a note in the comment to mark the diff --git a/doc/user/project/import/github.md b/doc/user/project/import/github.md index c0b13c76322..07a21d8b941 100644 --- a/doc/user/project/import/github.md +++ b/doc/user/project/import/github.md @@ -59,11 +59,10 @@ their GitHub authors and assignees in the database of the GitLab instance. Pull GitLab. For this association to succeed, each GitHub author and assignee in the repository -must meet one of the following conditions prior to the import: - -- Have previously logged in to a GitLab account using the GitHub icon. -- Have a GitHub account with a [public-facing email address](https://docs.github.com/en/account-and-profile/setting-up-and-managing-your-personal-account-on-github/managing-email-preferences/setting-your-commit-email-address) - that matches their GitLab account's email address. +must have a [public-facing email address](https://docs.github.com/en/account-and-profile/setting-up-and-managing-your-personal-account-on-github/managing-email-preferences/setting-your-commit-email-address) +on GitHub that matches their GitLab email address (regardless of how the account was created). +If their email address from GitHub is set as their secondary email address in GitLab, it must be +confirmed. GitLab content imports that use GitHub accounts require that the GitHub public-facing email address is populated. This means all comments and contributions are properly mapped to the same user in GitLab. GitHub Enterprise does not require this @@ -73,10 +72,8 @@ field to be populated so you may have to add it on existing accounts. ### Use the GitHub integration -Before you begin, ensure that any GitHub users who you want to map to GitLab users have either: - -- A GitLab account that has logged in using the GitHub icon. -- A GitLab account with an email address that matches the [publicly visible email address](https://docs.github.com/en/rest/users#get-a-user) in the profile of the GitHub user +Before you begin, ensure that any GitHub user you want to map to a GitLab user has a GitLab email address that matches their +[publicly visible email address](https://docs.github.com/en/rest/users#get-a-user) on GitHub. If you are importing to GitLab.com, you can alternatively import GitHub repositories using a [personal access token](#use-a-github-token). We do not recommend this method, as it does not associate all user activity (such as issues and pull requests) with matching GitLab users. @@ -96,7 +93,10 @@ If you are using a self-managed GitLab instance or if you are importing from Git ### Use a GitHub token -NOTE: +Prerequisite: + +- Authentication token with administrator access. + Using a personal access token to import projects is not recommended. If you are a GitLab.com user, you can use a personal access token to import your project from GitHub, but this method cannot associate all user activity (such as issues and pull requests) with matching GitLab users. @@ -222,21 +222,26 @@ References to pull requests and issues are preserved. Each imported repository m [visibility level is restricted](../../public_access.md#restrict-use-of-public-or-internal-projects), in which case it defaults to the default project visibility. -### Branch protection rules +### Branch protection rules and project settings + +When they are imported, supported GitHub branch protection rules are mapped to either: + +- GitLab branch protection rules. +- Project-wide GitLab settings. -Supported GitHub branch protection rules are mapped to GitLab branch protection rules or project-wide GitLab settings when they are imported: +| GitHub rule | GitLab rule | Introduced in | +| :---------------------------------------------------------------------------------- | :---------------------------------------------------------------------------------------------------------------------------------------------------------- | :------------------------------------------------------------------ | +| **Require conversation resolution before merging** for the project's default branch | **All threads must be resolved** [project setting](../../discussions/index.md#prevent-merge-unless-all-threads-are-resolved) | [GitLab 15.5](https://gitlab.com/gitlab-org/gitlab/-/issues/371110) | +| **Require a pull request before merging** | **No one** option in the **Allowed to push** list of [branch protection settings](../protected_branches.md#configure-a-protected-branch) | [GitLab 15.5](https://gitlab.com/gitlab-org/gitlab/-/issues/370951) | +| **Require signed commits** for the project's default branch | **Reject unsigned commits** GitLab [push rule](../repository/push_rules.md#prevent-unintended-consequences) **(PREMIUM)** | [GitLab 15.5](https://gitlab.com/gitlab-org/gitlab/-/issues/370949) | +| **Allow force pushes - Everyone** | **Allowed to force push** [branch protection setting](../protected_branches.md#allow-force-push-on-a-protected-branch) | [GitLab 15.6](https://gitlab.com/gitlab-org/gitlab/-/issues/370943) | +| **Require a pull request before merging - Require review from Code Owners** | **Require approval from code owners** [branch protection setting](../protected_branches.md#require-code-owner-approval-on-a-protected-branch) **(PREMIUM)** | [GitLab 15.6](https://gitlab.com/gitlab-org/gitlab/-/issues/376683) | -- GitHub rule **Require conversation resolution before merging** for the project's default branch is mapped to the [**All threads must be resolved** GitLab setting](../../discussions/index.md#prevent-merge-unless-all-threads-are-resolved). [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/371110) in GitLab 15.5. -- GitHub rule **Require a pull request before merging** is mapped to the **No one** option in the **Allowed to push** list of the branch protection rule. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/370951) in GitLab 15.5. -- GitHub rule **Require a pull request before merging - Require review from Code Owners** is mapped to the - [**Code owner approval** branch protection rule](../protected_branches.md#require-code-owner-approval-on-a-protected-branch). Requires GitLab Premium or higher. - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/376683) in GitLab 15.6. -- GitHub rule **Require signed commits** for the project's default branch is mapped to the **Reject unsigned commits** GitLab push rule. Requires GitLab Premium or higher. - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/370949) in GitLab 15.5. -- GitHub rule **Allow force pushes - Everyone** is mapped to the [**Allowed to force push** branch protection rule](../protected_branches.md#allow-force-push-on-a-protected-branch). [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/370943) in GitLab 15.6. -- GitHub rule **Allow force pushes - Specify who can force push** is proposed in issue [370945](https://gitlab.com/gitlab-org/gitlab/-/issues/370945). -- Support for GitHub rule **Require status checks to pass before merging** was proposed in issue [370948](https://gitlab.com/gitlab-org/gitlab/-/issues/370948). However, this rule cannot be translated during project import into GitLab due to technical difficulties. -You can still create [status checks](../merge_requests/status_checks.md) in GitLab yourself. +Mapping GitHub rule **Require status checks to pass before merging** to +[external status checks](../merge_requests/status_checks.md) was considered in issue +[370948](https://gitlab.com/gitlab-org/gitlab/-/issues/370948). However, this rule is not imported during project import +into GitLab due to technical difficulties. You can still create [external status checks](../merge_requests/status_checks.md) +manually. ## Alternative way to import notes and diff notes @@ -274,7 +279,7 @@ Feature.disable(:github_importer_lower_per_page_limit, group) ## Import from GitHub Enterprise on an internal network -If your GitHub Enterprise instance is on a internal network that is unaccessible to the internet, you can use a reverse proxy +If your GitHub Enterprise instance is on a internal network that is inaccessible to the internet, you can use a reverse proxy to allow GitLab.com to access the instance. The proxy needs to: diff --git a/doc/user/project/import/img/gitlab_import_history_page_v14_10.png b/doc/user/project/import/img/gitlab_import_history_page_v14_10.png Binary files differdeleted file mode 100644 index 812696a8faa..00000000000 --- a/doc/user/project/import/img/gitlab_import_history_page_v14_10.png +++ /dev/null diff --git a/doc/user/project/import/index.md b/doc/user/project/import/index.md index 1b5a658d209..208bce90453 100644 --- a/doc/user/project/import/index.md +++ b/doc/user/project/import/index.md @@ -5,76 +5,59 @@ group: Import info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- -# Migrate projects to a GitLab instance **(FREE)** - -See these documents to migrate to GitLab: - -- [From Bitbucket Cloud](bitbucket.md) -- [From Bitbucket Server (also known as Stash)](bitbucket_server.md) -- [From ClearCase](clearcase.md) -- [From CVS](cvs.md) -- [From FogBugz](fogbugz.md) -- [From GitHub.com or GitHub Enterprise](github.md) -- [From GitLab.com](gitlab_com.md) -- [From Gitea](gitea.md) -- [From Perforce](perforce.md) -- [From SVN](svn.md) -- [From TFVC](tfvc.md) -- [From repository by URL](repo_by_url.md) -- [By uploading a manifest file (AOSP)](manifest.md) -- [From Phabricator](phabricator.md) -- [From Jira (issues only)](jira.md) +# Import and migrate projects **(FREE)** -You can also import any Git repository through HTTP from the **New Project** page. Note that if the -repository is too large, the import can timeout. - -You can also [connect your external repository to get CI/CD benefits](../../../ci/ci_cd_for_external_repos/index.md). +If you want to bring existing projects to GitLab or copy GitLab projects to a different location, you can: -## Project import history - -You can view all project imports created by you. This list includes the following: +- Import projects from external systems using one of the [available importers](#available-project-importers). +- Migrate GitLab projects: + - Between two GitLab self-managed instances. + - Between a self-managed instance and GitLab.com in both directions. + - In the same GitLab instance. -- Source (without credentials for security reasons) -- Destination -- Status -- Error details if the import failed +For any type of source and target, you can migrate projects: -To view project import history: +- As part of a [GitLab group migration](../../group/import/index.md). You can't migrate only chosen projects, + but you can migrate many projects at once within a group. +- Using [file exports](../settings/import_export.md). With this method you can migrate projects one by one. No network + connection between instances is required. -1. Sign in to GitLab. -1. On the top bar, select **New** (**{plus}**). -1. Select **New project/repository**. -1. Select **Import project**. -1. Select **History**. +If you only need to migrate Git repositories, you can [import each project by URL](repo_by_url.md). However, you can't +import issues and merge requests this way. To retain metadata like issues and merge requests, either: - +- [Migrate projects with groups](../../group/import/index.md). +- Use [file exports](../settings/import_export.md) to import projects. -The history also includes projects created from [built-in](../working_with_projects.md#create-a-project-from-a-built-in-template) -or [custom](../working_with_projects.md#create-a-project-from-a-built-in-template) -templates. GitLab uses [import repository by URL](repo_by_url.md) -to create a new project from a template. +Keep in mind the limitations of [migrating using file exports](../settings/import_export.md#items-that-are-exported). +When migrating from self-managed to GitLab.com, user associations (such as comment author) +are changed to the user who is importing the projects. -## LFS authentication +## Available project importers -When importing a project that contains LFS objects, if the project has an [`.lfsconfig`](https://github.com/git-lfs/git-lfs/blob/master/docs/man/git-lfs-config.5.ronn) -file with a URL host (`lfs.url`) different from the repository URL host, LFS files are not downloaded. +You can import projects from: -## Migrate from self-managed GitLab to GitLab.com +- [Bitbucket Cloud](bitbucket.md) +- [Bitbucket Server (also known as Stash)](bitbucket_server.md) +- [ClearCase](clearcase.md) +- [CVS](cvs.md) +- [FogBugz](fogbugz.md) +- [GitHub.com or GitHub Enterprise](github.md) +- [GitLab.com](gitlab_com.md) +- [Gitea](gitea.md) +- [Perforce](perforce.md) +- [From SVN](https://git-scm.com/book/en/v2/Git-and-Other-Systems-Git-as-a-Client) +- [TFVC](tfvc.md) +- [Repository by URL](repo_by_url.md) +- [Uloading a manifest file (AOSP)](manifest.md) +- [Phabricator](phabricator.md) +- [Jira (issues only)](jira.md) -If you only need to migrate Git repositories, you can [import each project by URL](repo_by_url.md). -However, you can't import issues and merge requests this way. To retain all metadata like issues and -merge requests, use the [import/export feature](../settings/import_export.md) -to export projects from self-managed GitLab and import those projects into GitLab.com. All GitLab -user associations (such as comment author) are changed to the user importing the project. For more -information, see the prerequisites and important notes in these sections: +You can also import any Git repository through HTTP from the **New Project** page. Note that if the +repository is too large, the import can timeout. -- [Export a project and its data](../settings/import_export.md#export-a-project-and-its-data). -- [Import the project](../settings/import_export.md#import-a-project-and-its-data). +You can then [connect your external repository to get CI/CD benefits](../../../ci/ci_cd_for_external_repos/index.md). -NOTE: -When migrating to GitLab.com, you must create users manually unless [SCIM](../../../user/group/saml_sso/scim_setup.md) -will be used. Creating users with the API is limited to self-managed instances as it requires -administrator access. +## Migrate using the API To migrate all data from self-managed to GitLab.com, you can leverage the [API](../../../api/index.md). Migrate the assets in this order: @@ -83,17 +66,9 @@ Migrate the assets in this order: 1. [Projects](../../../api/projects.md) 1. [Project variables](../../../api/project_level_variables.md) -Keep in mind the limitations of the [import/export feature](../settings/import_export.md#items-that-are-exported). - You must still migrate your [Container Registry](../../packages/container_registry/index.md) over a series of Docker pulls and pushes. Re-run any CI pipelines to retrieve any build artifacts. -## Migrate from GitLab.com to self-managed GitLab - -The process is essentially the same as [migrating from self-managed GitLab to GitLab.com](#migrate-from-self-managed-gitlab-to-gitlabcom). -The main difference is that an administrator can create users on the self-managed GitLab instance -through the UI or the [users API](../../../api/users.md#user-creation). - ## Migrate between two self-managed GitLab instances To migrate from an existing self-managed GitLab instance to a new self-managed GitLab instance, it's @@ -105,12 +80,36 @@ The backups produced don't depend on the operating system running GitLab. You ca the restore method to switch between different operating system distributions or versions, as long as the same GitLab version [is available for installation](../../../administration/package_information/supported_os.md). -To instead merge two self-managed GitLab instances together, use the instructions in -[Migrate from self-managed GitLab to GitLab.com](#migrate-from-self-managed-gitlab-to-gitlabcom). -This method is useful when both self-managed instances have existing data that must be preserved. +Also note that administrators can use the [Users API](../../../api/users.md) to migrate users. + +## View project import history + +You can view all project imports created by you. This list includes the following: + +- Paths of source projects if projects were imported from external systems, or import method if GitLab projects were migrated. +- Paths of destination projects. +- Start date of each import. +- Status of each import. +- Error details if any errors occurred. + +To view project import history: + +1. Sign in to GitLab. +1. On the top bar, select **Create new...** (**{plus-square}**). +1. Select **New project/repository**. +1. Select **Import project**. +1. Select **History** in the upper right corner. +1. If there are any errors for a particular import, you can see them by selecting **Details**. + +The history also includes projects created from [built-in](../working_with_projects.md#create-a-project-from-a-built-in-template) +or [custom](../working_with_projects.md#create-a-project-from-a-built-in-template) +templates. GitLab uses [import repository by URL](repo_by_url.md) +to create a new project from a template. + +## LFS authentication -Also note that administrators can use the [Users API](../../../api/users.md) -to migrate users. +When importing a project that contains LFS objects, if the project has an [`.lfsconfig`](https://github.com/git-lfs/git-lfs/blob/master/docs/man/git-lfs-config.5.ronn) +file with a URL host (`lfs.url`) different from the repository URL host, LFS files are not downloaded. ## Project aliases **(PREMIUM SELF)** diff --git a/doc/user/project/import/svn.md b/doc/user/project/import/svn.md index 1687d621e2e..6730ef862e6 100644 --- a/doc/user/project/import/svn.md +++ b/doc/user/project/import/svn.md @@ -1,91 +1,11 @@ --- -type: howto -stage: Manage -group: Import -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments +redirect_to: 'index.md' +remove_date: '2023-03-15' --- -# Migrate from Subversion to GitLab **(FREE)** +This document was moved to [another location](index.md). -GitLab uses Git as its version control system. If you're using Subversion (SVN) as your version control system, -you can migrate to using a Git repository in GitLab using `svn2git`. - -You can follow the steps on this page to migrate to Git if your SVN repository: - -- Has a standard format (trunk, branches, and tags). -- Is not nested. - -For a non-standard repository see the [`svn2git` documentation](https://github.com/nirvdrum/svn2git). - -We recommend a hard cut over from SVN to Git and GitLab. Run the migration command once and then have all users use the -new GitLab repository immediately. - -## Install `svn2git` - -Install `svn2git` on a local workstation rather than the GitLab server: - -- On all systems you can install as a Ruby gem if you already have Ruby and Git installed: - - ```shell - sudo gem install svn2git - ``` - -- On Debian-based Linux distributions you can install the native packages: - - ```shell - sudo apt-get install git-core git-svn ruby - ``` - -## Prepare an authors file (recommended) - -Prepare an authors file so `svn2git` can map SVN authors to Git authors. If you choose not to create the authors file, -commits are not attributed to the correct GitLab user. - -To map authors, you must map every author present on changes in the SVN repository. If you don't, the -migration fails and you have to update the author file accordingly. - -1. Search through the SVN repository and output a list of authors: - - ```shell - svn log --quiet | grep -E "r[0-9]+ \| .+ \|" | cut -d'|' -f2 | sed 's/ //g' | sort | uniq - ``` - -1. Use the output from the last command to construct the authors file. Create a file called `authors.txt` and add one - mapping per line. For example: - - ```plaintext - sidneyjones = Sidney Jones <sidneyjones@example.com> - ``` - -## Migrate SVN repository to Git repository - -`svn2git` supports excluding certain file paths, branches, tags, and more. See -the [`svn2git` documentation](https://github.com/nirvdrum/svn2git) or run `svn2git --help` for full documentation on all of -the available options. - -For each repository to migrate: - -1. Create a new directory and change into it. -1. For repositories that: - - - Don't require a username and password, run: - - ```shell - svn2git https://svn.example.com/path/to/repo --authors /path/to/authors.txt - ``` - - - Do require a username and password, run: - - ```shell - svn2git https://svn.example.com/path/to/repo --authors /path/to/authors.txt --username <username> --password <password> - ``` - -1. Create a new GitLab project for your migrated code. -1. Copy the SSH or HTTP(S) repository URL from the GitLab project page. -1. Add the GitLab repository as a Git remote and push all the changes. This pushes all commits, branches, and tags. - - ```shell - git remote add origin git@gitlab.example.com:<group>/<project>.git - git push --all origin - git push --tags origin - ``` +<!-- This redirect file can be deleted after <2023-03-15>. --> +<!-- Redirects that point to other docs in the same project expire in three months. --> +<!-- Redirects that point to docs in a different project or site (link is not relative and starts with `https:`) expire in one year. --> +<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/redirects.html --> diff --git a/doc/user/project/integrations/bamboo.md b/doc/user/project/integrations/bamboo.md index fceec006a1a..db90bafaaa5 100644 --- a/doc/user/project/integrations/bamboo.md +++ b/doc/user/project/integrations/bamboo.md @@ -63,7 +63,7 @@ Bamboo. For example, `https://bamboo.example.com/browse/PROJ-PLAN`. ## Update Bamboo build status in GitLab -You can use a script that uses the [commit status API](../../../api/commits.md#post-the-build-status-to-a-commit) +You can use a script that uses the [commit status API](../../../api/commits.md#set-the-pipeline-status-of-a-commit) and Bamboo build variables to: - Update the commit with the build status. diff --git a/doc/user/project/integrations/gitlab_slack_application.md b/doc/user/project/integrations/gitlab_slack_application.md index 8d6fdf882b7..50b52421a5a 100644 --- a/doc/user/project/integrations/gitlab_slack_application.md +++ b/doc/user/project/integrations/gitlab_slack_application.md @@ -10,7 +10,7 @@ NOTE: The GitLab for Slack app is only configurable for GitLab.com. It does **not** work for on-premises installations where you can configure [Slack slash commands](slack_slash_commands.md) instead. See -[Slack application integration for self-managed instances](https://gitlab.com/gitlab-org/gitlab/-/issues/28164) +[Slack application integration for self-managed instances](https://gitlab.com/groups/gitlab-org/-/epics/1211) for our plans to make the app configurable for all GitLab installations. Slack provides a native application that you can enable with your project's @@ -36,12 +36,12 @@ workspace to be able to install a new application. See To enable the GitLab integration for your Slack workspace: -1. Go to your project's **Settings > Integration > Slack application** (only +1. Go to your project's **Settings > Integration > GitLab for Slack app** (only visible on GitLab.com). -1. Select **Install Slack app**. +1. Select **Install GitLab for Slack app**. 1. Select **Allow** on Slack's confirmation screen. -You can also select **Reinstall Slack app** to update the app in your Slack workspace +You can also select **Reinstall GitLab for Slack app** to update the app in your Slack workspace to the latest version. See [Version history](#version-history) for details. ## Create a project alias for Slack @@ -50,7 +50,7 @@ To create a project alias on GitLab.com for Slack integration: 1. Go to your project's home page. 1. Go to **Settings > Integrations** (only visible on GitLab.com). -1. On the **Integrations** page, select **Slack application**. +1. On the **Integrations** page, select **GitLab for Slack app**. 1. The current **Project Alias**, if any, is displayed. To edit this value, select **Edit**. 1. Enter your desired alias, and select **Save changes**. @@ -91,7 +91,7 @@ As a workaround, ensure your app is up to date. To update an existing Slack integration: 1. Go to your [chat settings](https://gitlab.com/-/profile/chat_names). -1. Next to your project, select **Slack application**. -1. Select **Reinstall Slack app**. +1. Next to your project, select **GitLab for Slack app**. +1. Select **Reinstall GitLab for Slack app**. Alternatively, you can [configure a new Slack integration](https://about.gitlab.com/solutions/slack/). diff --git a/doc/user/project/integrations/index.md b/doc/user/project/integrations/index.md index 77444570499..769a45fc6ff 100644 --- a/doc/user/project/integrations/index.md +++ b/doc/user/project/integrations/index.md @@ -58,7 +58,6 @@ You can configure the following integrations. | [Emails on push](emails_on_push.md) | Send commits and diff of each push by email. | **{dotted-circle}** No | | [EWM](ewm.md) | Use IBM Engineering Workflow Management as the issue tracker. | **{dotted-circle}** No | | [External wiki](../wiki/index.md#link-an-external-wiki) | Link an external wiki. | **{dotted-circle}** No | -| [Flowdock](../../../api/integrations.md#flowdock) | Send notifications from GitLab to Flowdock flows. | **{dotted-circle}** No | | [GitHub](github.md) | Obtain statuses for commits and pull requests. | **{dotted-circle}** No | | [Google Chat](hangouts_chat.md) | Send notifications from your GitLab project to a room in Google Chat. | **{dotted-circle}** No | | [Harbor](harbor.md) | Use Harbor as the container registry. | **{dotted-circle}** No | @@ -77,7 +76,7 @@ You can configure the following integrations. | Pushover | Get real-time notifications on your device. | **{dotted-circle}** No | | [Redmine](redmine.md) | Use Redmine as the issue tracker. | **{dotted-circle}** No | | [Shimo Workspace](shimo.md) | Use Shimo instead of the GitLab Wiki. | **{dotted-circle}** No | -| [Slack application](gitlab_slack_application.md) | Use Slack's official GitLab application. | **{dotted-circle}** No | +| [GitLab for Slack app](gitlab_slack_application.md) | Use Slack's official GitLab application. | **{dotted-circle}** No | | [Slack notifications](slack.md) | Send notifications about project events to Slack. | **{dotted-circle}** No | | [Slack slash commands](slack_slash_commands.md) | Enable slash commands in a workspace. | **{dotted-circle}** No | | [Unify Circuit](unify_circuit.md) | Send notifications about project events to Unify Circuit. | **{dotted-circle}** No | diff --git a/doc/user/project/integrations/mlflow_client.md b/doc/user/project/integrations/mlflow_client.md index 82bfd08e926..bd14021ab1c 100644 --- a/doc/user/project/integrations/mlflow_client.md +++ b/doc/user/project/integrations/mlflow_client.md @@ -19,9 +19,9 @@ Setting up your integrations requires minimal changes to existing code. GitLab plays the role of proxy server, both for artifact storage and tracking data. It reflects the MLFlow [Scenario 5](https://www.mlflow.org/docs/latest/tracking.html#scenario-5-mlflow-tracking-server-enabled-with-proxied-artifact-storage-access). -## Enable MFlow Client Integration +## Enable MLFlow Client Integration -Complete this task to enable MFlow Client Integration. +Complete this task to enable MLFlow Client Integration. Prerequisites: @@ -49,17 +49,17 @@ that can be explored by selecting an experiment. - The API GitLab supports is the one defined at MLFlow version 1.28.0. - API endpoints not listed above are not supported. -- During creation of experiments and runs, tags are ExperimentTags and RunTags are ignored. -- MLFLow Model Registry is not supported. +- During creation of experiments and runs, tags are ExperimentTags and RunTags are stored, even though they are not displayed. +- MLFlow Model Registry is not supported. ## Supported methods and caveats This is a list of methods we support from the MLFlow client. Other methods might be supported but were not tested. More information can be found in the [MLFlow Documentation](https://www.mlflow.org/docs/1.28.0/python_api/mlflow.html). -### `set_experiment` +### `set_experiment()` -Accepts both experiment_name and experiment_id +Accepts both `experiment_name` and `experiment_id` ### `start_run()` diff --git a/doc/user/project/integrations/slack.md b/doc/user/project/integrations/slack.md index d34c558ebbc..2ded5799c23 100644 --- a/doc/user/project/integrations/slack.md +++ b/doc/user/project/integrations/slack.md @@ -36,10 +36,6 @@ to control GitLab from Slack. Slash commands are configured separately. - *To send messages to channels,* enter the Slack channel names, separated by commas. - *To send direct messages,* use the Member ID found in the user's Slack profile. - - NOTE: - Usernames and private channels are not supported. - 1. In **Webhook**, enter the webhook URL you copied in the [Slack configuration](#configure-slack) step. 1. Optional. In **Username**, enter the username of the Slack bot that sends diff --git a/doc/user/project/integrations/webhook_events.md b/doc/user/project/integrations/webhook_events.md index 60187b9a682..63282d6ec6e 100644 --- a/doc/user/project/integrations/webhook_events.md +++ b/doc/user/project/integrations/webhook_events.md @@ -1019,7 +1019,7 @@ Payload example: ``` NOTE: -The fields `assignee_id`, `state`, `merge_status` are deprecated. +The fields `assignee_id`, `state`, `merge_status` are [deprecated](../../../api/merge_requests.md). ## Wiki page events @@ -1171,7 +1171,7 @@ Payload example: "project":{ "id": 41, "web_url": "https://gitlab.example.com/gitlab-org/upstream-project", - "path_with_namespace": "gitlab-org/upstream-project", + "path_with_namespace": "gitlab-org/upstream-project" }, "pipeline_id": 30, "job_id": 3401 @@ -1475,6 +1475,8 @@ Payload example: "deployable_id": 796, "deployable_url": "http://10.126.0.2:3000/root/test-deployment-webhooks/-/jobs/796", "environment": "staging", + "environment_slug": "staging", + "environment_external_url": "https://staging.example.com", "project": { "id": 30, "name": "test-deployment-webhooks", diff --git a/doc/user/project/integrations/webhooks.md b/doc/user/project/integrations/webhooks.md index ef6957ac2d8..3d45e947c4c 100644 --- a/doc/user/project/integrations/webhooks.md +++ b/doc/user/project/integrations/webhooks.md @@ -8,7 +8,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w [Webhooks](https://en.wikipedia.org/wiki/Webhook) are custom HTTP callbacks that you define. They are usually triggered by an -event, such as pushing code to a repository or posting a comment on a blog. +event, such as pushing code to a repository or posting a comment on an issue. When the event occurs, the source app makes an HTTP request to the URI configured for the webhook. The action to take may be anything. For example, you can use webhooks to: @@ -52,7 +52,7 @@ specific to a group, including: ## Configure a webhook in GitLab -You can configure a webhook for a group or a project. +To configure a webhook for a project or group: 1. In your project or group, on the left sidebar, select **Settings > Webhooks**. 1. In **URL**, enter the URL of the webhook endpoint. @@ -62,6 +62,40 @@ You can configure a webhook for a group or a project. 1. Optional. Clear the **Enable SSL verification** checkbox to disable [SSL verification](index.md#manage-ssl-verification). 1. Select **Add webhook**. +## Mask sensitive portions of webhook URLs + +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/99995) in GitLab 15.5 [with a flag](../../../administration/feature_flags.md) named `webhook_form_mask_url`. Disabled by default. +> - [Enabled on GitLab.com](https://gitlab.com/gitlab-org/gitlab/-/issues/376106) in GitLab 15.6. +> - [Generally available](https://gitlab.com/gitlab-org/gitlab/-/issues/376106) in GitLab 15.7. Feature flag `webhook_form_mask_url` removed. + +You can define and mask sensitive portions of webhook URLs and replace them +with configured values any number of times when webhooks are executed. +Sensitive portions do not get logged and are encrypted at rest in the database. + +To mask sensitive portions of the webhook URL: + +1. In your project or group, on the left sidebar, select **Settings > Webhooks**. +1. In **URL**, enter the full webhook URL. +1. Select **Mask portions of URL**. +1. In **Sensitive portion of URL**, enter the portion you want to mask. +1. In **How it looks in the UI**, enter the masking value. + +To interpolate sensitive portions for each webhook, use `url_variables`. +For example, if a webhook has the following URL: + +```plaintext +https://{subdomain}.example.com/{path}?key={value} +``` + +You must define the following variables: + +- `subdomain` +- `path` +- `value` + +Variable names can contain only lowercase letters (`a-z`), numbers (`0-9`), or underscores (`_`). +You can define URL variables directly using the REST API. + ## Configure your webhook receiver endpoint Webhook receiver endpoints should be fast and stable. @@ -86,27 +120,22 @@ Endpoints should follow these best practices: ### Failing webhooks -> Introduced in GitLab 13.12 [with a flag](../../../administration/feature_flags.md) named `web_hooks_disable_failed`. Disabled by default. - -FLAG: -On self-managed GitLab, by default this feature is not available. To make it available, -ask an administrator to [enable the feature flag](../../../administration/feature_flags.md) named `web_hooks_disable_failed`. -On GitLab.com, this feature is not available. -The feature is not ready for production use. +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/60837) in GitLab 13.12 [with a flag](../../../administration/feature_flags.md) named `web_hooks_disable_failed`. Disabled by default. +> - [Enabled on GitLab.com and self-managed](https://gitlab.com/gitlab-org/gitlab/-/issues/329849) in GitLab 15.7. +> - [Generally available](https://gitlab.com/gitlab-org/gitlab/-/issues/329849) in GitLab 15.7. Feature flag `web_hooks_disable_failed` removed. If a webhook fails repeatedly, it may be disabled automatically. Webhooks that return response codes in the `5xx` range are understood to be failing -intermittently, and are temporarily disabled. This lasts initially -for 10 minutes. If the hook continues to fail, the back-off period is -extended on each retry, up to a maximum disabled period of 24 hours. +intermittently and are temporarily disabled. These webhooks are initially disabled +for 1 minute, which is extended on each retry up to a maximum of 24 hours. -Webhooks that return failure codes in the `4xx` range are understood to be -misconfigured, and these are disabled until you manually re-enable -them. These webhooks are not automatically retried. +Webhooks that return response codes in the `4xx` range are understood to be +misconfigured and are permanently disabled until you manually re-enable +them yourself. -See [troubleshooting](#troubleshoot-webhooks) for information on -how to see if a webhook is disabled, and how to re-enable it. +See [Troubleshooting](#troubleshoot-webhooks) for more information on +disabled webhooks and how to re-enable them. ## Test a webhook @@ -190,10 +219,10 @@ You can filter push events by branch. Use one of the following options to filter - **All branches**: push events from all branches. - **Wildcard pattern**: push events from a branch that matches a wildcard pattern (for example, `*-stable` or `production/*`). -- **Regular expression**: push events from a branch that matches a regular expression (for example, `(feature|hotfix)/*`). +- **Regular expression**: push events from a branch that matches a regular expression (for example, `^(feature|hotfix)/`). -You can configure branch filtering -in the [webhook settings](#configure-a-webhook-in-gitlab) in your project. +To configure branch filtering for a project or group, see +[Configure a webhook in GitLab](#configure-a-webhook-in-gitlab). ## How image URLs are displayed in the webhook body @@ -302,13 +331,7 @@ GitLab expects a response in [10 seconds](../../../user/gitlab_com/index.md#othe ### Re-enable disabled webhooks > - Introduced in GitLab 15.2 [with a flag](../../../administration/feature_flags.md) named `webhooks_failed_callout`. Disabled by default. -> - The [`web_hooks_disable_failed` flag](#failing-webhooks) must also be enabled for this feature to work. Disabled by default. - -FLAG: -On self-managed GitLab, by default this feature is not available. To make it available, -ask an administrator to [enable the feature flags](../../../administration/feature_flags.md) named `webhooks_failed_callout` and `web_hooks_disable_failed`. -On GitLab.com, this feature is not available. -The feature is not ready for production use. +> - [Generally available](https://gitlab.com/gitlab-org/gitlab/-/issues/365535) in GitLab 15.7. Feature flag `webhooks_failed_callout` removed. If a webhook is failing, a banner displays at the top of the edit page explaining why it is disabled, and when it will be automatically re-enabled. For example: diff --git a/doc/user/project/issue_board.md b/doc/user/project/issue_board.md index c2952b23615..234faa893eb 100644 --- a/doc/user/project/issue_board.md +++ b/doc/user/project/issue_board.md @@ -463,6 +463,7 @@ You can edit the following issue attributes in the right sidebar: - Confidentiality - Due date - [Epic](../group/epics/index.md) +- [Health status](issues/managing_issues.md#health-status) - [Iteration](../group/iterations/index.md) - Labels - Milestone diff --git a/doc/user/project/issues/create_issues.md b/doc/user/project/issues/create_issues.md new file mode 100644 index 00000000000..3c2e20c1250 --- /dev/null +++ b/doc/user/project/issues/create_issues.md @@ -0,0 +1,221 @@ +--- +stage: Plan +group: Project Management +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments +--- + +# Create an issue **(FREE)** + +When you create an issue, you are prompted to enter the fields of the issue. +If you know the values you want to assign to an issue, you can use +[quick actions](../quick_actions.md) to enter them. + +You can create an issue in many ways in GitLab: + +- [From a project](#from-a-project) +- [From a group](#from-a-group) +- [From another issue or incident](#from-another-issue-or-incident) +- [From an issue board](#from-an-issue-board) +- [By sending an email](#by-sending-an-email) +- [Using a URL with prefilled values](#using-a-url-with-prefilled-values) +- [Using Service Desk](#using-service-desk) + +## From a project + +Prerequisites: + +- You must have at least the Guest role for the project. + +To create an issue: + +1. On the top bar, select **Main menu > Projects** and find your project. +1. Either: + + - On the left sidebar, select **Issues**, and then, in the top right corner, select **New issue**. + - On the top bar, select the plus sign (**{plus-square}**) and then, under **This project**, + select **New issue**. + +1. Complete the [fields](#fields-in-the-new-issue-form). +1. Select **Create issue**. + +The newly created issue opens. + +## From a group + +Issues belong to projects, but when you're in a group, you can access and create issues that belong +to the projects in the group. + +Prerequisites: + +- You must have at least the Guest role for the project in the group. + +To create an issue from a group: + +1. On the top bar, select **Main menu > Groups** and find your group. +1. On the left sidebar, select **Issues**. +1. In the top right corner, select **Select project to create issue**. +1. Select the project you'd like to create an issue for. The button now reflects the selected + project. +1. Select **New issue in `<project name>`**. +1. Complete the [fields](#fields-in-the-new-issue-form). +1. Select **Create issue**. + +The newly created issue opens. + +The project you selected most recently becomes the default for your next visit. +This can save you a lot of time, if you mostly create issues for the same project. + +## From another issue or incident + +> - New issue becoming linked to the issue of origin [introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/68226) in GitLab 14.3. +> - **Relate to…** checkbox [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/198494) in GitLab 14.9. + +You can create a new issue from an existing one. The two issues can then be marked as related. + +Prerequisites: + +- You must have at least the Guest role for the project. + +To create an issue from another issue: + +1. In an existing issue, select the vertical ellipsis (**{ellipsis_v}**). +1. Select **New related issue**. +1. Complete the [fields](#fields-in-the-new-issue-form). + The new issue form has a **Relate to issue #123** checkbox, where `123` is the ID of the + issue of origin. If you keep this checkbox checked, the two issues become + [linked](related_issues.md). +1. Select **Create issue**. + +The newly created issue opens. + +## From an issue board + +You can create a new issue from an [issue board](../issue_board.md). + +Prerequisites: + +- You must have at least the Guest role for the project. + +To create an issue from a project issue board: + +1. On the top bar, select **Main menu > Projects** and find your project. +1. Select **Issues > Boards**. +1. At the top of a board list, select **New issue** (**{plus-square}**). +1. Enter the issue's title. +1. Select **Create issue**. + +To create an issue from a group issue board: + +1. On the top bar, select **Main menu > Groups** and find your group. +1. Select **Issues > Boards**. +1. At the top of a board list, select **New issue** (**{plus-square}**). +1. Enter the issue's title. +1. Under **Projects**, select the project in the group that the issue should belong to. +1. Select **Create issue**. + +The issue is created and shows up in the board list. It shares the list's characteristic, so, for +example, if the list is scoped to a label `Frontend`, the new issue also has this label. + +## By sending an email + +> - Generated email address format changed in GitLab 11.7. +> - The older format is still supported, so existing aliases and contacts still work. + +You can send an email to create an issue in a project on the project's +**Issues List** page. + +Prerequisites: + +- Your GitLab instance must have [incoming email](../../../administration/incoming_email.md) + configured. +- There must be at least one issue in the issue list. +- You must have at least the Guest role for the project. + +To email an issue to a project: + +1. On the top bar, select **Main menu > Projects** and find your project. +1. Select **Issues**. +1. At the bottom of the page, select **Email a new issue to this project**. +1. To copy the email address, select **Copy** (**{copy-to-clipboard}**). +1. From your email client, send an email to this address. + The subject is used as the title of the new issue, and the email body becomes the description. + You can use [Markdown](../../markdown.md) and [quick actions](../quick_actions.md). + +A new issue is created, with your user as the author. +You can save this address as a contact in your email client to use it again. + +WARNING: +The email address you see is a private email address, generated just for you. +**Keep it to yourself**, because anyone who knows it can create issues or merge requests as if they +were you. + +To regenerate the email address: + +1. On the issues list, select **Email a new issue to this project**. +1. Select **reset this token**. + +## Using a URL with prefilled values + +> - Ability to use both `issuable_template` and `issue[description]` in the same URL [introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/80554) in GitLab 14.9. +> - Ability to specify `add_related_issue` [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/198494) in GitLab 14.9. + +To link directly to the new issue page with prefilled fields, use query +string parameters in a URL. You can embed a URL in an external +HTML page to create issues with certain fields prefilled. + +| Field | URL parameter | Notes | +| -------------------- | --------------------- | ------------------------------------------------------------------------------------------------------------------------------- | +| Title | `issue[title]` | Must be [URL-encoded](../../../api/index.md#namespaced-path-encoding). | +| Issue type | `issue[issue_type]` | Either `incident` or `issue`. | +| Description template | `issuable_template` | Must be [URL-encoded](../../../api/index.md#namespaced-path-encoding). | +| Description | `issue[description]` | Must be [URL-encoded](../../../api/index.md#namespaced-path-encoding). If used in combination with `issuable_template` or a [default issue template](../description_templates.md#set-a-default-template-for-merge-requests-and-issues), the `issue[description]` value is appended to the template. | +| Confidential | `issue[confidential]` | If `true`, the issue is marked as confidential. | +| Relate to… | `add_related_issue` | A numeric issue ID. If present, the issue form shows a [**Relate to…** checkbox](#from-another-issue-or-incident) to optionally link the new issue to the specified existing issue. | + +Adapt these examples to form your new issue URL with prefilled fields. +To create an issue in the GitLab project: + +- With a prefilled title and description: + + ```plaintext + https://gitlab.com/gitlab-org/gitlab/-/issues/new?issue[title]=Whoa%2C%20we%27re%20half-way%20there&issue[description]=Whoa%2C%20livin%27%20in%20a%20URL + ``` + +- With a prefilled title and description template: + + ```plaintext + https://gitlab.com/gitlab-org/gitlab/-/issues/new?issue[title]=Validate%20new%20concept&issuable_template=Feature%20Proposal%20-%20basic + ``` + +- With a prefilled title, description, and marked as confidential: + + ```plaintext + https://gitlab.com/gitlab-org/gitlab/-/issues/new?issue[title]=Validate%20new%20concept&issue[description]=Research%20idea&issue[confidential]=true + ``` + +## Using Service Desk + +To offer email support, enable [Service Desk](../service_desk.md) for your project. + +Now, when your customer sends a new email, a new issue can be created in +the appropriate project and followed up from there. + +### Fields in the new issue form + +> - Adding the new issue to an epic [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/13847) in GitLab 13.1. +> - Iteration field [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/233517) in GitLab 15.6. + +When you're creating a new issue, you can complete the following fields: + +- Title +- Type: either issue (default) or incident +- [Description template](../description_templates.md): overwrites anything in the Description text box +- Description: you can use [Markdown](../../markdown.md) and [quick actions](../quick_actions.md) +- Checkbox to make the issue [confidential](confidential_issues.md) +- [Assignees](managing_issues.md#assignee) +- [Weight](issue_weight.md) +- [Epic](../../group/epics/index.md) +- [Due date](due_dates.md) +- [Milestone](../milestones/index.md) +- [Labels](../labels.md) +- [Iteration](../../group/iterations/index.md) diff --git a/doc/user/project/issues/index.md b/doc/user/project/issues/index.md index 09067b69696..6c9a645d817 100644 --- a/doc/user/project/issues/index.md +++ b/doc/user/project/issues/index.md @@ -29,7 +29,7 @@ To learn how the GitLab Strategic Marketing department uses GitLab issues with [ ## Related topics -- [Create issues](managing_issues.md#create-an-issue) +- [Create issues](create_issues.md) - [Create an issue from a template](../../project/description_templates.md#use-the-templates) - [Edit issues](managing_issues.md#edit-an-issue) - [Move issues](managing_issues.md#move-an-issue) diff --git a/doc/user/project/issues/issue_weight.md b/doc/user/project/issues/issue_weight.md index 1ba5a4415e0..d852ad3262b 100644 --- a/doc/user/project/issues/issue_weight.md +++ b/doc/user/project/issues/issue_weight.md @@ -36,7 +36,7 @@ When you change the weight of an issue, the new value overwrites the previous va ### When you create an issue -To set the issue weight when you [create an issue](managing_issues.md#create-an-issue), enter a +To set the issue weight when you [create an issue](create_issues.md), enter a number under **Weight**. ### From an existing issue diff --git a/doc/user/project/issues/managing_issues.md b/doc/user/project/issues/managing_issues.md index 8cd211a51c7..ea90dda88f6 100644 --- a/doc/user/project/issues/managing_issues.md +++ b/doc/user/project/issues/managing_issues.md @@ -6,224 +6,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w # Manage issues **(FREE)** -[GitLab Issues](index.md) are the fundamental medium for collaborating on ideas and -planning work in GitLab. - -## Create an issue - -When you create an issue, you are prompted to enter the fields of the issue. -If you know the values you want to assign to an issue, you can use -[quick actions](../quick_actions.md) to enter them. - -You can create an issue in many ways in GitLab: - -- [From a project](#from-a-project) -- [From a group](#from-a-group) -- [From another issue or incident](#from-another-issue-or-incident) -- [From an issue board](#from-an-issue-board) -- [By sending an email](#by-sending-an-email) -- [Using a URL with prefilled values](#using-a-url-with-prefilled-values) -- [Using Service Desk](#using-service-desk) - -### From a project - -Prerequisites: - -- You must have at least the Guest role for the project. - -To create an issue: - -1. On the top bar, select **Main menu > Projects** and find your project. -1. Either: - - - On the left sidebar, select **Issues**, and then, in the top right corner, select **New issue**. - - On the top bar, select the plus sign (**{plus-square}**) and then, under **This project**, - select **New issue**. - -1. Complete the [fields](#fields-in-the-new-issue-form). -1. Select **Create issue**. - -The newly created issue opens. - -### From a group - -Issues belong to projects, but when you're in a group, you can access and create issues that belong -to the projects in the group. - -Prerequisites: - -- You must have at least the Guest role for the project in the group. - -To create an issue from a group: - -1. On the top bar, select **Main menu > Groups** and find your group. -1. On the left sidebar, select **Issues**. -1. In the top right corner, select **Select project to create issue**. -1. Select the project you'd like to create an issue for. The button now reflects the selected - project. -1. Select **New issue in `<project name>`**. -1. Complete the [fields](#fields-in-the-new-issue-form). -1. Select **Create issue**. - -The newly created issue opens. - -The project you selected most recently becomes the default for your next visit. -This can save you a lot of time, if you mostly create issues for the same project. - -### From another issue or incident - -> - New issue becoming linked to the issue of origin [introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/68226) in GitLab 14.3. -> - **Relate to…** checkbox [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/198494) in GitLab 14.9. - -You can create a new issue from an existing one. The two issues can then be marked as related. - -Prerequisites: - -- You must have at least the Guest role for the project. - -To create an issue from another issue: - -1. In an existing issue, select the vertical ellipsis (**{ellipsis_v}**). -1. Select **New related issue**. -1. Complete the [fields](#fields-in-the-new-issue-form). - The new issue form has a **Relate to issue #123** checkbox, where `123` is the ID of the - issue of origin. If you keep this checkbox checked, the two issues become - [linked](related_issues.md). -1. Select **Create issue**. - -The newly created issue opens. - -### From an issue board - -You can create a new issue from an [issue board](../issue_board.md). - -Prerequisites: - -- You must have at least the Guest role for the project. - -To create an issue from a project issue board: - -1. On the top bar, select **Main menu > Projects** and find your project. -1. Select **Issues > Boards**. -1. At the top of a board list, select **New issue** (**{plus-square}**). -1. Enter the issue's title. -1. Select **Create issue**. - -To create an issue from a group issue board: - -1. On the top bar, select **Main menu > Groups** and find your group. -1. Select **Issues > Boards**. -1. At the top of a board list, select **New issue** (**{plus-square}**). -1. Enter the issue's title. -1. Under **Projects**, select the project in the group that the issue should belong to. -1. Select **Create issue**. - -The issue is created and shows up in the board list. It shares the list's characteristic, so, for -example, if the list is scoped to a label `Frontend`, the new issue also has this label. - -### By sending an email - -> - Generated email address format changed in GitLab 11.7. -> - The older format is still supported, so existing aliases and contacts still work. - -You can send an email to create an issue in a project on the project's -**Issues List** page. - -Prerequisites: - -- Your GitLab instance must have [incoming email](../../../administration/incoming_email.md) - configured. -- There must be at least one issue in the issue list. -- You must have at least the Guest role for the project. - -To email an issue to a project: - -1. On the top bar, select **Main menu > Projects** and find your project. -1. Select **Issues**. -1. At the bottom of the page, select **Email a new issue to this project**. -1. To copy the email address, select **Copy** (**{copy-to-clipboard}**). -1. From your email client, send an email to this address. - The subject is used as the title of the new issue, and the email body becomes the description. - You can use [Markdown](../../markdown.md) and [quick actions](../quick_actions.md). - -A new issue is created, with your user as the author. -You can save this address as a contact in your email client to use it again. - -WARNING: -The email address you see is a private email address, generated just for you. -**Keep it to yourself**, because anyone who knows it can create issues or merge requests as if they -were you. - -To regenerate the email address: - -1. On the issues list, select **Email a new issue to this project**. -1. Select **reset this token**. - -### Using a URL with prefilled values - -> - Ability to use both `issuable_template` and `issue[description]` in the same URL [introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/80554) in GitLab 14.9. -> - Ability to specify `add_related_issue` [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/198494) in GitLab 14.9. - -To link directly to the new issue page with prefilled fields, use query -string parameters in a URL. You can embed a URL in an external -HTML page to create issues with certain fields prefilled. - -| Field | URL parameter | Notes | -| -------------------- | --------------------- | ------------------------------------------------------------------------------------------------------------------------------- | -| Title | `issue[title]` | Must be [URL-encoded](../../../api/index.md#namespaced-path-encoding). | -| Issue type | `issue[issue_type]` | Either `incident` or `issue`. | -| Description template | `issuable_template` | Must be [URL-encoded](../../../api/index.md#namespaced-path-encoding). | -| Description | `issue[description]` | Must be [URL-encoded](../../../api/index.md#namespaced-path-encoding). If used in combination with `issuable_template` or a [default issue template](../description_templates.md#set-a-default-template-for-merge-requests-and-issues), the `issue[description]` value is appended to the template. | -| Confidential | `issue[confidential]` | If `true`, the issue is marked as confidential. | -| Relate to… | `add_related_issue` | A numeric issue ID. If present, the issue form shows a [**Relate to…** checkbox](#from-another-issue-or-incident) to optionally link the new issue to the specified existing issue. | - -Adapt these examples to form your new issue URL with prefilled fields. -To create an issue in the GitLab project: - -- With a prefilled title and description: - - ```plaintext - https://gitlab.com/gitlab-org/gitlab/-/issues/new?issue[title]=Whoa%2C%20we%27re%20half-way%20there&issue[description]=Whoa%2C%20livin%27%20in%20a%20URL - ``` - -- With a prefilled title and description template: - - ```plaintext - https://gitlab.com/gitlab-org/gitlab/-/issues/new?issue[title]=Validate%20new%20concept&issuable_template=Feature%20Proposal%20-%20basic - ``` - -- With a prefilled title, description, and marked as confidential: - - ```plaintext - https://gitlab.com/gitlab-org/gitlab/-/issues/new?issue[title]=Validate%20new%20concept&issue[description]=Research%20idea&issue[confidential]=true - ``` - -### Using Service Desk - -To offer email support, enable [Service Desk](../service_desk.md) for your project. - -Now, when your customer sends a new email, a new issue can be created in -the appropriate project and followed up from there. - -### Fields in the new issue form - -> - Adding the new issue to an epic [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/13847) in GitLab 13.1. -> - Iteration field [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/233517) in GitLab 15.6. - -When you're creating a new issue, you can complete the following fields: - -- Title -- Type: either issue (default) or incident -- [Description template](../description_templates.md): overwrites anything in the Description text box -- Description: you can use [Markdown](../../markdown.md) and [quick actions](../quick_actions.md) -- Checkbox to make the issue [confidential](confidential_issues.md) -- [Assignees](#assignee) -- [Weight](issue_weight.md) -- [Epic](../../group/epics/index.md) -- [Due date](due_dates.md) -- [Milestone](../milestones/index.md) -- [Labels](../labels.md) -- [Iteration](../../group/iterations/index.md) +After you create an issue, you can start working with it. ## Edit an issue @@ -239,27 +22,7 @@ To edit an issue: 1. Edit the available fields. 1. Select **Save changes**. -### Reorder list items in the issue description - -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/15260) in GitLab 15.0. - -When you view an issue that has a list in the description, you can also reorder the list items. - -Prerequisites: - -- You must have at least the Reporter role for the project, be the author of the issue, or be - assigned to the issue. -- The issue's description must have an [ordered, unordered](../../markdown.md#lists), or - [task](../../markdown.md#task-lists) list. - -To reorder list items, when viewing an issue: - -1. Hover over the list item row to make the drag icon (**{drag-vertical}**) visible. -1. Select and hold the drag icon. -1. Drag the row to the new position in the list. -1. Release the drag icon. - -### Bulk edit issues from a project +## Bulk edit issues from a project > - Assigning epic [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/210470) in GitLab 13.2. > - Editing health status [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/218395) in GitLab 13.2. @@ -283,7 +46,7 @@ To edit multiple issues at the same time: When bulk editing issues in a project, you can edit the following attributes: - Status (open or closed) -- [Assignees](#assignee) +- [Assignees](managing_issues.md#assignee) - [Epic](../../group/epics/index.md) - [Milestone](../milestones/index.md) - [Labels](../labels.md) @@ -396,6 +159,26 @@ To do it: 1. To exit the Rails console, enter `quit`. +## Reorder list items in the issue description + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/15260) in GitLab 15.0. + +When you view an issue that has a list in the description, you can also reorder the list items. + +Prerequisites: + +- You must have at least the Reporter role for the project, be the author of the issue, or be + assigned to the issue. +- The issue's description must have an [ordered, unordered](../../markdown.md#lists), or + [task](../../markdown.md#task-lists) list. + +To reorder list items, when viewing an issue: + +1. Hover over the list item row to make the drag icon (**{drag-vertical}**) visible. +1. Select and hold the drag icon. +1. Drag the row to the new position in the list. +1. Release the drag icon. + ## Close an issue When you decide that an issue is resolved or no longer needed, you can close it. @@ -514,8 +297,7 @@ Prerequisites: - You must have [administrator access](../../../administration/index.md) to your GitLab instance. -To change the default issue closing pattern, edit the -[`gitlab.rb` or `gitlab.yml` file](../../../administration/issue_closing_pattern.md) +Learn how to change the default [issue closing pattern](../../../administration/issue_closing_pattern.md). of your installation. ## Change the issue type @@ -622,7 +404,7 @@ GitLab displays the results on-screen, but you can also ### Filter with the OR operator -> OR filtering for assignees was [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/23532) in GitLab 15.6 [with a flag](../../../administration/feature_flags.md) named `or_issuable_queries`. Disabled by default. +> OR filtering for author and assignee was [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/23532) in GitLab 15.6 [with a flag](../../../administration/feature_flags.md) named `or_issuable_queries`. Disabled by default. FLAG: On self-managed GitLab, by default this feature is not available. @@ -633,7 +415,7 @@ When this feature is enabled, you can use the OR operator (**is one of: `||`**) when you [filter the list of issues](#filter-the-list-of-issues). `is one of` represents an inclusive OR. For example, if you filter by `Assignee is one of Sidney Jones` and -`Assignee is one of Zhang Wei`, GitLab shows issues where either Sidney, Zhang, or both of them are assignees. +`Assignee is one of Zhang Wei`, GitLab shows issues where either `Sidney`, `Zhang`, or both of them are assignees. ### Filter issues by ID @@ -674,22 +456,6 @@ To copy the issue's email address: 1. Go to the issue. 1. On the right sidebar, next to **Issue email**, select **Copy Reference** (**{copy-to-clipboard}**). -## Real-time sidebar - -> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/17589) in GitLab 13.3. Disabled by default. -> - [Enabled on GitLab.com](https://gitlab.com/gitlab-com/gl-infra/production/-/issues/3413) in GitLab 13.9. -> - [Enabled on self-managed](https://gitlab.com/gitlab-org/gitlab/-/issues/17589) in GitLab 14.5. -> - [Generally available](https://gitlab.com/gitlab-org/gitlab/-/issues/17589) in GitLab 14.9. Feature flags `real_time_issue_sidebar` and `broadcast_issue_updates` removed. - -Some sections of the right sidebar are updated in real time. -When you're viewing an issue and somebody changes one of the values, -you can see the change without having to refresh the page. - -The following sections are updated in real time: - -- [Assignee](#assignee) -- [Labels](../labels.md#assign-and-unassign-labels) - ## Assignee An issue can be assigned to one or [more users](multiple_assignees_for_issues.md). @@ -708,6 +474,8 @@ To change the assignee on an issue: 1. From the dropdown list, select the user to add as an assignee. 1. Select any area outside the dropdown list. +The assignee is changed without having to refresh the page. + ## Similar issues To prevent duplication of issues on the same topic, GitLab searches for similar issues diff --git a/doc/user/project/issues/sorting_issue_lists.md b/doc/user/project/issues/sorting_issue_lists.md index 6a1a791645e..a2f90d5c444 100644 --- a/doc/user/project/issues/sorting_issue_lists.md +++ b/doc/user/project/issues/sorting_issue_lists.md @@ -101,6 +101,19 @@ title in this order: - Numbers - Letters: first Latin, then accented (for example, `ö`) +## Sorting by health status **(ULTIMATE)** + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/377841) in GitLab 15.7. + +When you sort by **Health**, the issue list changes to sort by the +[health status](managing_issues.md#health-status) of the issues +When in descending order, the issues are shown in the following order: + +1. **At risk** issues +1. **Needs attention** issues +1. **On track** issues +1. All other issues + ## Sorting by weight When you sort by **Weight**, the issue list changes to sort ascending by the diff --git a/doc/user/project/members/index.md b/doc/user/project/members/index.md index e8ec954df8f..a7627f12657 100644 --- a/doc/user/project/members/index.md +++ b/doc/user/project/members/index.md @@ -61,7 +61,7 @@ To add a user to a project: 1. Select **Invite members**. 1. Enter an email address and select a [role](../../permissions.md). 1. Optional. Select an **Access expiration date**. - From that date onwards, the user can no longer access the project. + From that date onward, the user can no longer access the project. 1. Select **Invite**. If the user has a GitLab account, they are added to the members list. @@ -110,7 +110,7 @@ To add a group to a project: 1. Select a group. 1. Select the highest [role](../../permissions.md) for users in the group. 1. Optional. Select an **Access expiration date**. - From that date onwards, the group can no longer access the project. + From that date onward, the group can no longer access the project. 1. Select **Invite**. The members of the group are not displayed on the **Members** tab. diff --git a/doc/user/project/members/share_project_with_groups.md b/doc/user/project/members/share_project_with_groups.md index 52cc9fc4f9b..b9887212be0 100644 --- a/doc/user/project/members/share_project_with_groups.md +++ b/doc/user/project/members/share_project_with_groups.md @@ -9,83 +9,77 @@ info: To determine the technical writer assigned to the Stage/Group associated w You can share projects with other [groups](../../group/index.md). This makes it possible to add a group of users to a project with a single action. -## Groups as collections of users +For example, if `Project A` belongs to `Group 1`, the members of `Group 1` have access to the project. +If `Project A` already belongs to another `Group 2`, the owner of `Group 2` can share `Project A` +with `Group 1`, so that both members of `Group 1` and `Group 2` have access to the project. -Groups are used primarily to [create collections of projects](../../group/index.md), but you can also -take advantage of the fact that groups define collections of _users_, namely the group -members. +When a project is shared with a group: -## Share a project with a group of users - -> - [Changed](https://gitlab.com/gitlab-org/gitlab/-/issues/247208) in GitLab 13.11 from a form to a modal - window [with a flag](../../feature_flags.md). Disabled by default. -> - Modal window [enabled on GitLab.com and self-managed](https://gitlab.com/gitlab-org/gitlab/-/issues/247208) - in GitLab 14.8. -> - [Generally available](https://gitlab.com/gitlab-org/gitlab/-/issues/352526) in GitLab 14.9. - [Feature flag `invite_members_group_modal`](https://gitlab.com/gitlab-org/gitlab/-/issues/352526) removed. - -You can share a project only with: - -- Groups for which you have an explicitly defined [membership](index.md). -- Groups that contain a nested subgroup or project for which you have an explicitly defined role. +- All group members, including members of subgroups or projects that belong to the group, +are assigned the same role in the project. +Each member's role is displayed in **Project information > Members > Max role**. +- The group is listed in the **Groups** tab. +- The project is listed on the group dashboard. -Administrators can share projects with any group in the instance. +Be aware of the restrictions that apply when you share projects with: -The primary mechanism to give a group of users, say 'Engineering', access to a project, -say 'Project Acme', in GitLab is to make the 'Engineering' group the owner of 'Project -Acme'. But what if 'Project Acme' already belongs to another group, say 'Open Source'? -This is where the group sharing feature can be of use. +- [Groups with a more restrictive visibility level](#share-projects-with-groups-with-a-more-restrictive-visibility-level). +- [Restricted sharing](#prevent-project-sharing). -To share 'Project Acme' with the 'Engineering' group: +## Share projects with groups with a more restrictive visibility level -1. For 'Project Acme' use the left navigation menu to go to **Project information > Members**. -1. Select **Invite a group**. -1. Add the 'Engineering' group with the maximum access level of your choice. -1. Optional. Select an **Access expiration date**. -1. Select **Invite**. +You can share projects only down the group's organization structure. +This means you can share a project with a group that has a more restrictive +[visibility level](../../public_access.md#project-and-group-visibility) than the project, +but not with a group that has a less restrictive visibility level. -After sharing 'Project Acme' with 'Engineering': +For example, you can share: -- The group is listed in the **Groups** tab. -- The project is listed on the group dashboard. -- All members, including members from the ancestors of the 'Engineering' group, gain access to 'Project Acme' with an access level based on the outcome of [maximum access level](#maximum-access-level). +- A public project with a private group. +- A public project with an internal group. +- An internal project with a private group. -When you share a project, be aware of the following restrictions and outcomes: +This restriction applies to subgroups as well. For example, `group/subgroup01/project`: -- [Maximum access level](#maximum-access-level) -- [Sharing projects with groups of a higher restrictive visibility level](#sharing-projects-with-groups-of-a-higher-restrictive-visibility-level) -- [Sharing project with group lock](#share-project-with-group-lock) +- Can't be shared with `group`. +- Can be shared with `group/subgroup02` or `group/subgroup01/subgroup03`. -## Maximum access level +When you share a project with a group that has a more restrictive visibility level than the project: -In the example above, the maximum access level of 'Developer' for members from 'Engineering' means that users with higher access levels in 'Engineering' ('Maintainer' or 'Owner') only have 'Developer' access to 'Project Acme'. - -### Share a project with a subgroup - -You can't share a project with a group that's an ancestor of a [subgroup](../../group/subgroups/index.md) the project is -in. That means you can only share down the hierarchy. For example, `group/subgroup01/project`: +- The group name is visible to all users that can view the project members page. +- Owners of the project have access to members of the group when they mention them in issues or merge requests. +- Project members who are direct or indirect members of the group can see +group members listed in addition to members of the project. -- Can not be shared with `group`. -- Can be shared with `group/subgroup02` or `group/subgroup01/subgroup03`. +## Share a project with a group -## Sharing projects with groups of a higher restrictive visibility level +> - [Changed](https://gitlab.com/gitlab-org/gitlab/-/issues/247208) in GitLab 13.11 from a form to a modal + window [with a flag](../../feature_flags.md). Disabled by default. +> - Modal window [enabled on GitLab.com and self-managed](https://gitlab.com/gitlab-org/gitlab/-/issues/247208) + in GitLab 14.8. +> - [Generally available](https://gitlab.com/gitlab-org/gitlab/-/issues/352526) in GitLab 14.9. + [Feature flag `invite_members_group_modal`](https://gitlab.com/gitlab-org/gitlab/-/issues/352526) removed. -There are several outcomes you must be aware of when you share a project with a group that has a more restrictive [visibility level](../../public_access.md#project-and-group-visibility) than the project. For example, when you: +You can share a project only with groups: -- Share a public project with a private group. -- Share a public project with an internal group. -- Share an internal project with a private group. +- Where you have an explicitly defined [membership](index.md). +- That contain a nested subgroup or project you have an explicitly defined role for. +- You are an administrator of. -The following outcomes occur: +To share a project with a group: -- The group name is visible to all users that can view the project members page. -- Owners of the project have access to members of the group when they mention them in issues or merge requests. -- Project members who are direct or indirect members of the group can see group members listed in addition to members of the project. +1. On the top bar, select **Main menu > Projects** and find your project. +1. In the left navigation menu, select **Project information > Members**. +1. Select **Invite a group**. +1. **Select a group** you want to add to the project. +1. **Select a role** you want to assign to the group. +1. Optional. Select an **Access expiration date**. +1. Select **Invite**. -## Share project with group lock +## Prevent project sharing -It is possible to prevent projects in a group from +You can prevent members of a group from [sharing a project with another group](../members/share_project_with_groups.md). -This allows for tighter control over project access. +This restriction allows for tighter control over project access. -Learn more about [Share with group lock](../../group/access_and_permissions.md#prevent-a-project-from-being-shared-with-groups). +For more information, see [Prevent a project from being shared with groups](../../group/access_and_permissions.md#prevent-a-project-from-being-shared-with-groups). diff --git a/doc/user/project/merge_requests/approvals/index.md b/doc/user/project/merge_requests/approvals/index.md index eb460225858..92ff78082e3 100644 --- a/doc/user/project/merge_requests/approvals/index.md +++ b/doc/user/project/merge_requests/approvals/index.md @@ -22,8 +22,10 @@ flexibility: - Specify a list of users who act as [code owners](../../code_owners.md) for specific files, and require their approval before work can merge. -You can configure merge request approvals on a per-project basis, and -[on the group level](../../../group/manage.md#group-merge-request-approval-settings). Administrators of +You can configure merge request approvals on a per-project basis, and some approvals can be configured +[on the group level](../../../group/manage.md#group-merge-request-approval-settings). Support for +group-level settings for merge request approval rules is tracked in this +[epic](https://gitlab.com/groups/gitlab-org/-/epics/4367). Administrators of [GitLab Premium](https://about.gitlab.com/pricing/) and [GitLab Ultimate](https://about.gitlab.com/pricing/) self-managed GitLab instances can also configure approvals diff --git a/doc/user/project/merge_requests/approvals/rules.md b/doc/user/project/merge_requests/approvals/rules.md index e09a1318981..5f81db10cf4 100644 --- a/doc/user/project/merge_requests/approvals/rules.md +++ b/doc/user/project/merge_requests/approvals/rules.md @@ -182,7 +182,7 @@ granting them push access: 1. [Create a new group](../../../group/manage.md#create-a-group). 1. [Add the user to the group](../../../group/manage.md#add-users-to-a-group), and select the Reporter role for the user. -1. [Share the project with your group](../../members/share_project_with_groups.md#share-a-project-with-a-group-of-users), +1. [Share the project with your group](../../members/share_project_with_groups.md#share-a-project-with-a-group), based on the Reporter role. 1. Go to your project and select **Settings > Merge requests**. 1. In the **Merge request approvals** section, scroll to **Approval rules**, and either: diff --git a/doc/user/project/merge_requests/approvals/settings.md b/doc/user/project/merge_requests/approvals/settings.md index a2a12b22c3b..a8acab3898b 100644 --- a/doc/user/project/merge_requests/approvals/settings.md +++ b/doc/user/project/merge_requests/approvals/settings.md @@ -21,22 +21,24 @@ To view or edit merge request approval settings: ### Approval settings -These settings limit who can approve merge requests. - -| Setting | Description | -| ------ | ------ | -| [Prevent approval by author](#prevent-approval-by-author) | When enabled, the author of a merge request cannot approve it. | -| [Prevent approvals by users who add commits](#prevent-approvals-by-users-who-add-commits) | When enabled, users who have committed to a merge request cannot approve it. | -| [Prevent editing approval rules in merge requests](#prevent-editing-approval-rules-in-merge-requests) | When enabled, users can't override the project's approval rules on merge requests. | -| [Require user password to approve](#require-user-password-to-approve) | Force potential approvers to first authenticate with a password. | - -You can further define what happens to existing approvals when commits are added to the merge request. - -| Setting | Description | -| ------ | ------ | -| Keep approvals | Do not remove approvals. | -| [Remove all approvals](#remove-all-approvals-when-commits-are-added-to-the-source-branch) | Remove all existing approvals. | -| [Remove approvals by Code Owners if their files changed](#remove-approvals-by-code-owners-if-their-files-changed) | If a Code Owner has approved the merge request, and the commit changes files they are the Code Owner for, their approval is removed. | +These settings limit who can approve merge requests: + +- [**Prevent approval by author**](#prevent-approval-by-author): + Prevents the author of a merge request from approving it. +- [**Prevent approvals by users who add commits**](#prevent-approvals-by-users-who-add-commits): + Prevents users who add commits to a merge request from also approving it. +- [**Prevent editing approval rules in merge requests**](#prevent-editing-approval-rules-in-merge-requests): + Prevents users from overriding project level approval rules on merge requests. +- [**Require user password to approve**](#require-user-password-to-approve): + Force potential approvers to first authenticate with a password. +- Code Owner approval removals: Define what happens to existing approvals when + commits are added to the merge request. + - **Keep approvals**: Do not remove any approvals. + - [**Remove all approvals**](#remove-all-approvals-when-commits-are-added-to-the-source-branch): + Remove all existing approvals. + - [**Remove approvals by Code Owners if their files changed**](#remove-approvals-by-code-owners-if-their-files-changed): + If a Code Owner approves a merge request, and a later commit changes files + they are a Code Owner for, their approval is removed. ## Prevent approval by author diff --git a/doc/user/project/merge_requests/changes.md b/doc/user/project/merge_requests/changes.md index 6e8b0cb1a75..f6e02dc0dfe 100644 --- a/doc/user/project/merge_requests/changes.md +++ b/doc/user/project/merge_requests/changes.md @@ -136,15 +136,10 @@ Files marked as viewed are not shown to you again unless either: - New changes are made to its content. - You clear the **Viewed** checkbox. -## Show merge request conflicts in diff **(FREE SELF)** +## Show merge request conflicts in diff -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/232484) in GitLab 13.5 [with a flag](../../../administration/feature_flags.md) named `display_merge_conflicts_in_diff`. Disabled by default. - -FLAG: -On self-managed GitLab, by default this feature is not available. To make it available, -ask an administrator to [enable the feature flag](../../../administration/feature_flags.md) -named `display_merge_conflicts_in_diff`. On GitLab.com, this feature is not available. -The feature is not ready for production use. +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/232484) in GitLab 13.5 [with a flag](../../../administration/feature_flags.md) named `display_merge_conflicts_in_diff`. Disabled by default. +> - [Enabled on GitLab.com and self-managed](https://gitlab.com/gitlab-org/gitlab/-/issues/276918) in GitLab 15.7. To avoid displaying the changes that are already on target branch in the diff, we compare the merge request's source branch with HEAD of the target branch. diff --git a/doc/user/project/merge_requests/commit_templates.md b/doc/user/project/merge_requests/commit_templates.md index 75c2bdffae8..a14d8bddd24 100644 --- a/doc/user/project/merge_requests/commit_templates.md +++ b/doc/user/project/merge_requests/commit_templates.md @@ -70,6 +70,7 @@ GitLab creates a squash commit message with this template: > - [Added](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/75639) `url`, `approved_by`, and `merged_by` variables in GitLab 14.7. > - [Added](https://gitlab.com/gitlab-org/gitlab/-/issues/20421) `co_authored_by` variable in GitLab 14.7. > - [Added](https://gitlab.com/gitlab-org/gitlab/-/issues/26303) `all_commits` variable in GitLab 14.9. +> - [Added](https://gitlab.com/gitlab-org/gitlab/-/issues/378352) `reviewed_by` variable in GitLab 15.7. Commit message templates support these variables: @@ -84,7 +85,8 @@ Commit message templates support these variables: | `%{first_commit}` | Full message of the first commit in merge request diff. | `Update README.md` | | `%{first_multiline_commit}` | Full message of the first commit that's not a merge commit and has more than one line in message body. Merge request title if all commits aren't multiline. | `Update README.md`<br><br>`Improved project description in readme file.` | | `%{url}` | Full URL to the merge request. | `https://gitlab.com/gitlab-org/gitlab/-/merge_requests/1` | -| `%{approved_by}` | Line-separated list of the merge request approvers. | `Approved-by: Sidney Jones <sjones@example.com>` <br> `Approved-by: Zhang Wei <zwei@example.com>` | +| `%{reviewed_by}` | Line-separated list of the merge request reviewers, based on users who submit a review via batch comments, in a `Reviewed-by` Git commit trailer format. | `Reviewed-by: Sidney Jones <sjones@example.com>` <br> `Reviewed-by: Zhang Wei <zwei@example.com>` | +| `%{approved_by}` | Line-separated list of the merge request approvers in a `Approved-by` Git commit trailer format. | `Approved-by: Sidney Jones <sjones@example.com>` <br> `Approved-by: Zhang Wei <zwei@example.com>` | | `%{merged_by}` | User who merged the merge request. | `Alex Garcia <agarcia@example.com>` | | `%{co_authored_by}` | Names and emails of commit authors in a `Co-authored-by` Git commit trailer format. Limited to authors of 100 most recent commits in merge request. | `Co-authored-by: Zane Doe <zdoe@example.com>` <br> `Co-authored-by: Blake Smith <bsmith@example.com>` | | `%{all_commits}` | Messages from all commits in the merge request. Limited to 100 most recent commits. Skips commit bodies exceeding 100KiB and merge commit messages. | `* Feature introduced` <br><br> `This commit implements feature` <br> `Changelog:added` <br><br> `* Bug fixed` <br><br> `* Documentation improved` <br><br>`This commit introduced better docs.`| diff --git a/doc/user/project/merge_requests/commits.md b/doc/user/project/merge_requests/commits.md index a6ae3ac80a5..a9f67c39ae8 100644 --- a/doc/user/project/merge_requests/commits.md +++ b/doc/user/project/merge_requests/commits.md @@ -1,56 +1,11 @@ --- -stage: Create -group: Code Review -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments -type: index, reference +redirect_to: '../merge_requests/index.md' +remove_date: '2023-03-12' --- -# Commits tab in merge requests **(FREE)** +This document was removed. -The **Commits** tab in a merge request displays a sequential list of commits -to the Git branch your merge request is based on. From this page, you can review -full commit messages and copy a commit's SHA when you need to -[cherry-pick changes](cherry_pick_changes.md). - -## Merge requests commit navigation - -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/18140) in GitLab 13.0. - -To seamlessly navigate among commits in a merge request: - -1. Select the **Commits** tab. -1. Select a commit to open it in the single-commit view. -1. Navigate through the commits by either: - - - Selecting **Prev** and **Next** buttons below the tab buttons. - - Using the <kbd>X</kbd> and <kbd>C</kbd> keyboard shortcuts. - - - -## View merge request commits in context - -> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/29274) in GitLab 13.12 [with a flag](../../../administration/feature_flags.md) named `context_commits`. Enabled by default. -> - [Enabled on GitLab.com](https://gitlab.com/gitlab-org/gitlab/-/issues/320757) in GitLab 14.8. -> - [Generally available](https://gitlab.com/gitlab-org/gitlab/-/issues/320757) in GitLab 14.9. [Feature flag `context_commits`](https://gitlab.com/gitlab-org/gitlab/-/issues/320757) removed. - -When reviewing a merge request, it helps to have more context about the changes -made. That includes unchanged lines in unchanged files, and previous commits -that have already merged that the change is built on. - -To add previously merged commits to a merge request for more context: - -1. Go to your merge request. -1. Select the **Commits** tab. -1. Scroll to the end of the list of commits, and select **Add previously merged commits**: - -  - -1. Select the commits that you want to add. -1. Select **Save changes**. - -To view the changes done on those previously merged commits: - -1. On your merge request, select the **Changes** tab. -1. Scroll to **(file-tree)** **Compare** and select **previously merged commits**: - -  +<!-- This redirect file can be deleted after <2023-03-12>. --> +<!-- Redirects that point to other docs in the same project expire in three months. --> +<!-- Redirects that point to docs in a different project or site (for example, link is not relative and starts with `https:`) expire in one year. --> +<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/redirects.html --> diff --git a/doc/user/project/merge_requests/conflicts.md b/doc/user/project/merge_requests/conflicts.md index 24f22924a08..6063d64a721 100644 --- a/doc/user/project/merge_requests/conflicts.md +++ b/doc/user/project/merge_requests/conflicts.md @@ -75,7 +75,7 @@ To resolve less-complex conflicts from the GitLab user interface: Resolving conflicts merges the target branch of the merge request into the source branch, using the version of the text you chose. If the source branch is `feature` and the target branch is `main`, these actions are similar to running -`git checkout feature; git merge main` locally. +`git switch feature; git merge main` locally. ## Resolve conflicts in the inline editor @@ -101,7 +101,7 @@ most control over each change: 1. Open the terminal and check out your feature branch. For example, `my-feature-branch`: ```shell - git checkout my-feature-branch + git switch my-feature-branch ``` 1. [Rebase your branch](../../../topics/git/git_rebase.md#regular-rebase) against the diff --git a/doc/user/project/merge_requests/creating_merge_requests.md b/doc/user/project/merge_requests/creating_merge_requests.md index df11d5a1d8d..eae4db2d4f7 100644 --- a/doc/user/project/merge_requests/creating_merge_requests.md +++ b/doc/user/project/merge_requests/creating_merge_requests.md @@ -102,7 +102,7 @@ You can create a merge request from your fork to contribute back to the main pro change the default target branch (which can be useful if you are working in a forked project). 1. Select **Compare branches and continue**. -1. Select **Submit merge request**. +1. Select **Create merge request**. After your work is merged, if you don't intend to make any other contributions to the upstream project, you can unlink your diff --git a/doc/user/project/merge_requests/img/add_previously_merged_commits_button_v14_1.png b/doc/user/project/merge_requests/img/add_previously_merged_commits_button_v14_1.png Binary files differdeleted file mode 100644 index e60e869f854..00000000000 --- a/doc/user/project/merge_requests/img/add_previously_merged_commits_button_v14_1.png +++ /dev/null diff --git a/doc/user/project/merge_requests/img/commit_nav_v13_11.png b/doc/user/project/merge_requests/img/commit_nav_v13_11.png Binary files differdeleted file mode 100644 index a9bc8fa6bee..00000000000 --- a/doc/user/project/merge_requests/img/commit_nav_v13_11.png +++ /dev/null diff --git a/doc/user/project/merge_requests/img/previously_merged_commits_v14_1.png b/doc/user/project/merge_requests/img/previously_merged_commits_v14_1.png Binary files differdeleted file mode 100644 index 4f49fad10ad..00000000000 --- a/doc/user/project/merge_requests/img/previously_merged_commits_v14_1.png +++ /dev/null diff --git a/doc/user/project/merge_requests/merge_request_dependencies.md b/doc/user/project/merge_requests/merge_request_dependencies.md deleted file mode 100644 index 6242a77e931..00000000000 --- a/doc/user/project/merge_requests/merge_request_dependencies.md +++ /dev/null @@ -1,11 +0,0 @@ ---- -redirect_to: 'dependencies.md' -remove_date: '2022-11-22' ---- - -This document was moved to [another location](dependencies.md). - -<!-- This redirect file can be deleted after <2022-11-22>. --> -<!-- Redirects that point to other docs in the same project expire in three months. --> -<!-- Redirects that point to docs in a different project or site (for example, link is not relative and starts with `https:`) expire in one year. --> -<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/redirects.html --> diff --git a/doc/user/project/merge_requests/methods/index.md b/doc/user/project/merge_requests/methods/index.md index e72c927198e..249a98f1779 100644 --- a/doc/user/project/merge_requests/methods/index.md +++ b/doc/user/project/merge_requests/methods/index.md @@ -10,47 +10,107 @@ type: reference, concepts The merge method you select for your project determines how the changes in your merge requests are merged into an existing branch. +The examples on this page assume a `main` branch with commits A, C, and E, and a +`feature` branch with commits B and D: + +```mermaid +gitGraph + commit id: "A" + branch feature + commit id: "B" + commit id: "D" + checkout main + commit id: "C" + commit id: "E" +``` + ## Configure a project's merge method 1. On the top bar, select **Main menu > Projects** and find your project. 1. On the left sidebar, select **Settings > Merge requests**. -1. In the **Merge method** section, select your desired merge method. +1. Select your desired **Merge method** from these options: + - Merge commit + - Merge commit with semi-linear history + - Fast-forward merge +1. In **Squash commits when merging**, select the default behavior for handling commits: + - **Do not allow**: Squashing is never performed, and the user cannot change the behavior. + - **Allow**: Squashing is off by default, but the user can change the behavior. + - **Encourage**: Squashing is on by default, but the user can change the behavior. + - **Require**: Squashing is always performed, and the user cannot change the behavior. 1. Select **Save changes**. ## Merge commit -This setting is the default. It always creates a separate merge commit, -even when using [squash](../squash_and_merge.md). An example commit graph generated using this merge method: +By default, GitLab creates a merge commit when a branch is merged into `main`. +A separate merge commit is always created, regardless of whether or not commits +are [squashed when merging](../squash_and_merge.md). This strategy can result +in both a squash commit and a merge commit being added to your `main` branch. + +These diagrams show how the `feature` branch merges into `main` if you use the +**Merge commit** strategy. They are equivalent to the command `git merge --no-ff <feature>`, +and selecting `Merge commit` as the **Merge method** in the GitLab UI: + +The merge strategy: ```mermaid +%%{init: { 'gitGraph': {'logLevel': 'debug', 'showBranches': true, 'showCommitLabel':true,'mainBranchName': 'main'}} }%% gitGraph - commit id: "Init" - branch mr-branch-1 - commit - checkout main - commit - branch mr-branch-2 - commit - checkout mr-branch-1 - commit - checkout main - branch squash-mr - commit id: "Squashed commits" - checkout main - merge squash-mr - merge mr-branch-1 - commit - merge mr-branch-2 + commit id: "A" + branch feature + commit id: "B" + commit id: "D" + checkout main + commit id: "C" + commit id: "E" + merge feature +``` + +After a feature branch is merged with the **Merge commit** method, your `main` branch +looks like this: + +```mermaid +%%{init: { 'gitGraph': {'logLevel': 'debug', 'showBranches': true, 'showCommitLabel':true,'mainBranchName': 'main'}} }%% +gitGraph + commit id: "A" + commit id: "C" + commit id: "E" + commit id: "squash commit" + commit id: "merge commit" ``` -- For regular merges, it is equivalent to the command `git merge --no-ff <source-branch>`. -- For squash merges, it squashes all commits in the source branch before merging it normally. It performs actions similar to: +In comparison, a **squash merge** constructs a squash commit, a virtual copy of all commits +from the `feature` branch. The original commits (B and D) remain unchanged +on the `feature` branch, and the squash commit is placed on the `main` branch: + +```mermaid +%%{init: { 'gitGraph': {'showBranches': true, 'showCommitLabel':true,'mainBranchName': 'main'}} }%% +gitGraph + commit id:"A" + branch feature + checkout main + commit id:"C" + checkout feature + commit id:"B" + commit id:"D" + checkout main + commit id:"E" + commit id:"squash commit" type: HIGHLIGHT +``` + +The squash merge graph is equivalent to these settings in the GitLab UI: + +- **Merge method**: Merge commit. +- **Squash commits when merging** should be set to either: + - Require. + - Either Allow or Encourage, and squashing must be selected on the merge request. + +The squash merge graph is also equivalent to these commands: ```shell - git checkout `git merge-base <source-branch> <target-branch>` - git merge --squash <source-branch> + git checkout `git merge-base feature main` + git merge --squash <feature> SOURCE_SHA=`git rev-parse HEAD` - git checkout <target-branch> + git checkout <main> git merge --no-ff $SOURCE_SHA ``` @@ -58,7 +118,8 @@ gitGraph A merge commit is created for every merge, but the branch is only merged if a fast-forward merge is possible. This ensures that if the merge request build -succeeded, the target branch build also succeeds after the merge. An example commit graph generated using this merge method: +succeeded, the target branch build also succeeds after the merge. An example +commit graph generated using this merge method: ```mermaid gitGraph @@ -113,8 +174,8 @@ This method is equivalent to `git merge --ff <source-branch>` for regular merges When the fast-forward merge ([`--ff-only`](https://git-scm.com/docs/git-merge#git-merge---ff-only)) setting -is enabled, no merge commits are created and all merges are fast-forwarded, -which means that merging is only allowed if the branch can be fast-forwarded. +is enabled, no merge commits are created and all merges are fast-forwarded. +Merging is only allowed if the branch can be fast-forwarded. When a fast-forward merge is not possible, the user is given the option to rebase, see [Rebasing in (semi-)linear merge methods](#rebasing-in-semi-linear-merge-methods). @@ -136,11 +197,16 @@ In these merge methods, you can merge only when your source branch is up-to-date - Fast-forward merge. If a fast-forward merge is not possible but a conflict-free rebase is possible, -GitLab offers you the [`/rebase` quick action](../../../../topics/git/git_rebase.md#rebase-from-the-gitlab-ui), -and the ability to select **Rebase** from the user interface. +GitLab provides: + +- The [`/rebase` quick action](../../../../topics/git/git_rebase.md#rebase-from-the-gitlab-ui). +- The option to select **Rebase** in the user interface. + +You must rebase the source branch locally before a fast-forward merge if both +conditions are true: -If the target branch is ahead of the source branch and a conflict-free rebase is -not possible, you must rebase the source branch locally before you can do a fast-forward merge. +- The target branch is ahead of the source branch. +- A conflict-free rebase is not possible.  diff --git a/doc/user/project/merge_requests/reviews/data_usage.md b/doc/user/project/merge_requests/reviews/data_usage.md index 3a3af7a24bc..f17015aef4e 100644 --- a/doc/user/project/merge_requests/reviews/data_usage.md +++ b/doc/user/project/merge_requests/reviews/data_usage.md @@ -27,7 +27,7 @@ This feature is designed as a progressive enhancement to the existing GitLab Rev ## Model Accuracy -Organizations use many different processes for code review. Some focus on senior engineers reviewing junior engineer's code, others have hierarchical organizational structure based reviews. Suggested Reviewers is focused on contextual reviewers based on historical merge request activity by users. While we will continue evolving the underlying ML model to better serve various code review use cases and processes Suggested Reviewers does not replace the usage of other code review features like Code Owners and [Approval Rules](../approvals/rules.md). Reviewer selection is highly subjective therefore, we do not expect Suggested Reviewers to provide perfect suggestions everytime. +Organizations use many different processes for code review. Some focus on senior engineers reviewing junior engineer's code, others have hierarchical organizational structure based reviews. Suggested Reviewers is focused on contextual reviewers based on historical merge request activity by users. While we will continue evolving the underlying ML model to better serve various code review use cases and processes Suggested Reviewers does not replace the usage of other code review features like Code Owners and [Approval Rules](../approvals/rules.md). Reviewer selection is highly subjective therefore, we do not expect Suggested Reviewers to provide perfect suggestions every time. Through analysis of beta customer usage, we find that the Suggested Reviewers ML model provides suggestions that are adopted in 60% of cases. We will be introducing a feedback mechanism into the Suggested Reviewers feature in the future to allow users to flag bad reviewer suggestions to help improve the model. Additionally we will be offering an opt-in feature in the future which will allow the model to use your project's data for training the underlying model. @@ -39,6 +39,6 @@ Suggested Reviewers is off by default and requires a Project Owner or Admin to e Suggested Reviewers operates completely within the GitLab.com infrastructure providing the same level of [privacy](https://about.gitlab.com/privacy/) and [security](https://about.gitlab.com/security/) of any other feature of GitLab.com. -No new additional data is collected to enable this feature. GitLab is inferencing your merge request against a trained machine learning model. The content of your source code is not used as training data. Your data also never leaves GitLab.com, all training and inference is done within GitLab.com infrastructure. +No new additional data is collected to enable this feature. GitLab infers your merge request against a trained machine learning model. The content of your source code is not used as training data. Your data also never leaves GitLab.com, all training and inference is done within GitLab.com infrastructure. [Read more about the security of GitLab.com](https://about.gitlab.com/security/faq/) diff --git a/doc/user/project/merge_requests/reviews/img/suggestion_code_block_editor_v12_8.png b/doc/user/project/merge_requests/reviews/img/suggestion_code_block_editor_v12_8.png Binary files differdeleted file mode 100644 index 927b4f812a5..00000000000 --- a/doc/user/project/merge_requests/reviews/img/suggestion_code_block_editor_v12_8.png +++ /dev/null diff --git a/doc/user/project/merge_requests/reviews/index.md b/doc/user/project/merge_requests/reviews/index.md index 4c503211513..9a75c038dbc 100644 --- a/doc/user/project/merge_requests/reviews/index.md +++ b/doc/user/project/merge_requests/reviews/index.md @@ -71,6 +71,52 @@ if you [approve a merge request](../approvals/index.md#approve-a-merge-request) are shown in the reviewer list, a green check mark **{check-circle-filled}** displays next to your name. +### Download merge request changes as a diff + +To download the changes included in a merge request as a diff: + +1. On the top bar, select **Main menu > Projects** and find your project. +1. On the left sidebar, select **Merge requests**. +1. Select your merge request. +1. On the top right, select **Code > Plain diff**. + +If you know the URL of the merge request, you can also download the diff from +the command line by appending `.diff` to the URL. This example downloads the diff +for merge request `000000`: + +```plaintext +https://gitlab.com/gitlab-org/gitlab/-/merge_requests/000000.diff +``` + +To download and apply the diff in a one-line CLI command: + +```shell +curl "https://gitlab.com/gitlab-org/gitlab/-/merge_requests/000000.diff" | git apply +``` + +### Download merge request changes as a patch file + +To download the changes included in a merge request as a patch file: + +1. On the top bar, select **Main menu > Projects** and find your project. +1. On the left sidebar, select **Merge requests**. +1. Select your merge request. +1. On the top right, select **Code > Email patches**. + +If you know the URL of the merge request, you can also download the patch from +the command line by appending `.patch` to the URL. This example downloads the patch +file for merge request `000000`: + +```plaintext +https://gitlab.com/gitlab-org/gitlab/-/merge_requests/000000.patch +``` + +To download and apply the patch in a one-line CLI command using [`git am`](https://git-scm.com/docs/git-am): + +```shell +curl "https://gitlab.com/gitlab-org/gitlab/-/merge_requests/000000.patch" | git am +``` + ### Submit a review You can submit your completed review in multiple ways: diff --git a/doc/user/project/merge_requests/reviews/suggestions.md b/doc/user/project/merge_requests/reviews/suggestions.md index 832f78d18a1..668dece9fda 100644 --- a/doc/user/project/merge_requests/reviews/suggestions.md +++ b/doc/user/project/merge_requests/reviews/suggestions.md @@ -74,7 +74,13 @@ To add a suggestion that includes a [fenced code block](../../../markdown.md#code-spans-and-blocks), wrap your suggestion in four backticks instead of three: - +~~~markdown +````suggestion:-0+2 +```shell +git config --global receive.advertisepushoptions true +``` +```` +~~~  diff --git a/doc/user/project/merge_requests/status_checks.md b/doc/user/project/merge_requests/status_checks.md index d330ccdefb6..74c3b3e24b6 100644 --- a/doc/user/project/merge_requests/status_checks.md +++ b/doc/user/project/merge_requests/status_checks.md @@ -6,7 +6,7 @@ type: reference, concepts disqus_identifier: 'https://docs.gitlab.com/ee/user/project/merge_requests/status_checks.html' --- -# External Status Checks **(ULTIMATE)** +# External status checks **(ULTIMATE)** > - [Introduced](https://gitlab.com/groups/gitlab-org/-/epics/3869) in GitLab 14.0, disabled behind the `:ff_external_status_checks` feature flag. > - [Feature flag removed](https://gitlab.com/gitlab-org/gitlab/-/issues/320783) in GitLab 14.1. diff --git a/doc/user/project/ml/experiment_tracking/img/candidate_v15_7.png b/doc/user/project/ml/experiment_tracking/img/candidate_v15_7.png Binary files differnew file mode 100644 index 00000000000..fb2e2e706d6 --- /dev/null +++ b/doc/user/project/ml/experiment_tracking/img/candidate_v15_7.png diff --git a/doc/user/project/ml/experiment_tracking/img/candidates.png b/doc/user/project/ml/experiment_tracking/img/candidates.png Binary files differdeleted file mode 100644 index df70a01a2bd..00000000000 --- a/doc/user/project/ml/experiment_tracking/img/candidates.png +++ /dev/null diff --git a/doc/user/project/ml/experiment_tracking/img/candidates_v15_7.png b/doc/user/project/ml/experiment_tracking/img/candidates_v15_7.png Binary files differnew file mode 100644 index 00000000000..58dfe94a108 --- /dev/null +++ b/doc/user/project/ml/experiment_tracking/img/candidates_v15_7.png diff --git a/doc/user/project/ml/experiment_tracking/img/experiments.png b/doc/user/project/ml/experiment_tracking/img/experiments.png Binary files differdeleted file mode 100644 index a6472406b90..00000000000 --- a/doc/user/project/ml/experiment_tracking/img/experiments.png +++ /dev/null diff --git a/doc/user/project/ml/experiment_tracking/img/experiments_v15_7.png b/doc/user/project/ml/experiment_tracking/img/experiments_v15_7.png Binary files differnew file mode 100644 index 00000000000..a7d4a3e559f --- /dev/null +++ b/doc/user/project/ml/experiment_tracking/img/experiments_v15_7.png diff --git a/doc/user/project/ml/experiment_tracking/index.md b/doc/user/project/ml/experiment_tracking/index.md index e274bd7f38e..a7096d633a0 100644 --- a/doc/user/project/ml/experiment_tracking/index.md +++ b/doc/user/project/ml/experiment_tracking/index.md @@ -16,9 +16,11 @@ engineering, and so on, to improve the performance of the model. Keeping track o artifacts so that the data scientist can later replicate the experiment is not trivial. Machine learning experiment tracking enables them to log parameters, metrics, and artifacts directly into GitLab, giving easy access later on. - + - + + + ## What is an experiment? @@ -53,18 +55,19 @@ integration. More information on how to use GitLab as a backend for MLFlow Clien ### Exploring model candidates To list the current active experiments, navigate to `https/-/ml/experiments`. To display all trials -that have been logged, along with their metrics and parameters, selecting an experiment. +that have been logged, along with their metrics and parameters, select an experiment. To display details for a candidate, +select **Details**. ### Logging artifacts Trial artifacts are saved as [generic packages](../../../packages/generic_packages/index.md), and follow all their conventions. After an artifact is logged for a candidate, all artifacts logged for the candidate are listed in the -package registry. The package name for a candidate is `ml_candidate_<candidate_id>`, with version `-`. +package registry. The package name for a candidate is `ml_candidate_<candidate_id>`, with version `-`. The link to the +artifacts can also be accessed from the **Experiment Candidates** list or **Candidate detail**. ### Limitations and future - Searching experiments, searching trials, visual comparison of trials, and creating, deleting and updating experiments and trials through GitLab UI is under development. -- No support for experiment and trial metadata that do not classify as parameters or metrics. ## Disabling or enabling the Feature @@ -73,4 +76,6 @@ On GitLab.com, this feature is currently on private testing. ## Feedback, roadmap and reports -For updates on the development, feedback and bug reports, refer to the [development epic](https://gitlab.com/groups/gitlab-org/-/epics/8560). +For updates on the development, refer to the [development epic](https://gitlab.com/groups/gitlab-org/-/epics/8560). + +For feedback, bug reports and feature requests, refer to the [feedback issue](https://gitlab.com/gitlab-org/gitlab/-/issues/381660). diff --git a/doc/user/project/pages/custom_domains_ssl_tls_certification/dns_concepts.md b/doc/user/project/pages/custom_domains_ssl_tls_certification/dns_concepts.md index 1d32091b294..197524f2fc5 100644 --- a/doc/user/project/pages/custom_domains_ssl_tls_certification/dns_concepts.md +++ b/doc/user/project/pages/custom_domains_ssl_tls_certification/dns_concepts.md @@ -35,7 +35,7 @@ for the most popular hosting services: - [123-reg](https://www.123-reg.co.uk/support/domains/domain-name-server-dns-management-guide/) - [Amazon](https://docs.aws.amazon.com/AmazonS3/latest/dev/website-hosting-custom-domain-walkthrough.html) - [Bluehost](https://www.bluehost.com/help/article/dns-management-add-edit-or-delete-dns-entries) -- [Cloudflare](https://support.cloudflare.com/hc/en-us/articles/201720164-Creating-a-Cloudflare-account-and-adding-a-website) +- [Cloudflare](https://developers.cloudflare.com/fundamentals/get-started/setup/) - [cPanel](https://documentation.cpanel.net/display/84Docs/Edit+DNS+Zone) - [DigitalOcean](https://docs.digitalocean.com/products/networking/dns/how-to/manage-records/) - [DreamHost](https://help.dreamhost.com/hc/en-us/articles/360035516812) diff --git a/doc/user/project/pages/custom_domains_ssl_tls_certification/img/add_certificate_to_pages.png b/doc/user/project/pages/custom_domains_ssl_tls_certification/img/add_certificate_to_pages.png Binary files differdeleted file mode 100644 index d92a981dc60..00000000000 --- a/doc/user/project/pages/custom_domains_ssl_tls_certification/img/add_certificate_to_pages.png +++ /dev/null diff --git a/doc/user/project/pages/custom_domains_ssl_tls_certification/index.md b/doc/user/project/pages/custom_domains_ssl_tls_certification/index.md index 6378d962ffe..dc23540bd1b 100644 --- a/doc/user/project/pages/custom_domains_ssl_tls_certification/index.md +++ b/doc/user/project/pages/custom_domains_ssl_tls_certification/index.md @@ -52,15 +52,14 @@ this document for an [overview on DNS records](dns_concepts.md). #### 1. Add a custom domain -Navigate to your project's **Setting > Pages** and select **+ New domain** -to add your custom domain to GitLab Pages. You can choose whether to: +To add your custom domain to GitLab Pages: -- Add an [SSL/TLS certificate](#adding-an-ssltls-certificate-to-pages). -- Leave it blank (it can be added later). - -Select **Create New Domain**. - - +1. On the top bar, select **Main menu > Projects** and find your project. +1. On the left sidebar, select **Settings > Pages**. +1. In the top right, select **New Domain**. +1. In **Domain**, enter your domain. +1. Optional. In **Certificate**, turn off the **Automatic certificate management using Let's Encrypt** toggle to add an [SSL/TLS certificate](#adding-an-ssltls-certificate-to-pages). You can also add the certificate and key later. +1. Select **Create New Domain**. #### 2. Get the verification code @@ -292,8 +291,6 @@ meet these requirements. - To add the certificate to a domain previously added, go to your project's **Settings > Pages**, locate your domain name, select **Details** and **Edit** to add the certificate. - - 1. Add the PEM certificate to its corresponding field. 1. If your certificate is missing its intermediate, copy and paste the root certificate (usually available from your CA website) diff --git a/doc/user/project/pages/getting_started/pages_ci_cd_template.md b/doc/user/project/pages/getting_started/pages_ci_cd_template.md index caf98e8a8a4..339ab239588 100644 --- a/doc/user/project/pages/getting_started/pages_ci_cd_template.md +++ b/doc/user/project/pages/getting_started/pages_ci_cd_template.md @@ -26,13 +26,12 @@ these steps, you may have to do additional configuration for the Pages site to g If everything is configured correctly, the site can take approximately 30 minutes to deploy. -You can watch the pipeline run by navigating to **CI/CD > Pipelines**. +To view the pipeline, go to **CI/CD > Pipelines**. When the pipeline is finished, go to **Settings > Pages** to find the link to your Pages website. -To view the HTML and other assets that were created for the site, -go to the **Pipelines** tab, view the job, and on the right side, -select **Download artifacts**. - For every change pushed to your repository, GitLab CI/CD runs a new pipeline that immediately publishes your changes to the Pages site. + +To view the HTML and other assets that were created for the site, +[download the job artifacts](../../../../ci/pipelines/job_artifacts.md#download-job-artifacts). diff --git a/doc/user/project/pages/getting_started/pages_forked_sample_project.md b/doc/user/project/pages/getting_started/pages_forked_sample_project.md index 69c60cab4b3..9841e52a089 100644 --- a/doc/user/project/pages/getting_started/pages_forked_sample_project.md +++ b/doc/user/project/pages/getting_started/pages_forked_sample_project.md @@ -29,32 +29,37 @@ When the pipeline is finished, go to **Settings > Pages** to find the link to yo For every change pushed to your repository, GitLab CI/CD runs a new pipeline that immediately publishes your changes to the Pages site. -To view the HTMl and other assets that were created for the site, -go to the **Pipelines** tab, view the job, and on the right side, -select **Download artifacts**. +## Remove the fork relationship -You can take some **optional** further steps: +If you want to contribute to the project you forked from, +you can keep the forked relationship. Otherwise: -- Remove the fork relationship. If you want to contribute to the project you forked from, - you can keep this relationship. Otherwise, go to your project's **Settings > General**, - expand **Advanced settings**, and scroll down to **Remove fork relationship**: +1. On the left sidebar, select **Settings > General**. +1. Expand **Advanced settings**. +1. Select **Remove fork relationship**. -  +## Change the URL -- Change the URL to match your namespace. If your Pages site is hosted on GitLab.com, - you can rename it to `<namespace>.gitlab.io`, where `<namespace>` is your GitLab namespace - (the one you chose when you forked the project). +You can change the URL to match your namespace. +If your Pages site is hosted on GitLab.com, +you can rename it to `<namespace>.gitlab.io`, where `<namespace>` is your GitLab namespace +(the one you chose when you forked the project). - - Go to your project's **Settings > General** and expand **Advanced**. Scroll down to - **Change path** and change the path to `<namespace>.gitlab.io`. +1. On the left sidebar, select **Settings > General**. +1. Expand **Advanced**. +1. In **Change path**, update the path to `<namespace>.gitlab.io`. - For example, if your project's URL is `gitlab.com/gitlab-tests/jekyll`, your namespace is - `gitlab-tests`. + For example, if your project's URL is `gitlab.com/gitlab-tests/jekyll`, your namespace is + `gitlab-tests`. - If you set the repository path to `gitlab-tests.gitlab.io`, - the resulting URL for your Pages website is `https://gitlab-tests.gitlab.io`. + If you set the repository path to `gitlab-tests.gitlab.io`, + the resulting URL for your Pages website is `https://gitlab-tests.gitlab.io`. -  +  - - Now go to your SSG's configuration file and change the [base URL](../getting_started_part_one.md#urls-and-base-urls) - from `"project-name"` to `""`. The project name setting varies by SSG and may not be in the configuration file. +1. Open your SSG configuration file and change the [base URL](../getting_started_part_one.md#urls-and-base-urls) + from `"project-name"` to `""`. The project name setting varies by SSG and may not be in the configuration file. + +## Related topics + +- [Download the job artifacts](../../../../ci/pipelines/job_artifacts.md#download-job-artifacts) diff --git a/doc/user/project/pages/getting_started/pages_from_scratch.md b/doc/user/project/pages/getting_started/pages_from_scratch.md index c0a1e8f16e0..ebadf39a984 100644 --- a/doc/user/project/pages/getting_started/pages_from_scratch.md +++ b/doc/user/project/pages/getting_started/pages_from_scratch.md @@ -420,9 +420,8 @@ Now GitLab CI/CD not only builds the website, but also: - **Caches** dependencies installed with Bundler. - **Continuously deploys** every push to the `main` branch. -To view the HTMl and other assets that were created for the site, -go to the **Pipelines** tab, view the job, and on the right side, -select **Download artifacts**. +To view the HTML and other assets that were created for the site, +[download the job artifacts](../../../../ci/pipelines/job_artifacts.md#download-job-artifacts). ## Related topics diff --git a/doc/user/project/pages/getting_started/pages_new_project_template.md b/doc/user/project/pages/getting_started/pages_new_project_template.md index e4890954d13..a0f9753a40c 100644 --- a/doc/user/project/pages/getting_started/pages_new_project_template.md +++ b/doc/user/project/pages/getting_started/pages_new_project_template.md @@ -29,6 +29,5 @@ your Pages website. For every change pushed to your repository, GitLab CI/CD runs a new pipeline that immediately publishes your changes to the Pages site. -To view the HTMl and other assets that were created for the site, -go to the **Pipelines** tab, view the job, and on the right side, -select **Download artifacts**. +To view the HTML and other assets that were created for the site, +[download the job artifacts](../../../../ci/pipelines/job_artifacts.md#download-job-artifacts). diff --git a/doc/user/project/pages/getting_started/pages_ui.md b/doc/user/project/pages/getting_started/pages_ui.md index ba97fcb8749..a6069b473e6 100644 --- a/doc/user/project/pages/getting_started/pages_ui.md +++ b/doc/user/project/pages/getting_started/pages_ui.md @@ -4,59 +4,69 @@ group: Incubation info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- -# Tutorial: Use the GitLab UI to deploy your static site **(FREE)** +# Create a Pages deployment for your static site **(FREE)** -This tutorial assumes you have a project that either: +If you already have a GitLab project that contains your static site or framework, +you can generate a GitLab Pages website from it. -- Generates static sites or a client-rendered single-page application (SPA), - such as [Eleventy](https://www.11ty.dev), [Astro](https://astro.build), or [Jekyll](https://jekyllrb.com). -- Contains a framework configured for static output, such as [Next.js](https://nextjs.org), - [Nuxt.js](https://nuxtjs.org), or [SvelteKit](https://kit.svelte.dev). +When you provide basic information in the UI, a `.gitlab-ci.yml` file is created +and a merge request opened. When you commit the merge request, +a pipeline deploys your Pages website. -## Update your app to output files to the `public` folder +## Prerequisites -GitLab Pages requires all files intended to be part of the published website to -be in a root-level folder called `public`. If you create this folder during the build -pipeline, committing it to Git is not required. +- Your app must [output files to the `public` folder](../public_folder.md). If you create + this folder during the build pipeline, you do not need to commit it to Git. -For detailed instructions, read [Configure the public files folder](../public_folder.md). + WARNING: + This step is important. Ensure your files are in a root-level `public` folder. -## Set up the `.gitlab-ci.yml` file +- You must have a project that either: + - Generates static sites or a client-rendered single-page application (SPA), + like [Eleventy](https://www.11ty.dev), [Astro](https://astro.build), or [Jekyll](https://jekyllrb.com). + - Contains a framework configured for static output, such as [Next.js](https://nextjs.org), + [Nuxt.js](https://nuxtjs.org), or [SvelteKit](https://kit.svelte.dev). +- GitLab Pages must be enabled for the project. (To enable, go to **Settings > General**, + expand **Visibility, project features, permissions**, and turn on the **Pages** toggle.) -GitLab helps you write the `.gitlab-ci.yml` needed to create your first GitLab Pages -deployment pipeline. Rather than building the file from scratch, it asks you to -provide the build commands, and creates the necessary boilerplate for you. +## Create the Pages deployment -To build your YAML file from the GitLab UI: +To complete the setup and generate a GitLab Pages deployment: 1. On the top bar, select **Main menu > Projects** and find your project. -1. On the left sidebar, select **Settings > Pages** to display the friendly - interface **Get Started With Pages**. -1. If your framework's build process does not need one of the provided build - commands, you can either: +1. On the left sidebar, select **Settings > Pages**. A **Get Started with Pages** form appears. + If this form is not available, see [Troubleshooting](#if-the-get-started-with-pages-form-is-not-available). +1. For **Step 1**, enter an image name and verify that your files are in a `public` folder. +1. Select **Next**. +1. For **Step 2**, enter your installation steps. If your framework's build process does not + need one of the provided build commands, you can either: - Skip the step by selecting **Next**. - Enter `:` (the bash "do nothing" command) if you still want to incorporate that step's boilerplate into your `.gitlab-ci.yml` file. -1. Optional. Edit and adjust the generated `.gitlab-ci.yml` file as needed. -1. Commit your `.gitlab-ci.yml` to your repository. This commit triggers your first +1. Select **Next**. +1. For **Step 3**, enter scripts that indicate how to build your application. +1. Select **Next**. +1. Optional. Edit the generated `.gitlab-ci.yml` file as needed. +1. For **Step 4**, add a commit message and select **Commit**. This commit triggers your first GitLab Pages deployment. -To view the HTMl and other assets that were created for the site, -go to **CI/CD > Pipelines**, view the job, and on the right side, -select **Download artifacts**. +To view the running pipeline, go to **CI/CD > Pipelines**. + +To view the artifacts that were created during the deployment, view the job, +and on the right side, select **Download artifacts**. ## Troubleshooting -### If you can't see the "Get Started with Pages" interface +### If the `Get Started with Pages` form is not available -GitLab doesn't show this interface if you have either: +When you go to **Settings > Pages**, the form is not available if you: - Deployed a GitLab Pages site before. -- Committed a `.gitlab-ci.yml` through this interface at least once. +- Committed a `.gitlab-ci.yml` through the forms at least one time. -To fix this problem: +To fix this issue: -- If you see the message **Waiting for the Pages Pipeline to complete**, select - **Start over** to start the wizard again. +- If the message **Waiting for the Pages Pipeline to complete** appears, select + **Start over** to start the form again. - If your project has previously deployed GitLab Pages successfully, - [manually update](pages_from_scratch.md) your `.gitlab-ci.yml`. + [manually update](pages_from_scratch.md) your `.gitlab-ci.yml` file. diff --git a/doc/user/project/pages/getting_started_part_one.md b/doc/user/project/pages/getting_started_part_one.md index 588d94729e2..a0c8073d6eb 100644 --- a/doc/user/project/pages/getting_started_part_one.md +++ b/doc/user/project/pages/getting_started_part_one.md @@ -87,7 +87,7 @@ Every Static Site Generator (SSG) default configuration expects to find your website under a (sub)domain (`example.com`), not in a subdirectory of that domain (`example.com/subdir`). Therefore, whenever you publish a project website (`namespace.gitlab.io/project-name`), -you must look for this configuration (base URL) on your SSG's +you must look for this configuration (base URL) on your static site generator's documentation and set it up to reflect this pattern. For example, for a Jekyll site, the `baseurl` is defined in the Jekyll diff --git a/doc/user/project/pages/img/remove_fork_relationship_v13_1.png b/doc/user/project/pages/img/remove_fork_relationship_v13_1.png Binary files differdeleted file mode 100644 index 84aa2e571c7..00000000000 --- a/doc/user/project/pages/img/remove_fork_relationship_v13_1.png +++ /dev/null diff --git a/doc/user/project/pages/public_folder.md b/doc/user/project/pages/public_folder.md index a19e296b954..8c9f1cbec86 100644 --- a/doc/user/project/pages/public_folder.md +++ b/doc/user/project/pages/public_folder.md @@ -2,40 +2,39 @@ description: 'Learn how to configure the build output folder for the most common static site generators' stage: Create -group: Incubation +group: Editor info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- # Configure the public files folder **(FREE)** -GitLab Pages requires all files you intend to be available in the published website to -be in a root-level folder called `public`. This page describe how -to set this up for some common static site generators. +All the files that should be accessible by the browser must be in a root-level folder called `public`. -## Guide by framework +Follow these instructions to configure the `public` folder +for the following frameworks. -### Eleventy +## Eleventy -For Eleventy, you should either: +For Eleventy, you should do one of the following: -1. Add the `--output=public` flag in Eleventy's build commands, for example: +- Add the `--output=public` flag in Eleventy's build commands, for example: - `npx @11ty/eleventy --input=path/to/sourcefiles --output=public` + `npx @11ty/eleventy --input=path/to/sourcefiles --output=public` -1. Add the following to your `.eleventy.js` file: +- Add the following to your `.eleventy.js` file: - ```javascript - // .eleventy.js - module.exports = function(eleventyConfig) { - return { - dir: { - output: "public" - } - } - }; - ``` + ```javascript + // .eleventy.js + module.exports = function(eleventyConfig) { + return { + dir: { + output: "public" + } + } + }; + ``` -### Astro +## Astro By default, Astro uses the `public` folder to store static assets. For GitLab Pages, rename that folder to a collision-free alternative first: @@ -65,11 +64,11 @@ rename that folder to a collision-free alternative first: }); ``` -### SvelteKit +## SvelteKit NOTE: GitLab Pages supports only static sites. For SvelteKit, -we recommend using [`adapter-static`](https://kit.svelte.dev/docs/adapters#supported-environments-static-sites). +you can use [`adapter-static`](https://kit.svelte.dev/docs/adapters#supported-environments-static-sites). When using `adapter-static`, add the following to your `svelte.config.js`: @@ -86,11 +85,11 @@ export default { }; ``` -### Next.js +## Next.js NOTE: -GitLab Pages supports only static sites. For Next.js, we -recommend using Next's [Static HTML export functionality](https://nextjs.org/docs/advanced-features/static-html-export) +GitLab Pages supports only static sites. For Next.js, you can use +Next's [Static HTML export functionality](https://nextjs.org/docs/advanced-features/static-html-export). Use the `-o public` flag after `next export` as the build command, for example: @@ -118,7 +117,7 @@ GitLab Pages supports only static sites. 1. Configure your Nuxt.js application for [Static Site Generation](https://nuxtjs.org/docs/features/deployment-targets/#static-hosting). -### Vite +## Vite Update your `vite.config.js` to include the following: @@ -131,7 +130,7 @@ export default { } ``` -### Webpack +## Webpack Update your `webpack.config.js` to include the following: @@ -147,9 +146,9 @@ module.exports = { ## Should you commit the `public` folder? Not necessarily. However, when the GitLab Pages deploy pipeline runs, it looks -for an [artifact](../../../ci/pipelines/job_artifacts.md) of that name. So +for an [artifact](../../../ci/pipelines/job_artifacts.md) of that name. If you set up a job that creates the `public` folder before deploy, such as by running `npm run build`, committing the folder isn't required. If you prefer to build your site locally, you can commit the `public` folder and -omit the build step during the job, instead. +omit the build step during the job instead. diff --git a/doc/user/project/pages/redirects.md b/doc/user/project/pages/redirects.md index 96de457c7f7..cf0c0dbff82 100644 --- a/doc/user/project/pages/redirects.md +++ b/doc/user/project/pages/redirects.md @@ -108,9 +108,8 @@ and an [HTTP status code](#http-status-codes): ## Rewrites -> - [Introduced](https://gitlab.com/gitlab-org/gitlab-pages/-/merge_requests/458) in GitLab 14.3. -> - Enabled on GitLab.com. -> - Disabled by default in self-managed GitLab behind the [`FF_ENABLE_PLACEHOLDERS` feature flag](#feature-flag-for-rewrites). +> - [Introduced](https://gitlab.com/gitlab-org/gitlab-pages/-/merge_requests/458) in GitLab 14.3 [with a flag](../../../administration/feature_flags.md) named `FF_ENABLE_PLACEHOLDERS`. Disabled by default. +> - [Enabled on GitLab.com and self-managed](https://gitlab.com/gitlab-org/gitlab-pages/-/issues/619) in GitLab 15.2. Provide a status code of `200` to serve the content of the `to` path when the request matches the `from`: @@ -267,28 +266,3 @@ However, there are some minor differences: - Netlify redirects to `/new/:placeholder` (with a literal `:placeholder`). - GitLab redirects to `/new/`. - -## Feature flag for rewrites - -FLAG: -Rewrites in GitLab Pages is under development, and is deployed behind a feature flag -that is **disabled by default**. - -To enable rewrites, for [Omnibus installations](../../../administration/pages/index.md), define the -`FF_ENABLE_PLACEHOLDERS` environment variable in the -[global settings](../../../administration/pages/index.md#global-settings). -Add the following line to `/etc/gitlab/gitlab.rb` and -[reconfigure the instance](../../../administration/restart_gitlab.md#omnibus-gitlab-reconfigure): - -```ruby -gitlab_pages['env']['FF_ENABLE_PLACEHOLDERS'] = 'true' -``` - -For [source installations](../../../administration/pages/source.md), define the -`FF_ENABLE_PLACEHOLDERS` environment variable, then -[restart GitLab](../../../administration/restart_gitlab.md#installations-from-source): - -```shell -export FF_ENABLE_PLACEHOLDERS="true" -/path/to/pages/bin/gitlab-pages -config gitlab-pages.conf -``` diff --git a/doc/user/project/push_options.md b/doc/user/project/push_options.md index b31ef858d59..9e5413b020e 100644 --- a/doc/user/project/push_options.md +++ b/doc/user/project/push_options.md @@ -61,10 +61,10 @@ time as pushing changes: | `merge_request.target=<branch_name>` | Set the target of the merge request to a particular branch or upstream project, such as: `git push -o merge_request.target=project_path/branch` | [11.10](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/26752) | | `merge_request.merge_when_pipeline_succeeds` | Set the merge request to [merge when its pipeline succeeds](merge_requests/merge_when_pipeline_succeeds.md). | [11.10](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/26752) | | `merge_request.remove_source_branch` | Set the merge request to remove the source branch when it's merged. | [12.2](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/64320) | -| `merge_request.title="<title>"` | Set the title of the merge request. Ex: `git push -o merge_request.title="The title I want"`. | [12.2](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/64320) | -| `merge_request.description="<description>"` | Set the description of the merge request. Ex: `git push -o merge_request.description="The description I want"`. | [12.2](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/64320) | -| `merge_request.draft` | Mark the merge request as a draft. Ex: `git push -o merge_request.draft`. | [15.0](https://gitlab.com/gitlab-org/gitlab/-/issues/296673) | -| `merge_request.milestone="<milestone>"` | Set the milestone of the merge request. Ex: `git push -o merge_request.milestone="3.0"`. | [14.1](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/63960) | +| `merge_request.title="<title>"` | Set the title of the merge request. For example: `git push -o merge_request.title="The title I want"`. | [12.2](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/64320) | +| `merge_request.description="<description>"` | Set the description of the merge request. For example: `git push -o merge_request.description="The description I want"`. | [12.2](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/64320) | +| `merge_request.draft` | Mark the merge request as a draft. For example: `git push -o merge_request.draft`. | [15.0](https://gitlab.com/gitlab-org/gitlab/-/issues/296673) | +| `merge_request.milestone="<milestone>"` | Set the milestone of the merge request. For example: `git push -o merge_request.milestone="3.0"`. | [14.1](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/63960) | | `merge_request.label="<label>"` | Add labels to the merge request. If the label does not exist, it is created. For example, for two labels: `git push -o merge_request.label="label1" -o merge_request.label="label2"`. | [12.3](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/31831) | | `merge_request.unlabel="<label>"` | Remove labels from the merge request. For example, for two labels: `git push -o merge_request.unlabel="label1" -o merge_request.unlabel="label2"`. | [12.3](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/31831) | | `merge_request.assign="<user>"` | Assign users to the merge request. Accepts username or user ID. For example, for two users: `git push -o merge_request.assign="user1" -o merge_request.assign="user2"`. | [13.10](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/25904), support for usernames added in [15.5](https://gitlab.com/gitlab-org/gitlab/-/issues/344276) | diff --git a/doc/user/project/quick_actions.md b/doc/user/project/quick_actions.md index 7b7619cfeb5..a73a5329688 100644 --- a/doc/user/project/quick_actions.md +++ b/doc/user/project/quick_actions.md @@ -72,13 +72,13 @@ threads. Some quick actions might not be available to all subscription tiers. | `/done` | **{check-circle}** Yes | **{check-circle}** Yes | **{check-circle}** Yes | Mark to do as done. | | `/draft` | **{dotted-circle}** No | **{check-circle}** Yes | **{dotted-circle}** No | Set the [draft status](merge_requests/drafts.md). Use for toggling the draft status ([deprecated](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/92654) in GitLab 15.4.) | | `/due <date>` | **{check-circle}** Yes | **{dotted-circle}** No | **{dotted-circle}** No | Set due date. Examples of valid `<date>` include `in 2 days`, `this Friday` and `December 31st`. | -| `/duplicate <#issue>` | **{check-circle}** Yes | **{dotted-circle}** No | **{dotted-circle}** No | Close this issue and mark as a duplicate of another issue. Also, mark both as related. | +| `/duplicate <#issue>` | **{check-circle}** Yes | **{dotted-circle}** No | **{dotted-circle}** No | Close this issue. Mark as a duplicate of, and related to, issue `<#issue>`. | | `/epic <epic>` | **{check-circle}** Yes | **{dotted-circle}** No | **{dotted-circle}** No | Add to epic `<epic>`. The `<epic>` value should be in the format of `&epic`, `group&epic`, or a URL to an epic. | | `/estimate <time>` or `/estimate_time <time>` | **{check-circle}** Yes | **{check-circle}** Yes | **{dotted-circle}** No | Set time estimate. For example, `/estimate 1mo 2w 3d 4h 5m`. Learn more about [time tracking](time_tracking.md). Alias `/estimate_time` [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/16501) in GitLab 15.6. | | `/health_status <value>` | **{check-circle}** Yes | **{dotted-circle}** No | **{dotted-circle}** No | Set [health status](issues/managing_issues.md#health-status). Valid options for `<value>` are `on_track`, `needs_attention`, and `at_risk` ([introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/213814) in GitLab 14.7). | | `/invite_email email1 email2` | **{check-circle}** Yes | **{dotted-circle}** No | **{dotted-circle}** No | Add up to six email participants. This action is behind feature flag `issue_email_participants` and is not yet supported in issue templates. | | `/iteration *iteration:"iteration name"` | **{check-circle}** Yes | **{dotted-circle}** No | **{dotted-circle}** No | Set iteration. For example, to set the `Late in July` iteration: `/iteration *iteration:"Late in July"` ([introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/196795) in GitLab 13.1). | -| `/label ~label1 ~label2` | **{check-circle}** Yes | **{check-circle}** Yes | **{check-circle}** Yes | Add one or more labels. Label names can also start without a tilde (`~`), but mixed syntax is not supported. | +| `/label ~label1 ~label2` or `/labels ~label1 ~label2` | **{check-circle}** Yes | **{check-circle}** Yes | **{check-circle}** Yes | Add one or more labels. Label names can also start without a tilde (`~`), but mixed syntax is not supported. | | `/lock` | **{check-circle}** Yes | **{check-circle}** Yes | **{dotted-circle}** No | Lock the discussions. | | `/link` | **{check-circle}** Yes | **{dotted-circle}** No | **{dotted-circle}** No | Add a link and description to [linked resources](../../operations/incident_management/linked_resources.md) in an incident ([introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/374964) in GitLab 15.5). | | `/merge` | **{dotted-circle}** No | **{check-circle}** Yes | **{dotted-circle}** No | Merge changes. Depending on the project setting, this may be [when the pipeline succeeds](merge_requests/merge_when_pipeline_succeeds.md), or adding to a [Merge Train](../../ci/pipelines/merge_trains.md). | diff --git a/doc/user/project/releases/index.md b/doc/user/project/releases/index.md index 75a25678125..6d5378bddb7 100644 --- a/doc/user/project/releases/index.md +++ b/doc/user/project/releases/index.md @@ -184,20 +184,20 @@ is not available. ## Edit a release -Only users with at least the Developer role can edit releases. -Read more about [Release permissions](#release-permissions). +To edit the details of a release after it's created, you can use the +[Update a release API](../../../api/releases/index.md#update-a-release) or the UI. -To edit the details of a release: +Prerequisites: + +- You must have at least the Developer role. + +In the UI: 1. On the left sidebar, select **Deployments > Releases**. 1. In the top-right corner of the release you want to modify, select **Edit this release** (the pencil icon). 1. On the **Edit Release** page, change the release's details. 1. Select **Save changes**. -You can edit the release title, notes, associated milestones, and asset links. -To change the release date use the -[Releases API](../../../api/releases/index.md#update-a-release). - ## Delete a release > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/213862) in GitLab 15.2 @@ -209,11 +209,15 @@ Prerequisites: - You must have at least the Developer role. Read more about [Release permissions](#release-permissions). -To delete a release in the UI: +To delete a release, use either the +[Delete a release API](../../../api/releases/index.md#delete-a-release) or the UI. + +In the UI: 1. On the top bar, select **Main menu > Projects** and find your project. 1. On the left sidebar, select **Deployments > Releases**. -1. In the top-right corner of the release you want to delete, select **Edit this release** (**{pencil}**). +1. In the top-right corner of the release you want to delete, select **Edit this release** + (**{pencil}**). 1. On the **Edit Release** page, select **Delete**. 1. Select **Delete release**. @@ -449,11 +453,11 @@ In the API: ## Release permissions -> [The permission model for create, update and delete actions was fixed](https://gitlab.com/gitlab-org/gitlab/-/issues/327505) in GitLab 14.1. +> Fixes to the permission model for create, update and delete actions [were introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/327505) in GitLab 14.1. ### View a release and download assets -> [Changes were made to the Guest role access](https://gitlab.com/gitlab-org/gitlab/-/issues/335209) in GitLab 14.5. +> Changes to the Guest role [were introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/335209) in GitLab 14.5. - Users with at least the Reporter role have read and download access to the project releases. diff --git a/doc/user/project/repository/branches/default.md b/doc/user/project/repository/branches/default.md index 6801899160d..87caeee73e3 100644 --- a/doc/user/project/repository/branches/default.md +++ b/doc/user/project/repository/branches/default.md @@ -36,11 +36,15 @@ the [Git commands you need](#update-the-default-branch-name-in-your-repository) ## Change the default branch name for a project -To update the default branch name for an individual [project](../../index.md): +Prerequisites: -1. Sign in to GitLab with at least the Maintainer role. +- You have the Owner or Maintainer role in the project. + +To update the default branch for an individual [project](../../index.md): + +1. On the top bar, select **Main menu > Projects** and find your project. 1. In the left navigation menu, go to **Settings > Repository**. -1. Expand **Default branch**, and select a new default branch. +1. Expand **Default branch**. For **Initial default branch name**, select a new default branch. 1. Optional. Select the **Auto-close referenced issues on default branch** checkbox to close issues when a merge request [uses a closing pattern](../../issues/managing_issues.md#closing-issues-automatically). @@ -66,8 +70,8 @@ groups and subgroups can override this instance-wide setting for their projects. 1. On the top bar, select **Main menu > Admin**. 1. On the left sidebar, select **Settings > Repository**. -1. Expand **Default initial branch name**. -1. Change the default initial branch to a custom name of your choice. +1. Expand **Default branch**. +1. For **Initial default branch name**, select a new default branch. 1. Select **Save changes**. Projects created on this instance after you change the setting use the @@ -78,11 +82,12 @@ overrides it. > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/221014) in GitLab 13.6. -Users with at least the Owner role of groups and subgroups can configure the default branch name for a group: +Users with the Owner role of groups and subgroups can configure the default branch name for a group: -1. Go to the group **Settings > Repository**. +1. On the top bar, select **Main menu > Group** and find your group. +1. On the left sidebar, select **Settings > Repository**. 1. Expand **Default branch**. -1. Change the default initial branch to a custom name of your choice. +1. For **Initial default branch name**, select a new default branch. 1. Select **Save changes**. Projects created in this group after you change the setting use the custom branch name, diff --git a/doc/user/project/repository/branches/index.md b/doc/user/project/repository/branches/index.md index 6cc7394e7b3..a86e32b4721 100644 --- a/doc/user/project/repository/branches/index.md +++ b/doc/user/project/repository/branches/index.md @@ -101,10 +101,13 @@ This feature allows you to search and select branches quickly. Search results ap - Branches with names that matched search terms exactly. - Other branches with names that include search terms, sorted alphabetically. -Sometimes when you have hundreds of branches you may want a more flexible matching pattern. In such cases you can use the following: +Sometimes when you have hundreds of branches you may want a more flexible matching pattern. In such cases you can use the following operators: -- `^feature` matches only branch names that begin with 'feature'. -- `feature$` matches only branch names that end with 'feature'. +- `^` matches beginning of branch name, for example `^feat` would match `feat/user-authentication` +- `$` matches end of branch name, for example `widget$` would match `feat/search-box-widget` +- `*` wildcard matcher, for example `branch*cache*` would match `fix/branch-search-cache-expiration` + +These operators can be mixed, for example `^chore/*migration$` would match `chore/user-data-migration` ## Swap revisions @@ -116,14 +119,26 @@ The Swap revisions feature allows you to swap the Source and Target revisions. W  -<!-- ## Troubleshooting +## Troubleshooting + +### Error: ambiguous `HEAD` branch exists + +In versions of Git earlier than 2.16.0, you could create a branch named `HEAD`. +This branch named `HEAD` collides with the internal reference (also named `HEAD`) +Git uses to describe the active (checked out) branch. This naming collision can +prevent you from updating the default branch of your repository: + +```plaintext +Error: Could not set the default branch. Do you have a branch named 'HEAD' in your repository? +``` + +To fix this problem: -Include any troubleshooting steps that you can foresee. If you know beforehand what issues -one might have when setting this up, or when something is changed, or on upgrading, it's -important to describe those, too. Think of things that may go wrong and include them here. -This is important to minimize requests for support, and to avoid doc comments with -questions that you know someone might ask. +1. On the top bar, select **Main menu > Projects** and find your project. +1. On the left sidebar, select **Repository > Branches**. +1. Search for a branch named `HEAD`. +1. Make sure the branch has no uncommitted changes. +1. Select **Delete branch**, then **Yes, delete branch**. -Each scenario can be a third-level heading, for example `### Getting error message X`. -If you have none to add when creating a doc, leave this section in place -but commented out to help encourage others to add to it in the future. --> +Git versions [2.16.0 and later](https://github.com/git/git/commit/a625b092cc59940521789fe8a3ff69c8d6b14eb2), +prevent you from creating a branch with this name. diff --git a/doc/user/project/repository/gpg_signed_commits/index.md b/doc/user/project/repository/gpg_signed_commits/index.md index a1f57f51f26..6b67ffd0e59 100644 --- a/doc/user/project/repository/gpg_signed_commits/index.md +++ b/doc/user/project/repository/gpg_signed_commits/index.md @@ -4,7 +4,7 @@ group: Source Code info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- -# Signing commits with GPG **(FREE)** +# Sign commits with GPG **(FREE)** You can sign the commits you make in a GitLab repository with a GPG ([GNU Privacy Guard](https://gnupg.org/)) key. When you add a cryptographic @@ -160,14 +160,6 @@ to use this key: git config --global user.signingkey <KEY ID> ``` -1. Optional. If Git uses `gpg` and you get errors like `secret key not available` - or `gpg: signing failed: secret key not available`, run this command to - use `gpg2` instead: - - ```shell - git config --global gpg.program gpg2 - ``` - ### Sign your Git commits After you [add your public key to your account](#add-a-gpg-key-to-your-account), @@ -246,6 +238,7 @@ If you must unverify both future and past commits, ## Related topics - [Sign commits and tags with X.509 certificates](../x509_signed_commits/index.md) +- [Sign commits with SSH keys](../ssh_signed_commits/index.md) - [Commits API](../../../../api/commits.md) - GPG resources: - [Git Tools - Signing Your Work](https://git-scm.com/book/en/v2/Git-Tools-Signing-Your-Work) @@ -269,3 +262,27 @@ or a GPG key. The verification process for both methods can fail for multiple re | `UNVERIFIED_KEY` | The key associated with the GPG signature has no verified email address associated with the committer. | Add and verify the email to your GitLab profile, [update the GPG key to include the email address](https://security.stackexchange.com/a/261468), or amend the commit to use a different committer email address. | | `UNKNOWN_KEY` | The GPG key associated with the GPG signature for this commit is unknown to GitLab. | [Add the GPG key](#add-a-gpg-key-to-your-account) to your GitLab profile. | | `MULTIPLE_SIGNATURES` | Multiple GPG or X.509 signatures have been found for the commit. | Amend the commit to use only one GPG or X.509 signature. | + +### Secret key not available + +If you receive the errors `secret key not available` +or `gpg: signing failed: secret key not available`, try using `gpg2` instead of `gpg`: + +```shell +git config --global gpg.program gpg2 +``` + +If your GPG key is password protected and the password entry prompt does not appear, +add `export GPG_TTY=$(tty)` to your shell's `rc` file (commonly `~/.bashrc` or `~/.zshrc`) + +### GPG failed to sign the data + +If your GPG key is password protected and you receive the error: + +```shell +error: gpg failed to sign the data +fatal: failed to write commit object +``` + +If the password entry prompt does not appear, add `export GPG_TTY=$(tty)` to your shell's `rc` file +(commonly `~/.bashrc` or `~/.zshrc`) and restart your terminal. diff --git a/doc/user/project/repository/ssh_signed_commits/index.md b/doc/user/project/repository/ssh_signed_commits/index.md new file mode 100644 index 00000000000..06affa54a51 --- /dev/null +++ b/doc/user/project/repository/ssh_signed_commits/index.md @@ -0,0 +1,174 @@ +--- +stage: Create +group: Source Code +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +--- + +# Sign commits with SSH keys **(FREE)** + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/343879) in GitLab 15.7 [with a flag](../../../../administration/feature_flags.md) named `ssh_commit_signatures`. Enabled by default. + +FLAG: +On self-managed GitLab, by default this feature is available. To hide the feature, +ask an administrator to [disable the feature flag](../../../../administration/feature_flags.md) named `ssh_commit_signatures`. +On GitLab.com, this feature is available. + +Use SSH keys to sign Git commits in the same manner as +[GPG signed commits](../gpg_signed_commits/index.md). When you sign commits +with SSH keys, GitLab uses the SSH public keys associated with your +GitLab account to cryptographically verify the commit signature. +If successful, GitLab displays a **Verified** label on the commit. + +You may use the same SSH keys for `git+ssh` authentication to GitLab +and signing commit signatures as long as their usage type is **Authentication & Signing**. +It can be verified on the page for [adding an SSH key to your GitLab account](../../../ssh.md#add-an-ssh-key-to-your-gitlab-account). + +To learn more about managing the SSH keys associated with your GitLab account, read +[use SSH keys to communicate with GitLab](../../../ssh.md). + +## Configure Git to sign commits with your SSH key + +After you [create an SSH key](../../../ssh.md#generate-an-ssh-key-pair) and +[add it to your GitLab account](../../../ssh.md#add-an-ssh-key-to-your-gitlab-account) +or [generate it using a password manager](../../../ssh.md#generate-an-ssh-key-pair-with-a-password-manager), +configure Git to begin using the key. + +Prerequisites: + +- Git 2.34.0 or newer. +- OpenSSH 8.0 or newer. + + NOTE: + OpenSSH 8.7 has broken signing functionality. If you are on OpenSSH 8.7, upgrade to OpenSSH 8.8. + +- A SSH key with the usage type of either **Authentication & Signing** or **Signing**. + The SSH key must be one of these types: + - [ED25519](../../../ssh.md#ed25519-ssh-keys) (recommended) + - [RSA](../../../ssh.md#rsa-ssh-keys) + +To configure Git to use your key: + +1. Configure Git to use SSH for commit signing: + + ```shell + git config --global gpg.format ssh + ``` + +1. Specify which SSH key should be used as the signing key, changing the filename + (here, `~/.ssh/examplekey`) to the location of your key. The filename may + differ, depending on how you generated your key: + + ```shell + git config --global user.signingkey ~/.ssh/examplekey + ``` + +## Sign commits with your SSH key + +Prerequisites: + +- You've [created an SSH key](../../../ssh.md#generate-an-ssh-key-pair). +- You've [added the key](../../../ssh.md#add-an-ssh-key-to-your-gitlab-account) to your GitLab account. +- You've [configured Git to sign commits](#configure-git-to-sign-commits-with-your-ssh-key) with your SSH key. + +To sign a commit: + +1. Use the `-S` flag when signing your commits: + + ```shell + git commit -S -m "My commit msg" + ``` + +1. Optional. If you don't want to type the `-S` flag every time you commit, tell + Git to sign your commits automatically: + + ```shell + git config --global commit.gpgsign true + ``` + +1. If your SSH key is protected, Git prompts you to enter your passphrase. +1. Push to GitLab. +1. Check that your commits [are verified](#verify-commits). + Signature verification uses the `allowed_signers` file to associate emails and SSH keys. + For help configuring this file, read [Verify commits locally](#verify-commits-locally). + +## Verify commits + +You can review commits for a merge request, or for an entire project, to confirm +they are signed: + +1. To review commits for a project: + 1. On the top bar, select **Main menu > Projects** and find your project. + 1. On the left sidebar, select **Repository > Commits**. +1. To review commits for a merge request: + 1. On the top bar, select **Main menu > Projects** and find your project. + 1. On the left sidebar, select **Merge requests**, then select your merge request. + 1. Select **Commits**. +1. Identify the commit you want to review. Signed commits show either a **Verified** + or **Unverified** badge, depending on the verification status of the signature. + Unsigned commits do not display a badge. +1. To display the signature details for a commit, select **Verified**. GitLab shows + the SSH key's fingerprint. + +## Verify commits locally + +To verify commits locally, create an +[allowed signers file](https://man7.org/linux/man-pages/man1/ssh-keygen.1.html#ALLOWED_SIGNERS) +for Git to associate SSH public keys with users: + +1. Create an allowed signers file: + + ```shell + touch allowed_signers + ``` + +1. Configure the `allowed_signers` file in Git: + + ```shell + git config gpg.ssh.allowedSignersFile "$(pwd)/allowed_signers" + ``` + +1. Add your entry to the allowed signers file. Use this command to add your + email address and public SSH key to the `allowed_signers` file. Replace `<MY_KEY>` + with the name of your key, and `~/.ssh/allowed_signers` + with the location of your project's `allowed_signers` file: + + ```shell + # Modify this line to meet your needs. + # Declaring the `git` namespace helps prevent cross-protocol attacks. + echo "$(git config --get user.email) namespaces=\"git\" $(cat ~/.ssh/<MY_KEY>.pub)" >> ~/.ssh/allowed_signers + ``` + + The resulting entry in the `allowed_signers` file contains your email address, key type, + and key contents, like this: + + ```plaintext + example@gitlab.com namespaces="git" ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIAmaTS47vRmsKyLyK1jlIFJn/i8wdGQ3J49LYyIYJ2hv + ``` + +1. Repeat the previous step for each user who you want to verify signatures for. + Consider checking this file in to your Git repository if you want to locally + verify signatures for many different contributors. + +1. Use `git log --show-signature` to view the signature status for the commits: + + ```shell + $ git log --show-signature + + commit e2406b6cd8ebe146835ceab67ff4a5a116e09154 (HEAD -> main, origin/main, origin/HEAD) + Good "git" signature for johndoe@example.com with ED25519 key SHA256:Ar44iySGgxic+U6Dph4Z9Rp+KDaix5SFGFawovZLAcc + Author: John Doe <johndoe@example.com> + Date: Tue Nov 29 06:54:15 2022 -0600 + + SSH signed commit + ``` + +## Revoke an SSH key for signing commits + +You can't revoke an SSH key used for signing commits. To learn more, read +[Add revocation for SSH keys](https://gitlab.com/gitlab-org/gitlab/-/issues/382984). + +## Related topics + +- [Sign commits and tags with X.509 certificates](../x509_signed_commits/index.md) +- [Sign commits with GPG](../gpg_signed_commits/index.md) +- [Commits API](../../../../api/commits.md) diff --git a/doc/user/project/repository/web_editor.md b/doc/user/project/repository/web_editor.md index 773662edb17..cc89ca0fb1a 100644 --- a/doc/user/project/repository/web_editor.md +++ b/doc/user/project/repository/web_editor.md @@ -73,7 +73,7 @@ To close the preview panel, do one of the following: > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/56159) in GitLab 13.11 for self-managed instances. Web Editor enables you to highlight a single line by adding specially formatted -hash information to the URL's file path segment. For example, the file path segment +hash information to the file path segment of the URL. For example, the file path segment `MY_FILE.js#L3` instructs the Web Editor to highlight line 3. The Web Editor also enables you to highlight multiple lines using a similar pattern. In diff --git a/doc/user/project/repository/x509_signed_commits/index.md b/doc/user/project/repository/x509_signed_commits/index.md index e16f5e4defe..42f7be30822 100644 --- a/doc/user/project/repository/x509_signed_commits/index.md +++ b/doc/user/project/repository/x509_signed_commits/index.md @@ -160,6 +160,8 @@ can start signing your tags: ## Related topics - [Rake task for X.509 signatures](../../../../raketasks/x509_signatures.md) +- [Sign commits with GPG](../gpg_signed_commits/index.md) +- [Sign commits with SSH keys](../ssh_signed_commits/index.md) ## Troubleshooting diff --git a/doc/user/project/service_desk.md b/doc/user/project/service_desk.md index 199f25f1122..0b52440d1e6 100644 --- a/doc/user/project/service_desk.md +++ b/doc/user/project/service_desk.md @@ -202,39 +202,52 @@ worker and it would not recognize `incoming_email` emails. To configure a custom mailbox for Service Desk with IMAP, add the following snippets to your configuration file in full: -- Example for installations from source: - - ```yaml - service_desk_email: - enabled: true - address: "project_contact+%{key}@example.com" - user: "project_contact@example.com" - password: "[REDACTED]" - host: "imap.gmail.com" - port: 993 - ssl: true - start_tls: false - log_path: "log/mailroom.log" - mailbox: "inbox" - idle_timeout: 60 - expunge_deleted: true - ``` +::Tabs -- Example for Omnibus GitLab installations: +:::TabTitle Linux package (Omnibus) - ```ruby - gitlab_rails['service_desk_email_enabled'] = true - gitlab_rails['service_desk_email_address'] = "project_contact+%{key}@gmail.com" - gitlab_rails['service_desk_email_email'] = "project_contact@gmail.com" - gitlab_rails['service_desk_email_password'] = "[REDACTED]" - gitlab_rails['service_desk_email_mailbox_name'] = "inbox" - gitlab_rails['service_desk_email_idle_timeout'] = 60 - gitlab_rails['service_desk_email_log_file'] = "/var/log/gitlab/mailroom/mail_room_json.log" - gitlab_rails['service_desk_email_host'] = "imap.gmail.com" - gitlab_rails['service_desk_email_port'] = 993 - gitlab_rails['service_desk_email_ssl'] = true - gitlab_rails['service_desk_email_start_tls'] = false - ``` +NOTE: +In GitLab 15.3 and later, Service Desk uses `webhook` (internal API call) by default instead of enqueuing a Sidekiq job. +To use `webhook` on an Omnibus installation running GitLab 15.3, you must generate a secret file. +For more context, visit [Omnibus GitLab MR 5927](https://gitlab.com/gitlab-org/omnibus-gitlab/-/merge_requests/5927). +In GitLab 15.4, reconfiguring an Omnibus installation generates this secret file automatically, so no secret file configuration setting is needed. +For details, visit [issue 1462](https://gitlab.com/gitlab-com/gl-infra/scalability/-/issues/1462). + +```ruby +gitlab_rails['service_desk_email_enabled'] = true +gitlab_rails['service_desk_email_address'] = "project_contact+%{key}@gmail.com" +gitlab_rails['service_desk_email_email'] = "project_contact@gmail.com" +gitlab_rails['service_desk_email_password'] = "[REDACTED]" +gitlab_rails['service_desk_email_mailbox_name'] = "inbox" +gitlab_rails['service_desk_email_idle_timeout'] = 60 +gitlab_rails['service_desk_email_log_file'] = "/var/log/gitlab/mailroom/mail_room_json.log" +gitlab_rails['service_desk_email_host'] = "imap.gmail.com" +gitlab_rails['service_desk_email_port'] = 993 +gitlab_rails['service_desk_email_ssl'] = true +gitlab_rails['service_desk_email_start_tls'] = false +``` + +:::TabTitle Self-compiled (source) + +```yaml +service_desk_email: + enabled: true + address: "project_contact+%{key}@example.com" + user: "project_contact@example.com" + password: "[REDACTED]" + host: "imap.gmail.com" + delivery_method: webhook + secret_file: .gitlab-mailroom-secret + port: 993 + ssl: true + start_tls: false + log_path: "log/mailroom.log" + mailbox: "inbox" + idle_timeout: 60 + expunge_deleted: true +``` + +::EndTabs The configuration options are the same as for configuring [incoming email](../../administration/incoming_email.md#set-it-up). @@ -360,12 +373,17 @@ does not count toward the license limit count. ### Moving a Service Desk issue +> [Changed](https://gitlab.com/gitlab-org/gitlab/-/issues/372246) in GitLab 15.7: customers continue receiving notifications when a Service Desk issue is moved. + Service Desk issues can be moved like any other issue in GitLab. You can move a Service Desk issue the same way you [move a regular issue](issues/managing_issues.md#move-an-issue) in GitLab. -If a Service Desk issue is moved to a different project the customer who created the issue stops receiving emails. +If a Service Desk issue is moved to a different project with Service Desk enabled, +the customer who created the issue continues to receive email notifications. +Because a moved issue is first closed, then copied, the customer is considered to be a participant +in both issues. They continue to receive any notifications in the old issue and the new one. ## Troubleshooting Service Desk diff --git a/doc/user/project/settings/import_export.md b/doc/user/project/settings/import_export.md index 3c12bb9b80f..d83a80ddb13 100644 --- a/doc/user/project/settings/import_export.md +++ b/doc/user/project/settings/import_export.md @@ -12,7 +12,22 @@ then imported into a new GitLab instance. You can also: - [Migrate groups](../../group/import/index.md) using the preferred method. - [Migrate groups using file exports](../../group/settings/import_export.md). -## Set up project import/export +GitLab maps user contributions correctly when an admin access token is used to perform the import. + +As a result, migrating projects using file exports does not map user contributions correctly when you are importing +projects from a self-managed instance to GitLab.com. + +Instead, all GitLab user associations (such as comment author) are changed to the user importing the project. For more +information, see the prerequisites and important notes in these sections: + +- [Export a project and its data](../settings/import_export.md#export-a-project-and-its-data). +- [Import the project](../settings/import_export.md#import-a-project-and-its-data). + +To preserve contribution history, [migrate using direct transfer](../../group/import/index.md#migrate-groups-by-direct-transfer-recommended). + +If you migrate from GitLab.com to self-managed GitLab, an administrator can create users on the self-managed GitLab instance. + +## Set up project to migrate using file exports Before you can import or export a project and its data, you must set it up. @@ -24,8 +39,7 @@ Before you can import or export a project and its data, you must set it up. ## Between CE and EE You can export projects from the [Community Edition to the Enterprise Edition](https://about.gitlab.com/install/ce-or-ee/) -and vice versa. This assumes [version history](#version-history) -requirements are met. +and vice versa. This assumes [version history](#version-history) requirements are met. If you're exporting a project from the Enterprise Edition to the Community Edition, you may lose data that is retained only in the Enterprise Edition. For more information, see @@ -37,8 +51,7 @@ Before you can import a project, you must export it. Prerequisites: -- Review the list of [items that are exported](#items-that-are-exported). - Not all items are exported. +- Review the list of [items that are exported](#items-that-are-exported). Not all items are exported. - You must have at least the Maintainer role for the project. To export a project and its data, follow these steps: @@ -134,7 +147,7 @@ To import a project: 1. Enter your project name and URL. Then select the file you exported previously. 1. Select **Import project** to begin importing. Your newly imported project page appears shortly. -To get the status of an import, you can query it through the [Project import/export API](../../../api/project_import_export.md#import-status). +To get the status of an import, you can query it through the [API](../../../api/project_import_export.md#import-status). As described in the API documentation, the query may return an import error or exceptions. ### Changes to imported items @@ -187,7 +200,7 @@ Imported users can be mapped by their public email addresses on self-managed ins For project migration imports performed over GitLab.com groups, preserving author information is possible through a [professional services engagement](https://about.gitlab.com/services/migration/). -## Rate Limits +## Rate limits To help avoid abuse, by default, users are rate limited to: @@ -197,9 +210,6 @@ To help avoid abuse, by default, users are rate limited to: | Download export | 1 download per group per minute | | Import | 6 projects per minute | -GitLab.com may have [different settings](../../gitlab_com/index.md#importexport) -from the defaults. - ## Version history ### 14.0+ @@ -211,8 +221,8 @@ is NDJSON. ### 13.0+ Starting with GitLab 13.0, GitLab can import bundles that were exported from a different GitLab deployment. -This ability is limited to two previous GitLab [minor](../../../policy/maintenance.md#versioning) -releases, which is similar to our process for [Security Releases](../../../policy/maintenance.md#security-releases). +**This ability is limited to two previous GitLab [minor](../../../policy/maintenance.md#versioning) +releases**, which is similar to our process for [Security Releases](../../../policy/maintenance.md#security-releases). For example: @@ -221,36 +231,9 @@ For example: | 13.0 | 13.0, 12.10, 12.9 | | 13.1 | 13.1, 13.0, 12.10 | -### 12.x - -Prior to 13.0 this was a defined compatibility table: - -| Exporting GitLab version | Importing GitLab version | -| -------------------------- | -------------------------- | -| 11.7 to 12.10 | 11.7 to 12.10 | -| 11.1 to 11.6 | 11.1 to 11.6 | -| 10.8 to 11.0 | 10.8 to 11.0 | -| 10.4 to 10.7 | 10.4 to 10.7 | -| 10.3 | 10.3 | -| 10.0 to 10.2 | 10.0 to 10.2 | -| 9.4 to 9.6 | 9.4 to 9.6 | -| 9.2 to 9.3 | 9.2 to 9.3 | -| 8.17 to 9.1 | 8.17 to 9.1 | -| 8.13 to 8.16 | 8.13 to 8.16 | -| 8.12 | 8.12 | -| 8.10.3 to 8.11 | 8.10.3 to 8.11 | -| 8.10.0 to 8.10.2 | 8.10.0 to 8.10.2 | -| 8.9.5 to 8.9.11 | 8.9.5 to 8.9.11 | -| 8.9.0 to 8.9.4 | 8.9.0 to 8.9.4 | - -Projects can be exported and imported only between versions of GitLab with matching Import/Export versions. - -For example, 8.10.3 and 8.11 have the same Import/Export version (0.1.3) -and the exports between them are compatible. - ## Related topics -- [Project import/export API](../../../api/project_import_export.md) -- [Project import/export administration Rake tasks](../../../administration/raketasks/project_import_export.md) -- [Group import/export](../../group/settings/import_export.md) -- [Group import/export API](../../../api/group_import_export.md) +- [Project import and export API](../../../api/project_import_export.md) +- [Project import and export administration Rake tasks](../../../administration/raketasks/project_import_export.md) +- [Migrating GitLab groups](../../group/import/index.md) +- [Group import and export API](../../../api/group_import_export.md) diff --git a/doc/user/project/settings/index.md b/doc/user/project/settings/index.md index a872a339433..a88427ab20b 100644 --- a/doc/user/project/settings/index.md +++ b/doc/user/project/settings/index.md @@ -1,7 +1,7 @@ --- stage: Manage group: Workspace -info: "To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments" +info: 'To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments' type: reference, index, howto --- @@ -75,41 +75,44 @@ To configure visibility, features, and permissions for a project: Use the toggles to enable or disable features in the project. -| Option | More access limit options | Description | -|:---------------------------------|:--------------------------|:--------------| -| **Issues** | ✓ | Activates the GitLab issues tracker. | -| **Repository** | ✓ | Enables [repository](../repository/index.md) functionality | +| Option | More access limit options | Description | +| :------------------------------- | :------------------------ | :----------------------------------------------------------------------------------------------------------------------------------------------------- | +| **Issues** | ✓ | Activates the GitLab issues tracker. | +| **Repository** | ✓ | Enables [repository](../repository/index.md) functionality | | **Merge requests** | ✓ | Enables [merge request](../merge_requests/index.md) functionality; also see [Merge request settings](#configure-merge-request-settings-for-a-project). | -| **Forks** | ✓ | Enables [forking](../repository/forking_workflow.md) functionality. | -| **Git Large File Storage (LFS)** | | Enables the use of [large files](../../../topics/git/lfs/index.md#git-large-file-storage-lfs). | -| **Packages** | | Supports configuration of a [package registry](../../../administration/packages/index.md#gitlab-package-registry-administration) functionality. | -| **CI/CD** | ✓ | Enables [CI/CD](../../../ci/index.md) functionality. | -| **Container Registry** | | Activates a [registry](../../packages/container_registry/index.md) for your Docker images. | -| **Analytics** | ✓ | Enables [analytics](../../analytics/index.md). | -| **Requirements** | ✓ | Control access to [Requirements Management](../requirements/index.md). | -| **Security & Compliance** | ✓ | Control access to [security features](../../application_security/index.md). | -| **Wiki** | ✓ | Enables a separate system for [documentation](../wiki/index.md). | -| **Snippets** | ✓ | Enables [sharing of code and text](../../snippets.md). | -| **Pages** | ✓ | Allows you to [publish static websites](../pages/index.md). | -| **Metrics Dashboard** | ✓ | Control access to [metrics dashboard](../integrations/prometheus.md). | -| **Releases** | ✓ | Control access to [Releases](../releases/index.md). | +| **Forks** | ✓ | Enables [forking](../repository/forking_workflow.md) functionality. | +| **Git Large File Storage (LFS)** | | Enables the use of [large files](../../../topics/git/lfs/index.md#git-large-file-storage-lfs). | +| **Packages** | | Supports configuration of a [package registry](../../../administration/packages/index.md#gitlab-package-registry-administration) functionality. | +| **CI/CD** | ✓ | Enables [CI/CD](../../../ci/index.md) functionality. | +| **Container Registry** | | Activates a [registry](../../packages/container_registry/index.md) for your Docker images. | +| **Analytics** | ✓ | Enables [analytics](../../analytics/index.md). | +| **Requirements** | ✓ | Control access to [Requirements Management](../requirements/index.md). | +| **Security & Compliance** | ✓ | Control access to [security features](../../application_security/index.md). | +| **Wiki** | ✓ | Enables a separate system for [documentation](../wiki/index.md). | +| **Snippets** | ✓ | Enables [sharing of code and text](../../snippets.md). | +| **Pages** | ✓ | Allows you to [publish static websites](../pages/index.md). | +| **Metrics Dashboard** | ✓ | Control access to [metrics dashboard](../integrations/prometheus.md). | +| **Releases** | ✓ | Control access to [Releases](../releases/index.md). | | **Environments** | ✓ | Control access to [Environments and Deployments](../../../ci/environments/index.md). | | **Feature flags** | ✓ | Control access to [Feature Flags](../../../operations/feature_flags.md). | -| **Monitor** | ✓ | Control access to [Monitor](../../../operations/index.md) features. | -| **Infrastructure** | ✓ | Control access to [Infrastructure](../../infrastructure/index.md) features. | +| **Monitor** | ✓ | Control access to [Monitor](../../../operations/index.md) features. | +| **Infrastructure** | ✓ | Control access to [Infrastructure](../../infrastructure/index.md) features. | When you disable a feature, the following additional features are also disabled: - If you disable the **Issues** feature, project users cannot use: + - **Issue Boards** - **Service Desk** - Project users can still access **Milestones** from merge requests. - If you disable **Issues** and **Merge Requests**, project users cannot use: + - **Labels** - **Milestones** - If you disable **Repository**, project users cannot access: + - **Merge requests** - **CI/CD** - **Container Registry** @@ -236,10 +239,8 @@ Prerequisites: - You must have at least the Maintainer role for the [group](../../group/manage.md#create-a-group) to which you are transferring. - You must be the Owner of the project you transfer. - The group must allow creation of new projects. -- The project must not contain any [container images](../../packages/container_registry/index.md#limitations). - - If you transfer a project to a different root namespace, - the project must not contain any - [NPM packages](../../packages/npm_registry/index.md#limitations). +- The project must not contain any [container images](../../packages/container_registry/index.md#known-issues). +- Remove any npm packages. If you transfer a project to a different root namespace, the project must not contain any npm packages. When you update the path of a user or group, or transfer a subgroup or project, you must remove any npm packages first. You cannot update the root namespace of a project with npm packages. Make sure you update your .npmrc files to follow the naming convention and run npm publish if necessary. To transfer a project: @@ -262,7 +263,7 @@ When you transfer a project from a namespace licensed for GitLab SaaS Premium or - [Project access tokens](../../../user/project/settings/project_access_tokens.md) are revoked - [Pipeline subscriptions](../../../ci/pipelines/index.md#trigger-a-pipeline-when-an-upstream-project-is-rebuilt) -and [test cases](../../../ci/test_cases/index.md) are deleted. + and [test cases](../../../ci/test_cases/index.md) are deleted. ## Delete a project @@ -270,7 +271,7 @@ You can mark a project to be deleted. Prerequisite: -- You must have at least the Owner role for a project. +- You must have the Owner role for a project. To delete a project: @@ -308,7 +309,7 @@ If you don't want to wait, you can delete a project immediately. Prerequisites: -- You must have at least the Owner role for a project. +- You must have the Owner role for a project. - You have [marked the project for deletion](#delete-a-project). To immediately delete a project marked for deletion: @@ -361,11 +362,11 @@ Configure [alert integrations](../../../operations/incident_management/integrati #### Alert integration -Automatically [create](../../../operations/incident_management/incidents.md#create-incidents-automatically), [notify on](../../../operations/incident_management/paging.md#email-notifications-for-alerts), and [resolve](../../../operations/incident_management/incidents.md#automatically-close-incidents-via-recovery-alerts) incidents based on GitLab alerts. +Automatically [create](../../../operations/metrics/alerts.md#trigger-actions-from-alerts), [notify on](../../../operations/incident_management/paging.md#email-notifications-for-alerts), and [resolve](../../../operations/incident_management/manage_incidents.md#automatically-close-incidents-via-recovery-alerts) incidents based on GitLab alerts. #### PagerDuty integration -[Create incidents in GitLab for each PagerDuty incident](../../../operations/incident_management/incidents.md#create-incidents-via-the-pagerduty-webhook). +[Create incidents in GitLab for each PagerDuty incident](../../../operations/incident_management/manage_incidents.md#using-the-pagerduty-webhook). #### Incident settings diff --git a/doc/user/project/settings/project_access_tokens.md b/doc/user/project/settings/project_access_tokens.md index f27672a1b07..cb7ba2a7df2 100644 --- a/doc/user/project/settings/project_access_tokens.md +++ b/doc/user/project/settings/project_access_tokens.md @@ -69,6 +69,11 @@ To create a project access token: A project access token is displayed. Save the project access token somewhere safe. After you leave or refresh the page, you can't view it again. +WARNING: +Project access tokens are treated as [internal users](../../../development/internal_users.md). +If an internal user creates a project access token, that token is able to access +all projects that have visibility level set to [Internal](../../public_access.md). + ## Revoke a project access token To revoke a project access token: diff --git a/doc/user/project/time_tracking.md b/doc/user/project/time_tracking.md index 71b8c911839..186ca0ba9f0 100644 --- a/doc/user/project/time_tracking.md +++ b/doc/user/project/time_tracking.md @@ -74,6 +74,25 @@ Prerequisites: - You must have at least the Reporter role for the project. +#### Using the user interface + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/101563) in GitLab 15.7. + +To add a time entry using the user interface: + +1. In the **Time tracking** section of the sidebar, select **Add time entry** (**{plus}**). A modal opens. +1. Enter: + + - The amount of time spent. + - Optional. When it was spent. + - Optional. A summary. + +1. Select **Save**. + +The **Spent** total in the sidebar is updated and you can view all entries in a [time tracking report](#view-a-time-tracking-report). + +#### Using a quick action + To enter time spent, use the `/spend` [quick action](quick_actions.md), followed by the time. For example, if you need @@ -90,15 +109,10 @@ Draft MR and respond to initial comments /spend 30m ``` -### Add time spent on a specific date - -Prerequisites: - -- You must have at least the Reporter role for the project. +To log when time was spent, enter a date after the time, using the `YYYY-MM-DD` format. -You can log time in the past by providing a date after the time. For example, to log 1 hour of time spent on 31 January 2021, -type `/spend 1h 2021-01-31`. +enter `/spend 1h 2021-01-31`. If you type a future date, no time is logged. diff --git a/doc/user/project/web_ide/index.md b/doc/user/project/web_ide/index.md index 0200e9c4e7d..fb100986df9 100644 --- a/doc/user/project/web_ide/index.md +++ b/doc/user/project/web_ide/index.md @@ -10,6 +10,10 @@ The Web Integrated Development Environment (IDE) editor streamlines the process to contribute changes to your projects, by providing an advanced editor with commit staging. +NOTE: +The Web IDE is being updated to use VS Code. For details, +see [Web IDE Beta](../web_ide_beta/index.md). + ## Open the Web IDE Use the <kbd>.</kbd> [keyboard shortcut](../../shortcuts.md) to open the Web IDE. @@ -80,7 +84,7 @@ If you are missing Syntax Highlighting support for any language, we prepared a s > - Full Solarized Dark Theme [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/219228) in GitLab 13.1. > - Full [Solarized Light](https://gitlab.com/gitlab-org/gitlab/-/issues/221035) and [Monokai](https://gitlab.com/gitlab-org/gitlab/-/issues/221034) Themes introduced in GitLab 13.6. -All the themes GitLab supports for syntax highlighting are applied to the Web IDE's entire screen. +All the themes GitLab supports for syntax highlighting are applied to the entire Web IDE screen. You can pick a theme from your [profile preferences](../../profile/preferences.md). | Solarized Dark Theme | Dark Theme | @@ -459,12 +463,3 @@ The Web IDE has a few limitations: - If the terminal displays **Connection Failure**, then the terminal could not connect to the runner. Try to stop and restart the terminal. If the problem persists, double check your runner configuration. - -## VSCode Reimplementation - -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/95169) in GitLab 15.4 [with a flag](../../../administration/feature_flags.md) named `vscode_web_ide`. Disabled by default. - -As announced in [this blog post](https://about.gitlab.com/blog/2022/05/23/the-future-of-the-gitlab-web-ide/), -the current implementation of the Web IDE will be replaced with a [VSCode inspired implementation](https://gitlab.com/groups/gitlab-org/-/epics/7683). - -This effort is currently under development. Follow [this epic](https://gitlab.com/groups/gitlab-org/-/epics/7683) for updates and more information. diff --git a/doc/user/project/web_ide_beta/img/fuzzy_finder_v15_7.png b/doc/user/project/web_ide_beta/img/fuzzy_finder_v15_7.png Binary files differnew file mode 100644 index 00000000000..66ebae15e98 --- /dev/null +++ b/doc/user/project/web_ide_beta/img/fuzzy_finder_v15_7.png diff --git a/doc/user/project/web_ide_beta/index.md b/doc/user/project/web_ide_beta/index.md new file mode 100644 index 00000000000..ef07cca465d --- /dev/null +++ b/doc/user/project/web_ide_beta/index.md @@ -0,0 +1,103 @@ +--- +stage: Create +group: Editor +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments +--- + +# Web IDE Beta **(FREE)** + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/95169) in GitLab 15.4 [with a flag](../../../administration/feature_flags.md) named `vscode_web_ide`. Disabled by default. + +FLAG: +On self-managed GitLab, by default this feature is not available. To make it available, ask an administrator to [enable the feature flag](../../../administration/feature_flags.md) named `vscode_web_ide`. On GitLab.com, this feature is available. + +As announced in [this blog post](https://about.gitlab.com/blog/2022/05/23/the-future-of-the-gitlab-web-ide/), +the current implementation of the Web IDE is being replaced with an +implementation inspired by Visual Studio Code. + +This effort is currently under development. For updates, +see [this epic](https://gitlab.com/groups/gitlab-org/-/epics/7683). + +## Enable the Web IDE Beta + +To use the Web IDE Beta on a self-managed GitLab instance, +ensure that the `vscode_web_ide` feature flag +[is enabled](../../../administration/feature_flags.md). + +On GitLab.com, this feature is available by default. However, you can +[stop using it if you choose](#stop-using-the-web-ide-beta). + +## Use the Web IDE Beta + +To open the Web IDE Beta from anywhere in the UI: + +- Use the <kbd>.</kbd> [keyboard shortcut](../../shortcuts.md). + +You can also open the Web IDE Beta when viewing a file, the repository file list, +and from merge requests. + +### Use when viewing a file or the repository file list + +To open the Web IDE Beta from a file or the repository file list: + +- On the top right of the page, select **Open in Web IDE**. + +If **Open in Web IDE** is not visible: + +1. Next to **Edit** or **Gitpod**, select the down arrow (**{chevron-lg-down}**). +1. From the list, select **Open in Web IDE**. +1. Select **Open in Web IDE**. + +### Use when viewing a merge request + +To open the Web IDE Beta from a merge request: + +1. Go to your merge request. +1. In the upper right corner, select **Code > Open in Web IDE**. + +## Open a file in the Web IDE Beta + +To open any file by its name: + +1. Type **Command** + **`P`** (<kbd>⌘</kbd> + <kbd>P</kbd>). +1. Type the name of your file. + + + +## Search across files + +You can use VS Code to quickly search all files in the currently opened folder. + +To enter your search term: + +1. Type **Shift** + **Command** + **`F`** (<kbd>⇧</kbd> + <kbd>⌘</kbd> + <kbd>F</kbd>). +1. Enter your search term. + +In the Web IDE Beta, only partial results from opened files are displayed. +Full file search is planned for a later date. + +## View list of changed files + +To view the list of files you changed in the Web IDE Beta: + +- On the VS Code Activity Bar, on the left, select the Source Control icon: + +Your `CHANGES`, `STAGED CHANGES` and `MERGE CHANGES` are displayed. + +For details, see [the VS Code documentation](https://code.visualstudio.com/docs/sourcecontrol/overview#_commit). + +## Known issues + +The [Web Terminal](../web_ide/index.md#interactive-web-terminals-for-the-web-ide) +and [Live Preview](../web_ide/index.md#live-preview) are not available in the Web IDE Beta. + +These features may become available at a later date. + +### Stop using the Web IDE Beta + +If you do not want to use the Web IDE Beta, you can change your personal preferences. + +1. On the top bar, in the top right corner, select your avatar. +1. Select **Preferences**. +1. In the **Web IDE** section, select the **Opt out of the Web IDE Beta** checkbox. +1. Select **Save changes**. diff --git a/doc/user/project/wiki/group.md b/doc/user/project/wiki/group.md index 0e9b76e943d..53aaa41319b 100644 --- a/doc/user/project/wiki/group.md +++ b/doc/user/project/wiki/group.md @@ -16,8 +16,6 @@ Group wikis are similar to [project wikis](index.md), with a few limitations: - [Git LFS](../../../topics/git/lfs/index.md) is not supported. - Group wikis are not included in [global search](../../search/advanced_search.md). - Changes to group wikis don't show up in the [group's activity feed](../../group/manage.md#group-activity-analytics). -- Group wikis are enabled by default for GitLab Premium and higher tiers. - You [can't turn them off from the GitLab user interface](https://gitlab.com/gitlab-org/gitlab/-/issues/208413). For updates, follow [the epic that tracks feature parity with project wikis](https://gitlab.com/groups/gitlab-org/-/epics/2782). diff --git a/doc/user/project/wiki/index.md b/doc/user/project/wiki/index.md index eedc44be3f9..04a6f9bee80 100644 --- a/doc/user/project/wiki/index.md +++ b/doc/user/project/wiki/index.md @@ -14,7 +14,7 @@ to keep it in the same project as your code, you can use the wiki GitLab provide in each GitLab project. Every wiki is a separate Git repository, so you can create wiki pages in the web interface, or [locally using Git](#create-or-edit-wiki-pages-locally). -GitLab wikis support Markdown, RDoc, AsciiDoc, and Org for content. +GitLab wikis support Markdown, Rdoc, AsciiDoc, and Org for content. Wiki pages written in Markdown support all [Markdown features](../../markdown.md), and also provide some [wiki-specific behavior](../../markdown.md#wiki-specific-markdown) for links. @@ -338,11 +338,12 @@ GitLab provides a WYSIWYG editing experience for GitLab Flavored Markdown in wik Support includes: -- Text formatting options, including bold, italics, block quotes, headings, and inline code. -- List formatting for unordered, numbered, and checklists. -- Creating and editing the structure of tables. +- Formatting text, including using bold, italics, block quotes, headings, and inline code. +- Formatting ordered lists, unordered lists, and checklists. +- Creating and editing table structure. - Inserting and formatting code blocks with syntax highlighting. -- Live preview of Mermaid, PlantUML, and Kroki diagrams ([introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/86701) in GitLab 15.2). +- Previewing Mermaid, PlantUML, and Kroki diagrams ([introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/86701) in GitLab 15.2). +- Creating and editing HTML comments ([introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/104084) in GitLab 15.7). ### Use the Content Editor diff --git a/doc/user/project/working_with_projects.md b/doc/user/project/working_with_projects.md index 067a0303277..77a7c48658e 100644 --- a/doc/user/project/working_with_projects.md +++ b/doc/user/project/working_with_projects.md @@ -347,10 +347,27 @@ You can sort projects by: You can also choose to hide or show archived projects. +## Change the visibility of individual features in a project + +You can change the visibility of individual features in a project. + +Prerequisite: + +- You must have the Owner role for the project. + +1. On the top bar, select **Main menu > Projects** and find your project. +1. On the left sidebar, select **Settings > General**. +1. Expand **Visibility, project features, permissions**. +1. Use the toggle by each feature you want to turn on or off, or change access for. +1. Select **Save changes**. + ## Leave a project -If you leave a project, you are no longer a project -member and cannot contribute. +When you leave a project: + +- You are no longer a project member and cannot contribute. +- All the issues and merge requests that were assigned + to you are unassigned. To leave a project: diff --git a/doc/user/public_access.md b/doc/user/public_access.md index bdc711f2098..acfc28a6f65 100644 --- a/doc/user/public_access.md +++ b/doc/user/public_access.md @@ -7,16 +7,17 @@ type: reference # Project and group visibility **(FREE)** -GitLab allows users with the Owner role to set a project's or group's visibility as: +If you have the Owner role, you can set a project's or group's visibility as: - **Public** -- **Internal** +- **[Internal](#internal-projects-and-groups)** - **Private** -These visibility levels affect who can see the project in the public access directory (`/public` -for your GitLab instance). For example, <https://gitlab.com/public>. -You can control the visibility of individual features with -[project feature settings](permissions.md#project-features). +These visibility levels affect who can see the project in the public access directory +(for example, <https://gitlab.com/public>). + +For more granular control, you can determine +[which features in a project are visible](project/working_with_projects.md#change-the-visibility-of-individual-features-in-a-project). The visibility setting of a project must be at least as restrictive as the visibility of its parent group. @@ -38,16 +39,16 @@ By default, `/public` is visible to unauthenticated users. However, if the [**Public** visibility level](admin_area/settings/visibility_and_access_controls.md#restrict-visibility-levels) is restricted, `/public` is visible only to signed-in users. -## Internal projects and groups +## Internal projects and groups **(FREE SELF)** Internal projects can be cloned by any signed-in user except -[external users](permissions.md#external-users). +[external users](admin_area/external_users.md). They are also listed in the public access directory (`/public`), but only for signed-in users. Internal groups can have internal or private subgroups. -Any signed-in users except [external users](permissions.md#external-users) have the +Any signed-in users except [external users](admin_area/external_users.md) have the Guest role on the repository. NOTE: @@ -66,6 +67,8 @@ Private groups can only have private subgroups. ## Change project visibility +You can change the visibility of a project. + Prerequisite: - You must have the Owner role for a project. @@ -78,6 +81,8 @@ Prerequisite: ## Change group visibility +You can change the visibility of all projects in a group. + Prerequisites: - You must have the Owner role for a group. diff --git a/doc/user/read_only_namespaces.md b/doc/user/read_only_namespaces.md new file mode 100644 index 00000000000..b11aac9b00c --- /dev/null +++ b/doc/user/read_only_namespaces.md @@ -0,0 +1,48 @@ +--- +stage: Growth +group: Acquisition +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments +noindex: true +--- + +# Read-only namespaces **(FREE SAAS)** + +In GitLab SaaS, a top-level namespace is placed in a read-only state when it either: + +- Exceeds the [free user limit](free_user_limit.md) when the namespace visibility is private. +- Exceeds the [storage usage quota](usage_quotas.md), regardless of namespace visibility. + +While a namespace is in a read-only state, a banner appears at the +top of the page. + +Your ability to write new data to read-only namespaces is restricted. For more +information, see [Restricted actions](#restricted-actions). + +## Remove the read-only state + +To restore a namespace to its normal state, you can: + +- For exceeded free user limits: + - [Reduce the number of members](free_user_limit.md#manage-members-in-your-namespace) in your namespace. + - [Start a free trial](https://gitlab.com/-/trial_registrations/new), which includes an unlimited number of members. + - [Purchase a paid tier](https://about.gitlab.com/pricing/). +- For exceeded storage quota: + - [Purchase more storage for the namespace](../subscriptions/gitlab_com/index.md#purchase-more-storage-and-transfer). + - [Manage your storage usage](usage_quotas.md#manage-your-storage-usage). + +## Restricted actions + +| Feature | Action restricted | +|---------|-------------------| +| Container Registry | Create, edit, and delete cleanup policies <br> Push an image to the container registry | +| Merge Requests | Create and update an MR | +| Package Registry | Publish a package | +| Repositories | Add tags <br> Create new branches <br> Create and update commit status <br> Push and force push to non-protected branches <br> Push and force push to protected branches <br> Upload files <br> Create merge requests | +| CI/CD | Create, edit, admin, and run pipelines <br> Create, edit, admin, and run builds <br> Create and edit admin environments <br> Create and edit admin deployments <br> Create and edit admin clusters <br> Create and edit admin releases | +| Namespaces | **For exceeded free user limits:** Invite new users | + +## Related topics + +- [Frequently Asked Questions - GitLab SaaS Free Tier](https://about.gitlab.com/pricing/faq-efficient-free-tier/) +- [Free user limit](free_user_limit.md) +- [Storage usage quotas](usage_quotas.md) diff --git a/doc/user/report_abuse.md b/doc/user/report_abuse.md index fab01973104..fde839c3ba4 100644 --- a/doc/user/report_abuse.md +++ b/doc/user/report_abuse.md @@ -1,6 +1,6 @@ --- -stage: none -group: unassigned +stage: Anti-Abuse +group: Anti-Abuse info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- @@ -27,7 +27,7 @@ You can report a user through their: To report abuse from a user's profile page: 1. Anywhere in GitLab, select the name of the user. -1. In the top right corner of the user's profile, select **Report abuse** (**{information-o}**). +1. In the top right corner of the user's profile, select **Report abuse to administrator** (**{information-o}**). 1. Complete an abuse report. 1. Select **Send report**. @@ -36,7 +36,7 @@ To report abuse from a user's profile page: To report abuse from a user's comment: 1. In the comment, in the top right corner, select **More actions** (**{ellipsis_v}**). -1. Select **Report abuse to admin**. +1. Select **Report abuse to administrator**. 1. Complete an abuse report. 1. Select **Send report**. @@ -47,14 +47,14 @@ A URL to the reported user's comment is pre-filled in the abuse report's ## Report abuse from an issue 1. On the issue, in the top right corner, select the vertical ellipsis (**{ellipsis_v}**). -1. Select **Report abuse**. +1. Select **Report abuse to administrator**. 1. Submit an abuse report. 1. Select **Send report**. ## Report abuse from a merge request 1. On the merge request, in the top right corner, select the vertical ellipsis (**{ellipsis_v}**). -1. Select **Report abuse**. +1. Select **Report abuse to administrator**. 1. Submit an abuse report. 1. Select **Send report**. diff --git a/doc/user/reserved_names.md b/doc/user/reserved_names.md index 1ab22fb846d..3eaf3178f3a 100644 --- a/doc/user/reserved_names.md +++ b/doc/user/reserved_names.md @@ -19,10 +19,13 @@ under the `TOP_LEVEL_ROUTES`, `PROJECT_WILDCARD_ROUTES` and `GROUP_ROUTES` lists ## Limitations on project and group names -- Special characters are not permitted at the start or end of project or group names. They are permitted in any other location of the name. -- Project or group names cannot end in `.git` or `.atom`. +- Project or group names must start with a letter, digit, emoji, or "_". - Project or group names can only contain letters, digits, emojis, "_", ".", "+", dashes, or spaces. -- Paths can only contain letters, digits, "_", "-", and "." +- Project or group slugs must start with a letter or digit. +- Project or group slugs can only contain letters, digits, '_', '.', '+', or dashes. +- Project or group slugs must not contain consecutive special characters. +- Project or group slugs cannot end with a special character. +- Project or group slugs cannot end in `.git` or `.atom`. ## Reserved project names diff --git a/doc/user/search/advanced_search.md b/doc/user/search/advanced_search.md index 925fc7e6036..ed1d3b1d290 100644 --- a/doc/user/search/advanced_search.md +++ b/doc/user/search/advanced_search.md @@ -29,11 +29,27 @@ You can use Advanced Search in: - Commits - Project wikis (not [group wikis](../project/wiki/group.md)) -## Configure Advanced Search +For Advanced Search: + +- You can only search files smaller than 1 MB. + For self-managed GitLab instances, an administrator can + [change this limit](../../integration/advanced_search/elasticsearch.md#advanced-search-configuration). +- You can't use any of the following characters in the search query: + + ```plaintext + . , : ; / ` ' = ? $ & ^ | ~ < > ( ) { } [ ] @ + ``` + +- Only the default branch of a project is indexed for code search. + In a non-default branch, basic search is used. +- Search results show only the first match in a file, + but there might be more results in that file. + +## Enable Advanced Search - On GitLab.com, Advanced Search is enabled for groups with paid subscriptions. - For self-managed GitLab instances, an administrator must - [configure Advanced Search](../../integration/advanced_search/elasticsearch.md). + [enable Advanced Search](../../integration/advanced_search/elasticsearch.md#enable-advanced-search). ## Syntax diff --git a/doc/user/search/img/basic_search_results_v15_1.png b/doc/user/search/img/basic_search_results_v15_1.png Binary files differdeleted file mode 100644 index 0de0b976d7d..00000000000 --- a/doc/user/search/img/basic_search_results_v15_1.png +++ /dev/null diff --git a/doc/user/search/img/basic_search_v15_1.png b/doc/user/search/img/basic_search_v15_1.png Binary files differdeleted file mode 100644 index 069d62ca80c..00000000000 --- a/doc/user/search/img/basic_search_v15_1.png +++ /dev/null diff --git a/doc/user/search/img/search_navbar_v15_7.png b/doc/user/search/img/search_navbar_v15_7.png Binary files differnew file mode 100644 index 00000000000..9175347b36f --- /dev/null +++ b/doc/user/search/img/search_navbar_v15_7.png diff --git a/doc/user/search/img/search_scope_v15_7.png b/doc/user/search/img/search_scope_v15_7.png Binary files differnew file mode 100644 index 00000000000..6395b5f1cda --- /dev/null +++ b/doc/user/search/img/search_scope_v15_7.png diff --git a/doc/user/search/index.md b/doc/user/search/index.md index 9bff2a91ec8..9536f7fe40a 100644 --- a/doc/user/search/index.md +++ b/doc/user/search/index.md @@ -53,12 +53,12 @@ Global search only flags with an error any search that includes more than: To start a search, type your search query in the search bar on the top-right of the screen. You must type at least two characters. - + After the results are displayed, you can modify the search, select a different type of data to search, or choose a specific group or project. - + ## Search in code @@ -150,6 +150,7 @@ To delete filter tokens one at a time, the <kbd>⌥</kbd> (Mac) / <kbd>Control</ In the search bar, you can view autocomplete suggestions for: - Projects and groups +- Users - Various help pages (try and type **API help**) - Project feature pages (try and type **milestones**) - Various settings pages (try and type **user settings**) diff --git a/doc/user/shortcuts.md b/doc/user/shortcuts.md index f9e61ad78ad..64f9b53f891 100644 --- a/doc/user/shortcuts.md +++ b/doc/user/shortcuts.md @@ -37,6 +37,7 @@ These shortcuts are available in most areas of GitLab: | <kbd>f</kbd> | Put cursor in the filter bar. | | <kbd>Shift</kbd> + <kbd>i</kbd> | Go to your Issues page. | | <kbd>Shift</kbd> + <kbd>m</kbd> | Go to your [Merge requests](project/merge_requests/index.md) page. | +| <kbd>Shift</kbd> + <kbd>r</kbd> | Go to your Review requests page. | | <kbd>Shift</kbd> + <kbd>t</kbd> | Go to your To-Do List page. | | <kbd>p</kbd>, then <kbd>b</kbd> | Show or hide the Performance Bar. | | <kbd>Escape</kbd> | Hide tooltips or popovers. | diff --git a/doc/user/snippets.md b/doc/user/snippets.md index 6a8184e9ca1..ee84f8a4169 100644 --- a/doc/user/snippets.md +++ b/doc/user/snippets.md @@ -267,3 +267,7 @@ creating a new snippet, use this workaround: 1. Enter any string into the text area for the second file. 1. Scroll back to the first filename, and select **Delete file**. 1. Create the rest of your file, and select **Create snippet** when done. + +## Related topics + +- [Configure snippet settings](../administration/snippets/index.md) on a self-managed GitLab instance diff --git a/doc/user/ssh.md b/doc/user/ssh.md index 85332446334..1d82d301e9a 100644 --- a/doc/user/ssh.md +++ b/doc/user/ssh.md @@ -270,6 +270,7 @@ You can use [1Password](https://1password.com/) and the [1Password browser exten 1. You can then select **Create SSH Key** or select an existing SSH key to fill in the public key. 1. In the **Title** box, type a description, like `Work Laptop` or `Home Workstation`. +1. Optional. Select the **Usage type** of the key. It can be used either for `Authentication` or `Signing` or both. `Authentication & Signing` is the default value. 1. Optional. Update **Expiration date** to modify the default expiration date. 1. Select **Add key**. @@ -314,6 +315,7 @@ To use SSH with GitLab, copy your public key to your GitLab account: `ssh-ed25519`, `sk-ecdsa-sha2-nistp256@openssh.com`, or `sk-ssh-ed25519@openssh.com`, and may end with a comment. 1. In the **Title** box, type a description, like `Work Laptop` or `Home Workstation`. +1. Optional. Select the **Usage type** of the key. It can be used either for `Authentication` or `Signing` or both. `Authentication & Signing` is the default value. 1. Optional. Update **Expiration date** to modify the default expiration date. In: - GitLab 13.12 and earlier, the expiration date is informational only. It doesn't prevent diff --git a/doc/user/tasks.md b/doc/user/tasks.md index 5fed887ca74..ffe7777fac0 100644 --- a/doc/user/tasks.md +++ b/doc/user/tasks.md @@ -49,7 +49,7 @@ the task opens in a full-page view. Prerequisites: -- You must have at least the Guest role for the project, or the project must be public. +- You must have at least the Reporter role for the project, or the project must be public. To create a task: @@ -207,10 +207,9 @@ To set a start date: ## Add a task to a milestone -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/367463) in GitLab 15.5 [with a flag](../administration/feature_flags.md) named `work_items_mvc_2`. Disabled by default. - -FLAG: -On self-managed GitLab, by default this feature is not available. To make it available per group, ask an administrator to [enable the feature flag](../administration/feature_flags.md) named `work_items_mvc_2`. On GitLab.com, this feature is not available. The feature is not ready for production use. +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/367463) in GitLab 15.5 [with a flag](../administration/feature_flags.md) named `work_items_mvc_2`. Disabled by default. +> - [Moved](https://gitlab.com/gitlab-org/gitlab/-/issues/367463) to feature flag named `work_items_mvc` in GitLab 15.7. Disabled by default. +> - [Enabled on GitLab.com and self-managed](https://gitlab.com/gitlab-org/gitlab/-/issues/367463) in GitLab 15.7. You can add a task to a [milestone](project/milestones/index.md). You can see the milestone title when you view a task. @@ -249,10 +248,13 @@ To set issue weight of a task: ## Add a task to an iteration **(PREMIUM)** -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/367456) in GitLab 15.5 [with a flag](../administration/feature_flags.md) named `work_items_mvc_2`. Disabled by default. +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/367456) in GitLab 15.5 [with a flag](../administration/feature_flags.md) named `work_items_mvc_2`. Disabled by default. +> - [Moved](https://gitlab.com/gitlab-org/gitlab/-/issues/367456) to feature flag named `work_items_mvc` in GitLab 15.7. Disabled by default. +> - [Enabled on GitLab.com and self-managed](https://gitlab.com/gitlab-org/gitlab/-/issues/367456) in GitLab 15.7. FLAG: -On self-managed GitLab, by default this feature is not available. To make it available per group, ask an administrator to [enable the feature flag](../administration/feature_flags.md) named `work_items_mvc_2`. On GitLab.com, this feature is not available. The feature is not ready for production use. +On self-managed GitLab, by default this feature is available. To hide the feature, ask an administrator to [disable the feature flag](../administration/feature_flags.md) named `work_items_mvc`. +On GitLab.com, this feature is available. You can add a task to an [iteration](group/iterations/index.md). You can see the iteration title and period only when you view a task. diff --git a/doc/user/todos.md b/doc/user/todos.md index d7deda33a65..4102d31cfc4 100644 --- a/doc/user/todos.md +++ b/doc/user/todos.md @@ -87,6 +87,7 @@ You can manually add an item to your To-Do List. - [Merge request](project/merge_requests/index.md) - [Epic](group/epics/index.md) - [Design](project/issues/design_management.md) + - [Incident](../operations/incident_management/incidents.md) 1. On the right sidebar, at the top, select **Add a to do**. @@ -135,7 +136,7 @@ You can manually mark a to-do item as done. There are two ways to do this: -- In the To-Do List, to the right of the to-do item, select **Done**. +- In the To-Do List, to the right of the to-do item, select **Mark as done** (**{check}**). - In the sidebar of an issue, merge request, or epic, select **Mark as done**.  diff --git a/doc/user/upgrade_email_bypass.md b/doc/user/upgrade_email_bypass.md index ef58d8aec66..afbdcbcdf09 100644 --- a/doc/user/upgrade_email_bypass.md +++ b/doc/user/upgrade_email_bypass.md @@ -76,7 +76,7 @@ Your primary email address is not confirmed. You can assure your users that they have not been [Blocked](admin_area/moderate_users.md#block-and-unblock-users) by an administrator. When affected users see this message, they must confirm their email address before they can commit code. -## What do I need to know as an administrator of a GitLab self-managed Instance? +## What do you need to know as an administrator of a GitLab self-managed Instance? You have the following options to help your users: @@ -85,7 +85,7 @@ You have the following options to help your users: As an administrator, you may also confirm a user in the [Admin Area](admin_area/index.md#administering-users). -## What do I do if I am an administrator and I am locked out? +## What do you do if you are an administrator and you're locked out? If you are an administrator and cannot otherwise verify your email address, sign in to your GitLab instance with a [Rails console session](../administration/operations/rails_console.md#starting-a-rails-console-session). @@ -97,7 +97,7 @@ admin.confirmed_at = Time.zone.now admin.save! ``` -## How do I force-confirm all users on my self-managed instance? +## How do you force-confirm all users on your self-managed instance? If you are an administrator and would like to force-confirm all users on your system, sign in to your GitLab instance with a [Rails console session](../administration/operations/rails_console.md#starting-a-rails-console-session). diff --git a/doc/user/usage_quotas.md b/doc/user/usage_quotas.md index 7df5ade215d..0e5703caffc 100644 --- a/doc/user/usage_quotas.md +++ b/doc/user/usage_quotas.md @@ -126,8 +126,7 @@ Storage types that add to the total namespace storage are: - Wiki - Snippets -If your total namespace storage exceeds the available namespace storage quota, all projects under the namespace are locked. -A locked project cannot push to the repository, run pipelines and jobs, or build and push packages. +If your total namespace storage exceeds the available namespace storage quota, all projects under the namespace become read-only. Your ability to write new data is restricted until the read-only state is removed. For more information, see [Restricted actions](../user/read_only_namespaces.md#restricted-actions). To prevent exceeding the namespace storage quota, you can: diff --git a/doc/user/workspace/index.md b/doc/user/workspace/index.md index 5bcd96cd4a5..4a8b083e3a2 100644 --- a/doc/user/workspace/index.md +++ b/doc/user/workspace/index.md @@ -26,12 +26,12 @@ everything you do as a GitLab administrator, including: Our goal is to reach feature parity between SaaS and self-managed installations, with all [Admin Area settings](/ee/user/admin_area/settings/index.md) moving to either: -- Groups. Available in the Workspace, top-level group namespaces, and subgroups. +- Groups. Available in the Workspace, top-level groups, and subgroups. - Hardware Controls. For functionality that does not apply to groups, Hardware Controls are only applicable to self-managed installations. There is one Hardware Controls section per installation. -To learn about the current state of workspace development, -see [epic 4257](https://gitlab.com/groups/gitlab-org/-/epics/4257). +For more information about the state of workspace development, +see [epic 9265](https://gitlab.com/groups/gitlab-org/-/epics/9265). <i class="fa fa-youtube-play youtube" aria-hidden="true"></i> For a video introduction to the new hierarchy concept for groups and projects for epics, see |
